Skip to content

feat: model hub visibility & features + review fixes v4 #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sheepdestroyer wants to merge 22 commits into from
Closed

feat: model hub visibility & features + review fixes v4 #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0f8c84a
Configure gated Ollama routing and set llm-routing-ollama as free tie…
sheepdestroyer Jun 19, 2026
b3ae575
Address code review comments: Fix annotations race condition and hand…
sheepdestroyer Jun 19, 2026
8338b23
Address new code reviews: Break Ollama fallback loop, use env key, an…
sheepdestroyer Jun 19, 2026
528eb8d
fix: sanitize triage router 429 detail and force immediate LiteLLM co…
sheepdestroyer Jun 19, 2026
e14a543
docs: update fallback diagrams and cooldown behavior for Ollama models
sheepdestroyer Jun 19, 2026
78f922d
fix: implement router-side Ollama cooldown to prevent crashloop
sheepdestroyer Jun 19, 2026
214c637
chore: tidy up repository, remove hello world dummies, move and docum…
sheepdestroyer Jun 19, 2026
12753bd
fix: resolve hardcoded worktree leaks and LiteLLM auth errors
sheepdestroyer Jun 19, 2026
bda8325
Address PR comments: robust httpx client teardown, dynamic path escap…
sheepdestroyer Jun 19, 2026
be67870
Implement Valkey global cooldown cache sync, standard HTTPX client li…
sheepdestroyer Jun 19, 2026
38e080c
Address PR#11 code reviews: multimodal message extraction, concurrent…
sheepdestroyer Jun 19, 2026
dcccf7f
Address PR #12 code review comments
sheepdestroyer Jun 19, 2026
caef8f9
feat: expose model capabilities, token limits, and costs in Model Hub…
sheepdestroyer Jun 19, 2026
c34bd9e
chore: address review feedback on PR #14 (DRY purge helper, safety-ne…
sheepdestroyer Jun 20, 2026
03b96f5
chore: address PR #15 code review fixes and fix CI workflow triggers
sheepdestroyer Jun 20, 2026
a7a5588
chore(deps): adjust dependabot root docker update time to 02:55 UTC (…
sheepdestroyer Jun 20, 2026
d22047a
chore: address code review feedback (YAML list indentation, raise_for…
sheepdestroyer Jun 20, 2026
296b957
chore: trigger CI
sheepdestroyer Jun 20, 2026
3466563
Address code reviews for PR #25
sheepdestroyer Jun 20, 2026
43b1987
Address new PR reviews and CodeRabbit feedback
sheepdestroyer Jun 20, 2026
07b035e
Address Gemini Code Assist review feedback on null text handling and …
sheepdestroyer Jun 20, 2026
eafa0fc
chore: address PR review feedback on DATABASE_URL, fallback password,…
sheepdestroyer Jun 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/dependabot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ updates:
directory: "/"
schedule:
interval: "daily"
time: "03:23"
time: "02:55"
open-pull-requests-limit: 5
labels:
- "dependencies"
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
test:
Expand All @@ -20,6 +19,9 @@ jobs:
with:
python-version: '3.11'

- name: Install dependencies
run: pip install httpx==0.28.1

- name: Run Circuit Breaker Tests
run: python3 test_circuit_breaker.py

Expand Down
168 changes: 123 additions & 45 deletions README.md
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion hello.py
View file
Open in desktop

This file was deleted.

141 changes: 117 additions & 24 deletions litellm/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
general_settings:
master_key: sk-lit...33bf
master_key: os.environ/LITELLM_MASTER_KEY
litellm_settings:
# -------------------------------------------------------------------------
# FALLBACK CHAINS (cascading, in order of escalation)
Expand All @@ -20,41 +20,76 @@ litellm_settings:
- langfuse
detailed_debug: false
drop_params: true
public_model_groups:
- openrouter-auto
- llm-routing-ollama
- ollama-deepseek-v4-pro
- ollama-deepseek-v4-flash
- agent-advanced-core
- agent-reasoning-core
- agent-complex-core
- agent-medium-core
- agent-simple-core
fallbacks:
- agent-simple-core:
- agent-medium-core
- agent-complex-core
- agent-reasoning-core
- agent-advanced-core
- llm-routing-ollama
- openrouter-auto
# - local-qwen-3.6 # DISABLED: 35B model unloaded (23GB GTT saved)
- agent-medium-core:
- agent-complex-core
- agent-reasoning-core
- agent-advanced-core
- llm-routing-ollama
- openrouter-auto
# - local-qwen-3.6 # DISABLED
- agent-complex-core:
- agent-reasoning-core
- agent-advanced-core
- llm-routing-ollama
- openrouter-auto
# - local-qwen-3.6 # DISABLED
- agent-reasoning-core:
- agent-advanced-core
- llm-routing-ollama
- openrouter-auto
# - local-qwen-3.6 # DISABLED
- agent-advanced-core:
- llm-routing-ollama
- openrouter-auto
# - local-qwen-3.6 # DISABLED
- ollama-deepseek-v4-pro:
- agent-advanced-core
- ollama-deepseek-v4-flash:
- agent-reasoning-core

model_list:
- litellm_params:
model: openrouter/openrouter/auto
request_timeout: 120
model_name: openrouter-auto
model_info:
supports_vision: true
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 2000000
max_input_tokens: 2000000
is_public_model_group: true
- litellm_params:
model: openai/llm-routing-ollama
api_base: http://127.0.0.1:5000/v1
api_key: os.environ/LITELLM_MASTER_KEY
request_timeout: 120
model_name: llm-routing-ollama
model_info:
supports_vision: true
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 524288
max_input_tokens: 524288
is_public_model_group: true

# DISABLED 2026-06-08 — 20GB on disk, 23GB GTT (system RAM as GPU buffer).
# Uncomment to re-enable once a lighter model replaces it.
#- litellm_params:
Expand All @@ -73,25 +108,48 @@ model_list:
# ================================================================================
# OLLAMA PAID TIER — ollama.com via LiteLLM's native ollama_chat provider.
# LiteLLM calls https://api.ollama.com/api/chat with Bearer auth (OLLAMA_API_KEY).
# Fallback: ollama-deepseek-v4-pro → agent-advanced-core → openrouter-auto.
# No LiteLLM-level fallbacks: failures propagate back to the triage router
# (router/main.py) which manages Ollama cooldowns internally. When Ollama fails,
# the router activates a 5-minute cooldown and skips Ollama on subsequent
# requests, returning 429 so LiteLLM falls through to openrouter-auto.
# ================================================================================
- model_name: ollama-deepseek-v4-pro
litellm_params:
model: ollama_chat/deepseek-v4-pro
api_base: https://api.ollama.com
api_key: os.environ/OLLAMA_API_KEY
request_timeout: 120
model_info:
supports_vision: true
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 524288
max_input_tokens: 524288
input_cost_per_token: 0.00000174
output_cost_per_token: 0.00000348
is_public_model_group: true

# ================================================================================
# OLLAMA FLASH TIER — lighter/faster model for reasoning-tier requests.
# Fallback: ollama-deepseek-v4-flash → agent-reasoning-core → openrouter-auto.
# Same cooldown architecture as the pro tier above.
# ================================================================================
- model_name: ollama-deepseek-v4-flash
litellm_params:
model: ollama_chat/deepseek-v4-flash
api_base: https://api.ollama.com
api_key: os.environ/OLLAMA_API_KEY
request_timeout: 120
model_info:
supports_vision: true
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 524288
max_input_tokens: 524288
input_cost_per_token: 0.00000014
output_cost_per_token: 0.00000028
is_public_model_group: true

# ================================================================================
# AGENT TIER DEPLOYMENTS — safety-net entries (one per tier)
Expand Down Expand Up @@ -119,43 +177,78 @@ model_list:

- model_name: agent-advanced-core
litellm_params:
model: openrouter/minimax/minimax-m2.5:free
request_timeout: 120
model: openrouter/google/gemma-4-31b-it:free
request_timeout: 20
model_info:
supports_vision: false
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 262144
max_input_tokens: 262144
is_public_model_group: true
- model_name: agent-reasoning-core
litellm_params:
model: openrouter/moonshotai/kimi-k2.6:free
request_timeout: 120
model: openrouter/google/gemma-4-26b-a4b-it:free
request_timeout: 20
model_info:
supports_vision: false
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 262144
max_input_tokens: 262144
is_public_model_group: true
- model_name: agent-complex-core
litellm_params:
model: openrouter/nvidia/nemotron-3-ultra-550b-a55b:free
request_timeout: 120
request_timeout: 20
model_info:
supports_vision: false
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 262144
max_input_tokens: 262144
is_public_model_group: true
- model_name: agent-medium-core
litellm_params:
model: openrouter/google/gemma-4-26b-a4b-it:free
request_timeout: 120
request_timeout: 20
model_info:
supports_vision: false
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 262144
max_input_tokens: 262144
is_public_model_group: true
- model_name: agent-simple-core
litellm_params:
model: openrouter/nvidia/nemotron-3-nano-30b-a3b:free
request_timeout: 120
request_timeout: 20
model_info:
supports_vision: false
supports_reasoning: true
supports_function_calling: true
mode: chat
max_tokens: 256000
max_input_tokens: 256000
Comment on lines +235 to +236

@coderabbitai coderabbitai Bot Jun 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Align simple-tier context metadata with the router clamp.

This advertises agent-simple-core as ~256K context, but router/main.py clamps agent-simple-core using a 32K minimum context. Clients relying on Model Hub metadata can submit prompts that the router/runtime path won’t actually honor. Pick one source of truth and update either this metadata or the router clamp.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@litellm/config.yaml` around lines 235 - 236, The max_tokens and
max_input_tokens values (256000) in the config.yaml for agent-simple-core are
misaligned with the router clamp in router/main.py which limits
agent-simple-core to a 32K minimum context, causing clients to see incorrect
metadata. Choose one source of truth and update either the max_tokens and
max_input_tokens values in this config.yaml file to reflect the 32K constraint,
or update the router clamp in router/main.py to honor the 256K values advertised
here, ensuring both paths enforce the same limits for agent-simple-core.

@sheepdestroyer sheepdestroyer Jun 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be extra mindful of any flow/logic changes to the tiering and fallback chains, and consider full implications.

@coderabbitai coderabbitai Bot Jun 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Look for max_tokens clamping / agent-simple-core context logic in router/main.py
rg -n "max_tokens|clamp|32768|256000|agent-simple|safe_context|context_length" router/main.py -A3 -B3

Length of output: 9692

@sheepdestroyer — confirmed the clamping logic is still active on this branch. Here's the full picture:

What the code does (Lines 1804–1821 of router/main.py)

_tier_min_ctx = {
    "agent-simple-core": 32768,      # worst-case floor over dynamic pool
    "ollama-deepseek-v4-pro": 524288,
    "ollama-deepseek-v4-flash": 524288,
}
_min_ctx  = _tier_min_ctx.get(model_name, 262144)
_safe_max = max(1024, _min_ctx - _est_input - 2048)   # 2K safety margin
if requested_max_tokens > _safe_max:
    body_to_send["max_tokens"] = _safe_max

For agent-simple-core with a near-empty prompt this yields _safe_max ≈ 30 720. config.yaml meanwhile advertises max_tokens: 256000.

Full implications

Scenario Effect
Client reads Model Hub (static entry), sends a 200 K-token prompt Router silently clamps output to ≤ 30 K tokens — client never knows
Dynamic simple-tier model with 32 K context receives a clamped but still large request Request may still fail with context-length error from the upstream provider
Change metadata in config.yaml to 32 K No routing/fallback logic changes; static entry becomes honest about worst-case; after roster sync the per-model entries get their own correct values anyway
Raise the router floor to 256 K Risky: dynamically added tiny models (liquid/dolphin variants) that land in simple-core would receive oversized requests and fail

Recommendation (minimal, safe)

The only change that avoids any flow impact is aligning the static metadata with the runtime floor already enforced by the router:

# litellm/config.yaml  — agent-simple-core
  model_info:
    max_tokens: 32768
    max_input_tokens: 32768

This does not change any routing decision, fallback chain, or LiteLLM router behavior — it only corrects what the Model Hub advertises to clients to match what the router actually enforces as its guaranteed minimum. The static entry is a cold-start safety net; once the roster sync runs, each dynamically registered deployment gets its own accurate per-model context from OpenRouter.

is_public_model_group: true

redis_settings:
redis_host: 127.0.0.1
redis_port: 6379
router_settings:
allowed_fails: 2
allowed_fails: 0
cooldown_time: 300
enable_pre_call_checks: false
num_retries: 1
routing_strategy: latency-based-routing
# Per-error-type cooldown thresholds (overrides allowed_fails for specific errors).
# Upstream rate limits ("temporarily rate-limited upstream") can last minutes —
# a 30s cooldown just wastes retries. 300s gives providers time to clear the limit.
# RateLimitError: cooldown on FIRST 429 (upstream rate limits persist for minutes —
# allowing even 1 retry wastes a request against a provider that's still throttling)
allowed_fails_policy:
RateLimitErrorAllowedFails: 0
TimeoutErrorAllowedFails: 3
BadRequestErrorAllowedFails: 1
enable_health_check_routing: false
# NOTE: allowed_fails_policy is an enterprise-only feature in LiteLLM.
# Ollama cooldowns are handled by the triage router itself (router/main.py),
# not by LiteLLM's deployment cooldown mechanism.
vector_store_settings:
collection_name: litellm_semantic_cache
connection_string: postgresql://postgres:***@127.0.0.1:5432/postgres
Expand Down
6 changes: 6 additions & 0 deletions pod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,10 @@ spec:
env:
- name: CONFIG_PATH
value: /config/router_dir/config.yaml
- name: LITELLM_CONFIG_PATH
value: /config/litellm_dir/config.yaml
- name: DATABASE_URL
value: postgresql://postgres:***@127.0.0.1:5432/postgres

@gemini-code-assist gemini-code-assist Bot Jun 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The DATABASE_URL environment variable contains a placeholder password ***, but the PostgreSQL container is configured with the password postgres-local-pw-2026 (on line 161). Since start-stack.sh does not replace *** at runtime, this will cause connection failures when the triage router tries to connect to the database. Furthermore, because DATABASE_URL is set in the environment, the fallback connection string in router/main.py (which contains the correct password) will be ignored. Please update the password in DATABASE_URL to postgres-local-pw-2026 or ensure it is dynamically replaced.

- name: DBUS_SESSION_BUS_ADDRESS
value: unix:path=/run/user/1000/bus
- name: LITELLM_MASTER_KEY
Expand Down Expand Up @@ -127,6 +131,8 @@ spec:
volumeMounts:
- mountPath: /config/router_dir
name: router-config
- mountPath: /config/litellm_dir
name: litellm-config
- mountPath: /app/data
name: dataset-data
- mountPath: /config/gemini_auth
Expand Down
2 changes: 1 addition & 1 deletion router/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ FROM python:3.14-slim
WORKDIR /app

# Install deps in a single layer (no pip cache, no extra files)
RUN pip install --no-cache-dir fastapi uvicorn httpx pyyaml python-multipart asyncpg langfuse
RUN pip install --no-cache-dir fastapi uvicorn httpx pyyaml python-multipart asyncpg langfuse redis

# Copy all source in one layer — removes dead config COPY (volume-mounted at runtime)
COPY main.py agy_proxy.py circuit_breaker.py aa_scores.json free_models_roster.json /app/
Expand Down
Loading