diff --git a/README.md b/README.md index b2838ecf..8c2ac516 100644 --- a/README.md +++ b/README.md @@ -141,16 +141,33 @@ sequenceDiagram end else Route = ollama (llm-routing-ollama, auto-ollama, auto-agy-ollama chain) - Note over Router: Proxy to LiteLLM as ollama-deepseek-v4-pro + Note over Router: Proxy to LiteLLM as ollama-deepseek-v4-pro / -flash Router->>Proxy: POST /v1/chat/completions (model=ollama-deepseek-v4-pro) Proxy->>Provider: Call api.ollama.com (ollama_chat provider) alt Ollama Succeeds Provider-->>Proxy: deepseek-v4-pro response Proxy-->>Router: Return response - Router-->>Client: Return chat completion + Router-->>Client: Return response else Ollama fails (rate-limited / unavailable) - Note over Proxy: Fallback: ollama-deepseek-v4-pro → agent-advanced-core - Proxy->>Provider: Call OpenRouter (agent-advanced-core tier) + Provider-->>Proxy: Error / HTTP 429 + Proxy-->>Router: Error Response + Note over Router: Activate 5-min router-side Ollama cooldown + alt Model = auto-ollama / auto-agy-ollama (triage-gated) + Note over Router: Catch error → Fall back to free tier + Router->>Proxy: POST /v1/chat/completions (model=original_target_model) + Proxy->>Provider: Call OpenRouter (free tier cascade) + Provider-->>Proxy: Response + Proxy-->>Router: Response + Router-->>Client: Return response + else Model = llm-routing-ollama (direct / fallback chain) + Note over Router: Return 429 (cooldown active) + Router-->>Proxy: HTTP 429 (Ollama cooled down) + Note over Proxy: Skip llm-routing-ollama → cascade to openrouter-auto + Proxy->>Provider: Call OpenRouter (openrouter-auto) + Provider-->>Proxy: Response + Proxy-->>Router: Response + Router-->>Client: Return response + end end else Route = LiteLLM (all other models) @@ -180,11 +197,11 @@ The gateway supports multiple routing modes controlled by the `model` field: | Model | Behavior | |-------|----------| | `llm-routing-auto-free` | **Full classifier pipeline** → routes to best free tier. Recommended default. | -| `llm-routing-auto-agy` | **Classifier + agy (gated)**: runs classifier, tries agy only if classified as advanced. | -| `llm-routing-auto-ollama` | **Classifier + Ollama (gated)**: runs classifier, tries Ollama only if advanced. | -| `llm-routing-auto-agy-ollama` | **Classifier → agy → ollama (gated)**: runs classifier, chains agy then Ollama only if advanced. | +| `llm-routing-auto-agy` | **Classifier + agy (gated)**: runs classifier, tries agy only if classified as advanced/reasoning. | +| `llm-routing-auto-ollama` | **Classifier + Ollama (gated)**: runs classifier, reasoning & advanced → `ollama-deepseek-v4-pro`, complex → `ollama-deepseek-v4-flash`, below (medium/simple) → bypasses Ollama to LiteLLM free tiers. | +| `llm-routing-auto-agy-ollama` | **Classifier → agy → ollama (gated)**: runs classifier, chains agy then Ollama only if advanced/reasoning. | | `llm-routing-agy` | **Direct agy**: skips classifier, agy proxy (Gemini/Claude) → LiteLLM fallback. | -| `llm-routing-ollama` | **Direct Ollama**: skips classifier, Ollama deepseek-v4-pro → LiteLLM fallback. | +| `llm-routing-ollama` | **Gated Ollama**: runs classifier, reasoning & advanced → `ollama-deepseek-v4-pro`, complex & below → `ollama-deepseek-v4-flash`. | | `agent-simple-core` / `agent-medium-core` / `agent-complex-core` / `agent-reasoning-core` / `agent-advanced-core` | **Direct routing**: bypasses classifier, goes straight to LiteLLM with that tier name. | | Anything else | Returns HTTP 400 with the list of available models | @@ -238,10 +255,10 @@ Exposes the entry endpoint (`http://localhost:5000/v1`) and evaluates prompt com |:---|---:|:---|:---| | `llm-routing-auto-free` | ✅ | — | LiteLLM with classified tier | 256K | | `llm-routing-auto-agy` | ✅ | agy (gated: reasoning → gemini-3.5-flash, advanced → gemini-3.5-flash → claude-opus-4.6) | LiteLLM with classified tier | 256K | -| `llm-routing-auto-ollama` | ✅ | Ollama (gated: reasoning → deepseek-v4-flash, advanced → deepseek-v4-pro) | LiteLLM agent-advanced-core | 256K | +| `llm-routing-auto-ollama` | ✅ | Ollama (gated: reasoning & advanced → ollama-deepseek-v4-pro, complex → ollama-deepseek-v4-flash, below → bypass) | LiteLLM with classified tier | 256K | | `llm-routing-auto-agy-ollama` | ✅ | agy → Ollama (gated: reasoning/advanced only) | LiteLLM with classified tier | 256K | | `llm-routing-agy` | ❌ | agy (Gemini/Claude) — unconditional | LiteLLM agent-advanced-core | 256K | -| `llm-routing-ollama` | ❌ | Ollama deepseek-v4-pro — unconditional | LiteLLM agent-advanced-core | 256K | +| `llm-routing-ollama` | ✅ | Ollama (gated: reasoning & advanced → ollama-deepseek-v4-pro, complex & below → ollama-deepseek-v4-flash) | LiteLLM agent-advanced-core / agent-reasoning-core | 256K | | `agent-advanced-core` | ❌ | — | LiteLLM openrouter-auto | | `agent-reasoning-core` | ❌ | — | LiteLLM fallback chain | | `agent-complex-core` | ❌ | — | LiteLLM fallback chain | @@ -259,14 +276,62 @@ Orchestrates routing fallback chains, Redis caching, and telemetry callbacks: - `embedding_model: "local-nomic-embed"` — uses the local nomic-embed model (no API costs) - `collection_name: "litellm_semantic_cache"` — stores embeddings for similarity-based cache lookups - **Cascading Fallback Chains** (configured in `litellm_settings.fallbacks`): - Each tier escalates through increasingly capable models. All chains end at `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT. - - **`agent-simple-core`**: medium-core → complex-core → reasoning-core → advanced-core → `openrouter-auto` - - **`agent-medium-core`**: complex-core → reasoning-core → advanced-core → `openrouter-auto` - - **`agent-complex-core`**: reasoning-core → advanced-core → `openrouter-auto` - - **`agent-reasoning-core`**: advanced-core → `openrouter-auto` - - **`agent-advanced-core`**: `openrouter-auto` - - **`ollama-deepseek-v4-pro`**: advanced-core → `openrouter-auto` - All tiers ultimately land on the local Ryzen APU MoE (`qwen-35b-q4ks` via llama-server on :8080) as the final safety net. + Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT. + + ```mermaid + graph TD + %% Define styles + classDef simple fill:#4F46E5,stroke:#312E81,stroke-width:2px,color:#fff; + classDef medium fill:#7C3AED,stroke:#4C1D95,stroke-width:2px,color:#fff; + classDef complex fill:#DB2777,stroke:#831843,stroke-width:2px,color:#fff; + classDef reasoning fill:#EA580C,stroke:#7C2D12,stroke-width:2px,color:#fff; + classDef advanced fill:#E11D48,stroke:#881337,stroke-width:2px,color:#fff; + classDef premium fill:#059669,stroke:#064E3B,stroke-width:2px,color:#fff; + classDef auto fill:#4B5563,stroke:#1F2937,stroke-width:2px,color:#fff; + + subgraph Simple["agent-simple-core Fallback Tree"] + S[agent-simple-core]:::simple --> SM[agent-medium-core]:::medium + SM --> SC[agent-complex-core]:::complex + SC --> SR[agent-reasoning-core]:::reasoning + SR --> SA[agent-advanced-core]:::advanced + SA --> SO1[llm-routing-ollama]:::premium + SO1 --> SAU[openrouter-auto]:::auto + end + + subgraph Medium["agent-medium-core Fallback Tree"] + M[agent-medium-core]:::medium --> MC[agent-complex-core]:::complex + MC --> MR[agent-reasoning-core]:::reasoning + MR --> MA[agent-advanced-core]:::advanced + MA --> MO1[llm-routing-ollama]:::premium + MO1 --> MAU[openrouter-auto]:::auto + end + + subgraph Complex["agent-complex-core Fallback Tree"] + C[agent-complex-core]:::complex --> CR[agent-reasoning-core]:::reasoning + CR --> CA[agent-advanced-core]:::advanced + CA --> CO1[llm-routing-ollama]:::premium + CO1 --> CAU[openrouter-auto]:::auto + end + + subgraph Reasoning["agent-reasoning-core Fallback Tree"] + R[agent-reasoning-core]:::reasoning --> RA[agent-advanced-core]:::advanced + RA --> RO1[llm-routing-ollama]:::premium + RO1 --> RAU[openrouter-auto]:::auto + end + + subgraph Advanced["agent-advanced-core Fallback Tree"] + A[agent-advanced-core]:::advanced --> AO1[llm-routing-ollama]:::premium + AO1 --> AAU[openrouter-auto]:::auto + end + ``` + + - **`agent-simple-core`**: medium-core → complex-core → reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto` + - **`agent-medium-core`**: complex-core → reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto` + - **`agent-complex-core`**: reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto` + - **`agent-reasoning-core`**: advanced-core → `llm-routing-ollama` → `openrouter-auto` + - **`agent-advanced-core`**: `llm-routing-ollama` → `openrouter-auto` + - **`llm-routing-ollama`** (classifier-gated proxy): `reasoning & advanced` → `ollama-deepseek-v4-pro`, `complex & below` → `ollama-deepseek-v4-flash`. Note: Ollama cooldowns are managed by the triage router internally (5-minute window on failure); during cooldown the router returns 429 immediately so LiteLLM skips to `openrouter-auto`. + All tiers ultimately land on OpenRouter auto/free model pools or the local Speculative MoE when enabled. *Note: Premium routing is controlled by the model name, not by the tier. `llm-routing-agy` and `llm-routing-auto-agy` trigger the agy proxy (Google/Claude via Cloud Code Assist) — but auto models only trigger agy if the classifier returns `agent-advanced-core`. `llm-routing-ollama` and `llm-routing-auto-ollama` route through Ollama.com (deepseek-v4-pro via LiteLLM's ollama_chat provider) — same gating for auto models. `llm-routing-auto-agy-ollama` chains both: agy first, then Ollama if agy is exhausted, both gated on advanced classification. The `agent-advanced-core` tier itself is a plain LiteLLM tier with no premium trigger. See §2 for the full routing table.* ### C. Valkey Caching (`redis_settings` in LiteLLM) @@ -444,6 +509,8 @@ This endpoint outputs plain-text metrics (`Content-Type: text/plain; version=0.0 | `circuit_breaker_tier` | gauge | Current circuit breaker cooldown tier (0=open, 1-3=cooldown) | | `circuit_breaker_agy_allowed` | gauge | Whether agy proxy requests are currently allowed (0/1) | | `circuit_breaker_cooldown_remaining_seconds` | gauge | Seconds until cooldown expires | +| `ollama_cooldown_active` | gauge | Whether Ollama is in router-side cooldown (1=active, 0=open) | +| `ollama_cooldown_remaining_seconds` | gauge | Seconds remaining in Ollama cooldown | | `circuit_breaker_total_trips` | counter | Total number of circuit breaker trips | | `circuit_breaker_probe_granted` | gauge | Whether a probe request has been granted (0/1) | @@ -676,29 +743,35 @@ LiteLLM calls `https://api.ollama.com/api/chat` with Bearer authentication using | Model | Ollama tag | Purpose | |-------|-----------|---------| -| `ollama-deepseek-v4-pro` | `deepseek-v4-pro` | Primary paid tier — 1.6T parameter model | +| `ollama-deepseek-v4-pro` | `deepseek-v4-pro` | Primary paid tier — 1.6T parameter reasoning & advanced model | +| `ollama-deepseek-v4-flash` | `deepseek-v4-flash` | Lightweight paid tier — fast complex & below model | Additional Ollama.com models can be added to `litellm/config.yaml` using the same `ollama_chat/` prefix pattern. -### Fallback Chain +### Fallback and Cooldown Behavior -``` -ollama-deepseek-v4-pro → agent-advanced-core → openrouter-auto -``` +To prevent cascading fallback loops where a rate-limited Ollama backend repeatedly receives requests from different tiers, the **Triage Router manages Ollama cooldowns internally** rather than relying on LiteLLM's deployment cooldown mechanism (which is unreliable for single-deployment model groups in Community Edition). + +The cooldown mechanism works as follows: -If Ollama.com is rate-limited or unavailable, LiteLLM automatically falls back through -the agent tier cascade to OpenRouter's free model pool. +1. **Failure Detection**: When calls to `ollama-deepseek-v4-pro` or `ollama-deepseek-v4-flash` fail (due to rate limiting, 429/502/503 errors, or connection issues), the failure is caught by the Triage Router. +2. **Router-Side Cooldown Activation**: The Triage Router activates an internal **5-minute cooldown** (configurable via `OLLAMA_COOLDOWN_SECONDS` env var). During this window, all subsequent Ollama requests are **immediately rejected** without making any LiteLLM calls. +3. **Direct / Fallback Requests (`llm-routing-ollama`)**: + - During cooldown, the Triage Router returns an HTTP `429` immediately. + - LiteLLM receives this 429, skips `llm-routing-ollama` in the fallback chain, and cascades directly to `openrouter-auto`. +4. **Auto-Routing Requests (`llm-routing-auto-ollama` or `llm-routing-auto-agy-ollama`)**: + - During cooldown, the Triage Router silently falls back to the original classified free tier model (e.g., `agent-advanced-core`), querying LiteLLM for a free model. ### Routing Modes | Model | Behavior | |-------|----------| -| `llm-routing-ollama` | Direct — skips classifier, goes straight to Ollama deepseek-v4-pro | -| `llm-routing-auto-ollama` | Auto — classifier runs first, then Ollama deepseek-v4-pro is tried regardless of tier | -| `llm-routing-auto-agy-ollama` | Chained — classifier runs, tries agy first, then chains to Ollama if agy exhausted | +| `llm-routing-ollama` | **Gated direct**: runs classifier, routes reasoning & advanced → `ollama-deepseek-v4-pro`, complex & below → `ollama-deepseek-v4-flash` | +| `llm-routing-auto-ollama` | **Gated auto**: runs classifier, reasoning & advanced → `ollama-deepseek-v4-pro`, complex → `ollama-deepseek-v4-flash`, below → bypasses Ollama to LiteLLM free tiers | +| `llm-routing-auto-agy-ollama` | **Gated chained**: runs classifier, tries agy first (advanced/reasoning only), then chains to Ollama if agy is exhausted | -All three modes ultimately fall back to LiteLLM's agent tier cascade if the premium backends fail. +For auto-routing modes, the Triage Router handles failures by silently falling back to the classified free tier cascade. For direct requests to `llm-routing-ollama`, the router returns `429` immediately during cooldown, allowing LiteLLM to skip this model group and cascade to `openrouter-auto`. The cooldown status is visible via the `/metrics` endpoint (`ollama_cooldown_active` and `ollama_cooldown_remaining_seconds` gauges). ## 9d. Hermes Cron Jobs & Health Check Alerting diff --git a/litellm/config.yaml b/litellm/config.yaml index c27b47c7..b731a2b7 100644 --- a/litellm/config.yaml +++ b/litellm/config.yaml @@ -1,5 +1,5 @@ general_settings: - master_key: sk-lit...33bf + master_key: os.environ/LITELLM_MASTER_KEY litellm_settings: # ------------------------------------------------------------------------- # FALLBACK CHAINS (cascading, in order of escalation) @@ -26,35 +26,44 @@ litellm_settings: - agent-complex-core - agent-reasoning-core - agent-advanced-core + - llm-routing-ollama - openrouter-auto # - local-qwen-3.6 # DISABLED: 35B model unloaded (23GB GTT saved) - agent-medium-core: - agent-complex-core - agent-reasoning-core - agent-advanced-core + - llm-routing-ollama - openrouter-auto # - local-qwen-3.6 # DISABLED - agent-complex-core: - agent-reasoning-core - agent-advanced-core + - llm-routing-ollama - openrouter-auto # - local-qwen-3.6 # DISABLED - agent-reasoning-core: - agent-advanced-core + - llm-routing-ollama - openrouter-auto # - local-qwen-3.6 # DISABLED - agent-advanced-core: + - llm-routing-ollama - openrouter-auto # - local-qwen-3.6 # DISABLED - - ollama-deepseek-v4-pro: - - agent-advanced-core - - ollama-deepseek-v4-flash: - - agent-reasoning-core + model_list: - litellm_params: model: openrouter/openrouter/auto request_timeout: 120 model_name: openrouter-auto +- litellm_params: + model: openai/llm-routing-ollama + api_base: http://127.0.0.1:5000/v1 + api_key: os.environ/LITELLM_MASTER_KEY + request_timeout: 120 + model_name: llm-routing-ollama + # DISABLED 2026-06-08 — 20GB on disk, 23GB GTT (system RAM as GPU buffer). # Uncomment to re-enable once a lighter model replaces it. #- litellm_params: @@ -73,7 +82,10 @@ model_list: # ================================================================================ # OLLAMA PAID TIER — ollama.com via LiteLLM's native ollama_chat provider. # LiteLLM calls https://api.ollama.com/api/chat with Bearer auth (OLLAMA_API_KEY). -# Fallback: ollama-deepseek-v4-pro → agent-advanced-core → openrouter-auto. +# No LiteLLM-level fallbacks: failures propagate back to the triage router +# (router/main.py) which manages Ollama cooldowns internally. When Ollama fails, +# the router activates a 5-minute cooldown and skips Ollama on subsequent +# requests, returning 429 so LiteLLM falls through to openrouter-auto. # ================================================================================ - model_name: ollama-deepseek-v4-pro litellm_params: @@ -84,7 +96,7 @@ model_list: # ================================================================================ # OLLAMA FLASH TIER — lighter/faster model for reasoning-tier requests. -# Fallback: ollama-deepseek-v4-flash → agent-reasoning-core → openrouter-auto. +# Same cooldown architecture as the pro tier above. # ================================================================================ - model_name: ollama-deepseek-v4-flash litellm_params: @@ -119,43 +131,38 @@ model_list: - model_name: agent-advanced-core litellm_params: - model: openrouter/minimax/minimax-m2.5:free - request_timeout: 120 + model: openrouter/google/gemma-4-31b-it:free + request_timeout: 20 - model_name: agent-reasoning-core litellm_params: - model: openrouter/moonshotai/kimi-k2.6:free - request_timeout: 120 + model: openrouter/google/gemma-4-26b-a4b-it:free + request_timeout: 20 - model_name: agent-complex-core litellm_params: model: openrouter/nvidia/nemotron-3-ultra-550b-a55b:free - request_timeout: 120 + request_timeout: 20 - model_name: agent-medium-core litellm_params: model: openrouter/google/gemma-4-26b-a4b-it:free - request_timeout: 120 + request_timeout: 20 - model_name: agent-simple-core litellm_params: model: openrouter/nvidia/nemotron-3-nano-30b-a3b:free - request_timeout: 120 + request_timeout: 20 redis_settings: redis_host: 127.0.0.1 redis_port: 6379 router_settings: - allowed_fails: 2 + allowed_fails: 0 cooldown_time: 300 enable_pre_call_checks: false num_retries: 1 routing_strategy: latency-based-routing - # Per-error-type cooldown thresholds (overrides allowed_fails for specific errors). - # Upstream rate limits ("temporarily rate-limited upstream") can last minutes — - # a 30s cooldown just wastes retries. 300s gives providers time to clear the limit. - # RateLimitError: cooldown on FIRST 429 (upstream rate limits persist for minutes — - # allowing even 1 retry wastes a request against a provider that's still throttling) - allowed_fails_policy: - RateLimitErrorAllowedFails: 0 - TimeoutErrorAllowedFails: 3 - BadRequestErrorAllowedFails: 1 + enable_health_check_routing: false + # NOTE: allowed_fails_policy is an enterprise-only feature in LiteLLM. + # Ollama cooldowns are handled by the triage router itself (router/main.py), + # not by LiteLLM's deployment cooldown mechanism. vector_store_settings: collection_name: litellm_semantic_cache connection_string: postgresql://postgres:***@127.0.0.1:5432/postgres diff --git a/router/agy_proxy.py b/router/agy_proxy.py index 918bfcb6..eda48dc3 100644 --- a/router/agy_proxy.py +++ b/router/agy_proxy.py @@ -220,7 +220,7 @@ async def try_agy_proxy(prompt: str, messages: list = None, context_parts = [] for msg in messages[-10:]: role = msg.get("role", "user") - content = msg.get("content", "") + content = msg.get("content") or "" if role == "user": context_parts.append(f"User: {content}") elif role == "assistant": diff --git a/router/free_models_roster.json b/router/free_models_roster.json index f7cedb5b..79b88eae 100644 --- a/router/free_models_roster.json +++ b/router/free_models_roster.json @@ -42,6 +42,12 @@ "score": 28.0, "context_length": 256000 }, + { + "id": "cohere/north-mini-code:free", + "name": "Cohere: North Mini Code (free)", + "score": 25.0, + "context_length": 256000 + }, { "id": "nex-agi/nex-n2-pro:free", "name": "Nex AGI: Nex-N2-Pro (free)", @@ -139,6 +145,6 @@ "context_length": 32768 } ], - "updated_at": "2026-06-17T20:17:56.981195Z", - "count": 23 + "updated_at": "2026-06-19T11:52:03.616141Z", + "count": 24 } \ No newline at end of file diff --git a/router/main.py b/router/main.py index 83c4127c..f9d92fce 100644 --- a/router/main.py +++ b/router/main.py @@ -130,6 +130,17 @@ async def push_aggregate_scores(): "timeline": [] } +# --------------------------------------------------------------------------- +# OLLAMA COOLDOWN — router-side cooldown for the Ollama backend. +# LiteLLM Community Edition's deployment cooldown is unreliable for single- +# deployment model groups (it bypasses cooldown when there's only 1 deployment) +# and doesn't reliably cooldown fallback-target model groups. Instead, the +# triage router tracks Ollama failures itself and returns 429 immediately +# during the cooldown window, skipping the LiteLLM call entirely. +# --------------------------------------------------------------------------- +_ollama_cooldown_until: float = 0.0 # monotonic timestamp when cooldown expires +OLLAMA_COOLDOWN_SECONDS: int = int(os.getenv("OLLAMA_COOLDOWN_SECONDS", "300")) # 5 min default + STATS_JSON_PATH = "/config/router_dir/router_stats.json" # Module-level set to hold references to fire-and-forget background tasks, @@ -350,7 +361,7 @@ def norm(s: float) -> float: for mid in model_ids: payload = { "model_name": tier_name, - "litellm_params": {"model": f"openrouter/{mid}", "request_timeout": 120} + "litellm_params": {"model": f"openrouter/{mid}", "request_timeout": 20} } try: r = await client.post(f"{admin_url}/model/new", headers=headers, json=payload) @@ -1090,7 +1101,7 @@ async def chat_completions(request: Request): last_user_message = "" for msg in reversed(messages): if msg.get("role") == "user": - last_user_message = msg.get("content", "") + last_user_message = msg.get("content") or "" break # Known tier names that can be routed directly (bypass classifier) @@ -1098,7 +1109,7 @@ async def chat_completions(request: Request): "agent-simple-core", "agent-medium-core", "agent-complex-core", "agent-reasoning-core", "agent-advanced-core", - "llm-routing-agy", "llm-routing-ollama", + "llm-routing-agy", } AUTO_MODELS = { @@ -1126,13 +1137,13 @@ async def chat_completions(request: Request): langfuse_trace_id = None parent_obs = None - if client_model in AUTO_MODELS: + if client_model in AUTO_MODELS or client_model == "llm-routing-ollama": # Full pipeline: classify → route to best tier bypass_cache = request.headers.get("x-bypass-cache") == "true" target_model, triage_latency, was_cache_hit, raw_classification = await classify_request( last_user_message, bypass_cache=bypass_cache, langfuse_trace_id=langfuse_trace_id ) - logger.info(f"Triage decision (auto): Routing to -> '{target_model}'") + logger.info(f"Triage decision (auto/gated): Routing to -> '{target_model}'") elif client_model in DIRECT_TIERS: # Direct routing: client knows what tier they want, skip classifier target_model = client_model @@ -1202,19 +1213,20 @@ async def chat_completions(request: Request): or (client_model in ("llm-routing-auto-agy", "llm-routing-auto-agy-ollama") and target_model in ("agent-advanced-core", "agent-reasoning-core")) ) should_try_ollama = ( - client_model == "llm-routing-ollama" # direct — always try - or (client_model in ("llm-routing-auto-ollama", "llm-routing-auto-agy-ollama") and target_model in ("agent-advanced-core", "agent-reasoning-core")) + client_model == "llm-routing-ollama" # always try (will map to flash for complex/below) + or (client_model in ("llm-routing-auto-ollama", "llm-routing-auto-agy-ollama") and target_model in ("agent-advanced-core", "agent-reasoning-core", "agent-complex-core")) ) # --- AGY PROXY --- if should_try_agy: + agy_span_obj = None try: from agy_proxy import try_agy_proxy last_prompt = "" for msg in reversed(messages): if msg.get("role") == "user": - last_prompt = msg.get("content", "") + last_prompt = msg.get("content") or "" break session_id = None @@ -1222,7 +1234,7 @@ async def chat_completions(request: Request): import hashlib fingerprint_parts = [] for msg in messages[:4]: - c = msg.get("content", "") or "" + c = msg.get("content") or "" if c: fingerprint_parts.append(c[:200]) fingerprint = "|".join(fingerprint_parts) @@ -1230,7 +1242,6 @@ async def chat_completions(request: Request): if last_prompt: # --- Langfuse child span: agy proxy --- - agy_span_obj = None if langfuse_trace_id: lf_agy = get_langfuse() if lf_agy: @@ -1301,7 +1312,7 @@ async def native_agy_stream_generator(stream_gen, model_name): # Success telemetry latency_ms = (time.time() - start_time) * 1000.0 # Approximate prompt tokens based on messages characters - prompt_chars = sum(len(m.get("content", "")) for m in messages) + prompt_chars = sum(len(m.get("content") or "") for m in messages) approx_prompt_tokens = max(1, prompt_chars // 4) record_tool_usage( @@ -1352,7 +1363,7 @@ async def native_agy_stream_generator(stream_gen, model_name): if is_stream_requested: # Robust fallback: simulate stream if we requested stream but got buffered response - content = agy_response.get("choices", [{}])[0].get("message", {}).get("content", "") + content = agy_response.get("choices", [{}])[0].get("message", {}).get("content") or "" async def agy_stream_generator(): """Asynchronous generator yielding simulated OpenAI-compatible streaming chunks from a static agy response.""" import uuid @@ -1412,169 +1423,240 @@ async def agy_stream_generator(): pass logger.error(f"agy proxy failed: {e}, falling back to LiteLLM") + original_target_model = target_model + # --- OLLAMA (via LiteLLM) --- # LiteLLM's ollama_chat provider handles the native Ollama API call. # We just proxy to LiteLLM with the appropriate model name. - # Reasoning tier → deepseek-v4-flash (lighter, faster) - # Advanced tier → deepseek-v4-pro (full power) # LiteLLM's fallback chain handles failures. if should_try_ollama: - if target_model == "agent-reasoning-core": - target_model = "ollama-deepseek-v4-flash" + if client_model in ("llm-routing-auto-ollama", "llm-routing-auto-agy-ollama"): + if target_model in ("agent-advanced-core", "agent-reasoning-core"): + target_model = "ollama-deepseek-v4-pro" + elif target_model == "agent-complex-core": + target_model = "ollama-deepseek-v4-flash" + elif client_model == "llm-routing-ollama": + if target_model in ("agent-advanced-core", "agent-reasoning-core"): + target_model = "ollama-deepseek-v4-pro" + else: + target_model = "ollama-deepseek-v4-flash" else: - target_model = "ollama-deepseek-v4-pro" + # Fallback (e.g. if LiteLLM fallback loops back with model: llm-routing-ollama) + if target_model in ("agent-advanced-core", "agent-reasoning-core"): + target_model = "ollama-deepseek-v4-pro" + else: + target_model = "ollama-deepseek-v4-flash" logger.info(f"Ollama route: proxying to LiteLLM as model={target_model}") - # Resolve backend connection parameters - backend_conf = backends.get(target_model) - if not backend_conf: - logger.error(f"Backend '{target_model}' not found in configuration backends.") - raise HTTPException(status_code=500, detail=f"Backend {target_model} misconfigured") - - backend_api_base = backend_conf["api_base"] - backend_api_key = backend_conf["api_key"] - if backend_api_key == "DYNAMIC_LITELLM_MASTER_KEY_PLACEHOLDER": - backend_api_key = os.getenv("LITELLM_MASTER_KEY", backend_api_key) - - # Delegate to LiteLLM which handles internal fallback chain - # Router sends model=agent-complex-core (or agent-simple-core) - # LiteLLM maps this to Nemotron → Kimi → GPT-OSS → local Qwen - logger.info(f"Proxying to LiteLLM as model={target_model}") - - # --- Langfuse child span: LiteLLM proxy --- - litellm_span_obj = None - if langfuse_trace_id: - lf_litellm = get_langfuse() - if lf_litellm: - try: - litellm_span_obj = lf_litellm.start_observation( - trace_context={"trace_id": langfuse_trace_id}, - name="litellm-proxy", - input=target_model, - metadata={"model": target_model}, - level="DEFAULT", - ) - except Exception: - pass + async def execute_proxy(model_name: str): + # Resolve backend connection parameters + backend_conf = backends.get(model_name) + if not backend_conf: + logger.error(f"Backend '{model_name}' not found in configuration backends.") + raise HTTPException(status_code=500, detail=f"Backend {model_name} misconfigured") + + backend_api_base = backend_conf["api_base"] + backend_api_key = backend_conf["api_key"] + if backend_api_key == "DYNAMIC_LITELLM_MASTER_KEY_PLACEHOLDER": + backend_api_key = os.getenv("LITELLM_MASTER_KEY", backend_api_key) + + logger.info(f"Proxying to LiteLLM as model={model_name}") + + # --- Langfuse child span: LiteLLM proxy --- + litellm_span_obj = None + if langfuse_trace_id: + lf_litellm = get_langfuse() + if lf_litellm: + try: + litellm_span_obj = lf_litellm.start_observation( + trace_context={"trace_id": langfuse_trace_id}, + name="litellm-proxy", + input=model_name, + metadata={"model": model_name}, + level="DEFAULT", + ) + except Exception: + pass - # Set up outgoing proxy request - client = httpx.AsyncClient(timeout=3600.0) - headers = {"Authorization": f"Bearer {backend_api_key}"} - if langfuse_trace_id: - headers["X-Langfuse-Trace-Id"] = langfuse_trace_id + # Set up outgoing proxy request + client = httpx.AsyncClient(timeout=3600.0) + headers = {"Authorization": f"Bearer {backend_api_key}"} + if langfuse_trace_id: + headers["X-Langfuse-Trace-Id"] = langfuse_trace_id - # Handle streaming vs non-streaming proxying (LiteLLM handles fallback internally) - proxy_start = time.time() - model_name = target_model # LiteLLM handles fallback internally - - # --- Pre-screening: clamp max_tokens to fit within downstream model context limits --- - # Hermes agents set max_tokens=65536 unconditionally, but some free models have - # context limits as low as 32K. Without clamping, input+output exceeds the limit - # and the request fails with a 400 BadRequest (context_length exceeded). - # We estimate input tokens and clamp max_tokens to leave a 2K safety margin. - try: - body_to_send = body.copy() - body_to_send["model"] = model_name - requested_max_tokens = body_to_send.get("max_tokens", 4096) - if requested_max_tokens > 32768: # Only intervene for unusually large max_tokens - # Tier-aware minimum context length (from actual roster data): - # - agent-simple-core: 32K (includes tiny liquid/dolphin models) - # - agent-medium-core+: 256K (smallest non-tiny model is nemotron-nano-omni at 256K) - # - ollama-deepseek-v4-*: 1M (DeepSeek V4 native context) - _tier_min_ctx = { - "agent-simple-core": 32768, - "ollama-deepseek-v4-pro": 1000000, - "ollama-deepseek-v4-flash": 1000000, - } - _min_ctx = _tier_min_ctx.get(model_name, 262144) - # Rough input token estimate: 1 token ≈ 4 chars of JSON - _est_input = len(json.dumps(body_to_send)) // 4 - _safe_max = max(1024, _min_ctx - _est_input - 2048) # 2K safety margin - if requested_max_tokens > _safe_max: - logger.warning( - f"⛔ Clamping max_tokens: {requested_max_tokens} → {_safe_max} " - f"(est_input={_est_input}, min_ctx={_min_ctx}, tier={model_name})" - ) - body_to_send["max_tokens"] = _safe_max - except Exception as e: - logger.warning(f"Pre-screening failed (non-fatal): {e}") - body_to_send = body.copy() - body_to_send["model"] = model_name - - try: - if "metadata" not in body_to_send or not isinstance(body_to_send["metadata"], dict): - body_to_send["metadata"] = {} - body_to_send["metadata"]["trace_name"] = "agent-completion" + # Handle streaming vs non-streaming proxying (LiteLLM handles fallback internally) + proxy_start = time.time() - if body.get("stream", False): - logger.info(f"Proxying streaming to LiteLLM as model={model_name}") - req = client.build_request("POST", f"{backend_api_base}/chat/completions", json=body_to_send, headers=headers) - r = await client.send(req, stream=True) - if r.status_code == 200: - async def stream_generator(): - """Asynchronous generator that yields streaming chunks from LiteLLM completions response and logs usage stats on completion.""" - completion_chars = 0 - request_tokens = len(json.dumps(body_to_send)) // 4 - try: - async for chunk in r.aiter_bytes(): - completion_chars += len(chunk) - yield chunk - proxy_latency = (time.time() - proxy_start) * 1000.0 - stats["total_proxy_time_ms"] += proxy_latency - stats["avg_proxy_latency_ms"] = stats["total_proxy_time_ms"] / stats["total_requests"] - record_tool_usage(active_tool, request_tokens, completion_chars // 4, model_name, proxy_latency, route="litellm_fallback") - # Finalize LiteLLM span (streaming path) - if litellm_span_obj: - try: - litellm_span_obj.end( - output={"model": model_name, "stream": True}, - metadata={"latency_ms": proxy_latency, "tokens": completion_chars // 4}, - ) - except Exception: - pass - except Exception as ex: - logger.error(f"Stream error: {ex}") - finally: - await r.aclose() - await client.aclose() - return StreamingResponse(stream_generator(), media_type="text/event-stream") + # --- Pre-screening: clamp max_tokens to fit within downstream model context limits --- + try: + body_to_send = body.copy() + body_to_send["model"] = model_name + requested_max_tokens = body_to_send.get("max_tokens", 4096) + if requested_max_tokens > 32768: # Only intervene for unusually large max_tokens + # Tier-aware minimum context length (from actual roster data): + # - agent-simple-core: 32K (includes tiny liquid/dolphin models) + # - agent-medium-core+: 256K (smallest non-tiny model is nemotron-nano-omni at 256K) + # - ollama-deepseek-v4-*: 1M (DeepSeek V4 native context) + _tier_min_ctx = { + "agent-simple-core": 32768, + "ollama-deepseek-v4-pro": 1000000, + "ollama-deepseek-v4-flash": 1000000, + } + _min_ctx = _tier_min_ctx.get(model_name, 262144) + # Rough input token estimate: 1 token ≈ 4 chars of JSON + _est_input = len(json.dumps(body_to_send)) // 4 + _safe_max = max(1024, _min_ctx - _est_input - 2048) # 2K safety margin + if requested_max_tokens > _safe_max: + logger.warning( + f"⛔ Clamping max_tokens: {requested_max_tokens} → {_safe_max} " + f"(est_input={_est_input}, min_ctx={_min_ctx}, tier={model_name})" + ) + body_to_send["max_tokens"] = _safe_max + except Exception as e: + logger.warning(f"Pre-screening failed (non-fatal): {e}") + body_to_send = body.copy() + body_to_send["model"] = model_name + + should_close_client = True + try: + if "metadata" not in body_to_send or not isinstance(body_to_send["metadata"], dict): + body_to_send["metadata"] = {} + body_to_send["metadata"]["trace_name"] = "agent-completion" + + if body.get("stream", False): + logger.info(f"Proxying streaming to LiteLLM as model={model_name}") + req = client.build_request("POST", f"{backend_api_base}/chat/completions", json=body_to_send, headers=headers) + r = await client.send(req, stream=True) + if r.status_code == 200: + should_close_client = False + async def stream_generator(): + """Asynchronous generator that yields streaming chunks from LiteLLM completions response and logs usage stats on completion.""" + completion_chars = 0 + request_tokens = len(json.dumps(body_to_send)) // 4 + try: + async for chunk in r.aiter_bytes(): + completion_chars += len(chunk) + yield chunk + proxy_latency = (time.time() - proxy_start) * 1000.0 + stats["total_proxy_time_ms"] += proxy_latency + stats["avg_proxy_latency_ms"] = stats["total_proxy_time_ms"] / stats["total_requests"] + record_tool_usage(active_tool, request_tokens, completion_chars // 4, model_name, proxy_latency, route="litellm_fallback") + # Finalize LiteLLM span (streaming path) + if litellm_span_obj: + try: + litellm_span_obj.end( + output={"model": model_name, "stream": True}, + metadata={"latency_ms": proxy_latency, "tokens": completion_chars // 4}, + ) + except Exception: + pass + except Exception as ex: + logger.error(f"Stream error: {ex}") + finally: + await r.aclose() + await client.aclose() + return StreamingResponse(stream_generator(), media_type="text/event-stream") + else: + error_body = await r.aread() if r else b"" + logger.warning(f"LiteLLM stream failed ({r.status_code}): {error_body[:300]}") + await r.aclose() + raise HTTPException(status_code=r.status_code, detail=f"LiteLLM failed: {error_body[:300].decode('utf-8', errors='ignore')}") else: - error_body = await r.aread() if r else b"" - logger.warning(f"LiteLLM stream failed ({r.status_code}): {error_body[:300]}") - await r.aclose(); await client.aclose() - raise HTTPException(status_code=502, detail=f"LiteLLM failed: {r.status_code}") - else: - logger.info(f"Proxying to LiteLLM as model={model_name}") - response = await client.post(f"{backend_api_base}/chat/completions", json=body_to_send, headers=headers) - await client.aclose() - if response.status_code == 200: - proxy_latency = (time.time() - proxy_start) * 1000.0 - stats["total_proxy_time_ms"] += proxy_latency - stats["avg_proxy_latency_ms"] = stats["total_proxy_time_ms"] / stats["total_requests"] - resp_json = response.json() - usage = resp_json.get("usage", {}) - prompt_tokens = usage.get("prompt_tokens", len(json.dumps(body_to_send)) // 4) - completion_tokens = usage.get("completion_tokens", len(json.dumps(resp_json)) // 4) - record_tool_usage(active_tool, prompt_tokens, completion_tokens, model_name, proxy_latency, route="litellm_fallback") - # Finalize LiteLLM span (non-streaming path) - if litellm_span_obj: - try: - litellm_span_obj.end( - output={"model": model_name, "tokens": completion_tokens}, - metadata={"latency_ms": proxy_latency}, - ) - except Exception: - pass - return resp_json + logger.info(f"Proxying to LiteLLM as model={model_name}") + response = await client.post(f"{backend_api_base}/chat/completions", json=body_to_send, headers=headers) + if response.status_code == 200: + proxy_latency = (time.time() - proxy_start) * 1000.0 + stats["total_proxy_time_ms"] += proxy_latency + stats["avg_proxy_latency_ms"] = stats["total_proxy_time_ms"] / stats["total_requests"] + resp_json = response.json() + usage = resp_json.get("usage", {}) + prompt_tokens = usage.get("prompt_tokens", len(json.dumps(body_to_send)) // 4) + completion_tokens = usage.get("completion_tokens", len(json.dumps(resp_json)) // 4) + record_tool_usage(active_tool, prompt_tokens, completion_tokens, model_name, proxy_latency, route="litellm_fallback") + # Finalize LiteLLM span (non-streaming path) + if litellm_span_obj: + try: + litellm_span_obj.end( + output={"model": model_name, "tokens": completion_tokens}, + metadata={"latency_ms": proxy_latency}, + ) + except Exception: + pass + return resp_json + else: + logger.warning(f"LiteLLM failed ({response.status_code}): {response.text[:300]}") + raise HTTPException(status_code=response.status_code, detail=f"LiteLLM failed: {response.text[:300]}") + except HTTPException: + raise + except Exception as exc: + logger.error(f"httpx call failed: {exc}") + raise HTTPException(status_code=502, detail=f"Proxy call failed: {exc}") from exc + finally: + if should_close_client: + await client.aclose() + + if should_try_ollama: + # --- Router-side Ollama cooldown check --- + # Skip Ollama entirely if we're in a cooldown window from a prior failure. + # This is the primary rate-limit protection: LiteLLM's deployment-level + # cooldown is unreliable for single-deployment model groups in CE. + global _ollama_cooldown_until + now_mono = time.monotonic() + if now_mono < _ollama_cooldown_until: + remaining = int(_ollama_cooldown_until - now_mono) + logger.warning( + f"⏳ Ollama cooldown active ({remaining}s remaining), " + f"skipping {target_model}" + ) + if client_model in ("llm-routing-auto-ollama", "llm-routing-auto-agy-ollama"): + # Auto mode: silently fall through to the free tier + logger.info(f"Auto-mode fallback: {target_model} → {original_target_model} (Ollama cooled down)") + return await execute_proxy(original_target_model) else: - logger.warning(f"LiteLLM failed ({response.status_code}): {response.text[:300]}") - raise HTTPException(status_code=502, detail=f"LiteLLM failed: {response.status_code}") - except HTTPException: - raise - except Exception as e: - logger.error(f"Exception during LiteLLM proxy: {e}") - await client.aclose() - raise HTTPException(status_code=502, detail="LiteLLM upstream failed") + # Direct/fallback llm-routing-ollama: return 429 so LiteLLM + # skips this model group and moves to openrouter-auto + raise HTTPException( + status_code=429, + detail=f"Ollama backend cooled down ({remaining}s remaining)" + ) + + try: + result = await execute_proxy(target_model) + return result + except HTTPException as e: + if e.status_code in (429, 503, 502): + # Ollama failure — activate router-side cooldown + _ollama_cooldown_until = time.monotonic() + OLLAMA_COOLDOWN_SECONDS + logger.error( + f"🧊 Ollama failed ({e.status_code}), activating {OLLAMA_COOLDOWN_SECONDS}s cooldown" + ) + if client_model in ("llm-routing-auto-ollama", "llm-routing-auto-agy-ollama"): + logger.warning(f"Ollama proxy failed ({e.detail}), falling back to free tier {original_target_model}") + return await execute_proxy(original_target_model) + else: + # Direct/fallback llm-routing-ollama request: return 429 to + # signal LiteLLM to skip this model group in the fallback chain + logger.error(f"Ollama proxy failed ({e.detail}) for direct/fallback request, returning 429") + raise HTTPException( + status_code=429, + detail="Ollama backend rate limited/unavailable" + ) from e + except Exception as e: + # Unexpected error — also cooldown to prevent hammering + _ollama_cooldown_until = time.monotonic() + OLLAMA_COOLDOWN_SECONDS + logger.error( + f"🧊 Ollama unexpected error ({e}), activating {OLLAMA_COOLDOWN_SECONDS}s cooldown" + ) + if client_model in ("llm-routing-auto-ollama", "llm-routing-auto-agy-ollama"): + logger.warning(f"Ollama proxy error ({e}), falling back to free tier {original_target_model}") + return await execute_proxy(original_target_model) + else: + raise HTTPException( + status_code=429, + detail="Ollama backend rate limited/unavailable" + ) from e + else: + return await execute_proxy(target_model) @app.get("/metrics") async def metrics(): @@ -1645,6 +1727,16 @@ async def metrics(): lines.append("# HELP circuit_breaker_total_trips Total trips across both breakers") lines.append("# TYPE circuit_breaker_total_trips counter") lines.append(f"circuit_breaker_total_trips {google['total_trips'] + vendor['total_trips']}") + + # Ollama router-side cooldown metrics + _now_mono = time.monotonic() + _ollama_remaining = max(0.0, _ollama_cooldown_until - _now_mono) + lines.append("# HELP ollama_cooldown_active Whether Ollama is in router-side cooldown (1=active)") + lines.append("# TYPE ollama_cooldown_active gauge") + lines.append(f"ollama_cooldown_active {int(_ollama_remaining > 0)}") + lines.append("# HELP ollama_cooldown_remaining_seconds Seconds remaining in Ollama cooldown") + lines.append("# TYPE ollama_cooldown_remaining_seconds gauge") + lines.append(f"ollama_cooldown_remaining_seconds {_ollama_remaining:.0f}") return Response(content="\n".join(lines), media_type="text/plain; version=0.0.4") @@ -2647,6 +2739,7 @@ class AnnotationItem(BaseModel): ts: Optional[str] = None VALID_TIERS = {"agent-simple-core", "agent-medium-core", "agent-complex-core", "agent-reasoning-core", "agent-advanced-core"} +annotations_lock = asyncio.Lock() @app.post("/dashboard/save-annotations") async def save_annotations(payload: Dict[str, AnnotationItem]): @@ -2696,19 +2789,20 @@ async def save_annotations(payload: Dict[str, AnnotationItem]): try: ann_path = DATA_DIR / "annotations.json" existing = {} - if ann_path.exists(): - try: - import json - with open(ann_path, "r", encoding="utf-8") as f: - existing = json.load(f) - except Exception as read_err: - logger.warning(f"Could not read existing annotations: {read_err}. Overwriting.") - - # Merge new annotations into existing - for k, item in payload.items(): - existing[k] = item.model_dump() if hasattr(item, "model_dump") else item.dict() + async with annotations_lock: + if ann_path.exists(): + try: + import json + with open(ann_path, "r", encoding="utf-8") as f: + existing = json.load(f) + except Exception as read_err: + logger.warning(f"Could not read existing annotations: {read_err}. Overwriting.") - await _atomic_write_json_async(str(ann_path), existing) + # Merge new annotations into existing + for k, item in payload.items(): + existing[k] = item.model_dump() if hasattr(item, "model_dump") else item.dict() + + await _atomic_write_json_async(str(ann_path), existing) return JSONResponse({"status": "ok", "saved": len(payload)}) except Exception as e: logger.error(f"Failed to save annotations: {e}") diff --git a/scripts/extract_gapfill.py b/scripts/extract_gapfill.py index e5b41fd3..b01a7bcd 100644 --- a/scripts/extract_gapfill.py +++ b/scripts/extract_gapfill.py @@ -46,8 +46,10 @@ def extract_user_prompt(obs): if not inp: return None if isinstance(inp, str): - try: inp = json.loads(inp) - except: return None + try: + inp = json.loads(inp) + except json.JSONDecodeError: + return None if not isinstance(inp, dict): return None messages = inp.get('messages', []) diff --git a/scripts/extract_prompts.py b/scripts/extract_prompts.py index 028baa8d..789da769 100644 --- a/scripts/extract_prompts.py +++ b/scripts/extract_prompts.py @@ -143,7 +143,7 @@ def extract_first_user_prompt(obs): lengths = [len(p['prompt']) for p in prompts] if lengths: print(f"Length: min={min(lengths)}, max={max(lengths)}, median={sorted(lengths)[len(lengths)//2]}, avg={sum(lengths)/len(lengths):.0f}") - print(f"Short (<100 chars): {sum(1 for l in lengths if l < 100)}") - print(f"\nSample (first 10):") + print(f"Short (<100 chars): {sum(1 for length in lengths if length < 100)}") + print("\nSample (first 10):") for p in prompts[:10]: print(f" [{p['timestamp'][:19]}] ({len(p['prompt'])}c) {p['prompt'][:120]}...") diff --git a/scripts/reclassify_all.py b/scripts/reclassify_all.py index 91ea6b61..c2bce257 100644 --- a/scripts/reclassify_all.py +++ b/scripts/reclassify_all.py @@ -38,14 +38,6 @@ def classify(prompt): with open(data_dir / 'classified_dataset.json') as f: dataset = json.load(f) -# Load raw prompts for full text -with open(data_dir / 'raw_prompts_hermes.json') as f: - all_prompts = json.load(f) - -# Build prompt lookup -prompt_map = {} -for p in all_prompts: - prompt_map[p['prompt']] = p print(f"Classifying {len(dataset['prompts'])} prompts with gemma4-26a4b (grammar-enforced)...") diff --git a/scripts/retry_errors.py b/scripts/retry_errors.py index eff8be8c..5568c698 100644 --- a/scripts/retry_errors.py +++ b/scripts/retry_errors.py @@ -1,5 +1,5 @@ """Retry the 94 failed prompts with 800-char truncation (safe for 4096-ctx model).""" -import json, urllib.request, time, subprocess +import json, urllib.request, time, subprocess, tempfile, os from pathlib import Path from collections import Counter @@ -65,8 +65,6 @@ def classify(prompt): data_dir = Path(__file__).resolve().parent.parent / "data" with open(data_dir / "classified_dataset.json") as f: dataset = json.load(f) -with open(data_dir / "raw_prompts_hermes.json") as f: - all_prompts = json.load(f) def is_error_val(val): if not val: @@ -110,8 +108,11 @@ def is_error_val(val): dataset['gaps'] = [t for t in ['agent-simple-core','agent-medium-core','agent-complex-core','agent-reasoning-core','agent-advanced-core'] if new_counts.get(t, 0) < 20] -with open(data_dir / "classified_dataset.json", 'w') as f: - json.dump(dataset, f, indent=2, ensure_ascii=False) +dest_path = data_dir / "classified_dataset.json" +with tempfile.NamedTemporaryFile("w", dir=str(data_dir), delete=False, encoding="utf-8") as tmp_f: + json.dump(dataset, tmp_f, indent=2, ensure_ascii=False) + tmp_name = tmp_f.name +os.replace(tmp_name, str(dest_path)) print(f"\nDone. Fixed: {fixed}, Errors: {errors}") for tier in sorted(new_counts.keys()): diff --git a/start-stack.sh b/start-stack.sh index 21ac9496..2d9e8ae5 100755 --- a/start-stack.sh +++ b/start-stack.sh @@ -9,7 +9,7 @@ set -e # (for router/Containerfile changes) # Set working directory -WORKDIR="/home/gpav/Vrac/LAB/AI/LLM-Routing" +WORKDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" cd "$WORKDIR" # Ensure local volume directories exist on the host for Podman mounts @@ -301,13 +301,13 @@ if podman pod exists agent-router-pod 2>/dev/null; then podman build -t localhost/llm-triage-router:latest -f router/Containerfile router safe_pod_teardown echo "🚀 Deploying fresh triage pod..." - podman play kube "$WORKDIR/pod.yaml" + cat "$WORKDIR/pod.yaml" | sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|$WORKDIR|g" | podman play kube - setup_minio_buckets verify_stack_health elif $REPLACE_MODE; then safe_pod_teardown echo "🚀 Deploying replacement pod from YAML..." - podman play kube "$WORKDIR/pod.yaml" + cat "$WORKDIR/pod.yaml" | sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|$WORKDIR|g" | podman play kube - setup_minio_buckets verify_stack_health else @@ -333,7 +333,7 @@ else podman build -t localhost/llm-triage-router:latest -f router/Containerfile router echo "🚀 No existing pod found. Deploying fresh triage pod..." - podman play kube "$WORKDIR/pod.yaml" + cat "$WORKDIR/pod.yaml" | sed "s|/home/gpav/Vrac/LAB/AI/LLM-Routing|$WORKDIR|g" | podman play kube - setup_minio_buckets verify_stack_health fi