fix: address all bot review comments -- structured content, rsync safety, exception hygiene

README.md

@@ -77,12 +77,12 @@ All core containers are configured with **Kubernetes-style liveness and readines
 | Container | Liveness Probe | Readiness Probe |
 |:---|---:|---:|
 | **valkey-cache** | `valkey-cli -p <port> ping` every 10s | Same, every 5s |
-| **litellm-gateway** | Python `urllib` GET `/ping` (port 4000) every 15s | Python `urllib` GET `/health/readiness` (port 4000) every 10s |
+| **litellm-gateway** | Python `urllib` GET `/health/liveness` (port 4000) every 15s | Python `urllib` GET `/health/readiness` (port 4000) every 10s |
 | **llm-triage-router** | Python `urllib` GET `/metrics` (port 5000) every 15s | Same, every 10s |
 | **postgres-db** | `pg_isready -U postgres -p <port>` every 10s | Same, every 5s |
 | **clickhouse-db** | `clickhouse-client --user clickhouse --password <generated> --query "SELECT 1"` every 15s | `clickhouse-client --user clickhouse --password <generated> --query "SELECT 1"` every 10s |
 | **valkey-lf** | `valkey-cli -p <port> -a <auth> ping` every 10s | Same, every 5s |
-| **langfuse-web** | `wget` GET `/api/health` (port 3001) every 15s | Same, every 10s |
+| **langfuse-web** | `wget` GET `/api/public/health` (port 3001) every 15s | Same, every 10s |
 | **langfuse-worker** | `pgrep node` every 15s | — |
 | **minio-s3** | `httpGet` `/minio/health/live` (port 9002) every 15s | `httpGet` `/minio/health/ready` (port 9002) every 10s |
 
@@ -234,6 +234,7 @@ All configurations, automation scripts, and databases are self-contained within
 │   ├── backup.sh        # Database backup with pg_isready retry logic
 │   ├── benchmark_classifier.py # Classifier accuracy & latency benchmarks
 │   ├── benchmark_tokens.py     # Token-count ground-truth comparisons
+│   ├── upgrade-prod.sh  # Release-driven prod sync & redeploy
 │   └── verification/    # Live-stack E2E verification scripts
 ├── tests/               # Project-wide unit & integration tests
 ├── data/                # Reference datasets for benchmarks & tests
@@ -284,7 +285,7 @@ Orchestrates routing fallback chains, Redis caching, and telemetry callbacks:
   - `embedding_model: "local-nomic-embed"` — uses the local nomic-embed model (no API costs)
   - `collection_name: "litellm_semantic_cache"` — stores embeddings for similarity-based cache lookups
 - **Cascading Fallback Chains** (configured in `litellm_settings.fallbacks`):
-  Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT.
+  Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`).
 
   ```mermaid
   graph TD
@@ -822,6 +823,32 @@ Execute the verification script:
 ./scripts/verification/verify_reasoning_tiers.py
 ```
 
+### Canonical Endpoint Verification
+
+A comprehensive endpoint health check that validates all services across both prod and dev environments:
+
+```bash
+# Prod (default)
+python scripts/verification/verify_canonical_endpoints.py
+
+# Dev
+python scripts/verification/verify_canonical_endpoints.py --dev
+```
+
+Tests cover:
+
+| Section | Endpoints |
+|---------|-----------|
+| Router API | `/v1/models`, `/metrics`, `/dashboard`, `/api/dashboard-stats`, `/visualizer` |
+| LiteLLM | `/health/liveness`, `/health/readiness`, `/v1/models`, `/llm-routing/litellm/ui/` |
+| Langfuse | `/api/public/health`, `/` (web UI) |
+| Infrastructure | MinIO `/minio/health/live`, ClickHouse `/ping` |
+| E2E chat | 3 completions through triage router |
+| LiteLLM direct | 1 completion directly to LiteLLM |
+| Canonical URLs | 6 GET + 1 POST through public HTTPS (graceful DNS skip) |
+
+Requires `PUBLIC_BASE_URL` in `.env` for canonical URL tests. Dev `.env.dev` already has it; prod `.env` should include `PUBLIC_BASE_URL="https://x570.vendeuvre.lan/llm-routing"`.
+
 ## 10. Performance Benchmarks
 
 Through our local benchmarks, the following performance characteristics have been achieved:

pod.yaml

@@ -53,6 +53,10 @@ spec:
       value: http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER
     - name: LITELLM_MASTER_KEY
       value: LITELLM_MASTER_KEY_PLACEHOLDER
+    - name: LITELLM_UI_USERNAME
+      value: LITELLM_UI_USERNAME_PLACEHOLDER
+    - name: LITELLM_UI_PASSWORD
+      value: LITELLM_UI_PASSWORD_PLACEHOLDER
     - name: OLLAMA_API_KEY
       value: OLLAMA_API_KEY_PLACEHOLDER
     - name: SERVER_ROOT_PATH
@@ -69,7 +73,7 @@ spec:
         command:
         - python3
         - -c
-        - import urllib.request; urllib.request.urlopen('http://localhost:LITELLM_PORT_PLACEHOLDER/ping')
+        - import urllib.request; urllib.request.urlopen('http://localhost:LITELLM_PORT_PLACEHOLDER/health/liveness')
       initialDelaySeconds: 240
       periodSeconds: 15
       timeoutSeconds: 5
@@ -318,7 +322,7 @@ spec:
     - name: NEXTAUTH_SECRET
       value: NEXTAUTH_SECRET_PLACEHOLDER
     - name: NEXTAUTH_URL
-      value: http://localhost:LANGFUSE_WEB_PORT_PLACEHOLDER
+      value: NEXTAUTH_URL_PLACEHOLDER
     - name: SALT
       value: SALT_PLACEHOLDER
     - name: ENCRYPTION_KEY
@@ -379,7 +383,7 @@ spec:
         - -q
         - -O
         - /dev/null
-        - http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/health
+        - http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/public/health
       initialDelaySeconds: 240
       periodSeconds: 15
       timeoutSeconds: 5
@@ -391,7 +395,7 @@ spec:
         - -q
         - -O
         - /dev/null
-        - http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/health
+        - http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/public/health
       initialDelaySeconds: 15
       periodSeconds: 10
       timeoutSeconds: 5

scripts/README.md

@@ -53,6 +53,7 @@ A simple HTTP server that returns `429 Rate Limit Exceeded` to simulate rate lim
 These tools are used to benchmark the prompt classifier and extract datasets from Langfuse traces:
 
 - **`benchmark_classifier.py`**: Benchmarks latency and precision metrics of the Ryzen PRO APU-offloaded classifier.
+- **`chat_helpers.py`**: Shared defensive chat completion response parser (`parse_chat_response`) used by classifier scripts, verification scripts, and canonical endpoint checks. Safely extracts content and reasoning_content with full isinstance guards.
 - **`classify_direct.py`**: Takes a string prompt argument and prints the classification decision directly.
 - **`extract_prompts.py` / `extract_complex.py` / `extract_gapfill.py`**: Mines prompt datasets from Langfuse PG/ClickHouse database traces for fine-tuning.
 - **`reclassify_all.py`**: Re-evaluates prompt classifications against updated models.

scripts/__init__.py

@@ -0,0 +1 @@
+# LLM-Routing scripts package

scripts/benchmark_classifier.py

@@ -6,6 +6,10 @@
 from collections import defaultdict, Counter
 from pathlib import Path
 
+# Shared chat response parser (used by verification scripts too)
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from scripts.chat_helpers import parse_chat_response
+
 # Load dataset
 dataset_path = Path(__file__).resolve().parent.parent / "data" / "classified_dataset.json"
 with open(dataset_path) as f:
@@ -42,10 +46,8 @@ def classify(prompt):
     )
     with urllib.request.urlopen(req, timeout=30) as resp:
         data = json.loads(resp.read())
-    choices = data.get("choices", [])
-    if not choices:
-        return "ERROR"
-    return choices[0].get("message", {}).get("content", "").strip()
+    content, _ = parse_chat_response(data)
+    return content if content else "ERROR"
 
 total = len(dataset.get("prompts", []))
 print(f"Benchmark: gemma4-26a4b-routing vs {total} labeled prompts\n")

scripts/chat_helpers.py

@@ -0,0 +1,66 @@
+"""Shared helpers for defensive chat completion response parsing.
+
+Used by the canonical endpoint verification script and the classifier scripts
+to safely extract content and reasoning_content from OpenAI-compatible API responses.
+"""
+
+from typing import Any
+
+
+def _normalize_chat_content(value: Any) -> str:
+    """Normalize structured content payloads into a plain string.
+
+    Handles plain strings, lists of strings, lists of dicts with
+    ``text`` / ``content`` keys, and dicts with ``text`` / ``content`` keys.
+    Returns an empty string for None or unrecognised shapes.
+    """
+    if value is None:
+        return ""
+    if isinstance(value, str):
+        return value.strip()
+    if isinstance(value, list):
+        parts: list[str] = []
+        for item in value:
+            if isinstance(item, str):
+                parts.append(item)
+            elif isinstance(item, dict):
+                text = item.get("text")
+                if isinstance(text, str):
+                    parts.append(text)
+                elif "content" in item:
+                    nested = _normalize_chat_content(item.get("content"))
+                    if nested:
+                        parts.append(nested)
+        return "".join(parts).strip()
+    if isinstance(value, dict):
+        text = value.get("text")
+        if isinstance(text, str):
+            return text.strip()
+        if "content" in value:
+            return _normalize_chat_content(value.get("content"))
+    return ""
+
+
+def parse_chat_response(data: Any) -> tuple[str, str]:
+    """Safely extract content and reasoning_content from a chat completion response.
+
+    Args:
+        data: Parsed JSON response from a chat completion endpoint (expected to be a dict).
+
+    Returns:
+        (content, reasoning_content) — both may be empty strings.
+    """
+    if not isinstance(data, dict):
+        return "", ""
+    choices = data.get("choices")
+    if not isinstance(choices, list) or not choices:
+        return "", ""
+    first_choice = choices[0]
+    if not isinstance(first_choice, dict):
+        return "", ""
+    message = first_choice.get("message")
+    if not isinstance(message, dict):
+        return "", ""
+    content = _normalize_chat_content(message.get("content"))
+    reasoning = _normalize_chat_content(message.get("reasoning_content"))
+    return content, reasoning

scripts/classify_direct.py

@@ -1,8 +1,12 @@
 """Direct classification of Hermes prompts using gemma4-26a4b-routing."""
 import os
-import json, urllib.request, time
+import json, urllib.request, time, sys
 from pathlib import Path
 
+# Shared chat response parser (used by verification scripts too)
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from scripts.chat_helpers import parse_chat_response
+
 PROMPT_TEMPLATE = """Classify the coding task complexity. Output ONLY the tier name.
 
 agent-simple-core: trivial one-liners, syntax fixes, single-line edits
@@ -33,7 +37,8 @@ def classify(prompt):
     )
     with urllib.request.urlopen(req, timeout=30) as resp:
         data = json.loads(resp.read())
-    return data["choices"][0]["message"].get("content", "").strip()
+    content, _ = parse_chat_response(data)
+    return content if content else "ERROR"
 
 # Load prompts
 data_dir = Path(__file__).resolve().parent.parent / "data"

scripts/reclassify_all.py

@@ -7,6 +7,10 @@
 from pathlib import Path
 from collections import Counter
 
+# Shared chat response parser (used by verification scripts too)
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from scripts.chat_helpers import parse_chat_response
+
 TIERS = ['agent-simple-core','agent-medium-core','agent-complex-core','agent-reasoning-core','agent-advanced-core']
 
 LLAMA_SERVER_URL = "http://127.0.0.1:8080/v1/chat/completions"
@@ -32,10 +36,8 @@ def classify(prompt):
     req = urllib.request.Request(LLAMA_SERVER_URL, data=json.dumps(payload).encode(), headers={'Content-Type':'application/json','Authorization': f'Bearer {os.environ.get("ROUTER_API_KEY", "local-token")}'})
     with urllib.request.urlopen(req, timeout=30) as resp:
         data = json.loads(resp.read())
-    choices = data.get('choices', [])
-    if not choices:
-        return "ERROR: empty response"
-    return choices[0].get('message', {}).get('content', '').strip()
+    content, _ = parse_chat_response(data)
+    return content if content else "ERROR: empty response"
 
 # Load existing dataset (kanban/llm evals)
 data_dir = Path(__file__).resolve().parent.parent / 'data'

scripts/retry_errors.py

@@ -1,8 +1,12 @@
 """Retry the 94 failed prompts with 800-char truncation (safe for 4096-ctx model)."""
-import json, urllib.request, time, tempfile, os
+import json, urllib.request, time, tempfile, os, sys
 from pathlib import Path
 from collections import Counter
 
+# Shared chat response parser (used by verification scripts too)
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from scripts.chat_helpers import parse_chat_response
+
 PROMPT_TEMPLATE = """Classify the coding task complexity. Output ONLY the tier name.
 
 agent-simple-core: trivial one-liners, syntax fixes, single-line edits
@@ -50,11 +54,10 @@ def classify(prompt):
     )
     with urllib.request.urlopen(req, timeout=30) as resp:
         data = json.loads(resp.read())
-    choices = data.get("choices", [])
-    content = ""
-    if choices:
-        content = choices[0].get("message", {}).get("content", "").strip()
-    # Normalize: strip "tier:" prefix, extract just the tier name
+    content, _ = parse_chat_response(data)
+    if not content:
+        return "ERROR"
+    # retry_errors specific: normalize tier name
     for tier in TIERS:
         if tier in content:
             return tier