🔒 Fix hardcoded default passwords in pod.yaml

.github/dependabot.yml

@@ -37,9 +37,9 @@ updates:
     #   - dockerhub
     ignore:
       # Example: ignore semver-major updates for Postgres and ClickHouse while testing
-      - dependency-name: "pgvector/pgvector"
+      - dependency-name: "docker.io/pgvector/pgvector"
         update-types: ["version-update:semver-major"]
-      - dependency-name: "clickhouse/clickhouse-server"
+      - dependency-name: "docker.io/clickhouse/clickhouse-server"
         update-types: ["version-update:semver-major"]
 
   # Dockerfiles (keeps Dockerfile FROM lines updated)

.github/workflows/test.yml

@@ -26,10 +26,10 @@ jobs:
         run: pip install httpx==0.28.1 pytest pytest-asyncio anyio pyyaml fastapi "pydantic>=2.0,<3.0" uvicorn python-multipart asyncpg langfuse redis
 
       - name: Run Unit Tests
-        run: CONFIG_PATH=router/config.yaml PYTHONPATH=.:router pytest --ignore=tests/test_agy_behavior.py --ignore=tests/test_agy_tiers.py
+        run: CONFIG_PATH=router/config.yaml PYTHONPATH=.:router pytest --ignore=test_agy_behavior.py --ignore=test_agy_tiers.py
 
       - name: Run Breaker Verification
-        run: python3 scripts/verification/verify_breaker.py
+        run: python3 verify_breaker.py
 
       - name: Run Integration Verification
-        run: python3 tests/test_a2_verify.py
+        run: python3 test_a2_verify.py

.gitignore

@@ -32,11 +32,8 @@ router/free_models_roster.json
 # agent artifacts
 .hermes/
 .jules/
-.local/
 workdirs/
 pr_description.txt
 
 # Dataset work in progress
 data/
-.cache/
-test_output*.log

README.md

@@ -66,6 +66,8 @@ graph TD
     style QwenLocal fill:#f0f0f0,stroke:#999,stroke-width:1px;
 ```
 
+> **Version Pin**: LiteLLM Gateway runs `ghcr.io/berriai/litellm:v1.88.0`. See §3B for pinning policy.
+
 ---
 
 ## 1b. Container Health Checks & Auto-Restart
@@ -74,15 +76,15 @@ All core containers are configured with **Kubernetes-style liveness and readines
 
 | Container | Liveness Probe | Readiness Probe |
 |:---|---:|---:|
-| **valkey-cache** | `tcpSocket` on port 6379 every 10s | Same, every 5s |
-| **litellm-gateway** | Python `urllib` GET `/ping` (port 4000) every 15s | Python `urllib` GET `/health/readiness` (port 4000) every 10s |
-| **llm-triage-router** | Python `urllib` GET `/metrics` (port 5000) every 15s | Same, every 10s |
+| **valkey-cache** (9.1.0-alpine) | `valkey-cli PING` every 10s | `valkey-cli PING` every 5s |
+| **litellm-gateway** | Python `urllib` GET `/health` (port 4000, accepts 200/401) every 15s | Same, every 10s |
+| **llm-triage-router** | Python `urllib` GET `/dashboard` (port 5000) every 15s | Same, every 10s |
 | **postgres-db** | `pg_isready -U postgres` every 10s | Same, every 5s |
-| **clickhouse-db** | `clickhouse-client --user clickhouse --password clickhouse --query "SELECT 1"` every 15s | `clickhouse-client --query "SELECT 1"` every 10s |
-| **valkey-lf** | `tcpSocket` on port 6380 every 10s | Same, every 5s |
-| **langfuse-web** | `wget` GET `/api/health` (port 3001) every 15s | Same, every 10s |
-| **langfuse-worker** | `pgrep node` every 15s | — |
-| **minio-s3** | `httpGet` `/minio/health/live` (port 9002) every 15s | `httpGet` `/minio/health/ready` (port 9002) every 10s |
+| **clickhouse-db** | `clickhouse-client --user clickhouse --password clickhouse --query "SELECT 1"` every 15s | Same, every 10s |
+| **valkey-lf** | `redis-cli -p 6380 -a langfuse-redis-2026 PING` every 10s | Same, every 5s |
+| **langfuse-web** | `wget -qO /dev/null http://127.0.0.1:3001/` every 15s | Same, every 10s |
+| **langfuse-worker** | `pgrep -f langfuse-worker` every 15s | — |
+| **minio-s3** | TCP socket check on port 9002 every 15s | Same, every 10s |
 
 The pod-level `restartPolicy: Always` combined with these probes means Podman will restart any container that fails its health check or exits unexpectedly, enabling true self-healing for the entire stack.
 
@@ -210,8 +212,8 @@ The gateway supports multiple routing modes controlled by the `model` field:
 All configurations, automation scripts, and databases are self-contained within this repository directory:
 
 ```
-LLM-Routing/
-├── .env                 # Environment file for API keys, passwords, and generated secrets (ignored by git)
+/home/gpav/Vrac/LAB/AI/LLM-Routing/
+├── .env                 # Environment file for OpenRouter API Key (ignored by git)
 ├── .gitignore           # Git ignore policy protecting secrets & database files
 ├── README.md            # In-depth system and operational guide
 ├── pod.yaml             # Podman Kubernetes template defining the 10-container stack
@@ -226,26 +228,16 @@ LLM-Routing/
 │   ├── agy_proxy.py     # 3-tier agy fallback with session continuation
 │   ├── circuit_breaker.py # Exponential cooldown breaker for agy proxy
 │   └── memory_mcp.py    # MCP bridge server for Goose memory integration
-├── scripts/             # Automation, maintenance, and verification scripts
-│   ├── backup.sh        # Database backup with pg_isready retry logic
-│   ├── host_agy_daemon.py # Real-time PTY-based streaming daemon for agy
-│   ├── sync_gemini_token.py # Extraction/sync script for keyring credentials
-│   ├── get_pr_status.py # PR state helper
-│   ├── watch_quota.sh   # Quota reset watcher
-│   ├── test_quota_reset.sh # Quota reset test simulator
-│   └── verification/    # Routing & cooldown verification tests
-│       ├── verify_breaker.py # Circuit breaker verification
-│       └── ...
-├── tests/               # Integration & unit test suite
-│   ├── test_agy_tiers.py # agy proxy model tier test suite
-│   ├── test_classifier_accuracy.py # Classifier accuracy benchmark
-│   └── ...
+├── scripts/
+│   └── backup.sh        # Database backup with pg_isready retry logic
 ├── backups/             # Timestamped PostgreSQL dumps + config snapshots
 ├── valkey-data/         # [Git Ignored] Persistent memory volumes for Valkey Cache
 ├── postgres-data/       # [Git Ignored] Persistent tables for PostgreSQL
 ├── clickhouse-data/     # [Git Ignored] Persistent traces for Langfuse v3
 ├── valkey-lf-data/      # [Git Ignored] Persistent job queues for Langfuse v3
-└── minio-data/          # [Git Ignored] S3-compatible event storage for Langfuse v3
+├── minio-data/          # [Git Ignored] S3-compatible event storage for Langfuse v3
+├── test_agy_tiers.py    # agy proxy model tier test suite
+└── test_classifier_accuracy.py # Classifier accuracy benchmark
 ```
 
 ---
@@ -277,8 +269,7 @@ Exposes the entry endpoint (`http://localhost:5000/v1`) and evaluates prompt com
 > Model capabilities, token limits, and costs are visible in LiteLLM's Model Hub Table at `http://localhost:4000/ui/?page=model-hub-table` (or port 4000 on the gateway host).
 
 ### B. LiteLLM Proxy Gateway (`litellm/config.yaml`)
-- **Version Pinning**: The tags are explicitly pinned in `pod.yaml` — never use `:latest`. Check available tags with `skopeo list-tags docker://ghcr.io/repo/image` before upgrading.
-
+- **Version Pinning**: The LiteLLM gateway runs `ghcr.io/berriai/litellm:v1.88.0` (latest stable as of June 2026). The tag is explicitly pinned in `pod.yaml` — never use `:latest`. Check available tags with `skopeo list-tags docker://ghcr.io/berriai/litellm` before upgrading. ClickHouse runs `docker.io/clickhouse/clickhouse-server:26.5.1` (upgraded from 24.8, June 2026). Valkey Cache runs `docker.io/valkey/valkey:9.1.0-alpine` (upgraded from 8, June 2026).
 Orchestrates routing fallback chains, Redis caching, and telemetry callbacks:
 - **`drop_params: true`**: Automatically strips unsupported arguments when transitioning to models that don't support them.
 - **Request Timeouts (`300s`)**: Provides ample padding to prevent connection aborts during dynamic RAM swapping operations on the local GPU `llama-server`.
@@ -288,7 +279,7 @@ Orchestrates routing fallback chains, Redis caching, and telemetry callbacks:
   - `embedding_model: "local-nomic-embed"` — uses the local nomic-embed model (no API costs)
   - `collection_name: "litellm_semantic_cache"` — stores embeddings for similarity-based cache lookups
 - **Cascading Fallback Chains** (configured in `litellm_settings.fallbacks`):
-  Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). 
+  Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT.
 
   ```mermaid
   graph TD
@@ -388,7 +379,7 @@ Run the startup script from the root of the repository:
                               #   health probes, env vars, containers — no rebuild)
 ./start-stack.sh --full-rebuild  # Full reset: rebuild image + recreate pod
 ```
-*Note: If running for the first time, the script will prompt you for your `OpenRouter API Key`, securely saving it inside `.env` with restrictive permissions (`chmod 600`). The script also automatically generates and persists secure random secrets (`LITELLM_MASTER_KEY`, `POSTGRES_PASSWORD`, `NEXTAUTH_SECRET`, `SALT`, `ENCRYPTION_KEY`, and `ROUTER_API_KEY`) to this file on startup if they are missing.*
+*Note: If running for the first time, the script will prompt you for your `OpenRouter API Key`, securely saving it inside `.env` with restrictive permissions (`chmod 600`).*
 
 ### 2. Verify Container Status
 Check that all **10 containers** inside `agent-router-pod` are up and running:
@@ -584,25 +575,19 @@ Without Minio, Langfuse v3 **will not start** — it validates S3 connectivity a
 |----------|-------|
 | `LANGFUSE_S3_EVENT_UPLOAD_BUCKET` | `langfuse-events` |
 | `LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT` | `http://127.0.0.1:9002` |
-| `LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID` | `minioadmin` |
-| `LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY` | `minioadmin` |
+| `LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID` | (auto-generated in `.env`) |
+| `LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY` | (auto-generated in `.env`) |
 | `S3_FORCE_PATH_STYLE` | `true` |
 
-Minio runs on ports **9001** (web console) and **9002** (S3 API). Credentials: `minioadmin` / `minioadmin`. Image pinned to `docker.io/minio/minio:RELEASE.2025-10-15T17-29-55Z`.
+Minio runs on ports **9001** (web console) and **9002** (S3 API). Credentials are automatically generated and stored in `.env`. Image pinned to `docker.io/minio/minio:RELEASE.2025-10-15T17-29-55Z`.
 
 ### Health Check
 
-MinIO's health is monitored using its native structured endpoints `/minio/health/live` (liveness) and `/minio/health/ready` (readiness) on port 9002:
+Minio's minimal Go image has no HTTP client tools. The probe uses a raw TCP socket check:
 
 ```yaml
-livenessProbe:
-  httpGet:
-    path: /minio/health/live
-    port: 9002
-readinessProbe:
-  httpGet:
-    path: /minio/health/ready
-    port: 9002
+exec:
+  command: [sh, -c, "exec 3<>/dev/tcp/127.0.0.1/9002 && echo ok"]
 ```
 
 ---
@@ -704,6 +689,15 @@ Additional mounts required in `pod.yaml`:
   mountPath: /root/.gemini   # agy expects config at ~/.gemini
 ```
 
+### Model Identifiers (found in agy binary)
+
+| Model | Env Var Value | Backend |
+|-------|---------------|---------|
+| Gemini 3.5 Flash | `""` (auto-select) | Cloud Code Assist (default) |
+| Claude Opus 4.6 | `claude-opus-4-6@default` | Anthropic premium tier |
+| Claude Sonnet 4.5 | `claude-sonnet-4-5@20250929` | Anthropic via Vertex AI |
+| Claude Haiku 4.5 | `claude-haiku-4-5@20251001` | Anthropic lightweight |
+
 ### Verification
 
 ```bash
@@ -719,15 +713,15 @@ agy --print "First message"                    # creates conversation
 agy --conversation <id> --print "Follow-up"    # continues same session
 
 # Run the full tier test suite
-python3 tests/test_agy_tiers.py
+python3 test_agy_tiers.py
 ```
 
 ### 9b. Streaming & Concurrency Optimizations
 
 To support production agentic environments (such as `goose-cli` or similar tools) that require low-latency streaming and high concurrent throughput, the following components were introduced:
 
 #### 1. Real-Time PTY-Based Streaming Bridge for `agy` Response
-To support low-latency streaming for agent clients (such as `goose-cli`), the host-side `scripts/host_agy_daemon.py` runs `agy --print` inside a pseudo-terminal (PTY) using `pty.openpty()`. 
+To support low-latency streaming for agent clients (such as `goose-cli`), the host-side `host_agy_daemon.py` runs `agy --print` inside a pseudo-terminal (PTY) using `pty.openpty()`.
 * Running `agy` inside a PTY disables internal buffering, forcing it to write generated characters/lines progressively.
 * The host daemon streams these chunks in real-time as `application/x-ndjson` lines to the Triage Router.
 * The Triage Router immediately transforms these incoming chunks into standard OpenAI Server-Sent Event (SSE) packets and yields them to the client. This results in a true, low-latency stream with minimal Time-To-First-Token (TTFT) and eliminates synthetic buffering.
@@ -789,15 +783,15 @@ For auto-routing modes, the Triage Router handles failures by silently falling b
 | Triage Evaluation Layer | Latency Footprint | Hardware Offload | Efficiency Ratio |
 | :--- | :---: | :---: | :---: |
 | **Cold-Run Triage** (First query) | ~15 - 24s | Dynamic HF Download | Includes GGUF fetch & initialization |
-| **Warm-Run Triage** (Local inference) | **~449 ms** | 100% GPU | **12x speedup** compared to 35B model |
+| **Warm-Run Triage** (Local inference) | **~449 ms** | 100% Vulkan GPU (Ryzen APU) | **12x speedup** compared to 35B model |
 | **Triage Cache Hit** (Repeat query) | **0.0 ms** | RAM In-Memory TTL | Infinite speedup, zero backend requests |
 | **Valkey Gateway Cache Hit** | **< 10 ms** | Redis RAM Cache | Zero provider cost, immediate response |
 
 ## 11. NotebookLM Companion Knowledge Base
 
 This project is supported by a dedicated NotebookLM companion notebook:
 * **Notebook Name:** `TriageGate-Architect-KB`
-* **Notebook ID:** llm-triage-gateway
+* **Notebook ID:** `llm-triage-gateway`
 * **URL:** [TriageGate-Architect-KB](https://notebooklm.google.com/notebook/826cbd87-7969-4b0e-a38e-5517b5ab7d28)
 
 This notebook contains a comprehensive semantic index of the system architecture, LiteLLM cascades, Langfuse telemetry pipelines, local model configurations, and integration guides. Agents and developers can query this notebook via the `notebooklm` MCP tools (e.g., using `notebook_ask` with `notebook_id: "llm-triage-gateway"`) to retrieve structured knowledge, check pitfalls, or get implementation examples for this gateway stack.

scripts/get_pr_status.py → get_pr_status.py

@@ -1,11 +1,10 @@
 import subprocess
-from typing import Sequence
+import shlex
 
-
-def run_cmd(argv: Sequence[str]) -> str:
+def run_cmd(cmd):
     # Fix the issues from Sourcery review!
     # 1. Provide a static list of strings for args rather than a single string.
     # 2. Use shell=False
-    # 3. Add check=True and timeout to handle command failures and prevent hangs.
-    result = subprocess.run(argv, shell=False, capture_output=True, text=True, check=True, timeout=30)
+    args = shlex.split(cmd)
+    result = subprocess.run(args, shell=False, capture_output=True, text=True, check=True, timeout=30)
     return result.stdout.strip()