Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
python-version: '3.11'

- name: Install dependencies
run: pip install httpx==0.28.1 pytest pytest-asyncio anyio pyyaml fastapi "pydantic>=2.0,<3.0" uvicorn python-multipart asyncpg langfuse redis aiofiles
run: pip install httpx==0.28.1 pytest pytest-asyncio anyio pyyaml fastapi "pydantic>=2.0,<3.0" uvicorn python-multipart asyncpg langfuse redis

- name: Run Unit Tests
run: CONFIG_PATH=router/config.yaml PYTHONPATH=.:router pytest --ignore=test_agy_behavior.py --ignore=test_agy_tiers.py
Expand Down
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,11 @@ router/free_models_roster.json
# agent artifacts
.hermes/
.jules/
.local/
workdirs/
pr_description.txt
/get_pr_status.py

# Dataset work in progress
data/
.cache/
test_output*.log
40 changes: 23 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,15 +76,15 @@ All core containers are configured with **Kubernetes-style liveness and readines

| Container | Liveness Probe | Readiness Probe |
|:---|---:|---:|
| **valkey-cache** (9.1.0-alpine) | `valkey-cli PING` every 10s | `valkey-cli PING` every 5s |
| **litellm-gateway** | Python `urllib` GET `/health` (port 4000, accepts 200/401) every 15s | Same, every 10s |
| **llm-triage-router** | Python `urllib` GET `/dashboard` (port 5000) every 15s | Same, every 10s |
| **valkey-cache** (9.1.0-alpine) | `tcpSocket` on port 6379 every 10s | Same, every 5s |
| **litellm-gateway** | Python `urllib` GET `/ping` (port 4000) every 15s | Python `urllib` GET `/health/readiness` (port 4000) every 10s |
| **llm-triage-router** | Python `urllib` GET `/metrics` (port 5000) every 15s | Same, every 10s |
| **postgres-db** | `pg_isready -U postgres` every 10s | Same, every 5s |
| **clickhouse-db** | `clickhouse-client --user clickhouse --password clickhouse --query "SELECT 1"` every 15s | Same, every 10s |
| **valkey-lf** | `redis-cli -p 6380 -a langfuse-redis-2026 PING` every 10s | Same, every 5s |
| **langfuse-web** | `wget -qO /dev/null http://127.0.0.1:3001/` every 15s | Same, every 10s |
| **langfuse-worker** | `pgrep -f langfuse-worker` every 15s | — |
| **minio-s3** | TCP socket check on port 9002 every 15s | Same, every 10s |
| **clickhouse-db** | `clickhouse-client --user clickhouse --password clickhouse --query "SELECT 1"` every 15s | `clickhouse-client --query "SELECT 1"` every 10s |
| **valkey-lf** (9.1.0-alpine) | `tcpSocket` on port 6380 every 10s | Same, every 5s |
| **langfuse-web** | `wget` GET `/api/health` (port 3001) every 15s | Same, every 10s |
| **langfuse-worker** | `pgrep node` every 15s | — |
| **minio-s3** | `httpGet` `/minio/health/live` (port 9002) every 15s | `httpGet` `/minio/health/ready` (port 9002) every 10s |

The pod-level `restartPolicy: Always` combined with these probes means Podman will restart any container that fails its health check or exits unexpectedly, enabling true self-healing for the entire stack.

Expand Down Expand Up @@ -213,7 +213,7 @@ All configurations, automation scripts, and databases are self-contained within

```
/home/gpav/Vrac/LAB/AI/LLM-Routing/
├── .env # Environment file for OpenRouter API Key (ignored by git)
├── .env # Environment file for API keys, passwords, and generated secrets (ignored by git)
├── .gitignore # Git ignore policy protecting secrets & database files
├── README.md # In-depth system and operational guide
├── pod.yaml # Podman Kubernetes template defining the 10-container stack
Expand Down Expand Up @@ -379,7 +379,7 @@ Run the startup script from the root of the repository:
# health probes, env vars, containers — no rebuild)
./start-stack.sh --full-rebuild # Full reset: rebuild image + recreate pod
```
*Note: If running for the first time, the script will prompt you for your `OpenRouter API Key`, securely saving it inside `.env` with restrictive permissions (`chmod 600`).*
*Note: If running for the first time, the script will prompt you for your `OpenRouter API Key`, securely saving it inside `.env` with restrictive permissions (`chmod 600`). The script also automatically generates and persists secure random secrets (`LITELLM_MASTER_KEY`, `POSTGRES_PASSWORD`, `NEXTAUTH_SECRET`, `SALT`, `ENCRYPTION_KEY`, and `ROUTER_API_KEY`) to this file on startup if they are missing.*

### 2. Verify Container Status
Check that all **10 containers** inside `agent-router-pod` are up and running:
Expand Down Expand Up @@ -575,19 +575,25 @@ Without Minio, Langfuse v3 **will not start** — it validates S3 connectivity a
|----------|-------|
| `LANGFUSE_S3_EVENT_UPLOAD_BUCKET` | `langfuse-events` |
| `LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT` | `http://127.0.0.1:9002` |
| `LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID` | (auto-generated in `.env`) |
| `LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY` | (auto-generated in `.env`) |
| `LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID` | `minioadmin` |
| `LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY` | `minioadmin` |
| `S3_FORCE_PATH_STYLE` | `true` |

Minio runs on ports **9001** (web console) and **9002** (S3 API). Credentials are automatically generated and stored in `.env`. Image pinned to `docker.io/minio/minio:RELEASE.2025-10-15T17-29-55Z`.
Minio runs on ports **9001** (web console) and **9002** (S3 API). Credentials: `minioadmin` / `minioadmin`. Image pinned to `docker.io/minio/minio:RELEASE.2025-10-15T17-29-55Z`.

### Health Check

Minio's minimal Go image has no HTTP client tools. The probe uses a raw TCP socket check:
MinIO's health is monitored using its native structured endpoints `/minio/health/live` (liveness) and `/minio/health/ready` (readiness) on port 9002:

```yaml
exec:
command: [sh, -c, "exec 3<>/dev/tcp/127.0.0.1/9002 && echo ok"]
livenessProbe:
httpGet:
path: /minio/health/live
port: 9002
readinessProbe:
httpGet:
path: /minio/health/ready
port: 9002
```

---
Expand Down Expand Up @@ -791,7 +797,7 @@ For auto-routing modes, the Triage Router handles failures by silently falling b

This project is supported by a dedicated NotebookLM companion notebook:
* **Notebook Name:** `TriageGate-Architect-KB`
* **Notebook ID:** `llm-triage-gateway`
* **Notebook ID:** llm-triage-gateway
* **URL:** [TriageGate-Architect-KB](https://notebooklm.google.com/notebook/826cbd87-7969-4b0e-a38e-5517b5ab7d28)

This notebook contains a comprehensive semantic index of the system architecture, LiteLLM cascades, Langfuse telemetry pipelines, local model configurations, and integration guides. Agents and developers can query this notebook via the `notebooklm` MCP tools (e.g., using `notebook_ask` with `notebook_id: "llm-triage-gateway"`) to retrieve structured knowledge, check pitfalls, or get implementation examples for this gateway stack.
11 changes: 11 additions & 0 deletions get_pr_status.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import subprocess
from typing import Sequence


def run_cmd(argv: Sequence[str]) -> str:
# Fix the issues from Sourcery review!
# 1. Provide a static list of strings for args rather than a single string.
# 2. Use shell=False
# 3. Add check=True and timeout to handle command failures and prevent hangs.
result = subprocess.run(argv, shell=False, capture_output=True, text=True, check=True, timeout=30)
return result.stdout.strip()
16 changes: 8 additions & 8 deletions pod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ spec:
value: sk-lit...33bf
- name: OLLAMA_API_KEY
value: 3cd542f8fca14fd08bedfd4f2ab36f9f.6G7Iukbvu1Keyi9x8eKckNEO
image: ghcr.io/berriai/litellm:v1.90.2
image: ghcr.io/berriai/litellm:v1.89.4
livenessProbe:
exec:
command:
Expand Down Expand Up @@ -273,9 +273,9 @@ spec:
- name: LANGFUSE_S3_EVENT_UPLOAD_REGION
value: us-east-1
- name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID
value: MINIO_USER_PLACEHOLDER
value: minioadmin
- name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY
value: MINIO_PASSWORD_PLACEHOLDER
value: minioadmin
- name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT
value: http://127.0.0.1:9002
- name: S3_FORCE_PATH_STYLE
Expand Down Expand Up @@ -307,7 +307,7 @@ spec:
- name: LANGFUSE_INIT_USER_EMAIL
value: admin@local.dev
- name: LANGFUSE_INIT_USER_PASSWORD
value: LANGFUSE_INIT_USER_PASSWORD_PLACEHOLDER
value: admin-local-pw-2026
- name: LANGFUSE_LOG_LEVEL
value: warn
image: docker.io/langfuse/langfuse:3
Expand Down Expand Up @@ -363,13 +363,13 @@ spec:
- name: LANGFUSE_S3_EVENT_UPLOAD_REGION
value: us-east-1
- name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID
value: MINIO_USER_PLACEHOLDER
value: minioadmin
- name: S3_FORCE_PATH_STYLE
value: 'true'
- name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT
value: http://127.0.0.1:9002
- name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY
value: MINIO_PASSWORD_PLACEHOLDER
value: minioadmin
- name: LANGFUSE_LOG_LEVEL
value: warn
image: docker.io/langfuse/langfuse-worker:3
Expand All @@ -393,9 +393,9 @@ spec:
- ":9001"
env:
- name: MINIO_ROOT_USER
value: MINIO_USER_PLACEHOLDER
value: minioadmin
- name: MINIO_ROOT_PASSWORD
value: MINIO_PASSWORD_PLACEHOLDER
value: minioadmin
image: docker.io/minio/minio:latest
livenessProbe:
httpGet:
Expand Down
2 changes: 1 addition & 1 deletion router/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ FROM python:3.14-slim
WORKDIR /app

# Install deps in a single layer (no pip cache, no extra files)
RUN pip install --no-cache-dir fastapi "pydantic>=2.0,<3.0" uvicorn httpx pyyaml python-multipart asyncpg langfuse redis aiofiles
RUN pip install --no-cache-dir fastapi "pydantic>=2.0,<3.0" uvicorn httpx pyyaml python-multipart asyncpg langfuse redis

# Copy all source in one layer — removes dead config COPY (volume-mounted at runtime)
COPY main.py agy_proxy.py circuit_breaker.py aa_scores.json free_models_roster.json /app/
Expand Down
Loading