Skip to content
3 changes: 3 additions & 0 deletions .env.dev
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,6 @@ ROUTER_IMAGE="localhost/llm-routing-dev:latest"
# reaches its local listener directly rather than any TLS-terminated hostname.
LLAMA_CLASSIFIER_URL="http://127.0.0.1:8083/v1"
LLAMA_SERVER_URL="http://127.0.0.1:8083"

# Distinct systemd Quadlet namespace; never share generated units with prod.
QUADLET_NAMESPACE="llm-routing-dev"
48 changes: 33 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ Exposes the entry endpoint (`http://localhost:5000/v1`) and evaluates prompt com
| `llm-routing-auto-agy-ollama` | ✅ | agy → Ollama (gated: reasoning/advanced/complex) | LiteLLM with classified tier | 512K |
| `llm-routing-agy` | ❌ | agy (Gemini/Claude) — unconditional | LiteLLM agent-advanced-core | 1M |
| `llm-routing-ollama` | ✅ | Ollama (gated: reasoning & advanced → ollama-deepseek-v4-pro, complex & below → ollama-deepseek-v4-flash) | LiteLLM openrouter-auto | 512K |
| `agent-advanced-core` | ❌ | — | LiteLLM openrouter-auto | 262K |
| `agent-advanced-core` | ❌ | — | LiteLLM `local-qwen-3.6` → `llm-routing-ollama` → `openrouter-auto` | 262K |
| `agent-reasoning-core` | ❌ | — | LiteLLM fallback chain | 262K |
| `agent-complex-core` | ❌ | — | LiteLLM fallback chain | 262K |
| `agent-medium-core` | ❌ | — | LiteLLM fallback chain | 262K |
Expand All @@ -285,7 +285,7 @@ Orchestrates routing fallback chains, Redis caching, and telemetry callbacks:
- `embedding_model: "local-nomic-embed"` — uses the local nomic-embed model (no API costs)
- `collection_name: "litellm_semantic_cache"` — stores embeddings for similarity-based cache lookups
- **Cascading Fallback Chains** (configured in `litellm_settings.fallbacks`):
Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`).
Each tier escalates through increasingly capable free models, then the local llama.cpp safety net (`local-qwen-3.6`), then the paid/remote Ollama tier, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`).

```mermaid
graph TD
Expand All @@ -303,42 +303,47 @@ Orchestrates routing fallback chains, Redis caching, and telemetry callbacks:
SM --> SC[agent-complex-core]:::complex
SC --> SR[agent-reasoning-core]:::reasoning
SR --> SA[agent-advanced-core]:::advanced
SA --> SO1[llm-routing-ollama]:::premium
SA --> SL[local-qwen-3.6]:::local
SL --> SO1[llm-routing-ollama]:::premium
SO1 --> SAU[openrouter-auto]:::auto
end

subgraph Medium["agent-medium-core Fallback Tree"]
M[agent-medium-core]:::medium --> MC[agent-complex-core]:::complex
MC --> MR[agent-reasoning-core]:::reasoning
MR --> MA[agent-advanced-core]:::advanced
MA --> MO1[llm-routing-ollama]:::premium
MA --> ML[local-qwen-3.6]:::local
ML --> MO1[llm-routing-ollama]:::premium
MO1 --> MAU[openrouter-auto]:::auto
end

subgraph Complex["agent-complex-core Fallback Tree"]
C[agent-complex-core]:::complex --> CR[agent-reasoning-core]:::reasoning
CR --> CA[agent-advanced-core]:::advanced
CA --> CO1[llm-routing-ollama]:::premium
CA --> CL[local-qwen-3.6]:::local
CL --> CO1[llm-routing-ollama]:::premium
CO1 --> CAU[openrouter-auto]:::auto
end

subgraph Reasoning["agent-reasoning-core Fallback Tree"]
R[agent-reasoning-core]:::reasoning --> RA[agent-advanced-core]:::advanced
RA --> RO1[llm-routing-ollama]:::premium
RA --> RL[local-qwen-3.6]:::local
RL --> RO1[llm-routing-ollama]:::premium
RO1 --> RAU[openrouter-auto]:::auto
end

subgraph Advanced["agent-advanced-core Fallback Tree"]
A[agent-advanced-core]:::advanced --> AO1[llm-routing-ollama]:::premium
A[agent-advanced-core]:::advanced --> AL[local-qwen-3.6]:::local
AL --> AO1[llm-routing-ollama]:::premium
AO1 --> AAU[openrouter-auto]:::auto
end
```

- **`agent-simple-core`**: medium-core → complex-core → reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto`
- **`agent-medium-core`**: complex-core → reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto`
- **`agent-complex-core`**: reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto`
- **`agent-reasoning-core`**: advanced-core → `llm-routing-ollama` → `openrouter-auto`
- **`agent-advanced-core`**: `llm-routing-ollama` → `openrouter-auto`
- **`agent-simple-core`**: medium-core → complex-core → reasoning-core → advanced-core → `local-qwen-3.6` → `llm-routing-ollama` → `openrouter-auto`
- **`agent-medium-core`**: complex-core → reasoning-core → advanced-core → `local-qwen-3.6` → `llm-routing-ollama` → `openrouter-auto`
- **`agent-complex-core`**: reasoning-core → advanced-core → `local-qwen-3.6` → `llm-routing-ollama` → `openrouter-auto`
- **`agent-reasoning-core`**: advanced-core → `local-qwen-3.6` → `llm-routing-ollama` → `openrouter-auto`
- **`agent-advanced-core`**: `local-qwen-3.6` → `llm-routing-ollama` → `openrouter-auto`
- **`llm-routing-ollama`** (classifier-gated proxy): `reasoning & advanced` → `ollama-deepseek-v4-pro`, `complex & below` → `ollama-deepseek-v4-flash`. Note: Ollama cooldowns are managed by the triage router internally (5-minute window on failure); during cooldown the router returns 429 immediately so LiteLLM skips to `openrouter-auto`.
All tiers ultimately land on OpenRouter auto/free model pools or the local Speculative MoE when enabled.
*Note: Premium routing is controlled by the model name, not by the tier. `llm-routing-agy` and `llm-routing-auto-agy` trigger the agy proxy (Google/Claude via Cloud Code Assist) — but auto models only trigger agy if the classifier returns `agent-advanced-core`. `llm-routing-ollama` and `llm-routing-auto-ollama` route through Ollama.com (deepseek-v4-pro via LiteLLM's ollama_chat provider) — same gating for auto models. `llm-routing-auto-agy-ollama` chains both: agy first, then Ollama if agy is exhausted, both gated on advanced classification. The `agent-advanced-core` tier itself is a plain LiteLLM tier with no premium trigger. See §2 for the full routing table.*
Expand Down Expand Up @@ -395,9 +400,9 @@ Run the startup script from the root of the repository:
./start-stack.sh --full-rebuild # Same as --replace plus rebuild the router image

# Inspect the generated systemd units and their logs
systemctl --user status llm-routing-pod.service --no-pager
systemctl --user list-units 'llm-routing-*' --no-pager
journalctl --user -u llm-routing-router.service -n 100 --no-pager
systemctl --user status llm-routing-prod-pod.service --no-pager # or llm-routing-dev-pod.service
systemctl --user list-units 'llm-routing-*' --no-pager # filter for dev/prod namespaces
journalctl --user -u llm-routing-prod-router.service -n 100 --no-pager # or llm-routing-dev-router.service
```
*Note: If running for the first time, the script will prompt you for your `OpenRouter API Key`, securely saving it inside `.env` with restrictive permissions (`chmod 600`). The script also automatically generates and persists secure random secrets (`LITELLM_MASTER_KEY`, `POSTGRES_PASSWORD`, `NEXTAUTH_SECRET`, `SALT`, `ENCRYPTION_KEY`, and `ROUTER_API_KEY`) to this file on startup if they are missing.*

Expand Down Expand Up @@ -855,6 +860,19 @@ Tests cover:

Requires `PUBLIC_BASE_URL` in `.env` for canonical URL tests. The router remains under its configured path (for example `https://x570.vendeuvre.lan/llm-routing`), while the verifier derives service URLs from its host: `https://litellm.<host>/ui/`, `https://langfuse.<host>/`, and `https://llama.<host>/health`. Dev `.env.dev` already has it; prod `.env` should include `PUBLIC_BASE_URL="https://x570.vendeuvre.lan/llm-routing"`. The dev local-model safety net uses the host-networked local listener: `LLAMA_CLASSIFIER_URL=http://127.0.0.1:8083/v1` and `LLAMA_SERVER_URL=http://127.0.0.1:8083`; it must not depend on TLS-terminated dev or production hostnames.

## Environment-isolated Quadlet deployment

Dev and production use distinct Quadlet namespaces because generated systemd
unit names are global within the user manager. Dev renders units under
`~/.config/containers/systemd/llm-routing-dev/` and uses
`llm-routing-dev-pod.service`; production uses
`~/.config/containers/systemd/llm-routing-prod/` and
`llm-routing-prod-pod.service`. Their pod/container names, ports, data roots,
and rendered configuration remain separate. Unless overridden explicitly,
`DATA_ROOT` is `${WORKDIR}/data`; because dev and production run from separate
`~/dev/` and `~/prod/` worktrees, their persistent data and rendered configs are
also physically separate.

## 10. Performance Benchmarks

Through our local benchmarks, the following performance characteristics have been achieved:
Expand Down
4 changes: 2 additions & 2 deletions scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ This directory and the repository root contain various scripts used for stack or
### `start-stack.sh` (Root Directory)
Unified startup and credential extraction script for the systemd Quadlet-managed Podman stack.
- **Usage**:
- `./start-stack.sh` (Restart the generated `llm-routing-pod.service`)
- `./start-stack.sh` (Restart the generated environment-specific pod service)
- `./start-stack.sh --replace` (Stop + clean ports + render/install Quadlets + daemon-reload + recreate stack)
- `./start-stack.sh --full-rebuild` (Same as `--replace` + rebuild the triage router image; required for code changes in `router/`)
- Quadlet templates live in `quadlets/`; rendered owner-only units are installed under `~/.config/containers/systemd/llm-routing/`. Use `systemctl --user status llm-routing-pod.service --no-pager` and `journalctl --user -u llm-routing-router.service --no-pager` for lifecycle diagnostics.
- Quadlet templates live in `quadlets/`; rendered owner-only units use environment-specific namespaces: dev under `~/.config/containers/systemd/llm-routing-dev/` and prod under `~/.config/containers/systemd/llm-routing-prod/`. Dev uses `llm-routing-dev-pod.service`; prod uses `llm-routing-prod-pod.service`. Use the matching `systemctl --user status <namespace>-pod.service --no-pager` and `journalctl --user -u <namespace>-router.service --no-pager` for lifecycle diagnostics.

### `scripts/backup.sh`
Automated database backup script that runs before every stack deployment. Uses `pg_isready` to safely wait for database connections and manages timestamped backups under `backups/`.
Expand Down
Loading
Loading