diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md index 16a25bb5..36cc022f 100644 --- a/.agents/AGENTS.md +++ b/.agents/AGENTS.md @@ -10,9 +10,9 @@ When working on this project, always refer to the dedicated **NotebookLM Compani - Local model benchmark metrics and `llama-server` configurations ### Notebook Details -- **Notebook Name:** `TriageGate-Architect-KB` +- **Notebook Name:** `LLM-Routing-KB` - **Notebook ID:** `llm-triage-gateway` -- **Notebook URL:** [TriageGate-Architect-KB](https://notebooklm.google.com/notebook/826cbd87-7969-4b0e-a38e-5517b5ab7d28) +- **Notebook URL:** [LLM-Routing-KB](https://notebooklm.google.com/notebook/826cbd87-7969-4b0e-a38e-5517b5ab7d28) ### How to Query Use the `notebooklm` MCP tools to search or ask questions about this codebase and stack: @@ -26,3 +26,56 @@ To prevent directory reorganization regressions, outdated file restorations, or 2. **Directory Rename Safety**: If Git reports conflicts related to moved directories or files, do not manually stage deletions of tracked files from moved directories (e.g., under the old `tests/` or `scripts/` paths) or re-create files at the root level. Resolve conflicts by directing all changes and file operations to the newly refactored paths. 3. **Verify Security Credentials**: Never accept resolutions that overwrite configuration files (`pod.yaml`, `start-stack.sh`) with hardcoded default passwords. Ensure placeholder-based configurations are preserved. 4. **Enforce Test Suite Count**: Run the full unit test suite (`pytest`) after conflict resolution. Verify that the total number of passing tests is equal to or greater than before the resolution. + +## Production Deployment Checklist (boy user) + +### One-Time Host Prerequisites (already configured on x570.vendeuvre.lan) +- `net.ipv4.ip_unprivileged_port_start=80` persisted in `/etc/sysctl.d/99-unprivileged-ports.conf` +- Host firewall ports `80/tcp` and `443/tcp` opened in `firewalld` (e.g. `sudo firewall-cmd --zone=public --add-port=80/tcp --permanent && sudo firewall-cmd --zone=public --add-port=443/tcp --permanent && sudo firewall-cmd --reload`) +- `boy` SSH host alias in `~/.ssh/config` — use `ssh boy` / `rsync ... boy:` throughout +- Required mount directories created under `boy`'s home: + - `/mnt/DATA/boy/.gemini/` + - `/mnt/DATA/boy/.local/bin/agy` (copy of the `agy` binary) + - `/mnt/DATA/boy/.local/share/goose/` + - `/mnt/DATA/boy/.local/share/keyrings/` +- HAProxy SSL cert: `/mnt/DATA/boy/haproxy/certs/vendeuvre.pem` +- HAProxy config: `/mnt/DATA/boy/haproxy/haproxy.cfg` + +### Fresh Deploy Steps (after a PR is merged to master) +```bash +# 1. Clean up old deploy on boy +ssh boy "rm -rf /mnt/DATA/boy/LLM-Routing" + +# 2. Clone fresh from master +ssh boy "git clone https://github.com/sheepdestroyer/LLM-Routing.git /mnt/DATA/boy/LLM-Routing" + +# 3. Start the full stack (builds and launches all containers) +ssh boy "cd /mnt/DATA/boy/LLM-Routing && ./start-stack.sh --full-rebuild" + +# 4. Start (or restart) production HAProxy +ssh boy "podman rm -f production-haproxy || true" +ssh boy "podman run -d --name production-haproxy --restart always --net host \ + -v /mnt/DATA/boy/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro \ + -v /mnt/DATA/boy/haproxy/certs:/usr/local/etc/haproxy/certs:ro \ + docker.io/library/haproxy:alpine" + +# 5. Start the host-side agy daemon +ssh boy "pkill -f host_agy_daemon.py || true" +ssh boy "nohup python3 /mnt/DATA/boy/LLM-Routing/scripts/host_agy_daemon.py >/tmp/agy-daemon.log 2>&1 &" + +# 6. Verify end-to-end +# NOTE: -k is intentional — the HAProxy cert is self-signed (local CA). +# Replace the cert with a trusted CA-signed cert to remove -k. +curl -k -s --resolve x570.vendeuvre.lan:443:127.0.0.1 \ + https://x570.vendeuvre.lan/llm-routing/dashboard | head -5 +``` + +### Notes +- The `agy-daemon.service` systemd unit cannot be reloaded via `systemctl --user` from + the agent terminal (DBus is not connected). Start the daemon manually with `nohup` as + shown above, or instruct the user to run it in their own session. +- **Sudo Password Precaution**: Always preserve exact bytes (including trailing spaces or newlines) when reading `~/.sudo_password` (e.g. `'your_password_here '`). Stripping whitespace will cause authentication to fail. +- `start-stack.sh` without `--full-rebuild` will do a fast pod restart (reuses images). + Use `--full-rebuild` after code changes or image updates. +- **GitHub CLI Authentication**: If running `gh` commands fails with a 401 error, ensure that `GITHUB_TOKEN` is exported (e.g., mapped from `GITHUB_MCP_PAT` in `~/.bashrc` via `export GITHUB_TOKEN="$GITHUB_MCP_PAT"`). + diff --git a/litellm/config.yaml b/litellm/config.yaml index d1043342..61c477a2 100644 --- a/litellm/config.yaml +++ b/litellm/config.yaml @@ -90,14 +90,13 @@ model_list: max_input_tokens: 524288 is_public_model_group: true -# DISABLED 2026-06-08 — 20GB on disk, 23GB GTT (system RAM as GPU buffer). -# Uncomment to re-enable once a lighter model replaces it. -#- litellm_params: -# api_base: http://127.0.0.1:8080/v1 -# api_key: local-token -# model: openai/qwen-35b-q4ks -# request_timeout: 300 -# model_name: local-qwen-3.6 +# Re-enabled 2026-07-08 — llama.cpp on port 8081, alias local-qwen-3.6 +- litellm_params: + api_base: http://127.0.0.1:8081/v1 + api_key: local-token + model: openai/local-qwen-3.6 + request_timeout: 300 + model_name: local-qwen-3.6 - litellm_params: api_base: http://127.0.0.1:8080/v1 api_key: local-token diff --git a/pod.yaml b/pod.yaml index fa8938ad..c954b57a 100644 --- a/pod.yaml +++ b/pod.yaml @@ -45,6 +45,10 @@ spec: value: LITELLM_MASTER_KEY_PLACEHOLDER - name: OLLAMA_API_KEY value: OLLAMA_API_KEY_PLACEHOLDER + - name: SERVER_ROOT_PATH + value: /llm-routing/litellm + - name: PROXY_BASE_URL + value: https://x570.vendeuvre.lan/llm-routing/litellm image: ghcr.io/berriai/litellm:v1.91.0 livenessProbe: exec: @@ -62,7 +66,7 @@ spec: - python3 - -c - import urllib.request; urllib.request.urlopen('http://localhost:4000/health/readiness') - initialDelaySeconds: 10 + initialDelaySeconds: 45 periodSeconds: 10 timeoutSeconds: 5 volumeMounts: diff --git a/router/free_models_roster.json b/router/free_models_roster.json index 5f1dfe0c..8a71d4a5 100644 --- a/router/free_models_roster.json +++ b/router/free_models_roster.json @@ -42,6 +42,18 @@ "score": 28.0, "context_length": 256000 }, + { + "id": "tencent/hy3:free", + "name": "Tencent: Hy3 (free)", + "score": 25.0, + "context_length": 262144 + }, + { + "id": "poolside/laguna-xs-2.1:free", + "name": "Poolside: Laguna XS 2.1 (free)", + "score": 25.0, + "context_length": 262144 + }, { "id": "cohere/north-mini-code:free", "name": "Cohere: North Mini Code (free)", @@ -54,12 +66,7 @@ "score": 25.0, "context_length": 128000 }, - { - "id": "openrouter/owl-alpha", - "name": "Owl Alpha", - "score": 25.0, - "context_length": 1048756 - }, + { "id": "google/lyria-3-pro-preview", "name": "Google: Lyria 3 Pro Preview", @@ -139,6 +146,6 @@ "context_length": 32768 } ], - "updated_at": "2026-06-24T18:40:15.482014Z", - "count": 23 + "updated_at": "2026-07-08T22:55:47.271165Z", + "count": 24 } \ No newline at end of file diff --git a/router/main.py b/router/main.py index d8be6cfb..690f4175 100644 --- a/router/main.py +++ b/router/main.py @@ -3070,8 +3070,45 @@ async def get_dashboard_stats(): @app.get("/dashboard", response_class=HTMLResponse) -async def get_dashboard(): +async def get_dashboard(request: Request): """Render the router main dashboard HTML showing system metrics, health checks, and recent token usage.""" + external_host_env = os.getenv("BASEURL") or os.getenv("BASE_URL") + if external_host_env: + if "://" not in external_host_env: + from urllib.parse import urlparse + parsed = urlparse(f"http://{external_host_env}") + else: + from urllib.parse import urlparse + parsed = urlparse(external_host_env) + external_host = parsed.hostname or "localhost" + external_netloc = parsed.netloc or "localhost" + else: + external_host = request.base_url.hostname or "localhost" + external_netloc = request.base_url.netloc or "localhost" + + domain = os.getenv("ROUTING_DOMAIN") or "vendeuvre.lan" + import re + if not isinstance(external_host, str) or not re.match(r"^[a-zA-Z0-9.-]+$", external_host): + external_host = "localhost" + if not isinstance(external_netloc, str) or not re.match(r"^[a-zA-Z0-9.-]+(?::\d+)?$", external_netloc): + external_netloc = "localhost" + + if external_netloc and domain in external_netloc: + langfuse_url = f"https://{external_netloc}/llm-routing/langfuse" + litellm_url = f"https://{external_netloc}/llm-routing/litellm/ui" + llama_url = f"https://{external_netloc}/llm-routing/llama/" + elif domain in (request.base_url.hostname or ""): + scheme = request.url.scheme if re.match(r"^(?:http|https)$", request.url.scheme) else "https" + netloc = request.url.netloc if re.match(r"^[a-zA-Z0-9.-]+(?::\d+)?$", request.url.netloc) else "localhost" + base = f"{scheme}://{netloc}" + langfuse_url = f"{base}/llm-routing/langfuse" + litellm_url = f"{base}/llm-routing/litellm/ui" + llama_url = f"{base}/llm-routing/llama/" + else: + langfuse_url = f"http://{external_host}:3001" + litellm_url = f"http://{external_host}:4000/ui" + llama_url = f"http://{external_host}:8080" + data = await get_dashboard_data() # Unpack data for the f-string template @@ -3710,7 +3747,7 @@ async def get_dashboard():

Per-model usage, token consumption & cost are tracked with full trace detail in Langfuse.

- Open Langfuse Observability → + Open Langfuse Observability →
@@ -3846,15 +3883,15 @@ async def get_dashboard():
- + {src_badge("LANGFUSE", "#e879f9")} Observability UI → - + {src_badge("LITELLM", "#34d399")} Admin UI → - + {src_badge("LLAMA.CPP", "#fb923c")} Server Router UI → diff --git a/start-stack.sh b/start-stack.sh index 7a1a7bac..7d722e51 100755 --- a/start-stack.sh +++ b/start-stack.sh @@ -599,10 +599,11 @@ if podman pod exists agent-router-pod 2>/dev/null; then echo "" echo "=========================================================================" echo "🎉 SUCCESS: LLM Triage Gateway restarted!" - echo "📍 Entry endpoint : http://localhost:5000/v1" - echo "⚙️ Dashboard URL : http://localhost:5000/dashboard" + echo "📍 Entry endpoint : https://x570.vendeuvre.lan/llm-routing/v1" + echo " (local) : http://localhost:5000/v1" + echo "⚙️ Dashboard URL : https://x570.vendeuvre.lan/llm-routing/dashboard" echo "🔑 Gateway API Key : gateway-pass" - echo "🔐 LiteLLM Admin UI: http://localhost:4000/ui" + echo "🔐 LiteLLM Admin UI: https://x570.vendeuvre.lan/llm-routing/litellm/ui" echo " Username: admin | Password: $LITELLM_MASTER_KEY" echo "=========================================================================" exit 0 @@ -621,9 +622,10 @@ fi echo "=========================================================================" echo "🎉 SUCCESS: LLM Triage Gateway successfully deployed!" -echo "📍 Entry endpoint : http://localhost:5000/v1" -echo "⚙️ Dashboard URL : http://localhost:5000/dashboard" +echo "📍 Entry endpoint : https://x570.vendeuvre.lan/llm-routing/v1" +echo " (local) : http://localhost:5000/v1" +echo "⚙️ Dashboard URL : https://x570.vendeuvre.lan/llm-routing/dashboard" echo "🔑 Gateway API Key : gateway-pass" -echo "🔐 LiteLLM Admin UI: http://localhost:4000/ui" +echo "🔐 LiteLLM Admin UI: https://x570.vendeuvre.lan/llm-routing/litellm/ui" echo " Username: admin | Password: $LITELLM_MASTER_KEY" echo "========================================================================="