feat(deploy): migrate stack management from play kube to systemd Quadlets #343
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| [Unit] | ||
| Description=ClickHouse — Langfuse v3 OLAP trace store (POD_NAME_PLACEHOLDER) | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-clickhouse-db | ||
| Image=clickhouse/clickhouse-server:26.6.2.81 | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| User=101 | ||
| Group=101 | ||
| Environment=CLICKHOUSE_DB=default | ||
| Environment=CLICKHOUSE_USER=clickhouse | ||
| Environment=CLICKHOUSE_PASSWORD=CLICKHOUSE_PASSWORD_PLACEHOLDER | ||
| Volume=DATA_ROOT_PLACEHOLDER/clickhouse-data:/var/lib/clickhouse | ||
| Volume=DATA_ROOT_PLACEHOLDER/clickhouse-config/port-override.xml:/etc/clickhouse-server/config.d/port-override.xml | ||
| HealthCmd=clickhouse-client --host 127.0.0.1 --port CLICKHOUSE_TCP_PORT_PLACEHOLDER --user clickhouse --password CLICKHOUSE_PASSWORD_PLACEHOLDER --query "SELECT 1" | ||
| HealthInterval=15s | ||
| HealthTimeout=5s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=20s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=120 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| [Unit] | ||
| Description=Langfuse Web — observability dashboard (POD_NAME_PLACEHOLDER) | ||
| After=llm-routing-postgres.service llm-routing-clickhouse.service llm-routing-valkey-lf.service llm-routing-minio.service | ||
| Wants=llm-routing-postgres.service llm-routing-clickhouse.service llm-routing-valkey-lf.service llm-routing-minio.service | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-langfuse-web | ||
| Image=langfuse/langfuse:3.222.0 | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| Environment=DATABASE_URL=postgresql://postgres:POSTGRES_PASSWORD_ENCODED_PLACEHOLDER@127.0.0.1:POSTGRES_PORT_PLACEHOLDER/langfuse | ||
| Environment=NEXTAUTH_SECRET=NEXTAUTH_SECRET_PLACEHOLDER | ||
| Environment=NEXTAUTH_URL=NEXTAUTH_URL_PLACEHOLDER | ||
| Environment=SALT=SALT_PLACEHOLDER | ||
| Environment=ENCRYPTION_KEY=ENCRYPTION_KEY_PLACEHOLDER | ||
| Environment=HOSTNAME=0.0.0.0 | ||
| Environment=PORT=LANGFUSE_WEB_PORT_PLACEHOLDER | ||
| Environment=TELEMETRY_ENABLED=false | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse-events | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_REGION=us-east-1 | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=MINIO_USER_PLACEHOLDER | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=MINIO_PASSWORD_PLACEHOLDER | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://127.0.0.1:MINIO_S3_PORT_PLACEHOLDER | ||
| Environment=S3_FORCE_PATH_STYLE=true | ||
| Environment=CLICKHOUSE_URL=http://127.0.0.1:CLICKHOUSE_HTTP_PORT_PLACEHOLDER | ||
| Environment=CLICKHOUSE_MIGRATION_URL=clickhouse://127.0.0.1:CLICKHOUSE_TCP_PORT_PLACEHOLDER | ||
| Environment=CLICKHOUSE_USER=clickhouse | ||
| Environment=CLICKHOUSE_PASSWORD=CLICKHOUSE_PASSWORD_PLACEHOLDER | ||
| Environment=CLICKHOUSE_CLUSTER_ENABLED=false | ||
| Environment=REDIS_HOST=127.0.0.1 | ||
| Environment=REDIS_PORT=VALKEY_LF_PORT_PLACEHOLDER | ||
| Environment=REDIS_AUTH=REDIS_AUTH_PLACEHOLDER | ||
| Environment=LANGFUSE_INIT_ORG_ID=org-local-dev-id | ||
| Environment=LANGFUSE_INIT_ORG_NAME=Local Development | ||
| Environment=LANGFUSE_INIT_PROJECT_ID=proj-triage-gateway-id | ||
| Environment=LANGFUSE_INIT_PROJECT_NAME=Triage Gateway | ||
| Environment=LANGFUSE_INIT_USER_EMAIL=admin@local.dev | ||
| Environment=LANGFUSE_INIT_USER_PASSWORD=LANGFUSE_INIT_USER_PASSWORD_PLACEHOLDER | ||
| Environment=LANGFUSE_LOG_LEVEL=warn | ||
| HealthCmd=wget -q -O /dev/null http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER/api/public/health | ||
| HealthInterval=15s | ||
| HealthTimeout=5s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=240s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=300 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| [Unit] | ||
| Description=Langfuse Worker — background job processor (POD_NAME_PLACEHOLDER) | ||
| After=llm-routing-postgres.service llm-routing-clickhouse.service llm-routing-valkey-lf.service llm-routing-minio.service | ||
| Wants=llm-routing-postgres.service llm-routing-clickhouse.service llm-routing-valkey-lf.service llm-routing-minio.service | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-langfuse-worker | ||
| Image=langfuse/langfuse-worker:3.222.0 | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| Environment=DATABASE_URL=postgresql://postgres:POSTGRES_PASSWORD_ENCODED_PLACEHOLDER@127.0.0.1:POSTGRES_PORT_PLACEHOLDER/langfuse | ||
| Environment=CLICKHOUSE_URL=http://127.0.0.1:CLICKHOUSE_HTTP_PORT_PLACEHOLDER | ||
| Environment=CLICKHOUSE_USER=clickhouse | ||
| Environment=CLICKHOUSE_PASSWORD=CLICKHOUSE_PASSWORD_PLACEHOLDER | ||
| Environment=CLICKHOUSE_CLUSTER_ENABLED=false | ||
| Environment=REDIS_HOST=127.0.0.1 | ||
| Environment=REDIS_PORT=VALKEY_LF_PORT_PLACEHOLDER | ||
| Environment=REDIS_AUTH=REDIS_AUTH_PLACEHOLDER | ||
| Environment=HOSTNAME=0.0.0.0 | ||
| Environment=PORT=LANGFUSE_WORKER_PORT_PLACEHOLDER | ||
| Environment=TELEMETRY_ENABLED=false | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse-events | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_REGION=us-east-1 | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=MINIO_USER_PLACEHOLDER | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=MINIO_PASSWORD_PLACEHOLDER | ||
| Environment=LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://127.0.0.1:MINIO_S3_PORT_PLACEHOLDER | ||
| Environment=S3_FORCE_PATH_STYLE=true | ||
| Environment=LANGFUSE_LOG_LEVEL=warn | ||
| HealthCmd=pgrep node | ||
| HealthInterval=15s | ||
| HealthTimeout=5s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=240s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=300 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| [Unit] | ||
| Description=LiteLLM Gateway — upstream proxy with adaptive routing (POD_NAME_PLACEHOLDER) | ||
| After=llm-routing-postgres.service llm-routing-valkey-cache.service | ||
| Wants=llm-routing-postgres.service llm-routing-valkey-cache.service | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-litellm-gateway | ||
| Image=ghcr.io/berriai/litellm:v1.93.0 | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| # Image entrypoint is docker/prod_entrypoint.sh (litellm CLI); pod.yaml used | ||
| # command: python3 /app/entrypoint.py which replaced it. Quadlet Exec= only sets | ||
| # args, so override Entrypoint= with the venv python (matches play-kube behavior). | ||
| Entrypoint=/app/.venv/bin/python3 | ||
| Exec=/app/entrypoint.py | ||
| Environment=DATABASE_URL=postgresql://postgres:POSTGRES_PASSWORD_ENCODED_PLACEHOLDER@127.0.0.1:POSTGRES_PORT_PLACEHOLDER/postgres | ||
| Environment=STORE_MODEL_IN_DB=True | ||
| Environment=LITELLM_LOG=INFO | ||
| Environment=LANGFUSE_HOST=http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER | ||
| Environment=LITELLM_MASTER_KEY=LITELLM_MASTER_KEY_PLACEHOLDER | ||
| Environment=LITELLM_UI_USERNAME=LITELLM_UI_USERNAME_PLACEHOLDER | ||
| Environment=LITELLM_UI_PASSWORD=LITELLM_UI_PASSWORD_PLACEHOLDER | ||
| Environment=OLLAMA_API_KEY=OLLAMA_API_KEY_PLACEHOLDER | ||
| Environment=OPENROUTER_API_KEY=OPENROUTER_API_KEY_PLACEHOLDER | ||
| Environment=SERVER_ROOT_PATH=/llm-routing/litellm | ||
| Environment=PROXY_BASE_URL=PROXY_BASE_URL_PLACEHOLDER | ||
| Environment=LITELLM_PORT=LITELLM_PORT_PLACEHOLDER | ||
| Environment=POSTGRES_PORT=POSTGRES_PORT_PLACEHOLDER | ||
| Volume=DATA_ROOT_PLACEHOLDER/litellm-rendered/config.yaml:/app/config.yaml | ||
| Volume=DATA_ROOT_PLACEHOLDER/litellm-rendered/entrypoint.py:/app/entrypoint.py | ||
| Volume=WORKDIR_PLACEHOLDER/.env:/config/.env | ||
| Volume=HOME_PLACEHOLDER/.gemini:/config/gemini_auth | ||
| HealthCmd=python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:LITELLM_PORT_PLACEHOLDER/health/liveness')" | ||
| HealthInterval=15s | ||
| HealthTimeout=5s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=240s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=300 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| [Unit] | ||
| Description=MinIO S3 — Langfuse v3 event storage (POD_NAME_PLACEHOLDER) | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-minio-s3 | ||
| Image=minio/minio:RELEASE.2025-09-07T16-13-09Z | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| Environment=MINIO_ROOT_USER=MINIO_USER_PLACEHOLDER | ||
| Environment=MINIO_ROOT_PASSWORD=MINIO_PASSWORD_PLACEHOLDER | ||
| # pod.yaml used args: server ... with no command:, so image entrypoint | ||
| # /usr/bin/docker-entrypoint.sh ran and consumed them. Quadlet Exec= sets args | ||
| # the same way — image entrypoint stays, Exec provides the server args. | ||
| Exec=server --quiet /data --address :MINIO_S3_PORT_PLACEHOLDER --console-address :MINIO_CONSOLE_PORT_PLACEHOLDER | ||
| Volume=DATA_ROOT_PLACEHOLDER/minio-data:/data | ||
| HealthCmd=curl -f http://127.0.0.1:MINIO_S3_PORT_PLACEHOLDER/minio/health/live | ||
|
Contributor
There was a problem hiding this comment. 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "Files mentioning MinIO health endpoints or quadlet HealthCmd:"
rg -n -C 3 "MinIO|MINIO_S3_PORT_PLACEHOLDER|/minio/health/(live|ready)|HealthCmd|health/live|health/ready" . || true
echo
echo "Quadlet file:"
cat -n quadlets/llm-routing-minio.container || true
echo
echo "README health endpoint sections:"
if [ -f README.md ]; then
nl -ba README.md | sed -n '/minio/health/p; /MinIO/p; /health/p' | head -120
fiRepository: sheepdestroyer/LLM-Routing Length of output: 20677 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "Locate service dependency/usages of this quadlet or container services:"
rg -n -C 3 "llm-routing-minio|minio|MINIO|S3_ADDRESS|Langfuse|language-model|routing" . --glob '!/.git/**' || trueRepository: sheepdestroyer/LLM-Routing Length of output: 50385 Use MinIO readiness for the component healthcheck.
🤖 Prompt for AI Agents |
||
| HealthInterval=15s | ||
| HealthTimeout=5s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=10s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=90 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| [Unit] | ||
| Description=PostgreSQL — Langfuse + LiteLLM metadata store (POD_NAME_PLACEHOLDER) | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-postgres-db | ||
| Image=pgvector/pgvector:0.8.5-pg18 | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| Environment=POSTGRES_USER=postgres | ||
| Environment=POSTGRES_PASSWORD=POSTGRES_PASSWORD_RAW_PLACEHOLDER | ||
| Environment=POSTGRES_DB=langfuse | ||
| Environment=PGDATA=/var/lib/postgresql/data/pg18 | ||
| Environment=PGPORT=POSTGRES_PORT_PLACEHOLDER | ||
| Volume=DATA_ROOT_PLACEHOLDER/postgres-data:/var/lib/postgresql | ||
| HealthCmd=pg_isready -U postgres -p POSTGRES_PORT_PLACEHOLDER | ||
|
Contributor
There was a problem hiding this comment. 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== locate file =="
git ls-files | rg '(^|/)llvm-routing-postgres\.container$|llm-routing-postgres\.container$' || true
echo "== file contents =="
if [ -f quadlets/llm-routing-postgres.container ]; then
cat -n quadlets/llm-routing-postgres.container
fi
echo "== pg_isready occurrences =="
rg -n "pg_isready|HealthCmd|POSTGRES_PORT_PLACEHOLDER|127\.0\.0\.1|postgres" quadlets . 2>/dev/null | head -200Repository: sheepdestroyer/LLM-Routing Length of output: 21577 🌐 Web query:
💡 Result: For the PostgreSQL utility pg_isready, the -h (or --host) option specifies the database server host. If the provided value begins with a forward slash (/), it is interpreted as the directory containing the Unix-domain socket [1][2]. If no -h option is provided, pg_isready defaults to the behavior defined by the underlying libpq connection library [3]. Typically, this default target is the standard Unix-domain socket directory, which is defined at compile time (often /tmp or /var/run/postgresql on many Unix-like systems) [1][4][3]. Regarding other connection options: - Port: If the -p (or --port) option is not specified, it defaults to the value of the PGPORT environment variable. If that variable is not set, it defaults to the port specified at compile time, which is usually 5432 [1][2][5]. In summary, when no host is specified, pg_isready relies on the default connection parameters established by libpq, which prioritizes the compile-time default Unix-domain socket path [3]. Citations:
🌐 Web query:
💡 Result: The --host (or -h) option for the pg_isready utility is used to specify the database server host [1][2]. Crucially, if the value provided to the --host option begins with a forward slash (/), pg_isready interprets it as the directory path for the Unix-domain socket [1][3][2]. For example, providing /tmp to the host option instructs the utility to look for the socket file within that directory [1][2]. The default behavior when no host is specified generally follows standard PostgreSQL client connection rules, often defaulting to the local Unix-domain socket directory if applicable (typically /tmp on many Unix-like systems) [1]. You can also use the --port (or -p) option to specify the TCP port or the local Unix-domain socket file extension [1][2]. Citations:
Probe the TCP endpoint used by the app containers. LiteLLM, Langfuse, and the router all connect via 🤖 Prompt for AI Agents |
||
| HealthInterval=10s | ||
| HealthTimeout=5s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=15s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=120 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| [Unit] | ||
| Description=LLM Triage Router — classification + routing gateway (POD_NAME_PLACEHOLDER) | ||
| After=llm-routing-litellm.service | ||
| Wants=llm-routing-litellm.service | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-llm-triage-router | ||
| Image=ROUTER_IMAGE_PLACEHOLDER | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| # Router image has no entrypoint (EP=null, CMD=uvicorn...); pod.yaml used | ||
| # command: /bin/sh + args: -c ... — Entrypoint=/bin/sh, Exec=-c "..." matches. | ||
| Entrypoint=/bin/sh | ||
| Exec=-c "set -a && . /config/.env && set +a && LITELLM_PORT=LITELLM_PORT_PLACEHOLDER VALKEY_CACHE_PORT=VALKEY_CACHE_PORT_PLACEHOLDER exec uvicorn main:app --host 0.0.0.0 --port ROUTER_PORT_PLACEHOLDER --log-level $${LOG_LEVEL:-warning}" | ||
| Environment=CONFIG_PATH=/config/router_dir/config.yaml | ||
| Environment=LITELLM_CONFIG_PATH=/config/litellm_dir/config.yaml | ||
| Environment=DATABASE_URL=postgresql://postgres:POSTGRES_PASSWORD_ENCODED_PLACEHOLDER@127.0.0.1:POSTGRES_PORT_PLACEHOLDER/postgres | ||
| Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=RUN_USER_PLACEHOLDER/bus | ||
| Environment=LITELLM_MASTER_KEY=LITELLM_MASTER_KEY_PLACEHOLDER | ||
| Environment=LANGFUSE_PUBLIC_KEY=LANGFUSE_PUBLIC_KEY_PLACEHOLDER | ||
| Environment=LANGFUSE_SECRET_KEY=LANGFUSE_SECRET_KEY_PLACEHOLDER | ||
| Environment=LANGFUSE_HOST=http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER | ||
| Environment=OLLAMA_API_KEY=OLLAMA_API_KEY_PLACEHOLDER | ||
| Environment=PUBLIC_BASE_URL=PUBLIC_BASE_URL_PLACEHOLDER | ||
| Environment=ROUTING_DOMAIN=ROUTING_DOMAIN_PLACEHOLDER | ||
| Environment=LITELLM_PORT=LITELLM_PORT_PLACEHOLDER | ||
| Environment=VALKEY_CACHE_PORT=VALKEY_CACHE_PORT_PLACEHOLDER | ||
| Environment=ROUTER_PORT=ROUTER_PORT_PLACEHOLDER | ||
| Environment=LANGFUSE_WEB_PORT=LANGFUSE_WEB_PORT_PLACEHOLDER | ||
| Environment=LOG_LEVEL=info | ||
| Volume=DATA_ROOT_PLACEHOLDER/router-rendered:/config/router_dir | ||
| Volume=DATA_ROOT_PLACEHOLDER/litellm-rendered:/config/litellm_dir | ||
| Volume=DATA_ROOT_PLACEHOLDER/datasets:/app/data | ||
| Volume=HOME_PLACEHOLDER/.gemini:/config/gemini_auth | ||
| Volume=HOME_PLACEHOLDER/.gemini:/root/.gemini | ||
| Volume=HOME_PLACEHOLDER/.local/share/goose:/config/goose_sessions | ||
| Volume=HOME_PLACEHOLDER/.local/bin/agy:/usr/local/bin/agy | ||
| Volume=RUN_USER_PLACEHOLDER/bus:RUN_USER_PLACEHOLDER/bus | ||
| Volume=HOME_PLACEHOLDER/.local/share/keyrings:/root/.local/share/keyrings | ||
| Volume=RUN_USER_PLACEHOLDER:RUN_USER_PLACEHOLDER | ||
| Volume=WORKDIR_PLACEHOLDER/.env:/config/.env | ||
| HealthCmd=python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:ROUTER_PORT_PLACEHOLDER/metrics')" | ||
| HealthInterval=15s | ||
| HealthTimeout=5s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=240s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=300 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| [Unit] | ||
| Description=Valkey Cache — LiteLLM response cache (POD_NAME_PLACEHOLDER) | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-valkey-cache | ||
| Image=valkey/valkey:9.1.1 | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| # pod.yaml used command: (replaces entrypoint). Image entrypoint docker-entrypoint.sh | ||
| # would also work, but explicit Entrypoint= matches play-kube semantics exactly. | ||
| Entrypoint=valkey-server | ||
| Exec=--bind 127.0.0.1 --port VALKEY_CACHE_PORT_PLACEHOLDER --loglevel warning | ||
| Volume=DATA_ROOT_PLACEHOLDER/valkey-data:/data | ||
| HealthCmd=valkey-cli -p VALKEY_CACHE_PORT_PLACEHOLDER ping | ||
| HealthInterval=10s | ||
| HealthTimeout=2s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=5s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=90 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| [Unit] | ||
| Description=Valkey LF — Langfuse v3 BullMQ job queue (POD_NAME_PLACEHOLDER) | ||
|
|
||
| [Container] | ||
| ContainerName=POD_NAME_PLACEHOLDER-valkey-lf | ||
| Image=valkey/valkey:9.1.1 | ||
| Pod=llm-routing.pod | ||
| # play-kube injected <pod-name> -> 127.0.0.1 into /etc/hosts; replicate for apps | ||
| # (langfuse 3.222+) that resolve the pod hostname for self/intra-pod calls. | ||
| AddHost=POD_NAME_PLACEHOLDER:127.0.0.1 | ||
| Entrypoint=redis-server | ||
| Exec=--bind 127.0.0.1 --port VALKEY_LF_PORT_PLACEHOLDER --requirepass REDIS_AUTH_PLACEHOLDER --maxmemory-policy noeviction --loglevel warning | ||
| Volume=DATA_ROOT_PLACEHOLDER/redis-lf-data:/data | ||
| HealthCmd=valkey-cli -p VALKEY_LF_PORT_PLACEHOLDER -a REDIS_AUTH_PLACEHOLDER ping | ||
|
Contributor
There was a problem hiding this comment. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Locate quadlets files =="
fd -a 'llm-routing-(valkey-lf|clickhouse)\.container$' . || true
echo "== Relevant lines =="
for f in $(fd 'llm-routing-(valkey-lf|clickhouse)\.container$' .); do
echo "--- $f ---"
nl -ba "$f" | sed -n '1,80p'
done
echo "== References to placeholders =="
rg -n "REDIS_AUTH_PLACEHOLDER|CLICKHOUSE_PASSWORD_PLACEHOLDER|HealthCmd|VALKEYCLI_AUTH|CLICKHOUSE" . || true
echo "== Repository files around quadlets =="
git ls-files | rg '(^|/)quadlets/|\.env|container|compose' | head -200Repository: sheepdestroyer/LLM-Routing Length of output: 427 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Relevant lines =="
for f in ./quadlets/llm-routing-clickhouse.container ./quadlets/llm-routing-valkey-lf.container; do
echo "--- $f ---"
awk '{printf "%5d %s\n", NR, $0}' "$f" | sed -n '1,120p'
done
echo "== References to placeholders in quadlets =="
rg -n "REDIS_AUTH_PLACEHOLDER|CLICKHOUSE_PASSWORD_PLACEHOLDER|HealthCmd|VALKEYCLI_AUTH|CLICKHOUSE" ./quadlets || true
echo "== Nearby quadlets files =="
awk '{printf "%5d %s\n", NR, $0}' ./quadlets/llm-routing-clickhouse.containerRepository: sheepdestroyer/LLM-Routing Length of output: 6616 Do not pass credentials as healthcheck command-line arguments. Both templates render secrets into
📍 Affects 2 files
🤖 Prompt for AI Agents |
||
| HealthInterval=10s | ||
| HealthTimeout=2s | ||
| HealthOnFailure=kill | ||
| HealthRetries=5 | ||
| HealthStartPeriod=5s | ||
|
|
||
| [Service] | ||
| Restart=always | ||
| TimeoutStartSec=90 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # LLM Triage Gateway pod — Quadlet template (rendered by start-stack.sh) | ||
| # Pod name is rendered from POD_NAME (prod-router-pod / dev-router-pod). | ||
| [Unit] | ||
| Description=LLM Triage Gateway Pod (POD_NAME_PLACEHOLDER) | ||
| Wants=network-online.target | ||
| After=network-online.target | ||
|
|
||
| [Pod] | ||
| PodName=POD_NAME_PLACEHOLDER | ||
| Network=host | ||
|
|
||
| [Install] | ||
| WantedBy=default.target |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: sheepdestroyer/LLM-Routing
Length of output: 446
🏁 Script executed:
Repository: sheepdestroyer/LLM-Routing
Length of output: 402
🏁 Script executed:
Repository: sheepdestroyer/LLM-Routing
Length of output: 238
🏁 Script executed:
Repository: sheepdestroyer/LLM-Routing
Length of output: 318
🏁 Script executed:
Repository: sheepdestroyer/LLM-Routing
Length of output: 17615
🏁 Script executed:
Repository: sheepdestroyer/LLM-Routing
Length of output: 4640
🌐 Web query:
LiteLLM langfuse callback environment variables LANGFUSE_PUBLIC_KEY LANGFUSE_SECRET_KEY LANGFUSE_HOST💡 Result:
When using LiteLLM with Langfuse, the environment variables and configuration depend on whether you are using the standard Langfuse integration or the Langfuse OpenTelemetry (OTEL) integration [1][2][3]. Standard Langfuse Integration For the standard callback integration (enabled via litellm.success_callback = ["langfuse"]), the following environment variables are used [1][4]: - LANGFUSE_PUBLIC_KEY: Your Langfuse project public key [1][4]. - LANGFUSE_SECRET_KEY (or LANGFUSE_SECRET): Your Langfuse project secret key [1][4]. - LANGFUSE_HOST: Optional. The host URL for your Langfuse instance (e.g., https://us.cloud.langfuse.com) [1][4]. If omitted, it defaults to the Langfuse cloud [1][4]. You can also pass these credentials as keyword arguments directly to the completion or acompletion functions to override environment variables on a per-request basis [1][4]. Langfuse OpenTelemetry (OTEL) Integration When using the OpenTelemetry integration (enabled via litellm.callbacks = ["langfuse_otel"]), the variable for the host changes [2][3]: - LANGFUSE_PUBLIC_KEY: Required [5][3]. - LANGFUSE_SECRET_KEY: Required [5][3]. - LANGFUSE_OTEL_HOST: Optional. The endpoint host for Langfuse OTEL (e.g., https://us.cloud.langfuse.com) [2][3]. This variable is specific to the OTEL integration [5][3]. Summary Table Integration Type | Public Key Var | Secret Key Var | Host Var --- | --- | --- | --- Standard | LANGFUSE_PUBLIC_KEY | LANGFUSE_SECRET_KEY | LANGFUSE_HOST OTEL | LANGFUSE_PUBLIC_KEY | LANGFUSE_SECRET_KEY | LANGFUSE_OTEL_HOST For detailed setup, consult the official LiteLLM documentation for Langfuse integration [4] or the Langfuse OpenTelemetry integration documentation [3].
Citations:
Add the missing Langfuse credentials to the LiteLLM Quadlet.
litellm/config.yamlenablescallbacks: [- langfuse], and LiteLLM’s standard Langfuse callback needsLANGFUSE_PUBLIC_KEYandLANGFUSE_SECRET_KEYin addition toLANGFUSE_HOST; without them, LiteLLM traces won’t authenticate. Matchpod.yaml/llm-routing-router.containerby adding the two placeholder env vars to this container.🤖 Prompt for AI Agents