Skip to content

fix: replace tcpSocket probes with valkey-cli exec probes, switch to non-alpine valkey image #249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sheepdestroyer wants to merge 10 commits into from
Closed

fix: replace tcpSocket probes with valkey-cli exec probes, switch to non-alpine valkey image #249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3084c68
feat: parameterize all ports and pod name for parallel prod/dev deplo…
Jul 11, 2026
3bcf555
fix: default DATA_ROOT to WORKDIR/data instead of WORKDIR
Jul 11, 2026
6907d02
fix: move dataset mount under DATA_ROOT as datasets/
Jul 11, 2026
a4a3940
fix: remove dev-data workaround — DATA_ROOT now defaults to data/
Jul 11, 2026
a5fcaf1
fix: consolidate gitignore — all data dirs now under data/
Jul 11, 2026
cc5b4e3
fix: router reads ports from env vars (LITELLM_PORT, VALKEY_CACHE_POR…
Jul 11, 2026
9564d2c
fix: router image configurable via ROUTER_IMAGE env var; router reads…
Jul 11, 2026
ea36046
fix: render router config.yaml with LITELLM_PORT placeholder at deplo…
Jul 11, 2026
9f9a3e4
fix: replace tcpSocket probes with valkey-cli exec probes, switch to …
Jul 11, 2026
bba9418
fix: add -p VALKEY_CACHE_PORT_PLACEHOLDER to valkey-cache probes
Jul 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .env.dev
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Dev environment overlay — sourced AFTER .env to override prod values
# Secrets are inherited from .env; only env-specific config is set here.

# Pod identity
POD_NAME="dev-router-pod"

# Public URLs
BASEURL="dev.vendeuvre.lan"
BASE_URL="dev.vendeuvre.lan"
PUBLIC_BASE_URL="https://dev.vendeuvre.lan/llm-routing"

# Dev port assignments (+10 offset from production)
ROUTER_PORT="5010"
LITELLM_PORT="4010"
LANGFUSE_WEB_PORT="3011"
LANGFUSE_WORKER_PORT="3040"
POSTGRES_PORT="5442"
VALKEY_CACHE_PORT="6389"
VALKEY_LF_PORT="6390"
CLICKHOUSE_HTTP_PORT="8133"
CLICKHOUSE_TCP_PORT="9010"
CLICKHOUSE_INTERSERVER_PORT="9019"
MINIO_S3_PORT="9012"
MINIO_CONSOLE_PORT="9011"

# Dev uses locally-built router image (for testing code changes)
ROUTER_IMAGE="localhost/llm-routing-dev:latest"
12 changes: 3 additions & 9 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,9 @@
# Exclude Podman runtime file generated dynamically at startup
pod-run.yaml

# Exclude persistent database and cache storages
postgres-data/
langfuse-data/
valkey-data/
clickhouse-data/
minio-data/
redis-lf-data/
# Exclude persistent database and cache storages (all under data/)
data/


# Exclude auto-generated backups
backups/
Expand All @@ -36,7 +32,5 @@ router/free_models_roster.json
workdirs/
pr_description.txt

# Dataset work in progress
data/
.cache/
test_output*.log
6 changes: 3 additions & 3 deletions litellm/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ litellm_settings:
cache_params:
host: 127.0.0.1
mode: default
port: 6379
port: VALKEY_CACHE_PORT_PLACEHOLDER
ttl: 3600
type: redis
caching_backend: redis
Expand Down Expand Up @@ -77,7 +77,7 @@ model_list:
is_public_model_group: true
- litellm_params:
model: openai/llm-routing-ollama
api_base: http://127.0.0.1:5000/v1
api_base: http://127.0.0.1:ROUTER_PORT_PLACEHOLDER/v1
api_key: os.environ/LITELLM_MASTER_KEY
request_timeout: 120
model_name: llm-routing-ollama
Expand Down Expand Up @@ -237,7 +237,7 @@ model_list:

redis_settings:
redis_host: 127.0.0.1
redis_port: 6379
redis_port: VALKEY_CACHE_PORT_PLACEHOLDER
router_settings:
allowed_fails: 0
cooldown_time: 300
Expand Down
11 changes: 6 additions & 5 deletions litellm/entrypoint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@
line = line.strip()
if line and not line.startswith("#") and "=" in line:
key, _, val = line.partition("=")
val = val.strip().strip('"').strip("'")
os.environ[key] = val
os.environ.setdefault(key, val)
Comment on lines 18 to +19

@gemini-code-assist gemini-code-assist Bot Jul 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The quote and whitespace stripping was accidentally removed during the refactoring to os.environ.setdefault. Without stripping, values in .env that are wrapped in quotes (e.g., POSTGRES_PORT="5442") will be loaded with literal quotes, causing parsing failures (such as ValueError when converting POSTGRES_PORT to an integer on line 48).

Suggested change
key, _, val = line.partition("=")
val = val.strip().strip('"').strip("'")
os.environ[key] = val
os.environ.setdefault(key, val)
key, _, val = line.partition("=")
val = val.strip().strip('"').strip("'")
os.environ.setdefault(key, val)

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect the .env parsing logic in litellm/entrypoint.py lines 12-20
sed -n '12,20p' litellm/entrypoint.py

Repository: sheepdestroyer/LLM-Routing

Length of output: 469

Strip .env values before storing
line.strip() only trims the whole line; val is still assigned raw here, so quoted values and trailing spaces will be preserved in os.environ. Strip/unwrap val before setdefault.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@litellm/entrypoint.py` at line 19, Update the environment-loading logic
around os.environ.setdefault to normalize val before storing it: trim
surrounding whitespace and unwrap surrounding quotes, while preserving the
existing setdefault behavior and key handling.


# Load Gemini OAuth token from credentials JSON
creds_path = "/config/gemini_auth/oauth_creds.json"
Expand Down Expand Up @@ -46,9 +45,10 @@ def check_tcp_port(ip: str, port: int) -> bool:
return False

max_wait = 60
print(f"🔌 Waiting for PostgreSQL on :5432 (max {max_wait}s)...")
postgres_port = int(os.environ.get("POSTGRES_PORT", "5432"))
print(f"🔌 Waiting for PostgreSQL on :{postgres_port} (max {max_wait}s)...")
for i in range(max_wait):
if check_tcp_port("127.0.0.1", 5432):
if check_tcp_port("127.0.0.1", postgres_port):
print(f"✅ PostgreSQL ready after {i+1}s")
break
time.sleep(1)
Expand Down Expand Up @@ -127,5 +127,6 @@ def _serialize_dt(dt):
# Start LiteLLM Proxy
import litellm
from litellm.proxy.proxy_cli import run_server
sys.argv = ["litellm", "--config", "/app/config.yaml", "--port", "4000"]
litellm_port = os.environ.get("LITELLM_PORT", os.environ.get("PORT", "4000"))
sys.argv = ["litellm", "--config", "/app/config.yaml", "--port", litellm_port]
run_server()
Loading