Skip to content

feat: parameterize all ports and pod name for parallel prod/dev deployments #248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sheepdestroyer wants to merge 12 commits into from
Closed

feat: parameterize all ports and pod name for parallel prod/dev deployments #248

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
3084c68
feat: parameterize all ports and pod name for parallel prod/dev deplo…
Jul 11, 2026
143d8ef
Merge branch 'master' into chore/learn-deployment-guidelines
sheepdestroyer Jul 11, 2026
9e86358
fix: default DATA_ROOT to WORKDIR/data instead of WORKDIR
Jul 11, 2026
38d03b4
fix: move dataset mount under DATA_ROOT as datasets/
Jul 11, 2026
bce02a6
fix: remove dev-data workaround — DATA_ROOT now defaults to data/
Jul 11, 2026
24a7012
fix: consolidate gitignore — all data dirs now under data/
Jul 11, 2026
2fe9545
fix: router reads ports from env vars (LITELLM_PORT, VALKEY_CACHE_POR…
Jul 11, 2026
fff4592
fix: router image configurable via ROUTER_IMAGE env var; router reads…
Jul 11, 2026
02182ff
fix: render router config.yaml with LITELLM_PORT placeholder at deplo…
Jul 11, 2026
f4befbb
fix: replace tcpSocket probes with valkey-cli exec probes, switch to …
Jul 11, 2026
5f4c87e
fix: add -p VALKEY_CACHE_PORT_PLACEHOLDER to valkey-cache probes
Jul 11, 2026
1afa7c1
fix: address PR #248 and #249 review comments
Jul 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .env.dev
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Dev environment overlay — sourced AFTER .env to override prod values
# Secrets are inherited from .env; only env-specific config is set here.

# Pod identity
POD_NAME="dev-router-pod"

# Public URLs
BASEURL="dev.vendeuvre.lan"
BASE_URL="dev.vendeuvre.lan"
PUBLIC_BASE_URL="https://dev.vendeuvre.lan/llm-routing"

# Dev port assignments (+10 offset from production)
ROUTER_PORT="5010"
LITELLM_PORT="4010"
LANGFUSE_WEB_PORT="3011"
LANGFUSE_WORKER_PORT="3040"
POSTGRES_PORT="5442"
VALKEY_CACHE_PORT="6389"
VALKEY_LF_PORT="6390"
CLICKHOUSE_HTTP_PORT="8133"
CLICKHOUSE_TCP_PORT="9010"
CLICKHOUSE_INTERSERVER_PORT="9019"
MINIO_S3_PORT="9012"
MINIO_CONSOLE_PORT="9011"

# Dev uses locally-built router image (for testing code changes)
ROUTER_IMAGE="localhost/llm-routing-dev:latest"
12 changes: 3 additions & 9 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,9 @@
# Exclude Podman runtime file generated dynamically at startup
pod-run.yaml

# Exclude persistent database and cache storages
postgres-data/
langfuse-data/
valkey-data/
clickhouse-data/
minio-data/
redis-lf-data/
# Exclude persistent database and cache storages (all under data/)
data/


# Exclude auto-generated backups
backups/
Expand All @@ -36,7 +32,5 @@ router/free_models_roster.json
workdirs/
pr_description.txt

# Dataset work in progress
data/
.cache/
test_output*.log
6 changes: 3 additions & 3 deletions litellm/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ litellm_settings:
cache_params:
host: 127.0.0.1
mode: default
port: 6379
port: VALKEY_CACHE_PORT_PLACEHOLDER
ttl: 3600
type: redis
caching_backend: redis
Expand Down Expand Up @@ -77,7 +77,7 @@ model_list:
is_public_model_group: true
- litellm_params:
model: openai/llm-routing-ollama
api_base: http://127.0.0.1:5000/v1
api_base: http://127.0.0.1:ROUTER_PORT_PLACEHOLDER/v1
api_key: os.environ/LITELLM_MASTER_KEY
request_timeout: 120
model_name: llm-routing-ollama
Expand Down Expand Up @@ -237,7 +237,7 @@ model_list:

redis_settings:
redis_host: 127.0.0.1
redis_port: 6379
redis_port: VALKEY_CACHE_PORT_PLACEHOLDER
router_settings:
allowed_fails: 0
cooldown_time: 300
Expand Down
10 changes: 6 additions & 4 deletions litellm/entrypoint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
if line and not line.startswith("#") and "=" in line:
key, _, val = line.partition("=")
val = val.strip().strip('"').strip("'")

@sourcery-ai sourcery-ai Bot Jul 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Preserving quotes/whitespace in .env values can break numeric parsing (e.g. POSTGRES_PORT).

With the updated logic, a line like POSTGRES_PORT="5432" now sets os.environ['POSTGRES_PORT'] to the literal string "5432", so int(os.environ.get("POSTGRES_PORT", "5432")) will raise ValueError. Leading/trailing spaces cause the same issue. Please restore at least val.strip() (and optionally quote stripping) while keeping the new setdefault behavior.

os.environ[key] = val
os.environ.setdefault(key, val)
Comment on lines 18 to +20

@gemini-code-assist gemini-code-assist Bot Jul 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Without stripping quotes from the environment variable values parsed from the .env file, any quoted values (e.g., POSTGRES_PORT="5442") will retain their literal quotes (e.g., '"5442"'). This will cause a ValueError when attempting to cast them to integers later in the script (e.g., int(os.environ.get("POSTGRES_PORT", "5432"))).

We should restore the quote-stripping logic before calling os.environ.setdefault().

Suggested change
key, _, val = line.partition("=")
val = val.strip().strip('"').strip("'")
os.environ[key] = val
os.environ.setdefault(key, val)
key, _, val = line.partition("=")
val = val.strip().strip('"').strip("'")
os.environ.setdefault(key, val)


# Load Gemini OAuth token from credentials JSON
creds_path = "/config/gemini_auth/oauth_creds.json"
Expand Down Expand Up @@ -46,9 +46,10 @@ def check_tcp_port(ip: str, port: int) -> bool:
return False

max_wait = 60
print(f"🔌 Waiting for PostgreSQL on :5432 (max {max_wait}s)...")
postgres_port = int(os.environ.get("POSTGRES_PORT", "5432"))
print(f"🔌 Waiting for PostgreSQL on :{postgres_port} (max {max_wait}s)...")
for i in range(max_wait):
if check_tcp_port("127.0.0.1", 5432):
if check_tcp_port("127.0.0.1", postgres_port):
print(f"✅ PostgreSQL ready after {i+1}s")
break
time.sleep(1)
Expand Down Expand Up @@ -127,5 +128,6 @@ def _serialize_dt(dt):
# Start LiteLLM Proxy
import litellm
from litellm.proxy.proxy_cli import run_server
sys.argv = ["litellm", "--config", "/app/config.yaml", "--port", "4000"]
litellm_port = os.environ.get("LITELLM_PORT", os.environ.get("PORT", "4000"))
sys.argv = ["litellm", "--config", "/app/config.yaml", "--port", litellm_port]
run_server()
Loading