Skip to content

feat: add upgrade-prod.sh for release-to-prod deployments#283

Closed
sheepdestroyer wants to merge 4 commits into
masterfrom
feat/add-upgrade-prod-script
Closed

feat: add upgrade-prod.sh for release-to-prod deployments#283
sheepdestroyer wants to merge 4 commits into
masterfrom
feat/add-upgrade-prod-script

Conversation

@sheepdestroyer

Copy link
Copy Markdown
Owner

Summary

Adds scripts/upgrade-prod.sh — syncs runtime files from a GitHub release tag into ~/prod/LLM-Routing and redeploys.

What it does

  • Clones a release tag, rsyncs pod.yaml, start-stack.sh, litellm/, router/, scripts/ into prod
  • Self-copy guard: re-execs from /tmp so rsync can safely update the script itself
  • Pod stays running during file sync — only rendered configs in DATA_ROOT are used at runtime; start-stack.sh --pull handles graceful shutdown with pre-deploy backup
  • Per-directory rsync with --delete (clean stale files within each dir, no data/ contamination)
  • Pre-flight env var validation before touching anything
  • Non-interactive auto-proceed when stdin is not a TTY
  • .env and data/ are never touched

Location

Under scripts/ per repo convention — root is for primary orchestration (start-stack.sh), scripts/ for supporting tools (backup.sh, etc.). README already documents it at this path.

Why

Prod deployments previously required manual file syncing. This standardizes the workflow and prevents the single-rsync corruption seen during the v0.1.5 deployment.

boy added 3 commits July 12, 2026 23:37
- Clones a release tag to a temp dir, rsyncs pod.yaml, start-stack.sh,
  litellm/, router/, scripts/ into ~/prod/LLM-Routing
- Self-copy guard: re-execs from /tmp so the script can safely update itself
- Per-directory rsync (no data/ contamination from --delete)
- Pre-flight env var validation: checks OPENROUTER_API_KEY, OLLAMA_API_KEY,
  LLAMA_CLASSIFIER_URL, PUBLIC_BASE_URL before stopping the pod
- Non-interactive auto-proceed when stdin is not a TTY
- .env and data/ are never touched
- Remove manual pod stop before rsync (let start-stack.sh --pull handle
  graceful shutdown, which includes pre-deploy database backup)
- Guard cleanup rm commands with -n checks to prevent noisy errors on
  early exit when TEMP_DIR/UPGRADE_PROD_SELF_PATH are unset
- Use git clone -q instead of piping to tail -1 (preserves full error
  output from stderr on clone failure)
- Delete duplicate scripts/upgrade-prod.sh (root-level is canonical)
Root is for primary orchestration (start-stack.sh); supporting tools
live under scripts/ (backup.sh, upgrade-prod.sh, etc.). README already
documents it at this path.
@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@sheepdestroyer, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 32 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c327c863-fb09-4615-aaad-394fe5db671e

📥 Commits

Reviewing files that changed from the base of the PR and between 3b203a4 and a349e97.

📒 Files selected for processing (1)
  • scripts/upgrade-prod.sh
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/add-upgrade-prod-script

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @sheepdestroyer, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the scripts/upgrade-prod.sh script by improving cleanup logic, silencing the git clone command, removing the inline pod shutdown step, and adding a pre-flight check to source and validate required environment variables from .env. The review feedback highlights a critical syntax error in the environment variable validation loop where combining indirect expansion with parameter expansion modifiers (${!var:-}) causes a script crash under set -u. Additionally, it suggests avoiding the use of the leaked loop variable $var when printing the example error message, recommending referencing the first missing variable explicitly instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread scripts/upgrade-prod.sh
Comment on lines +117 to +121
for var in OPENROUTER_API_KEY OLLAMA_API_KEY LLAMA_CLASSIFIER_URL PUBLIC_BASE_URL; do
if [ -z "${!var:-}" ]; then
missing_vars+=("$var")
fi
done

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

In Bash, combining indirect expansion (${!var}) with parameter expansion modifiers like :- (i.e., ${!var:-}) is a syntax error (bad substitution) and will cause the script to crash immediately when executed.

Additionally, because set -u is active (via set -euo pipefail on line 15), referencing an unset variable indirectly via ${!var} will trigger an unbound variable error and terminate the script.

To safely check if a dynamically named variable is set and non-empty under set -u, you can use the [[ -v var_name ]] built-in to check if it is defined, and only evaluate ${!var} if it is set.

Suggested change
for var in OPENROUTER_API_KEY OLLAMA_API_KEY LLAMA_CLASSIFIER_URL PUBLIC_BASE_URL; do
if [ -z "${!var:-}" ]; then
missing_vars+=("$var")
fi
done
if ! [[ -v $var ]] || [ -z "${!var}" ]; then
missing_vars+=("$var")
fi

Comment thread scripts/upgrade-prod.sh Outdated
done
echo ""
echo "Add them before upgrading. Example:"
echo " echo '$var=your_value' >> $PROD_DIR/.env"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Relying on the loop variable $var leaking outside of the for loop is a code smell and can lead to unexpected behavior if the loop structure changes. Since var will always hold the last element of missing_vars after the loop, showing it as the only example might also be confusing if multiple variables are missing.

Instead, you can explicitly reference the first missing variable using ${missing_vars[0]} to provide a concrete and accurate example.

Suggested change
echo " echo '$var=your_value' >> $PROD_DIR/.env"
echo " echo '${missing_vars[0]}=your_value' >> $PROD_DIR/.env"

- Replace ${!var:-} (syntax error under set -u) with [[ -v var ]] + ${!var}
- Fix leaked loop variable in error example: use ${missing_vars[0]} instead of $var

Both flagged by Gemini Code Assist on PR #283.
@sheepdestroyer

Copy link
Copy Markdown
Owner Author

Closing in favor of a fresh PR with all Gemini review fixes applied. See #284.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant