Skip to content

fix(deploy): isolate dev and prod Quadlet namespaces#348

Closed
sheepdestroyer wants to merge 3 commits into
masterfrom
fix/quadlet-environment-isolation
Closed

fix(deploy): isolate dev and prod Quadlet namespaces#348
sheepdestroyer wants to merge 3 commits into
masterfrom
fix/quadlet-environment-isolation

Conversation

@sheepdestroyer

@sheepdestroyer sheepdestroyer commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Isolate generated systemd Quadlet files and unit names between dev and production. Dev now uses llm-routing-dev-* units and ~/.config/containers/systemd/llm-routing-dev; prod uses llm-routing-prod-* and its own namespace. This prevents a dev deployment from stopping or overwriting production. Added regression coverage and updated deployment documentation. Validation: full pytest suite, bash syntax checks, and successful dev full rebuild with isolated units.

Summary by Sourcery

Isolate systemd Quadlet deployment between dev and production by introducing environment-specific namespaces for rendered units and pod services while maintaining compatibility with existing deployments.

New Features:

  • Introduce QUADLET_NAMESPACE to parameterize Quadlet unit names and directories per environment.
  • Namespace Quadlet-generated unit filenames and internal dependencies so dev and prod use distinct llm-routing-* identifiers.

Bug Fixes:

  • Prevent dev Quadlet deployments from stopping, overwriting, or conflicting with production systemd units by separating their namespaces.
  • Ensure Quadlet stack ownership checks and lifecycle operations handle both namespaced and legacy llm-routing-pod.service units.

Enhancements:

  • Add helper functions for stopping and resetting Quadlet units to centralize lifecycle handling.
  • Refine Quadlet rendering logic to safely rewrite only unit-related llm-routing identifiers without touching image names, URLs, or other configuration values.

Documentation:

  • Document environment-isolated Quadlet deployment behavior and namespaces in the main README.
  • Update scripts README to describe dev/prod-specific Quadlet directories and unit names and how to use them for diagnostics.

Tests:

  • Add regression tests covering environment-specific Quadlet namespaces, including dev/prod .env values and namespaced rendering behavior.

Summary by CodeRabbit

  • New Features

    • Added environment-specific Quadlet namespaces for development and production deployments.
    • Isolated generated service units, configuration files, containers, ports, and data paths between environments.
    • Improved deployment handling for both current and legacy service units.
  • Documentation

    • Added setup and troubleshooting guidance for environment-isolated deployments, including updated service status and log commands.
  • Tests

    • Added coverage verifying distinct development and production namespaces and correctly generated service names.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@sourcery-ai

sourcery-ai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

This PR introduces environment-specific Quadlet namespaces so dev and prod deployments render to separate systemd unit directories and use distinct unit names, while preserving backwards compatibility with the previous single-namespace setup and documenting the new behavior.

File-Level Changes

Change Details Files
Introduce QUADLET_NAMESPACE and wire it through Quadlet rendering and unit management so dev/prod use separate systemd unit names and directories.
  • Default QUADLET_NAMESPACE to llm-routing-prod in start-stack.sh and export it for downstream tooling.
  • Derive LLM_ROUTING_POD_UNIT from QUADLET_NAMESPACE and set QUADLET_DIR to an environment-specific systemd Quadlet directory under the user config.
  • Pass QUADLET_NAMESPACE into the embedded Python renderer and use it to namespace Quadlet identifiers and filenames, including pod names, service names, and internal dependencies.
start-stack.sh
Maintain compatibility with legacy non-namespaced Quadlet units during teardown and restart.
  • Track a LEGACY_LLM_ROUTING_POD_UNIT alongside the namespaced unit.
  • Add stop_quadlet_units and reset_quadlet_units helpers that operate on both the new namespaced unit and the legacy unit when they differ.
  • Use the new helpers in safe_pod_teardown and Quadlet restart paths so both legacy and namespaced deployments are reconciled cleanly.
  • Extend stack_ownership() to recognize both namespaced and legacy PODMAN_SYSTEMD_UNIT labels as Quadlet-owned.
start-stack.sh
Add regression tests and documentation for environment-isolated Quadlet behavior.
  • Extend Quadlet template tests to assert the new QUADLET_NAMESPACE usage, derived LLM_ROUTING_POD_UNIT, and QUADLET_DIR wiring, plus the namespace regex and filename rewriting in the renderer.
  • Add a new test ensuring .env.dev and .env define distinct QUADLET_NAMESPACE values for dev and prod.
  • Update top-level and scripts README files to describe the environment-specific Quadlet namespaces, directories, and unit names, and how to use them with systemctl and journalctl.
tests/test_quadlet_templates.py
README.md
scripts/README.md
.env.dev
.env

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@github-actions github-actions Bot added documentation Improvements or additions to documentation scripts tests labels Jul 23, 2026
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@sheepdestroyer, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 53 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 91bbe9d4-681a-4268-a693-3cd0c47dd97d

📥 Commits

Reviewing files that changed from the base of the PR and between e4e5780 and 919f22a.

📒 Files selected for processing (1)
  • tests/test_quadlet_templates.py
📝 Walkthrough

Walkthrough

Quadlet deployment now derives systemd unit names, render directories, template references, and lifecycle operations from QUADLET_NAMESPACE. Development and production use separate namespaces, while legacy unit handling remains supported and tests verify the generated names and paths.

Changes

Quadlet namespace isolation

Layer / File(s) Summary
Namespace configuration and deployment contract
.env.dev, start-stack.sh, README.md, scripts/README.md
Development sets llm-routing-dev; the script derives namespaced Quadlet units and directories, and documentation describes the dev/prod layouts and diagnostics commands.
Quadlet lifecycle and legacy compatibility
start-stack.sh
Ownership, stop, reset, teardown, and restart paths handle both namespaced and legacy Quadlet pod units.
Namespaced rendering and validation
start-stack.sh, tests/test_quadlet_templates.py
Rendering rewrites Quadlet identifiers and output filenames using the namespace, while tests verify environment-specific namespaces, units, paths, and replacements.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Environment
  participant start-stack.sh
  participant QuadletTemplates
  participant UserSystemd
  Environment->>start-stack.sh: Set QUADLET_NAMESPACE
  start-stack.sh->>QuadletTemplates: Rewrite identifiers and filenames
  QuadletTemplates->>UserSystemd: Install namespaced Quadlet units
  start-stack.sh->>UserSystemd: Stop or reset namespaced and legacy units
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: isolating dev and prod Quadlet namespaces for deployment.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/quadlet-environment-isolation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The hardcoded regex list of Quadlet identifiers in render_quadlets (pod|clickhouse|langfuse|...) is brittle; consider centralizing this list or deriving it from the templates so adding new units doesn’t silently skip namespacing.
  • The new stop_quadlet_units and reset_quadlet_units helpers duplicate the legacy-unit branching logic; consider parameterizing or consolidating this pattern to avoid divergence if more unit variants are introduced.
  • Several tests assert exact substrings of the shell script (e.g., specific QUADLET_NAMESPACE and regex lines), which can make refactors noisy; consider tightening these to focus on behavior (namespace wiring and isolation) rather than precise implementation text where possible.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The hardcoded regex list of Quadlet identifiers in `render_quadlets` (pod|clickhouse|langfuse|...) is brittle; consider centralizing this list or deriving it from the templates so adding new units doesn’t silently skip namespacing.
- The new `stop_quadlet_units` and `reset_quadlet_units` helpers duplicate the legacy-unit branching logic; consider parameterizing or consolidating this pattern to avoid divergence if more unit variants are introduced.
- Several tests assert exact substrings of the shell script (e.g., specific `QUADLET_NAMESPACE` and regex lines), which can make refactors noisy; consider tightening these to focus on behavior (namespace wiring and isolation) rather than precise implementation text where possible.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
start-stack.sh (1)

552-561: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Preserve the actual Quadlet owner during teardown.

Ownership detection loses whether the pod belongs to the namespaced or legacy unit, while cleanup then stops and resets both. During migration, a development teardown can therefore stop production’s legacy llm-routing-pod.service; conversely, a failed/stopped legacy unit is missed when no pod object remains.

  • start-stack.sh#L552-L561: probe both unit names when the pod is absent and return the specific owning unit.
  • start-stack.sh#L564-L577: make stop/reset operate only on that detected owner instead of unconditionally touching the legacy unit.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@start-stack.sh` around lines 552 - 561, Update the ownership detection logic
around the pod-unit checks at start-stack.sh lines 552-561 to probe both
namespaced and legacy unit names when no pod object exists, returning the
specific detected owner rather than a generic Quadlet result; update the
teardown logic at start-stack.sh lines 564-577 to stop and reset only that
detected owner, preserving the existing behavior when neither unit exists.
🧹 Nitpick comments (1)
tests/test_quadlet_templates.py (1)

47-48: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy lift

Test rendered behavior instead of source strings.

These assertions only verify literal text inside start-stack.sh; they do not render dev/prod templates or exercise lifecycle ownership. Add behavioral coverage that checks generated filenames, internal dependencies, preservation of values containing llm-routing-*, and legacy versus namespaced teardown behavior.

Also applies to: 93-101

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_quadlet_templates.py` around lines 47 - 48, Replace the
source-string assertions in the relevant test with behavioral coverage that
renders both dev and prod templates and validates the generated filenames,
internal dependencies, and preservation of values containing “llm-routing-*”.
Also verify lifecycle ownership by covering both legacy teardown behavior and
namespaced teardown behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@start-stack.sh`:
- Around line 789-799: The namespace substitutions around the rendered Quadlet
text currently rewrite arbitrary image names, URLs, and configuration values.
Update the relevant generation logic to namespace only known Quadlet identifier
and unit-reference fields, preserving values such as image names and URLs
containing “llm-routing”; ensure coverage verifies those values remain
unchanged.
- Around line 82-83: Validate the inherited or default QUADLET_NAMESPACE
immediately after it is assigned and before it is used to construct QUADLET_DIR,
rendered filenames, systemd unit names, or regex replacements. Accept only a
safe namespace format that excludes path traversal, slashes, whitespace,
backslashes, and other characters invalid for these identifiers; reject invalid
values with a clear error and terminate the script.

---

Outside diff comments:
In `@start-stack.sh`:
- Around line 552-561: Update the ownership detection logic around the pod-unit
checks at start-stack.sh lines 552-561 to probe both namespaced and legacy unit
names when no pod object exists, returning the specific detected owner rather
than a generic Quadlet result; update the teardown logic at start-stack.sh lines
564-577 to stop and reset only that detected owner, preserving the existing
behavior when neither unit exists.

---

Nitpick comments:
In `@tests/test_quadlet_templates.py`:
- Around line 47-48: Replace the source-string assertions in the relevant test
with behavioral coverage that renders both dev and prod templates and validates
the generated filenames, internal dependencies, and preservation of values
containing “llm-routing-*”. Also verify lifecycle ownership by covering both
legacy teardown behavior and namespaced teardown behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 74a04139-4bf2-4836-b918-4b3915b78c35

📥 Commits

Reviewing files that changed from the base of the PR and between 8e7eb90 and e4e5780.

📒 Files selected for processing (5)
  • .env.dev
  • README.md
  • scripts/README.md
  • start-stack.sh
  • tests/test_quadlet_templates.py

Comment thread start-stack.sh
Comment on lines +82 to +83
QUADLET_NAMESPACE="${QUADLET_NAMESPACE:-llm-routing-prod}"
export QUADLET_NAMESPACE

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Validate QUADLET_NAMESPACE before using it as a path and unit-name fragment.

Lines 82-83 accept arbitrary inherited values, which are later interpolated into QUADLET_DIR, rendered filenames, systemd unit names, and a regex replacement. Values containing / or .. can escape the intended namespace directory; whitespace or backslashes can also produce invalid or altered unit identifiers.

Proposed validation
 QUADLET_NAMESPACE="${QUADLET_NAMESPACE:-llm-routing-prod}"
+if [[ ! "$QUADLET_NAMESPACE" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then
+    echo "❌ Error: QUADLET_NAMESPACE must contain only lowercase letters, digits, and hyphens" >&2
+    exit 1
+fi
 export QUADLET_NAMESPACE
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
QUADLET_NAMESPACE="${QUADLET_NAMESPACE:-llm-routing-prod}"
export QUADLET_NAMESPACE
QUADLET_NAMESPACE="${QUADLET_NAMESPACE:-llm-routing-prod}"
if [[ ! "$QUADLET_NAMESPACE" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then
echo "❌ Error: QUADLET_NAMESPACE must contain only lowercase letters, digits, and hyphens" >&2
exit 1
fi
export QUADLET_NAMESPACE
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@start-stack.sh` around lines 82 - 83, Validate the inherited or default
QUADLET_NAMESPACE immediately after it is assigned and before it is used to
construct QUADLET_DIR, rendered filenames, systemd unit names, or regex
replacements. Accept only a safe namespace format that excludes path traversal,
slashes, whitespace, backslashes, and other characters invalid for these
identifiers; reject invalid values with a clear error and terminate the script.

Comment thread start-stack.sh
Comment on lines +789 to +799
# Quadlet generated unit names are global in the user systemd manager.
# Namespace both filenames and internal dependencies to isolate dev/prod.
# Namespace Quadlet identifiers only. Do not rewrite image names, URL
# paths, or other configuration values containing "llm-routing".
text = re.sub(
r"llm-routing-(?=(?:pod|clickhouse|langfuse|litellm|minio|postgres|router|valkey))",
namespace + "-",
text,
)
text = text.replace("llm-routing.pod", namespace + ".pod")
text = text.replace("llm-routing-pod.service", namespace + "-pod.service")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Scope namespace substitutions to Quadlet identifier fields.

These substitutions run over the entire rendered file after placeholder values are inserted. For example, an image value such as registry/llm-routing-router:latest would become registry/llm-routing-dev-router:latest, despite the comment saying image names and configuration values must not be rewritten. Apply replacements only to known unit-reference fields, or perform them before injecting arbitrary configuration values and add coverage for image/URL values containing llm-routing-*.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@start-stack.sh` around lines 789 - 799, The namespace substitutions around
the rendered Quadlet text currently rewrite arbitrary image names, URLs, and
configuration values. Update the relevant generation logic to namespace only
known Quadlet identifier and unit-reference fields, preserving values such as
image names and URLs containing “llm-routing”; ensure coverage verifies those
values remain unchanged.

@sheepdestroyer

Copy link
Copy Markdown
Owner Author

Superseded by a clean replacement PR with all review fixes, exact-owner teardown, namespace validation, behavioral coverage, and live dev/prod validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation scripts tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant