Skip to content

fix(deploy): finish Quadlet isolation hardening#350

Closed
sheepdestroyer wants to merge 7 commits into
masterfrom
fix/quadlet-environment-isolation
Closed

fix(deploy): finish Quadlet isolation hardening#350
sheepdestroyer wants to merge 7 commits into
masterfrom
fix/quadlet-environment-isolation

Conversation

@sheepdestroyer

@sheepdestroyer sheepdestroyer commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Supersedes #349.

Addresses all valid review findings:

  • Restart failures now report the exact namespaced or legacy owner unit.
  • Namespace validation regression coverage checks the complete shell expression.
  • Quadlet identifier suffixes are centralized in the embedded renderer.
  • Removed unused lifecycle helpers.
  • Added behavioral renderer coverage for dev/prod unit names and preservation of image names and URLs containing llm-routing.
  • Updated README and scripts documentation to use environment-specific service names.
  • Confirmed the local dev safety net remains on reachable host-networked http://127.0.0.1:8083 and does not use TLS-terminated host routes.

Validation:

  • Full test suite: 208 passed, one existing Starlette/httpx deprecation warning.
  • bash -n start-stack.sh passed.
  • git diff --check passed.
  • Dev full rebuild succeeded.
  • Dev canonical verification: 29/29 passed.
  • llama-router safety net remains reachable with the approved local models loaded.
  • Dev and production continue running under separate llm-routing-dev-* and llm-routing-prod-* Quadlet namespaces.

Summary by Sourcery

Harden Quadlet-based deployment isolation between dev and prod environments and clarify ownership and restart behavior for generated systemd units.

New Features:

  • Introduce an environment-specific Quadlet namespace variable to segregate dev and prod rendered units and systemd service names.

Bug Fixes:

  • Ensure Quadlet ownership detection and restart/teardown paths report and operate on the exact namespaced or legacy pod unit instead of a single hard-coded service name.
  • Fix namespace rendering to avoid rewriting image names and URLs that contain llm-routing while still namespacing Quadlet identifiers and dependencies.

Enhancements:

  • Validate Quadlet namespace values early in the startup script to prevent invalid unit names.
  • Centralize Quadlet identifier suffix handling in the embedded renderer and consistently namespace pod/container filenames and internal dependencies based on the active environment.
  • Scope DATA_ROOT to the current worktree by default so dev and prod stacks maintain separate persistent data and rendered configuration directories.

Documentation:

  • Update README and scripts documentation to describe environment-specific Quadlet namespaces, unit names, and data roots, including examples for dev and prod service management.

Tests:

  • Expand Quadlet template tests to cover environment-specific namespaces, renderer behavior for dev/prod unit names, preservation of llm-routing image names and URLs, namespace validation, and updated documentation expectations.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions github-actions Bot added documentation Improvements or additions to documentation scripts tests labels Jul 23, 2026
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@sheepdestroyer, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 56 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7910f53e-8c33-4139-b16a-692b18b6d98f

📥 Commits

Reviewing files that changed from the base of the PR and between 8e7eb90 and 49584f6.

📒 Files selected for processing (5)
  • .env.dev
  • README.md
  • scripts/README.md
  • start-stack.sh
  • tests/test_quadlet_templates.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/quadlet-environment-isolation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sheepdestroyer

Copy link
Copy Markdown
Owner Author

Superseded by #351 with the remaining review fixes: exact-unit diagnostics, centralized identifier suffixes, behavioral renderer coverage, worktree-scoped DATA_ROOT documentation/tests, and corrected operator commands.

@sourcery-ai

sourcery-ai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Hardens Quadlet deployment isolation by introducing environment-specific namespaces for dev/prod, updating the embedded Python renderer to namespace only Quadlet identifiers while preserving image/URL values, tightening restart and ownership diagnostics, and aligning documentation and tests with the new behavior.

Sequence diagram for Quadlet-owned stack restart with environment-specific unit names

sequenceDiagram
    actor User
    participant start_stack_sh as start_stack.sh
    participant stack_ownership
    participant systemd_user as systemd_user

    User->>start_stack_sh: ./start-stack.sh
    start_stack_sh->>stack_ownership: stack_ownership()
    stack_ownership-->>start_stack_sh: quadlet:<owner_unit>

    alt Quadlet-owned stack
        start_stack_sh->>systemd_user: systemctl --user restart owner_unit
        note over systemd_user: owner_unit may be
        note over systemd_user: llm-routing-prod-pod.service
        note over systemd_user: llm-routing-dev-pod.service
        note over systemd_user: llm-routing-pod.service (legacy)
    else Non-Quadlet or absent
        start_stack_sh->>start_stack_sh: deploy_fresh_pod()
    end
Loading

File-Level Changes

Change Details Files
Introduce environment-specific Quadlet namespace configuration and validation in the deployment script and use it for pod unit naming and render output paths.
  • Default QUADLET_NAMESPACE to llm-routing-prod and validate its format with a strict shell regex
  • Export QUADLET_NAMESPACE for use by the embedded Python renderer and other stack logic
  • Derive LLM_ROUTING_POD_UNIT from QUADLET_NAMESPACE and add LEGACY_LLM_ROUTING_POD_UNIT for backwards compatibility
  • Scope QUADLET_DIR to ~/.config/containers/systemd/${QUADLET_NAMESPACE} instead of a single shared llm-routing directory
start-stack.sh
tests/test_quadlet_templates.py
Refine Quadlet ownership detection and restart/teardown flows to track the exact pod unit name (including legacy units) and surface precise diagnostics.
  • Change stack_ownership to return quadlet: for both namespaced and legacy units
  • Update safe_pod_teardown to parse the owner unit from STACK_OWNERSHIP and operate on that unit instead of a fixed name
  • Update restart handling to use the parsed owner_unit in systemctl restart and status hint messages
start-stack.sh
tests/test_quadlet_templates.py
Centralize and harden Quadlet namespace rendering logic in the embedded Python renderer, ensuring only Quadlet identifiers are rewritten while preserving images and URLs that contain llm-routing.
  • Read QUADLET_NAMESPACE in the renderer and define a constrained list of identifier suffixes
  • Add a regex-based identifier_prefix for llm-routing-* names scoped to known unit/service suffixes
  • Introduce namespace_identifier to rewrite Pod/After/Wants/BindsTo/Requires/PartOf lines to namespaced values, including special-casing llm-routing.pod and llm-routing-pod.service
  • Change rendered file names and rendered_names sets to replace llm-routing with the namespace, isolating dev/prod unit filenames
  • Add tests that execute the embedded renderer via a temporary directory to verify that image names and PUBLIC_BASE_URL containing llm-routing are not rewritten while internal service dependencies are namespaced
start-stack.sh
tests/test_quadlet_templates.py
Document the environment-specific Quadlet namespaces and service names for dev and prod, and ensure examples and script docs use the correct units.
  • Update README systemd/journalctl examples to reference llm-routing-prod-* and llm-routing-dev-* units
  • Add a README section describing environment-isolated Quadlet deployment, including namespace-specific unit directories and DATA_ROOT scoping to worktrees
  • Update scripts/README to describe dev/prod-specific Quadlet directories and unit names and generic restart usage
README.md
scripts/README.md
Extend tests to cover environment-specific namespaces, ownership semantics, documentation expectations, and default DATA_ROOT scoping.
  • Add tests asserting QUADLET_NAMESPACE usage in .env.dev and start-stack.sh
  • Add tests verifying namespace rendering preserves non-identifiers such as image names and URLs containing llm-routing
  • Add tests ensuring restart/ownership messaging uses the exact owner unit name and that namespace validation is present
  • Add tests ensuring documentation examples no longer reference generic llm-routing-pod.service and that DATA_ROOT defaults to a worktree-scoped path distinct between dev and prod
tests/test_quadlet_templates.py
.env.dev
README.md
scripts/README.md

Possibly linked issues

  • #N/A: PR makes dev safety net reachable and achieves 29/29 canonical verification, directly satisfying the issue’s requirements.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • In the embedded Quadlet renderer, namespace_identifier has an unreachable elif field == "Pod" branch because "Pod" is already in the earlier set check; consider restructuring this logic so Pod-specific handling is clearly separated and the dead branch removed.
  • The tests that extract the embedded Python renderer from start-stack.sh by splitting on heredoc markers are quite fragile to formatting changes; consider moving the renderer into a separate Python module that start-stack.sh invokes so tests can import it directly.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In the embedded Quadlet renderer, `namespace_identifier` has an unreachable `elif field == "Pod"` branch because `"Pod"` is already in the earlier set check; consider restructuring this logic so Pod-specific handling is clearly separated and the dead branch removed.
- The tests that extract the embedded Python renderer from `start-stack.sh` by splitting on heredoc markers are quite fragile to formatting changes; consider moving the renderer into a separate Python module that `start-stack.sh` invokes so tests can import it directly.

## Individual Comments

### Comment 1
<location path="start-stack.sh" line_range="793-801" />
<code_context>
+        def namespace_identifier(match):
</code_context>
<issue_to_address>
**suggestion:** The `elif field == "Pod"` branch in `namespace_identifier` is unreachable and can be simplified.

Because `field` is constrained to `Pod|After|Wants|BindsTo|Requires|PartOf`, and the initial `if field in {...}` set already includes `"Pod"`, the `elif field == "Pod"` can never run. To avoid dead code and clarify control flow, either remove `"Pod"` from the initial set and keep the `elif`, or remove the `elif` and handle all `Pod`-specific replacements in the first branch.

```suggestion
        def namespace_identifier(match):
            field, value = match.group(1), match.group(2)
            if field in {"Pod", "After", "Wants", "BindsTo", "Requires", "PartOf"}:
                value = identifier_prefix.sub(namespace + "-", value)
                value = value.replace("llm-routing.pod", namespace + ".pod")
                value = value.replace("llm-routing-pod.service", namespace + "-pod.service")
            return f"{field}={value}"
```
</issue_to_address>

### Comment 2
<location path="tests/test_quadlet_templates.py" line_range="147-153" />
<code_context>
+        assert "Pod=llm-routing-dev.pod" in rendered
+
+
+def test_namespace_is_validated_and_ownership_preserves_exact_unit():
+    script = (ROOT / "start-stack.sh").read_text()
+    assert '[[ ! "$QUADLET_NAMESPACE" =~ ^[a-z0-9][a-z0-9-]*$ ]]' in script
+    assert "printf 'quadlet:%s\\n' \"$infra_unit\"" in script
+    assert 'owner_unit="${STACK_OWNERSHIP#quadlet:}"' in script
+    assert 'failed to restart ${owner_unit}' in script
+    assert 'status ${owner_unit} --no-pager' in script
+
+
</code_context>
<issue_to_address>
**suggestion (testing):** Add a behavioral test for invalid namespace values triggering validation failure

This test only checks that the validation regex and ownership/restart logic are present in `start-stack.sh`, but not that they actually work. Given this validation is part of hardening, please add a behavioral test that executes the script (or an extracted shell fragment) with an invalid `QUADLET_NAMESPACE` (e.g. `Invalid_Namespace` or one starting with `-`) and asserts it exits non‑zero and emits the expected validation error message. That way we verify the regex is enforced, not just defined.

Suggested implementation:

```python
ROOT = Path(__file__).resolve().parent.parent


def test_start_stack_script_hardening():
    script = (ROOT / "start-stack.sh").read_text()
    assert 'chmod 700 "$QUADLET_DIR"' in script
    assert "os.chmod(staged_path, 0o600)" in script
    assert 'LLM_ROUTING_POD_UNIT="${QUADLET_NAMESPACE}-pod.service"' in script
    assert 'QUADLET_DIR="${HOME}/.config/containers/systemd/${QUADLET_NAMESPACE}"' in script
    assert "systemctl --user daemon-reload" in script
    assert "stack_ownership()" in script


def test_namespace_is_validated_and_ownership_preserves_exact_unit():
    script = (ROOT / "start-stack.sh").read_text()
    assert '[[ ! "$QUADLET_NAMESPACE" =~ ^[a-z0-9][a-z0-9-]*$ ]]' in script
    assert "printf 'quadlet:%s\\n' \"$infra_unit\"" in script
    assert 'owner_unit="${STACK_OWNERSHIP#quadlet:}"' in script
    assert 'failed to restart ${owner_unit}' in script
    assert 'status ${owner_unit} --no-pager' in script


def test_invalid_namespace_is_rejected(tmp_path):
    script_path = ROOT / "start-stack.sh"

    env = os.environ.copy()
    env["HOME"] = str(tmp_path)
    # Intentionally invalid: contains uppercase and an underscore
    env["QUADLET_NAMESPACE"] = "Invalid_Namespace"

    result = subprocess.run(
        ["bash", str(script_path)],
        env=env,
        capture_output=True,
        text=True,
    )

    # The script should fail fast on invalid namespace
    assert result.returncode != 0

    output = result.stdout + result.stderr
    # The script is expected to emit a validation error message
    assert "Invalid QUADLET_NAMESPACE" in output

```

1. Ensure `os` is imported at the top of `tests/test_quadlet_templates.py`:
   `import os`
2. If `start-stack.sh` uses a different error message string for invalid namespaces,
   update the assertion in `test_invalid_namespace_is_rejected` to match the exact
   substring actually emitted by the script.
3. If the project prefers `shell=True` or a different interpreter (e.g. `["/usr/bin/env", "bash"]`)
   for executing shell scripts in tests, adjust the `subprocess.run` invocation accordingly.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread start-stack.sh
Comment on lines +793 to +801
def namespace_identifier(match):
field, value = match.group(1), match.group(2)
if field in {"Pod", "After", "Wants", "BindsTo", "Requires", "PartOf"}:
value = identifier_prefix.sub(namespace + "-", value)
value = value.replace("llm-routing.pod", namespace + ".pod")
value = value.replace("llm-routing-pod.service", namespace + "-pod.service")
elif field == "Pod":
value = value.replace("llm-routing.pod", namespace + ".pod")
return f"{field}={value}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: The elif field == "Pod" branch in namespace_identifier is unreachable and can be simplified.

Because field is constrained to Pod|After|Wants|BindsTo|Requires|PartOf, and the initial if field in {...} set already includes "Pod", the elif field == "Pod" can never run. To avoid dead code and clarify control flow, either remove "Pod" from the initial set and keep the elif, or remove the elif and handle all Pod-specific replacements in the first branch.

Suggested change
def namespace_identifier(match):
field, value = match.group(1), match.group(2)
if field in {"Pod", "After", "Wants", "BindsTo", "Requires", "PartOf"}:
value = identifier_prefix.sub(namespace + "-", value)
value = value.replace("llm-routing.pod", namespace + ".pod")
value = value.replace("llm-routing-pod.service", namespace + "-pod.service")
elif field == "Pod":
value = value.replace("llm-routing.pod", namespace + ".pod")
return f"{field}={value}"
def namespace_identifier(match):
field, value = match.group(1), match.group(2)
if field in {"Pod", "After", "Wants", "BindsTo", "Requires", "PartOf"}:
value = identifier_prefix.sub(namespace + "-", value)
value = value.replace("llm-routing.pod", namespace + ".pod")
value = value.replace("llm-routing-pod.service", namespace + "-pod.service")
return f"{field}={value}"

Comment on lines +147 to +153
def test_namespace_is_validated_and_ownership_preserves_exact_unit():
script = (ROOT / "start-stack.sh").read_text()
assert '[[ ! "$QUADLET_NAMESPACE" =~ ^[a-z0-9][a-z0-9-]*$ ]]' in script
assert "printf 'quadlet:%s\\n' \"$infra_unit\"" in script
assert 'owner_unit="${STACK_OWNERSHIP#quadlet:}"' in script
assert 'failed to restart ${owner_unit}' in script
assert 'status ${owner_unit} --no-pager' in script

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (testing): Add a behavioral test for invalid namespace values triggering validation failure

This test only checks that the validation regex and ownership/restart logic are present in start-stack.sh, but not that they actually work. Given this validation is part of hardening, please add a behavioral test that executes the script (or an extracted shell fragment) with an invalid QUADLET_NAMESPACE (e.g. Invalid_Namespace or one starting with -) and asserts it exits non‑zero and emits the expected validation error message. That way we verify the regex is enforced, not just defined.

Suggested implementation:

ROOT = Path(__file__).resolve().parent.parent


def test_start_stack_script_hardening():
    script = (ROOT / "start-stack.sh").read_text()
    assert 'chmod 700 "$QUADLET_DIR"' in script
    assert "os.chmod(staged_path, 0o600)" in script
    assert 'LLM_ROUTING_POD_UNIT="${QUADLET_NAMESPACE}-pod.service"' in script
    assert 'QUADLET_DIR="${HOME}/.config/containers/systemd/${QUADLET_NAMESPACE}"' in script
    assert "systemctl --user daemon-reload" in script
    assert "stack_ownership()" in script


def test_namespace_is_validated_and_ownership_preserves_exact_unit():
    script = (ROOT / "start-stack.sh").read_text()
    assert '[[ ! "$QUADLET_NAMESPACE" =~ ^[a-z0-9][a-z0-9-]*$ ]]' in script
    assert "printf 'quadlet:%s\\n' \"$infra_unit\"" in script
    assert 'owner_unit="${STACK_OWNERSHIP#quadlet:}"' in script
    assert 'failed to restart ${owner_unit}' in script
    assert 'status ${owner_unit} --no-pager' in script


def test_invalid_namespace_is_rejected(tmp_path):
    script_path = ROOT / "start-stack.sh"

    env = os.environ.copy()
    env["HOME"] = str(tmp_path)
    # Intentionally invalid: contains uppercase and an underscore
    env["QUADLET_NAMESPACE"] = "Invalid_Namespace"

    result = subprocess.run(
        ["bash", str(script_path)],
        env=env,
        capture_output=True,
        text=True,
    )

    # The script should fail fast on invalid namespace
    assert result.returncode != 0

    output = result.stdout + result.stderr
    # The script is expected to emit a validation error message
    assert "Invalid QUADLET_NAMESPACE" in output
  1. Ensure os is imported at the top of tests/test_quadlet_templates.py:
    import os
  2. If start-stack.sh uses a different error message string for invalid namespaces,
    update the assertion in test_invalid_namespace_is_rejected to match the exact
    substring actually emitted by the script.
  3. If the project prefers shell=True or a different interpreter (e.g. ["/usr/bin/env", "bash"])
    for executing shell scripts in tests, adjust the subprocess.run invocation accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation scripts tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant