Skip to content

fix(deploy): abort foreign legacy pod conflicts#353

Merged
sheepdestroyer merged 13 commits into
masterfrom
fix/quadlet-environment-isolation
Jul 23, 2026
Merged

fix(deploy): abort foreign legacy pod conflicts#353
sheepdestroyer merged 13 commits into
masterfrom
fix/quadlet-environment-isolation

Conversation

@sheepdestroyer

@sheepdestroyer sheepdestroyer commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Supersedes #352.

Final review fixes:

  • A mismatched existing legacy pod is now classified as a hard conflict, not as absent.
  • Deployment aborts before cleanup, replacement, or Quadlet start when a foreign legacy unit owns the pod name.
  • Added an isolated conflict harness covering the existing-pod mismatch path.
  • Updated the backend target table and fallback prose to document the local llama.cpp safety net (local-qwen-3.6) before the Ollama and OpenRouter fallbacks.

Validation:

  • 15 targeted Quadlet tests passed.
  • bash -n start-stack.sh passed.
  • git diff --check passed.
  • Existing mismatched legacy pod resolves to conflict:llm-routing-pod.service and aborts safely.
  • Dev canonical verification previously passed 29/29.
  • http://127.0.0.1:8083 remains reachable with the approved local models.

Summary by Sourcery

Isolate Quadlet deployment between dev and prod environments and harden stack ownership detection so foreign legacy units cannot be torn down or reused, while updating routing documentation for the local llama.cpp safety net.

New Features:

  • Introduce environment-specific Quadlet namespaces and unit names for dev and prod deployments.
  • Namespace rendered Quadlet templates and internal unit dependencies based on the configured QUADLET_NAMESPACE.
  • Add explicit conflict handling that detects mismatched legacy pod owners and aborts deployment with guidance.

Bug Fixes:

  • Treat existing pods attached to a foreign legacy unit as hard conflicts instead of absent ownership to avoid tearing down unrelated stacks.

Enhancements:

  • Validate QUADLET_NAMESPACE format and use environment-scoped Quadlet directories and data roots to keep dev/prod configuration and state separate.
  • Refine stack ownership and teardown logic to operate on the exact owning unit and distinguish Quadlet vs legacy units.
  • Update documentation and scripts README to describe environment-specific unit names and diagnostics for dev/prod.
  • Document the local-qwen-3.6 llama.cpp safety net in the LiteLLM routing table and fallback chain diagrams.

Tests:

  • Extend Quadlet template tests to cover namespace rendering, ownership/conflict detection, environment-specific documentation, and worktree-scoped data roots.

Summary by CodeRabbit

  • New Features

    • Added environment-isolated dev and production deployments with distinct systemd service namespaces and data locations.
    • Added automatic namespace-aware service rendering and management.
    • Added safeguards to detect conflicting or legacy stack ownership before deployment.
    • Existing stacks can now be restarted through their specific owning service.
  • Documentation

    • Updated routing fallback behavior and operational commands for environment-specific services.
    • Documented environment-isolated deployment and troubleshooting workflows.
  • Bug Fixes

    • Improved protection against accidentally sharing or disrupting dev and production resources.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@sourcery-ai

sourcery-ai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

This PR hardens Quadlet deployment by introducing environment-specific namespaces for generated systemd units, refining stack ownership detection to distinguish safe legacy reuse from foreign conflicts, and updating documentation/tests to cover the new isolation and local llama.cpp fallback behavior.

Sequence diagram for Quadlet stack ownership and conflict-aware deployment

sequenceDiagram
    actor Dev
    participant start_stack_sh
    participant podman
    participant systemd_user

    Dev->>start_stack_sh: run start_stack.sh
    start_stack_sh->>podman: podman pod exists POD_NAME
    podman-->>start_stack_sh: infra container id
    start_stack_sh->>podman: podman inspect PODMAN_SYSTEMD_UNIT
    podman-->>start_stack_sh: infra_unit
    alt infra_unit equals LLM_ROUTING_POD_UNIT
        start_stack_sh-->>start_stack_sh: stack_ownership returns quadlet:LLM_ROUTING_POD_UNIT
    else infra_unit equals LEGACY_LLM_ROUTING_POD_UNIT
        start_stack_sh->>systemd_user: legacy_unit_owns_pod LEGACY_LLM_ROUTING_POD_UNIT
        alt legacy unit owns POD_NAME
            systemd_user-->>start_stack_sh: true
            start_stack_sh-->>start_stack_sh: stack_ownership returns quadlet:LEGACY_LLM_ROUTING_POD_UNIT
        else legacy unit does not own POD_NAME
            systemd_user-->>start_stack_sh: false
            start_stack_sh-->>start_stack_sh: stack_ownership returns conflict:LEGACY_LLM_ROUTING_POD_UNIT
        end
    else no matching infra_unit
        start_stack_sh-->>start_stack_sh: stack_ownership returns legacy or absent
    end

    start_stack_sh-->>start_stack_sh: STACK_OWNERSHIP=$(stack_ownership)
    alt STACK_OWNERSHIP starts with conflict:
        start_stack_sh-->>Dev: print error and exit 1
    else STACK_OWNERSHIP starts with quadlet:
        start_stack_sh->>systemd_user: require_user_systemd
        systemd_user-->>start_stack_sh: ok
        start_stack_sh-->>start_stack_sh: owner_unit=${STACK_OWNERSHIP#quadlet:}
        start_stack_sh->>systemd_user: systemctl --user stop owner_unit
        start_stack_sh->>systemd_user: systemctl --user reset-failed owner_unit
        start_stack_sh->>podman: podman pod rm -f POD_NAME
        start_stack_sh->>systemd_user: systemctl --user restart owner_unit
        systemd_user-->>Dev: Quadlet stack restarted
    else STACK_OWNERSHIP is legacy or absent
        start_stack_sh-->>Dev: deploy_fresh_pod
    end
Loading

File-Level Changes

Change Details Files
Introduce environment-specific Quadlet namespaces and validate them, replacing the single llm-routing-pod.service unit with dev/prod-specific units and directories.
  • Default QUADLET_NAMESPACE to llm-routing-prod and validate it via a strict lowercase/digit/hyphen regex, exporting it for subsequent use.
  • Derive LLM_ROUTING_POD_UNIT from QUADLET_NAMESPACE and set QUADLET_DIR to the namespace-specific systemd directory.
  • Adjust stack ownership, teardown, and restart flows to operate on the resolved owner unit instead of a hardcoded unit name.
  • Ensure DATA_ROOT defaults to the worktree-scoped data directory, aligning dev/prod separation.
start-stack.sh
README.md
scripts/README.md
tests/test_quadlet_templates.py
Refine stack ownership detection to distinguish Quadlet-owned pods from foreign legacy units and abort deployment on conflicts.
  • Add legacy_unit_owns_pod helper to confirm a legacy unit explicitly owns the current POD_NAME via PodName= matching.
  • Extend stack_ownership to return quadlet: for both current and qualifying legacy units, return conflict: for mismatched legacy units, and treat only loaded matching units as Quadlet-owned.
  • Update safe_pod_teardown and restart logic to use the specific owner unit extracted from STACK_OWNERSHIP.
  • Abort deployment early when STACK_OWNERSHIP indicates a conflict, emitting guidance to inspect the conflicting legacy unit.
start-stack.sh
tests/test_quadlet_templates.py
Namespace Quadlet rendering logic so that filenames and internal dependencies are environment-specific while non-identifier strings remain untouched.
  • Pass QUADLET_NAMESPACE into the embedded Python renderer and define identifier_suffixes plus an identifier_prefix regex to target Quadlet identifiers.
  • Rewrite Pod/After/Wants/BindsTo/Requires/PartOf lines to prepend the namespace to llm-routing-* identifiers and to map llm-routing.pod and llm-routing-pod.service to namespace-specific names.
  • Derive rendered_name and rendered_names by replacing llm-routing with the current namespace in template basenames, preserving owner-only file semantics.
  • Add tests that run the embedded renderer in a temporary directory to assert that identifiers are namespaced while image names and URLs containing llm-routing are preserved.
start-stack.sh
tests/test_quadlet_templates.py
Update documentation to reflect environment-specific units and the local llama.cpp (local-qwen-3.6) safety net in LiteLLM fallback chains.
  • Change CLI examples to use llm-routing-prod-pod.service and llm-routing-dev-pod.service, and explain environment-specific namespaces in README and scripts/README.
  • Document dev/prod Quadlet directories, pod unit names, and the worktree-scoped DATA_ROOT behavior for data/config separation.
  • Describe the local-qwen-3.6 safety net in the routing table and fallback chain prose, including its position before Ollama and OpenRouter in advanced-tier chains.
  • Update mermaid diagrams and bullet lists for each agent tier to include local-qwen-3.6 in the fallback sequence.
README.md
scripts/README.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@sheepdestroyer, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 37 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fb569b15-f127-4b0e-9769-b6b0fc7c199b

📥 Commits

Reviewing files that changed from the base of the PR and between f9caaf6 and a24842b.

📒 Files selected for processing (8)
  • README.md
  • litellm/entrypoint.py
  • pod.yaml
  • quadlets/llm-routing-litellm.container
  • quadlets/llm-routing-router.container
  • scripts/README.md
  • start-stack.sh
  • tests/test_quadlet_templates.py
📝 Walkthrough

Walkthrough

The deployment script now namespaces Quadlet units per environment, detects exact or conflicting ownership, and restarts or tears down the corresponding systemd unit. Dev configuration, documentation, fallback diagrams, and deployment-contract tests were updated for the new naming and isolation behavior.

Changes

Environment-isolated Quadlet deployment

Layer / File(s) Summary
Namespace configuration and Quadlet rendering
.env.dev, start-stack.sh, tests/test_quadlet_templates.py
Dev configuration defines a separate namespace; rendering rewrites relevant Quadlet identifiers and filenames while preserving unrelated values and isolating data roots.
Ownership detection and lifecycle control
start-stack.sh, tests/test_quadlet_templates.py
Deployment identifies current or legacy Quadlet owners, rejects conflicting units, and restarts or tears down the exact owning unit.
Deployment documentation and contract coverage
README.md, scripts/README.md, tests/test_quadlet_templates.py
Documentation describes the updated fallback chains, environment-specific unit commands, and isolated deployment layout, with tests covering the documented contracts.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant start_stack as start-stack.sh
  participant systemd as systemctl --user
  participant Podman
  Operator->>start_stack: Start or replace stack
  start_stack->>systemd: Inspect namespaced and legacy units
  systemd-->>start_stack: Return owner or conflict
  alt Existing Quadlet owner
    start_stack->>systemd: Restart exact owner unit
    systemd->>Podman: Manage namespaced pod
  else No conflict
    start_stack->>Podman: Render and deploy namespaced Quadlets
  end
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main change: aborting deployment when a foreign legacy pod conflict is detected.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/quadlet-environment-isolation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added documentation Improvements or additions to documentation scripts tests labels Jul 23, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • In the embedded Python namespace_identifier helper, the elif field == "Pod" branch is unreachable because Pod is already handled in the first if clause; consider simplifying this logic and avoiding the duplicated llm-routing.pod replacement to make the namespacing rules clearer.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In the embedded Python `namespace_identifier` helper, the `elif field == "Pod"` branch is unreachable because `Pod` is already handled in the first `if` clause; consider simplifying this logic and avoiding the duplicated `llm-routing.pod` replacement to make the namespacing rules clearer.

## Individual Comments

### Comment 1
<location path="start-stack.sh" line_range="808-814" />
<code_context>
+        # paths, or other configuration values containing "llm-routing".
+        # Namespace only Quadlet identifier lines and values. This preserves
+        # arbitrary image names, URLs, and credentials containing llm-routing.
+        def namespace_identifier(match):
+            field, value = match.group(1), match.group(2)
+            if field in {"Pod", "After", "Wants", "BindsTo", "Requires", "PartOf"}:
+                value = identifier_prefix.sub(namespace + "-", value)
+                value = value.replace("llm-routing.pod", namespace + ".pod")
+                value = value.replace("llm-routing-pod.service", namespace + "-pod.service")
+            elif field == "Pod":
+                value = value.replace("llm-routing.pod", namespace + ".pod")
+            return f"{field}={value}"
</code_context>
<issue_to_address>
**issue:** The `namespace_identifier` function has an unreachable `elif field == "Pod"` branch.

Because the first `if` already matches `"Pod"` (and other fields), the subsequent `elif field == "Pod"` is dead code. This is confusing and could mask issues if the condition set changes later. Either separate the handling for `"Pod"` (remove it from the first set and keep a distinct branch) or remove the redundant `elif` if the extra behavior is no longer needed.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread start-stack.sh Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@start-stack.sh`:
- Around line 558-562: Update legacy_unit_owns_pod to inspect the generated
ExecStartPre command and match the pod name passed to podman pod create,
supporting both --name= and space-separated --name forms instead of the PodName
source line. Extend the regression test to cover ownership detection from
generated service output.

In `@tests/test_quadlet_templates.py`:
- Around line 130-132: Split the semicolon-separated src.mkdir() and out.mkdir()
statements in the TemporaryDirectory setup into separate statements, preserving
the existing directory creation behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e8131469-8db6-48fb-9bba-d6e9e3e918fb

📥 Commits

Reviewing files that changed from the base of the PR and between 8e7eb90 and f9caaf6.

📒 Files selected for processing (5)
  • .env.dev
  • README.md
  • scripts/README.md
  • start-stack.sh
  • tests/test_quadlet_templates.py

Comment thread start-stack.sh
Comment thread tests/test_quadlet_templates.py Outdated
@sheepdestroyer
sheepdestroyer merged commit 3a3dcbe into master Jul 23, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation litellm scripts tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant