Wayfinder: CodeQ deployment — seamless staging/prod updates without stale binaries/configs

[Kenny-Heitritter]

Destination

A charted route with every remediation decision locked so that, in a follow-up execution effort, a push to qBraid/opencode staging/main reliably reaches all Lab users on that environment with no stale codeq binary and no stale codeq config — and so the deployment path is correctly documented.

Planning only (destination (a)). This map decides and specs; it does not implement the Terraform/script changes. The map is done when nothing is left to decide before an implementer can execute.

Anchor artifact: CODEQ_DEPLOYMENT.md in this repo.

Notes

• Domain: qBraid CodeQ (rebranded sst/opencode fork) delivery to JupyterLab singleuser pods on GKE staging/prod.
• The 3-hop pipeline (see anchor doc): push → binary in GCS (auto) → Lab image rebuild (not auto) → user pod (auto for new pods only).
• The three staleness layers: (1) binary→image decoupling, (2) image→running-pod, (3) config persistence.
• Cross-repo source of truth for the GCP side is NOT this repo: Terraform in qbraid-infrastructure owns the Cloud Build triggers (opencode-{staging-branch,prod-main}, build-lab-base[-prod]); qbraid-Dstacks owns the Lab image + runtime codeq scripts.
• Read before deciding: qbraid-infrastructure/docs/architecture/v2-ci-cd-strategy.md (ADR Q2/Q4), .../docs/contributing/opencode-fork-maintenance.md, and qbraid-Dstacks/CLAUDE.md §5 — the 2026-06-11 chained-trigger cascade post-mortem (a mis-scoped chain pushed a wrong-repo image to prod; any layer-1 automation must not repeat it).
• Skills per session: /grilling + /domain-modeling for decision tickets; /research for AFK investigation.
• Planning-by-default is IN EFFECT — tickets resolve decisions, not deliverables.

Decisions so far

• Design & compare layer-1 auto-rebuild mechanisms (codeq binary → Lab image) — compared Pub/Sub-chain / GCS-notify→Cloud-Function / hardened-manual; recommends Pub/Sub chain for staging + manual for prod, with narrow-trigger + per-env-topic + explicit-branch guardrails against the 2026-06-11 cascade, and stamp git SHA/build-id into manifest.json + codeq --version first. Full comparison in the ticket. Feeds Decide & lock the layer-1 rebuild mechanism #19.
• Decide running-pod refresh policy (layer 2) — layer-2 = passive convergence (idle-cull + re-login) for routine updates; documented manual break-glass drain for security-critical; no in-Lab banner, no automated drain. Only execution artifact is documentation (convergence window + break-glass runbook line → feeds Validate & complete the CodeQ deployment documentation #17).
• Decide config-migration systematization (layer 3) — layer-3 = one uniform high-authority re-assert stays in update-codeq-token.sh (qbraid-Dstacks); rejected versioned-migration framework, .well-known defaults channel, and binary-side migration. Config is .optional() + binary-defaulted, so only explicit persisted values go stale. The opencode→Dstacks contract is scoped + registry-backed: obligation bites only on force-correct-worthy changes (rename / retire-id / forced-default / new-injected-field); a System-owned config fields registry in CODEQ_DEPLOYMENT.md is the single source of truth the Dstacks block mirrors. Execution = two doc/organization edits, no Terraform. ⚠️ overlay copy of the script has drifted to the pre-migration version — reconcile during execution (lab-base is canonical).
• Decide & lock the layer-1 rebuild mechanism — locked mechanism A both envs, realized as a direct gcloud builds triggers run build-lab-base[-prod] step on the codeq build (no Pub/Sub topic, no chain-rebuild-* trigger — those are being retired; anchor only on qbraid-Dstacks triggers). Staging auto now; prod wired but the firing step stays off until lab-base Trivy is back to --exit-code=1 (+ smoke-test + verify prod bucket versioning). SHA/build-id stamping is Phase 0. Rollback = forward-fix (revert+push) primary, GCS-versioning restore as fast break-glass, Decide running-pod refresh policy (layer 2) #21 drain for pods. Full ordered spec + anti-cascade guardrails in the ticket. Graduated the smoke-test fog → Decide & spec the rollout smoke-test (did a codeq change reach all users?) #22.
• Decide & spec the rollout smoke-test (did a codeq change reach all users?) — smoke-test asserts one claim: lab-base:latest carries the expected codeq SHA (manifest.json sha vs in-image codeq --version; pass-key sha, no image label). Two identical checks both envs: (A) fail-closed assert-codeq-sha step in build-lab-base[-prod] before the deferred images: push (mismatch aborts the build → :latest never poisoned; a guard on Decide & lock the layer-1 rebuild mechanism #19's auto-rebuild) + (B) operator one-liner verify-codeq-rollout.sh <env> against the live Artifact-Registry image (registry-read only, no pod shell). "Live for all users" ≙ image source correct; running-pod convergence self-corrects on restart per Decide running-pod refresh policy (layer 2) #21 — not measured/gated; rejected cron + stale-pod enumeration. Satisfies Decide & lock the layer-1 rebuild mechanism #19 Phase-2 go-live item 2; feeds CODEQ_DEPLOYMENT.md.

Not yet specified

• Staging/dev codeq-gcs-upload.yml inconsistency. On non-main branches it's still the upstream-shaped workflow_dispatch build that includes linux-x64 and targets the prod gs://qbraid-codeq bucket. Whether/how to converge it is a possible cleanup decision.
• BMA / multi-cloud (non-GKE) instances. The same three layers apply via the AMI-bake path (qbraid-arch/qbraid-BMA/IMAGES-AND-AMIS.md). Whether this map extends there or defers to that runbook.

Out of scope

• Implementing the fixes (writing/landing Terraform, a Cloud Function, or script changes). Destination is planning-only; execution is a separate follow-up map.
• Mac/Windows/linux-arm64 desktop & CLI codeq distribution (the codeq-gcs-upload.yml GitHub Actions path). Not the staging/prod Lab-instance path this effort targets.