Decide & lock the layer-1 rebuild mechanism

[Kenny-Heitritter]

Part of #16

Question

Given the comparison from the layer-1 design ticket, which mechanism do we commit to for auto-rebuilding the Lab image when a new codeq binary ships — and what exactly gets specced for the execution follow-up (trigger/topic names, env isolation, guardrails against the June cascade, rollback story)?

HITL. Resolves the biggest staleness gap. Output: chosen approach + an execution-ready spec (no open questions left for the implementer).

[Kenny-Heitritter]

Resolution — layer-1 rebuild mechanism: LOCKED

Resolves #19. Decided via /grilling with @Kenny-Heitritter, choosing from the comparison in #18. Ground truth re-verified against the live Terraform in qbraid-infrastructure (local clone).

Decision (headline)

• Mechanism A (chain on codeq-build success), BOTH envs — realized as a direct gcloud builds triggers run step, not a Pub/Sub topic + chain-rebuild-* trigger. Rationale: the chain-rebuild-* triggers (chain-rebuild-docker-stacks) are outdated and being deleted; we do not add another one. This anchors only on qbraid-Dstacks triggers (build-lab-base[-prod]), never the legacy docker-stacks repo triggers.
• Staging: auto now. Prod: wired now but the firing step stays disabled until lab-base Trivy is restored to --exit-code=1. This reconciles "both envs" with Design & compare layer-1 auto-rebuild mechanisms (codeq binary -> Lab image) #18's guardrails 4–5 (don't let a Trivy-clean-but-CVE-bearing image auto-promote to prod :latest while the lab-base scan is advisory).
• SHA/build-id stamping = Phase 0, ships first (it's the observability substrate + the prerequisite for the rollout smoke-test).
• Rollback = forward-fix + GCS-versioning break-glass + Decide running-pod refresh policy (layer 2) #21 drain (detail below).

Execution spec (ordered — no open questions for the implementer)

Phase 0 — SHA/build-id stamping (prerequisite, both envs)

In opencode-staging-branch / opencode-prod-main (terraform/environments/{staging,prod}/gcp/cloud-build-codeq.tf):

• Stamp the opencode git SHA + Cloud Build $SHORT_SHA/$BUILD_ID into codeq --version and add "sha" + "build_id" keys to the emitted manifest.json (today it carries only version/timestamp/binaries).
• Cheap: the build-binary step already installs git and sets OPENCODE_BUMP=1.

Phase 1 — staging auto (enable immediately)

• Append a final step to opencode-staging-branch, after the gsutil cp + manifest steps succeed:
  gcloud builds triggers run build-lab-base --branch=staging
• IAM: grant the staging codeq build SA permission to run only the staging build-lab-base trigger (roles/cloudbuild.builds.editor at the staging project, or a custom role limited to cloudbuild.builds.create + cloudbuild.builds.get). Do not grant it in prod.
• No topic, no chain-rebuild trigger.

Phase 2 — prod wired-but-gated

• Add the same final step to opencode-prod-main:
  gcloud builds triggers run build-lab-base-prod --branch=main
• Keep it disabled (step commented out or IAM grant withheld) until the prod go-live checklist passes:
  1. lab-base Trivy restored to --exit-code=1 (tracked cross-repo: infra#331 / Dstacks#34/Wire Azure terraform into CI/CD (IaC runner: staging auto-apply, prod gated) #35/Add billing pricing + quota for the 3 models (qbraid-api) #38). Both build-foundation and build-lab-base[-prod] are currently --exit-code=0 (advisory, temporary per infra#284).
  2. Rollout smoke-test exists (graduated to a new map ticket — see Follow-ups).
  3. Confirm/enable versioning on the prod qbraid-codeq bucket. Staging (qbraid-codeq-staging) has versioning { enabled = true }; the prod bucket is not declared in this repo's Terraform (referenced by IAM only), so its versioning must be verified directly. The fast break-glass rollback depends on it.

Anti-cascade guardrails (the contract every step honors)

1. Narrow target only: run build-lab-base[-prod] and nothing else — never build-foundation, never any docker-stacks-* trigger. (This is what the 2026-06-11 cascade violated: a chain fanned out to docker-stacks-prod-main → all 9 legacy images.)
2. Per-env + explicit branch: staging step → staging trigger --branch=staging; prod step → prod trigger --branch=main. The codeq build SA holds run-permission on only its own env's lab-base trigger. Keep the prod SA's existing read-only objectViewer on the staging codeq bucket from ever becoming a trigger-run path.
3. Idempotent: builds triggers run is fire-and-forget and a rebuild re-pulls latest, so at-least-once (e.g. a codeq-build retry firing it twice) is safe. No debounce needed.
4. Retire, don't extend, the chain-rebuild pattern: deliberately avoids adding a pubsub_config chain trigger.

Rollback story

• Primary — forward-fix: git revert the bad codeq commit on staging/main → push → the same auto-pipeline rebuilds a clean binary and re-fires build-lab-base[-prod].
• Fast — break-glass: restore the prior latest/linux-x64/codeq object generation (bucket versioning on for staging) and the prior manifest.json generation (so the stamped SHA stays consistent), then manually gcloud builds triggers run build-lab-base[-prod] --branch=….
• Existing pods: routine → passive convergence; security-critical → the documented break-glass drain (both per Decide running-pod refresh policy (layer 2) #21).

Verified ground truth (qbraid-infrastructure, local clone)

• codeq build: opencode-staging-branch (google_cloudbuild_trigger.opencode_staging, push ^staging$) → gsutil cp to gs://qbraid-codeq-staging/latest/linux-x64/codeq + writes manifest.json. Prod: opencode-prod-main → gs://qbraid-codeq.
• lab-base build: build-lab-base / build-lab-base-prod (google_cloudbuild_trigger.qbraid_dstacks_lab_base, qbraid-Dstacks repo) via repository_event_config on ^staging$/^main$ + included_files=["qbraid-lab-base/**"]; its download step gsutil cps the codeq binary in. This is the HOP-2 gap: a new codeq object touches no qbraid-lab-base/** file, so the trigger never fires on its own.
• Trivy: codeq binary build genuinely gated (module.trivy_gate_opencode, --exit-code=1); lab-base image scans advisory (--exit-code=0) — the basis of the prod gate.
• qbraid-codeq-staging has object versioning; prod qbraid-codeq bucket not TF-declared here (verify).

Follow-ups

• Graduates the rollout smoke-test fog → new ticket (frontier; the SHA-stamping decision it needed is locked here).
• Prod go-live gated on cross-repo Trivy restoration (infra#331 / Dstacks#34/Wire Azure terraform into CI/CD (IaC runner: staging auto-apply, prod gated) #35/Add billing pricing + quota for the 3 models (qbraid-api) #38) — tracked there, not a wayfinder ticket.
• CODEQ_DEPLOYMENT.md (Validate & complete the CodeQ deployment documentation #17) to be updated with this mechanism during the execution effort.