Design & compare layer-1 auto-rebuild mechanisms (codeq binary -> Lab image)

[Kenny-Heitritter]

Part of #16

Question

What are the viable mechanisms to make a new codeq binary in gs://qbraid-codeq[-staging]/latest/linux-x64/codeq automatically trigger a lab-base image rebuild (closing HOP 2 / staleness layer 1), and what are their trade-offs? Produce a comparison an operator can decide from.

Cover at least:

• Pub/Sub chain: codeq build publishes on success → a Cloud Build trigger on build-lab-base[-prod] subscribed to the topic. How to scope it so it CANNOT repeat the 2026-06-11 cascade (qbraid-Dstacks CLAUDE.md §5): correct source repo/ref, env isolation (staging topic ≠ prod topic), idempotency.
• GCS bucket notification / Eventarc on the latest/linux-x64/codeq object → Cloud Function → gcloud builds triggers run build-lab-base[-prod].
• Manual release checklist (status quo hardened): the documented builds triggers run step as the deliberate gate.
• Version/observability angle: whether to stamp git SHA / build number into the binary version + manifest.json so "did it roll out?" is answerable without shelling into a pod.

For each: blast radius, staging/prod isolation, failure mode (does a bad codeq build become :latest lab image?), Terraform surface, and how it interacts with the Trivy gate (ADR Q5).

AFK. Output: a comparison doc (linked) with a recommended default + guardrails. Feeds the layer-1 decision.

[Kenny-Heitritter]

Layer-1 auto-rebuild mechanisms: codeq binary → Lab image

Resolves #18 (feeds the decision in #19). AFK research. Grounded in the live Terraform in qbraid-infrastructure (terraform/environments/{staging,prod}/gcp/cloud-build-*.tf), the V2 CI/CD ADR (docs/architecture/v2-ci-cd-strategy.md), and the OpenCode fork-maintenance doc.

---

1. What the gap actually is (verified against Terraform)

Current staging pipeline (prod is identical with staging→main, -staging buckets → prod buckets):

1. Push to opencode staging → trigger opencode-staging-branch (^staging$) builds the branded codeq binary, runs its Trivy fs-gate, then gsutil cp → gs://qbraid-codeq-staging/latest/linux-x64/codeq and writes latest/manifest.json.
2. Trigger build-lab-base (in qbraid-Dstacks, ^staging$) has included_files = ["qbraid-lab-base/**"]. Its download-artifacts step does gsutil cp gs://qbraid-codeq-staging/latest/linux-x64/codeq … and bakes it into lab-base:latest (+ retag lab-standard:latest).
3. :latest + imagePullPolicy: Always (ADR Q4) → new pods pull the new image.

The layer-1 defect (HOP 2): a fresh codeq object in GCS touches no qbraid-lab-base/** file, so build-lab-base never fires. The new binary sits in the bucket; lab-base:latest keeps the old binary baked in until either (a) something pushes to qbraid-lab-base/**, or (b) an operator runs gcloud builds triggers run build-lab-base[-prod] --branch=…. That manual run is today's only bridge. Layer 2 (existing pods keep the old image until restart) is #21; layer 3 (config) is #20.

The 2026-06-11 cascade this must not repeat (qbraid-Dstacks/CLAUDE.md §5; shape confirmed in cloud-build-docker-stacks.tf): the prod chain-rebuild-docker-stacks trigger subscribes Pub/Sub topic cloud-build-extension-updated and runs gcloud builds triggers run docker-stacks-prod-main --branch=main. docker-stacks-prod-main rebuilds all 9 images from the legacy docker-stacks repo and pushes :latest/:prod. A mis-scoped chain pushed a wrong-repo image straight to prod :latest. Three root causes any layer-1 automation must avoid: wrong source repo/ref, no staging/prod topic isolation, unbounded fan-out.

Reusable primitives already in the repo (these lower the cost of the options below):

• Per-env Pub/Sub chain-build topics already exist: cloud-build-extension-updated, cloud-build-docker-image-updated, with roles/pubsub.publisher granted to each env's Cloud Build SA. chain-rebuild-docker-stacks is a working pubsub_config → builds triggers run trigger — a direct template.
• Cloud Functions Gen2 are an established pattern here (preview_command, domain_config_manager, billing_leakage_monitor; nodejs20, GCS source object) — the Eventarc option is not novel infra.
• qbraid-codeq-staging has versioning { enabled = true } (supports object-notification + rollback). The trivy-gate module encapsulates the ADR Q5 γ-pattern.

Trivy posture (ADR Q5), which every option inherits:

• The codeq binary build itself is really gated — module.trivy_gate_opencode runs an fs scan of the isolated shipped-CLI dependency closure with --exit-code=1. A binary with a fixable CRITICAL/HIGH in its shipped deps never reaches GCS :latest.
• The lab-base image scans are currently advisory (--exit-code=0, temporary per infra#284, tracked infra#331, pending Dstacks#34/Wire Azure terraform into CI/CD (IaC runner: staging auto-apply, prod gated) #35/Add billing pricing + quota for the 3 models (qbraid-api) #38 + jlab#22). So today the image layer does not block. Auto-rebuild runs the same build-lab-base build, so it neither adds nor removes a gate — but while lab-base is advisory, "new codeq → new :latest" ships with only advisory image scanning.

---

2. The mechanisms

A — Pub/Sub chain (codeq build publishes on success → build-lab-base subscribes)

Append a gcloud pubsub topics publish <codeq-updated-topic> step to opencode-staging-branch / opencode-prod-main; add a new pubsub_config trigger that runs gcloud builds triggers run build-lab-base[-prod] --branch=staging|main. Mirrors chain-rebuild-docker-stacks.

• Blast radius: exactly one downstream trigger → 2 images (lab-base + lab-standard retag). Bounded iff it targets the narrow build-lab-base, not a full-chain trigger.
• Isolation: requires a new topic per env and explicit --branch. Staging event must never reach a prod trigger (note: prod Cloud Build SA already has objectViewer on the staging codeq bucket for cutover bridging — keep that read-only bridge from becoming an event bridge).
• Failure mode → bad codeq as :latest? A Trivy-clean-but-functionally-broken binary does auto-promote (removes the human "should this ship?" gate). Trivy-failing binary never publishes.
• Terraform surface: small — 1 topic/env (+publisher IAM, pattern exists), 1 pubsub_config trigger/env, 1 publish step per codeq trigger. All in existing files. No new deployable to own.
• Cascade-safety: the safe re-do of the pattern that failed — narrow trigger, correct repo (qbraid-Dstacks), per-env topic, explicit branch.

B — GCS notification / Eventarc → Cloud Function → builds triggers run

google_storage_notification on qbraid-codeq[-staging] for OBJECT_FINALIZE, prefix-filtered to latest/linux-x64/codeq → Pub/Sub → Cloud Function (Gen2) → gcloud builds triggers run build-lab-base[-prod].

• Blast radius: same one downstream trigger; the Function is one extra place for a bug.
• Isolation: clean by construction — notification is bucket-scoped (staging vs prod buckets are already separate); Function per env calls the env-correct trigger+branch.
• Distinct advantage: fires on the artifact actually landing (the object is the source of truth), so it also catches out-of-band uploads (manual gsutil cp), and needs no edit to the opencode build. In-place overwrite of latest/linux-x64/codeq still fires OBJECT_FINALIZE. Prefix filter must exclude manifest.json.
• Failure mode → bad codeq as :latest? Same as A (no human gate).
• Terraform surface: largest — google_storage_notification + dedicated topic + google_cloudfunctions2_function + source object + SA + IAM (cloudbuild.builds.editor, invoker) + Eventarc SA. Not novel (3 Gen2 functions already), but it is a new deployable to patch/own.
• Cascade-safety: the Function is the natural place to enforce narrow-trigger + per-env + idempotency/debounce.

C — Manual release checklist (status quo, hardened)

Keep gcloud builds triggers run build-lab-base[-prod] --branch=… as the deliberate gate; make it a required, documented step (in CODEQ_DEPLOYMENT.md — see #17 — and the fork-maintenance forward-port flow, which already gates promotion by human review). Harden with the SHA-stamping below + a one-liner helper.

• Blast radius: zero automation; operator-scoped. Safest.
• Isolation: inherent — operator names the env.
• Failure mode: the failure is omission (forget → stale binary — today's problem), never a bad auto-rollout. Preserves the human gate.
• Terraform surface: none (doc + optional script).
• Cascade-safety: cannot cascade.
• Fit: consistent with ADR Q4 / fork-maintenance ("no automation merges main→staging; the forward-port is deliberate"), but leaves binary→image decoupling as standing toil.

Cross-cutting — Version/observability (git SHA / build number stamping)

Today manifest.json = {version:"latest", timestamp, binaries:{linux-x64:url}} — no SHA, no build id, so "did my push roll out?" needs shelling into a pod. Stamp the opencode git SHA + Cloud Build $SHORT_SHA/$BUILD_ID into codeq --version and into manifest.json ("sha", "build_id"). Cheap (the build-binary step already installs git and sets OPENCODE_BUMP=1), orthogonal to A/B/C, and the prerequisite for the rollout smoke-test still in the map's Not yet specified. Adopt regardless of which mechanism wins.

---

3. Comparison

Axis	A — Pub/Sub chain	B — GCS notify → Cloud Function	C — Manual (hardened)
Trigger source	codeq build success	codeq object landing (incl. out-of-band)	operator
Blast radius	2 images (if narrow)	2 images (if narrow)	operator-scoped
Staging/prod isolation	new topic/env + explicit branch	bucket-scoped (inherent) + branch	inherent
Bad-but-Trivy-clean binary → :latest?	yes (auto)	yes (auto)	no (human gate)
New deployable to own	none	Cloud Function	none
Terraform surface	small (reuses chain pattern)	largest	none
Cascade-repeat risk	low if narrow+per-env	low if narrow+per-env	none
Catches out-of-band uploads	no	yes	n/a
Edits opencode build	yes (publish step)	no	no

---

4. Recommendation + guardrails

Default: A (Pub/Sub chain) for staging now; keep C (manual) for prod; adopt SHA-stamping immediately. Promote prod to automation only after the rollout smoke-test exists and lab-base Trivy is back to --exit-code=1.

Why: A reuses an already-understood primitive (chain-build topics + the chain-rebuild-docker-stacks template) for the lowest net-new surface among the automation options and no new deployable (vs B's Function). Staging-auto/prod-manual kills the staleness toil where iteration is frequent while keeping the human "should this ship?" gate where it costs the most (prod). B is the better long-term design if out-of-band uploads or artifact-as-source-of-truth matter — recommend it as the upgrade path, not the default, because of the Function it adds to own.

Guardrails for whichever automation is chosen (these are the anti-cascade contract):

1. Aim only at the narrow build-lab-base[-prod] trigger — never a docker-stacks-* full-chain trigger.
2. Per-env topic/notification + explicit --branch=staging|main; no cross-env fan-out. Keep the staging→prod codeq bucket read-grant from becoming an event path.
3. Idempotency/debounce: carry the object generation / git SHA in the event; rebuild is naturally idempotent (re-pulls latest), so at-least-once is fine, but debounce rapid pushes.
4. Keep prod's human gate until the smoke-test (map Not yet specified) and blocking Trivy both exist.
5. Restore lab-base Trivy to --exit-code=1 before enabling prod auto-rebuild (else a codeq-introduced image CVE reaches :latest unblocked).
6. Ship SHA/build stamping first (manifest.json + codeq --version) — unlocks observability and the smoke-test for every option.

This is the comparison #19 decides from; #19 also inherits the open question of whether prod ever goes auto (guardrails 4–5) and whether SHA-stamping is folded in there or split out.