Validate & complete the CodeQ deployment documentation

[Kenny-Heitritter]

Part of #16

Question

Is CODEQ_DEPLOYMENT.md accurate and complete as the anchor for this map's decisions? Cross-check its claims against the live source of truth and fix any drift, so downstream decision tickets build on correct facts.

Verify specifically:

• Trigger names / branch filters / GCS buckets in qbraid-infrastructure/terraform/environments/{staging,prod}/gcp/cloud-build-codeq.tf and cloud-build-qbraid-dstacks.tf (opencode-staging-branch, opencode-prod-main, build-lab-base, build-lab-base-prod, included_files = ["qbraid-lab-base/**"]).
• The gsutil cp gs://qbraid-codeq[-staging]/latest/linux-x64/codeq step really is inlined in the trigger and NOT chained to a codeq upload.
• Helm singleuser.image tag: latest + pullPolicy: Always in values-{staging,prod}.yaml.
• The runtime scripts' behaviour: setup-codeq-config.sh (only-if-missing) and update-codeq-token.sh (fields it re-asserts), and the GCS home-sync persistence claim.
• Whether codeq --version on a pod is currently the only way to tell which binary a user is on.

AFK. Output: corrections applied to CODEQ_DEPLOYMENT.md (linked), plus a note of anything that couldn't be verified without live GCP access.

[Kenny-Heitritter]

Resolution — validated against source of truth

Headline finding: CODEQ_DEPLOYMENT.md did not exist on any branch of
qBraid/opencode (checked all local + remote branches and GitHub code search;
blob/dev/CODEQ_DEPLOYMENT.md is a 404). So this ticket's "validate &
complete" resolved to authoring the anchor from scratch against the live
source of truth, not correcting an existing file's drift. Every factual claim
carried in the map turned out to be accurate — the work was verification +
adding precision, with no corrections needed.

Verified against: qbraid-infrastructure main @ 01b278b; qbraid-Dstacks
origin/staging (bbf58c3) and origin/main (69999a9) — the branches that
deploy to staging/prod. (Note: qbraid-Dstacks origin/staging's
setup-codeq-config.sh has 6 lines more than origin/main — a .local/state
chown fix; the only-if-missing logic is identical on both.)

Verification checklist (all ✅)

1. Trigger names / branch filters / buckets / included_files — ✅
   • opencode-staging-branch → push ^staging$ → gs://qbraid-codeq-staging/latest/linux-x64/codeq
   • opencode-prod-main → push ^main$ → gs://qbraid-codeq/latest/linux-x64/codeq
   • build-lab-base (^staging$) / build-lab-base-prod (^main$), both included_files = ["qbraid-lab-base/**"]
2. gsutil cp …/codeq inlined in trigger, NOT chained to a codeq upload — ✅
   The download is an inline gsutil cp in the lab-base trigger's
   download-artifacts step. There is no google_storage_notification,
   Pub/Sub, Cloud Function, or Eventarc on the codeq bucket, and nothing
   programmatically runs build-lab-base[-prod] after an upload. The rebuild
   fires only on a qbraid-lab-base/** push (or manual trigger run). This is
   the Layer-1 gap, confirmed. (The build-lab-base hits in
   cloud-build-docker-stacks.tf/extensions.tf are step-ids inside the legacy
   V1 docker-stacks monolith — unrelated to the V2 trigger.)
3. Helm singleuser.image tag: latest + pullPolicy: Always — ✅ in both
   values-staging.yaml and values-prod.yaml (singleuser image = lab-base).
4. Runtime scripts + home-sync persistence — ✅
   • setup-codeq-config.sh is only-if-missing (if [ ! -f "$CONFIG_FILE" ]).
   • update-codeq-token.sh re-asserts (always) apiKey, baseURL,
     mcp.pod_mcp.command; sets mcp.pod_mcp.type/enabled only if absent;
     migrates dead model ids → current and defaults empty → qbraid/gemini-3.5-flash,
     preserving valid user models. It touches nothing else (not the permission
     block).
   • GCS home-sync confirmed: data-restore initContainer runs
     rclone sync gcs:qbraid-<env>-users/$JUPYTERHUB_USER/ /home/jovyan/ on every
     pod start, so ~/.config/codeq/config.json persists → only-if-missing means
     returning users never get template updates (Layer-3 gap).
5. codeq --version the only on-pod way to identify the binary — ✅ (with
   nuance). Image is always :latest; no version file is baked in; manifest.json
   lives in the bucket, not the pod. codeq --version (wrapper → codeq-bin --version) is the only on-pod signal; OPENCODE_BUMP=1 makes builds report
   distinct versions, so it's a usable discriminator — but there is no
   at-a-glance external marker.

Added precision (beyond the map's gist)

• The codeq binary is also baked by build-lab-intel[-prod] and the disabled
  build-bma-lab-base trigger — any Layer-1 fix must cover every consumer.
• Prod Cloud Build SA holds objectViewer on gs://qbraid-codeq-staging for
  cutover bridging (Phase B) — envs aren't strictly bucket-isolated during cutover.
• Binary + wrapper are baked to /usr/local/bin (outside home) explicitly to
  dodge GCS home-sync — so the binary is only ever stale via Layers 1–2, never
  via home persistence; config (in home) is the only home-synced piece.

Could not verify without live GCP access

• Current contents/timestamps of gs://qbraid-codeq[-staging]/latest/ (is a newer
  binary sitting un-baked right now?).
• Which digest lab-base:latest currently points to / last build time (Cloud Build history).
• Whether triggers are currently enabled/connected (local.github_connection_ready).
• The actual codeq version any running pod reports.
• Whether a manual gcloud builds triggers run recently forced a rebuild.

Deliverable

The anchor doc is authored and committed locally as CODEQ_DEPLOYMENT.md on
branch docs/codeq-deployment-anchor (off dev, commit acd919c). It is
not pushed — say the word and I'll push it / open a PR into dev. Full
content below.

CODEQ_DEPLOYMENT.md (authored anchor)

CodeQ deployment path (staging / prod Lab instances)

How a change to qBraid/opencode reaches a JupyterLab singleuser pod as an
updated codeq binary, and where staleness can creep in. This is the anchor
document for the CodeQ-deployment wayfinder effort
(map #16): every fact below is
cross-checked against the live source of truth so downstream decision tickets
build on correct facts.

Scope: the GKE staging (qbraid-staging) and prod (qbraid-prod)
Lab-instance path only. Mac/Windows/arm64 desktop & CLI distribution (the
codeq-gcs-upload.yml GitHub Actions path) and BMA / multi-cloud AMI-bake
instances are out of scope here.

Sources of truth

The GCP side is not owned by this repo. The authoritative files are:

Concern	Repo · path
Build codeq binary → upload to GCS	qbraid-infrastructure · terraform/environments/{staging,prod}/gcp/cloud-build-codeq.tf
Download binary → bake into Lab image	qbraid-infrastructure · terraform/environments/{staging,prod}/gcp/cloud-build-qbraid-dstacks.tf
Which image singleuser pods run	qbraid-infrastructure · kubernetes/clusters/gcp/gke_{staging,prod}_cluster/helm/jupyterhub/values-{staging,prod}.yaml
Runtime config + token scripts, binary placement	qbraid-Dstacks · qbraid-lab-base/scripts/{setup-codeq-config,update-codeq-token,codeq-wrapper}.sh, qbraid-lab-base/Dockerfile

Verified against: qbraid-infrastructure main @ 01b278b; qbraid-Dstacks
origin/staging (bbf58c3) and origin/main (69999a9) — the branches that
deploy to staging and prod respectively.

The 3-hop pipeline

push to opencode           push to qbraid-Dstacks              new pod spawn
   staging/main    ─▶  (qbraid-lab-base/** changed)   ─▶     (pullPolicy: Always)
        │                        │                                  │
   HOP 1 · AUTO            HOP 2 · NOT AUTO                    HOP 3 · AUTO
   build binary,          download binary from GCS,           pull lab-base:latest,
   upload to GCS          bake into lab-base:latest           run baked binary

Hop 1 — push → binary in GCS · automatic

• Trigger opencode-staging-branch fires on push to ^staging$ of
  qBraid/opencode; opencode-prod-main fires on push to ^main$.
• Each is a single Cloud Build: install deps → apply qBraid branding → build the
  single-file codeq binary (bun run build --single, OPENCODE_BUMP=1) →
  Trivy CLI-dependency scan → upload-gcs step (gsutil cp the binary) →
  upload-manifest.
• Output:
  • staging → gs://qbraid-codeq-staging/latest/linux-x64/codeq (+ latest/manifest.json)
  • prod → gs://qbraid-codeq/latest/linux-x64/codeq (+ latest/manifest.json)
• Only linux-x64 is produced here. manifest.json records
  {"version":"latest","timestamp":"…"} — it lives in the bucket, not on any pod.

Hop 2 — binary in GCS → Lab image · NOT automatic (Layer 1 gap)

• Trigger build-lab-base (staging, ^staging$) and
  build-lab-base-prod (prod, ^main$) on qBraid/qbraid-Dstacks, each
  narrowed by included_files = ["qbraid-lab-base/**"].
• Their download-artifacts step inlines a plain
  gsutil cp gs://qbraid-codeq-staging/latest/linux-x64/codeq … (staging) /
  gs://qbraid-codeq/latest/linux-x64/codeq … (prod) into
  qbraid-lab-base/codeq_bin/, chmod +x, then the build step bakes it into
  …/jupyterhub/lab-base:latest (re-tagged lab-standard:latest).
• This gsutil cp is an inline pull at build time — it is not chained to the
  codeq upload. There is no storage notification, Pub/Sub subscription, Cloud
  Function, or Eventarc watching the codeq bucket, and nothing programmatically
  runs the build-lab-base[-prod] trigger after an upload. The image rebuild
  fires only on a push to qbraid-Dstacks staging/main touching
  qbraid-lab-base/** (or a manual
  gcloud builds triggers run build-lab-base[-prod]).
• Consequence (Layer 1): a freshly built codeq binary can sit in GCS
  indefinitely. It only reaches an image when some unrelated qbraid-lab-base/**
  change happens to trigger a rebuild, or someone manually re-runs the trigger.
  This is the core decoupling this effort exists to close.
• The same binary is also baked by build-lab-intel[-prod]
  (foundation:py310 → lab-base:py310 → lab-intel; its included_files include
  qbraid-lab-base/**) and by the disabled build-bma-lab-base trigger — so
  the layer-1 fix must consider every consumer of the binary, not just lab-base.
• Cutover nuance: the prod Cloud Build SA holds objectViewer on
  gs://qbraid-codeq-staging "so prod lab-base can consume the staging binary
  during cutover bridging (Phase B)." During cutover the environments are not
  strictly bucket-isolated.

Hop 3 — Lab image → user pod · automatic for new pods only (Layer 2 gap)

• singleuser.image in values-{staging,prod}.yaml:
  name: …/jupyterhub/lab-base, tag: latest, pullPolicy: Always.
• Every newly-spawned pod pulls lab-base:latest and therefore runs whatever
  binary the current image baked in.
• Consequence (Layer 2): already-running pods are not refreshed. A user
  keeps their old binary until they restart their server (new spawn → fresh
  pull). "Reached all users on an env" is not true until the last long-running
  pod is recycled.

Runtime config & the binary on the pod

Binary placement — outside home, never stale-from-sync

The Dockerfile copies the binary to /usr/local/bin/codeq-bin and the wrapper
to /usr/local/bin/codeq — explicitly not under ~/.local/bin, with the
comment "which gets overwritten by GCS home sync." So the binary a user runs is
always the image's baked binary; it is only ever stale via Layers 1–2, never via
home persistence.

codeq (the wrapper, codeq-wrapper.sh) runs update-codeq-token.sh silently,
then execs /usr/local/bin/codeq-bin "$@".

Home persistence — GCS sync (why config goes stale)

The data-restore initContainer runs
rclone sync gcs:qbraid-<env>-users/$JUPYTERHUB_USER/ /home/jovyan/ on every
pod start. So ~/.config/codeq/config.json (and anything else in home) persists
across pod restarts, restored from GCS.

Layer 3 — config staleness

On pod start, before-notebook.d/setup-qbraid.sh runs setup-codeq-config.sh
then update-codeq-token.sh:

• setup-codeq-config.sh — only-if-missing. It writes
  ~/.config/codeq/config.json only if the file does not already exist
  (if [ ! -f "$CONFIG_FILE" ]), else it logs "already exists" and leaves it
  untouched. Because the config is GCS-persisted (above), a returning user's
  config is never regenerated. New template fields added to this script
  (permissions, provider options, new MCP servers, model default) do not reach
  existing users. This is the Layer-3 gap. The template it would write pins
  model: qbraid/gemini-3.5-flash, baseURL: <ACCOUNT_URL>/api/ai/v1,
  apiKey seeded from QBRAID_ACCESS_TOKEN, a full permission allow-block, and
  the pod_mcp MCP server.

• update-codeq-token.sh — the only self-healing path for existing configs.
  Runs on every pod start and every codeq launch (via the wrapper). On the
  existing config it re-asserts, via jq (sed fallback):

  • .provider.qbraid.options.apiKey ← QBRAID_ACCESS_TOKEN (fallback: ~/.qbraid/qbraidrc) — always
  • .provider.qbraid.options.baseURL ← <ACCOUNT_URL>/api/ai/v1 (fallback: qbraidrc url map; default https://account.qbraid.com) — always
  • .mcp.pod_mcp.command ← ["/usr/bin/python3","/opt/qbraid-mcp-server/server/main.py"] — always
  • .mcp.pod_mcp.type / .enabled ← "local" / true — only if absent (//=)
  • .model — migrates now-dead ids to current (e.g. gemini-3-flash→gemini-3.5-flash, gemini-3-pro→gemini-3.1-pro, claude-opus-4-6→claude-opus-4-8, claude-sonnet-4-5→claude-sonnet-4-6, grok-4.1-fast→gemini-3.5-flash); empty → qbraid/gemini-3.5-flash; a valid user-chosen model is preserved.

  It does not touch the permission block or any field outside those above.
  So the only config drift that self-heals for existing users is token, gateway
  URL, MCP command, and dead-model migration — nothing else.

Version detection

• The image is always tagged :latest; there is no per-build image tag, no pod
  label, and no version file baked into the image. manifest.json records a
  build timestamp but lives in the GCS bucket, not on the pod.
• On a running pod, codeq --version is effectively the only way to tell which
  binary build a user is on. (codeq --version → wrapper → codeq-bin --version.) Because every build runs with OPENCODE_BUMP=1, distinct builds
  report distinct versions, so --version is a usable discriminator — but there
  is no at-a-glance external marker (image digest aside, which requires cluster
  access, not a user-visible signal).

The three staleness layers (summary)

Layer	Where	Mechanism	Auto-heals?
1 binary → image	GCS → lab-base:latest	inline gsutil cp at image-build time; no chain from upload	❌ needs a qbraid-lab-base/** push or manual trigger run
2 image → running pod	lab-base:latest → pod	pullPolicy: Always on spawn	⚠️ new pods only; running pods need a restart
3 config persistence	GCS home-sync → config.json	setup-codeq-config.sh only-if-missing; update-codeq-token.sh re-asserts a fixed field set	⚠️ only token/baseURL/MCP-command/dead-model self-heal

Verified but not confirmable without live GCP access

The following depend on live-project state and could not be checked from source:

• Current contents/timestamps of gs://qbraid-codeq[-staging]/latest/ — i.e.
  whether a newer binary is sitting un-baked right now.
• Which image digest lab-base:latest currently points to and when it was last
  built (Cloud Build history).
• Whether the triggers are currently enabled/connected in the live projects
  (local.github_connection_ready).
• The actual codeq version any given running pod reports.
• Whether any manual gcloud builds triggers run … has recently forced a rebuild.