mind: the compile-axis arc — re-scoped, issued, shipped, recorded

complete/2026/08/compile-axis-campaign-coverage.md

@@ -0,0 +1,123 @@
+# Compile-axis phase 1 — `campaign --axis compile`
+
+- shipped: 2026-08-10
+- issue: https://github.com/PyAutoLabs/PyAutoBrain/issues/218
+- pr: https://github.com/PyAutoLabs/PyAutoBrain/pull/219 (squash cd79005)
+- repos: PyAutoBrain
+- arc: phase 1 of 3 (see compile-warm-baseline-dashboard, compile-axis-triage-drift)
+
+## Summary
+
+The Profiling Agent could not see the compile corpus `autolens_profiling` was
+already producing: `AGENTS.md` listed compile-time profiling under *Future modes*,
+`_profiling.py` had zero occurrences of "compile", and all three modes read only
+`results/runtime/` — while 93 committed probe records sat uncross-referenced.
+
+`campaign --axis compile` answers coverage: **2 of 21 grid cells**, `hst` only, no
+interferometer / datacube / `jwst` / `ao` row anywhere. That number is what
+justified phases 2 and 3.
+
+## Traps and findings
+
+- **The tier vocabularies do not interchange.** Runtime buckets by sweep *config*
+  name (`local_cpu_fp64`), folding precision into the name; a compile record has a
+  raw `hardware` string plus a **separate** `mixed_precision` bool. Reusing
+  `TIER_CONFIGS` would have mis-bucketed every row.
+- **Records must be placed by their own fields, not their path.** Results are filed
+  under `<hardware>/<model_type>`, which drops class and instrument entirely.
+- **`export_probe.py` / `trace_profile.py` share the results tree** with a different
+  schema. The first implementation reported their 4 records as "malformed", which
+  would have sent someone to fix two files that work correctly. Missing the whole
+  `(hardware, dataset_class, instrument)` identity triple means *sibling
+  instrument*; missing only some key fields is corruption.
+- Profiling was the **only conductor without a test file**, so "runtime axis
+  unchanged" had nothing to assert against. `tests/test_profiling_conductor.py` was
+  part of the work, not a bonus.
+
+## Original prompt
+
+# Profiling Agent phase 1 — `campaign --axis compile`: what compile coverage do we actually have?
+
+Type: feature
+Target: PyAutoBrain
+Repos:
+- PyAutoBrain
+Difficulty: small
+Autonomy: supervised
+Priority: high
+Status: formalised
+
+Phase 1 of `profiling_agent_jax_compile_time_scope.md` (read that first — it
+carries the re-scope, the evidence and the comparability constraint).
+
+## Why this leg first
+
+It is the cheapest of the three, it is pure-read, and it produces the number that
+justifies the other two. Right now nobody can say how much of the science grid the
+compile corpus covers, because nothing cross-references the two trees.
+
+Measured by hand 2026-08-10 — this is what the mode should compute automatically:
+
+- **93 records**, all under `scripts/misc/jax_compile/results/<hardware>/<model_type>.json`.
+- Distinct `(dataset_class, model_type, instrument)` touched: **`imaging/mge/hst`,
+  `imaging/pixelization/hst`, `imaging/delaunay_matern/hst`, `imaging/knn/hst`**,
+  plus two synthetic multi-band classes (`datacube_img`, `datacube_img_hetero`).
+- The runtime grid (`scripts/misc/likelihood_runtime/sweep.py::CELLS`) is **21**
+  `(class, model, instrument)` cells across `imaging` / `interferometer` /
+  `datacube` × `hst,jwst,ao` / `sma,alma,alma_high,jvla`.
+- So: **no interferometer row, no datacube row, no `jwst`, no `ao`** — one
+  instrument out of seven, and `delaunay_matern` / `knn` are mesh variants that are
+  not grid cells at all.
+- Hardware: `local_cpu` (82 rows), A100 (9, all `pixelization`/`mge` × `jit`/`vag`),
+  RTX 2060 (2). Plus **4 malformed rows** with `hardware`/`dataset_class`/
+  `instrument` all `null`.
+
+## Scope
+
+Add `--axis compile` to the existing `campaign` mode in
+`agents/conductors/profiling/_profiling.py`. The runtime axis stays the default and
+is untouched.
+
+1. **Read the compile corpus.** Load `scripts/misc/jax_compile/results/*/*.json`
+   (append-only lists of flat records). Stdlib `json` only — the same
+   never-import-the-workspace rule the runtime path already follows via `ast`.
+2. **Resolve each record to a grid cell** using its in-record
+   `(dataset_class, model_type, instrument)`, not its file path — the results tree
+   is filed by `<hardware>/<model_type>` and drops class/instrument from the path.
+3. **Report coverage** over cell × transform × hardware-tier. The transform axis is
+   `probe.py`'s seven: `jit`, `grad`, `vag`, `vmap`, `vmap_vag`, `laxmap_vag`,
+   `pyloop_vag`. Cells not in `CELLS` are reported in their own **off-grid** bucket
+   rather than silently counted or silently dropped — `knn` and `delaunay_matern`
+   are real measurements from the Prodigy census and must not read as noise.
+4. **Report malformed records** (missing `hardware` / `dataset_class` /
+   `instrument`) as their own bucket with file + index, so they can be fixed or
+   deleted deliberately.
+5. **Emit a dispatch plan** of concrete `probe.py` invocations for what is missing,
+   mirroring how the runtime campaign emits `sweep.py` lines:
+   `python jax_compile/probe.py --model-type <m> --transforms <list>` for local,
+   and the `hpc/batch_gpu/submit_*` form for `--tier a100`.
+
+## Explicitly not in scope
+
+- Any judgement about whether a compile time is *good* — that is phase 3.
+- Any pin, baseline or dashboard — that is phase 2.
+- Running `probe.py`. The agent reasons and emits a plan; it never executes the
+  workspace (`ProfilingDecision` contract).
+
+## Acceptance
+
+- `pyauto-brain profiling campaign --axis compile` and
+  `... --axis compile --tier a100` both run against a real `autolens_profiling`
+  checkout and report done / missing / off-grid / malformed counts.
+- The reported on-grid coverage matches the hand count above on the corpus as it
+  stands. MEASURED once built: **11 of 147** cell×transform runs on the `local`
+  tier and **3 of 147** on `a100`, i.e. only `imaging/mge/hst` and
+  `imaging/pixelization/hst` are on-grid. (The pre-build estimate said "3–4 cells
+  of 24" — the grid is 21 cells, and `knn`/`delaunay_matern` are off-grid rather
+  than partial coverage.)
+- `--json` emits the same structure, consistent with the existing modes.
+- The runtime axis output is byte-identical to before the change (regression test).
+- No file in `autolens_profiling` is written or executed by the agent.
+
+<!-- filed 2026-08-10 as phase 1 of the compile-axis arc; coverage figures measured
+     against autolens_profiling main at clone time. -->

complete/2026/08/compile-axis-triage-drift.md

@@ -0,0 +1,100 @@
+# Compile-axis phase 3 — `triage --axis compile`, and the arc closed
+
+- shipped: 2026-08-10
+- issue: https://github.com/PyAutoLabs/PyAutoBrain/issues/221
+- pr: https://github.com/PyAutoLabs/PyAutoBrain/pull/222 (squash a50efc3)
+- repos: PyAutoBrain
+- arc: phase 3 of 3 — closes it
+
+## Summary
+
+`ingest` says a warm compile moved; that alone is useless, because a dead cache, a
+flag that stopped reaching XLA, a busy laptop and a real library regression are the
+same number. `triage --axis compile` separates them into seven classifications,
+three actionable, and names the owner.
+
+`AGENTS.md` moves compile-time profiling out of **Future modes** into the Modes
+table; all three modes now serve `--axis compile`.
+
+## Traps and findings
+
+- **The cold-scale comparison makes `cache-regression` a measurement, not a
+  guess.** 25 of 32 cell/transform keys carry both a warm and a cold row, so the
+  yardstick is real data from the same machine. Verified by injecting a synthetic
+  regression into a copy of the real workspace (warm `vag` 1.622s → its own
+  34.592s cold cost).
+- **Two of the prompt's five categories cannot reach triage by construction.** A
+  `jax_version` bump or a changed host is a *different comparability key*, so
+  `ingest` reports it as unpinned, never drifted. Classified as bookkeeping so
+  nothing vanishes; never regressions.
+- **`host-load` was added in their place** — not in the prompt, but host load alone
+  has produced 7x errors here, and a classifier that cannot say "your laptop was
+  busy" sends people chasing phantoms.
+- Boundaries now records that **release-validation script cost stayed with the
+  hygiene conductor**. It had been moved out of this agent once already; the note
+  exists to stop a third round-trip.
+
+## Original prompt
+
+# Profiling Agent phase 3 — `triage --axis compile`: classify compile drift, route the real ones
+
+Type: feature
+Target: PyAutoBrain
+Repos:
+- PyAutoBrain
+Difficulty: small
+Autonomy: supervised
+Priority: high
+Status: formalised
+
+> Depends on phase 2, `draft/feature/profiling/compile_warm_baseline_dashboard.md`
+> (its pins). Not declared as `Blocked-by:` yet because that key grades **issue**
+> refs and phase 2 has no issue — add the real ref at `/start_dev` time.
+
+Phase 3, and the point of the arc: phases 1–2 make compile drift *visible*; this
+makes it *actionable*. Read `profiling_agent_jax_compile_time_scope.md` first.
+
+## Scope
+
+Extend `triage` with `--axis compile`, classifying each drifted pin from phase 2
+into one of four outcomes — the same shape as the runtime triage's per-finding
+classification.
+
+| classification | signal | action |
+|---|---|---|
+| **cache regression** | a `warm` row's compile reverts toward its `cold` scale on an unchanged key | **the alarm this arc exists for** — the persistent cache is not being hit; route to config/stack, not to the library |
+| **autotune regression** | GPU compile jumps ~an order of magnitude on the pathological shapes | `--xla_gpu_autotune_level=0` is not reaching XLA — the exact PyAutoNerves#127 `XLA_FLAGS`-clobbering failure, which went undetected for two months |
+| **expected recompile** | `jax_version` moved | **not drift** — cache keys include jax version, so one recompile is by design. Re-pin, do not report |
+| **stale pin** | measurement conditions changed (host, core count, `mixed_precision`) | re-pin here |
+| **library regression** | compile grows on an unchanged key with no config explanation | route to `bug/` via intake — *classify and route only* |
+
+## The two constraints inherited from the agent's existing contract
+
+- **Profiling records and flags; it never adjudicates library correctness.** A
+  suspected library regression is routed to `bug/` via intake; the debug is never
+  planned inside the profiling repo. This is the existing `triage` boundary and it
+  applies unchanged.
+- **Never compare across the comparability key.** `(hardware, jax_version,
+  mixed_precision, cache_state)`. Compile timings are host-load-sensitive — the
+  7×-wrong measurements (851 s vs 117 s, same compile) are the standing reminder.
+
+## Close the arc
+
+- Move "JAX compilation-time profiling of likelihood functions" from **Future
+  modes** to the **Modes** table in `agents/conductors/profiling/AGENTS.md`.
+- Record in **Boundaries** that release-validation script cost stayed with the
+  hygiene conductor's `perf` mode, so the question does not get re-opened a third
+  time.
+
+## Acceptance
+
+- `pyauto-brain profiling triage --axis compile` classifies every phase-2 drift row
+  into exactly one of the five outcomes, with the evidence that drove it.
+- A synthetic warm-reverting-to-cold row classifies as **cache regression**.
+- A drift explained solely by a `jax_version` bump classifies as **expected
+  recompile** and is not reported as a regression.
+- Nothing is written to `autolens_profiling`; library findings are emitted as
+  intake-routable `bug/` candidates only.
+- `AGENTS.md` Modes/Boundaries updated as above.
+
+<!-- filed 2026-08-10 as phase 3 of the compile-axis arc. -->

complete/2026/08/compile-warm-baseline-dashboard.md

@@ -0,0 +1,144 @@
+# Compile-axis phase 2 — warm compile made identifiable, pinned, dashboarded
+
+- shipped: 2026-08-10
+- issues: https://github.com/PyAutoLabs/autolens_profiling/issues/103
+- prs: autolens_profiling#104 (squash 355d555) · PyAutoBrain#220 (squash 74d7b1b)
+- repos: autolens_profiling, PyAutoBrain
+- arc: phase 2 of 3
+
+## Summary
+
+Tracking **warm** compile is the point of the arc — the persistent cache turned
+117.0s into 2.3s (CPU MGE `vag`) and 5517.8s into 937.1s (A100 end-to-end), and
+nothing watched for that reverting. Both wins are *settings*.
+
+`cache_state` is now derived from what the compile DID (cache entries counted
+either side of `lowered.compile()`), 25 warm pins are committed, the dashboard
+renders through the existing sentinel-block mechanism, and
+`ingest --axis compile` reports drift.
+
+## Traps and findings
+
+- **`cache_dir` is non-empty on COLD rows too** — the cold run is the one that
+  populates the cache. Any truthiness check on it, or substring match on the tag,
+  gets warmness exactly backwards. The corpus carries ~40 ad-hoc tag spellings.
+- **`hardware` alone pools different machines.** It is only ever `local_cpu` /
+  `local_gpu_<dev>`, so one `local_cpu` label spanned a laptop (66 records) and a
+  32-core RAL node (12). The first rendered dashboard put pins from both side by
+  side under one heading — the 7x host-load hazard, live. `hostname` joined the
+  comparability key.
+- **Pins must be STICKY.** "Most recent warm row wins" meant re-deriving pins after
+  a cache regression would move the pin *onto* the regressed value and report
+  all-clear forever — exactly inverted from the purpose. `--repin` is now required.
+- **Rows predating a pin are not drift.** The first `ingest` run flagged four, all
+  measurements the pin had been *chosen over*.
+- Backfill used **end-anchored** tag matching: `mb_homo_cold_laxmap_gpu` contains
+  "cold" mid-tag and is left `unknown` rather than mislabelled.
+- `scripts/misc/test/` **ran in no workflow at all**; a pytest step was added to
+  `lint.yml` (47 tests, ~2s). Its apparent failure outside CI was a missing
+  `matplotlib` (imported at module scope by `aggregate.py`), not a defect.
+
+## Original prompt
+
+# Profiling Agent phase 2 — make warm compile machine-identifiable, pinnable, and dashboarded
+
+Type: feature
+Target: autolens_profiling
+Repos:
+- autolens_profiling
+- PyAutoBrain
+Difficulty: medium
+Autonomy: supervised
+Priority: high
+Status: formalised
+
+> Depends on phase 1, `draft/feature/profiling/compile_axis_campaign_coverage.md`
+> (its cell-resolution logic). Not declared as `Blocked-by:` yet because that key
+> grades **issue** refs and phase 1 has no issue — add the real ref at
+> `/start_dev` time, once phase 1 is issued.
+
+Phase 2 of `profiling_agent_jax_compile_time_scope.md`. **Absorbs**
+`draft/feature/autolens_profiling/jax_compile_time_profiling.md`, whose surviving
+scope (2026-07-28 re-scope: "the recurring cell-grid compile **dashboard** — track
+warm compile per cell so cache regressions are caught") is this prompt's workspace
+leg.
+
+## The blocker this prompt exists to remove
+
+The whole arc's purpose is to catch the persistent-compilation-cache win
+(117.0 s → 2.3 s CPU; 5517.8 s → 937.1 s A100 end-to-end) silently reverting.
+That requires tracking **warm** compile per cell.
+
+**Warm rows are not machine-identifiable today.** Warmness is encoded only in a
+free-text `tag`. Across the 93 committed records there are ~40 distinct ad-hoc
+values: `census-warm`, `census-warm2`, `cache-warm`, `a100-census-warm`,
+`prodigy-census-warm`, `prodigy-census-warm-retry`, `prodigy-census-ral32-warm`,
+`mb_homo_warm`, `mb_hetero_warm`, against colds spelled a dozen other ways, plus
+`idle-check`, `smoke`, `matrix`, `pix-first`, `a100-at0`, `flag-parallel-codegen`.
+No parser should be asked to guess at that, and a substring match on `"warm"`
+would be a trap: `cache_dir` is non-empty on rows tagged `*-cold` too (the cache is
+*configured*, and the cold row is the one that populates it).
+
+`cache_dir` is recorded but as a machine-specific path
+(`/tmp/jax_cache` vs the RAL path), so it answers "a cache was configured", never
+"this row hit it".
+
+## Workspace leg — `autolens_profiling`
+
+1. **Add an explicit `cache_state` field to `probe.py`'s record**: `cold` | `warm` |
+   `none` (no cache configured). Derive it from what the probe actually did, not
+   from the tag. `--cache-dir` with a pre-existing populated entry for the shape is
+   `warm`; a fresh/empty cache dir is `cold`; no `--cache-dir` is `none`.
+2. **Add `host_state`** (or equivalent) capturing the idle/loaded provenance the
+   README currently carries in prose. This is load-bearing, not bookkeeping: the
+   first measurements were wrong by **7×** (851 s vs 117 s for the same compile)
+   purely from host load, and `prodigy-census-ral32-*` rows sit in the same tree at
+   a different core count. Record at least core count and load average; leave `tag`
+   as the free-text human note it already is.
+3. **Backfill** the existing 93 records where the mapping is unambiguous from tag +
+   README provenance, and leave the rest `unknown` rather than guessing. Fix or
+   delete the **4 malformed records** (`hardware`/`dataset_class`/`instrument` all
+   `null`) as a deliberate call.
+4. **Pin warm compile per cell** — the compile-axis equivalent of the runtime
+   results' `pinned_expected`, which is the hook `ingest`/`triage` are already
+   built around. A pin is only meaningful within one comparability key
+   `(hardware, jax_version, mixed_precision, cache_state)`, so the pin store must be
+   keyed on it.
+5. **Dashboard rows** through the existing `build_baseline.py` / `build_readme.py`
+   pattern, so warm compile per cell appears alongside the runtime tables.
+
+## Brain leg — `PyAutoBrain`
+
+6. **`ingest --axis compile`**: report compile records that are unpinned, and
+   records whose warm compile has moved away from its pin — *only ever within one
+   comparability key*. Cross-key pairs are not a regression and must not be
+   reported as one. Mirror the runtime `ingest`'s freshness discipline (it skips
+   probes older than the table file for a documented reason).
+7. Emit the same shape as the runtime axis: rows to apply, plus `steps` and
+   `next_action`.
+
+## The trap
+
+Do **not** let the dashboard compare a `cold` row against a `warm` pin, or an
+A100 row against a CPU pin, or rows across a `jax_version` bump. Cache keys include
+jax version and shapes, so a version bump recompiles once **by design** — that is
+expected behaviour, not drift, and phase 3 classifies it as such. A tool that flags
+it is a tool people learn to ignore.
+
+## Acceptance
+
+- `probe.py` records carry `cache_state` derived from probe behaviour; a fresh
+  cold/warm pair on one cell produces exactly one `cold` and one `warm` record with
+  no tag parsing anywhere in the pipeline.
+- Warm compile per cell is pinned and rendered in the workspace dashboard.
+- `pyauto-brain profiling ingest --axis compile` reports unpinned and drifted rows,
+  and **never** pairs rows across `(hardware, jax_version, mixed_precision,
+  cache_state)`.
+- A synthetic warm row at cold-scale timing is reported as drift; the same row
+  under a bumped `jax_version` is not.
+- The 4 malformed records are resolved.
+- Runtime-axis behaviour unchanged.
+
+<!-- filed 2026-08-10 as phase 2 of the compile-axis arc; absorbs the workspace-side
+     jax_compile_time_profiling.md. Tag/cache_state findings measured against the 93
+     committed probe records on autolens_profiling main. -->