diff --git a/complete/2026/08/compile-axis-campaign-coverage.md b/complete/2026/08/compile-axis-campaign-coverage.md new file mode 100644 index 00000000..f671355c --- /dev/null +++ b/complete/2026/08/compile-axis-campaign-coverage.md @@ -0,0 +1,123 @@ +# Compile-axis phase 1 — `campaign --axis compile` + +- shipped: 2026-08-10 +- issue: https://github.com/PyAutoLabs/PyAutoBrain/issues/218 +- pr: https://github.com/PyAutoLabs/PyAutoBrain/pull/219 (squash cd79005) +- repos: PyAutoBrain +- arc: phase 1 of 3 (see compile-warm-baseline-dashboard, compile-axis-triage-drift) + +## Summary + +The Profiling Agent could not see the compile corpus `autolens_profiling` was +already producing: `AGENTS.md` listed compile-time profiling under *Future modes*, +`_profiling.py` had zero occurrences of "compile", and all three modes read only +`results/runtime/` — while 93 committed probe records sat uncross-referenced. + +`campaign --axis compile` answers coverage: **2 of 21 grid cells**, `hst` only, no +interferometer / datacube / `jwst` / `ao` row anywhere. That number is what +justified phases 2 and 3. + +## Traps and findings + +- **The tier vocabularies do not interchange.** Runtime buckets by sweep *config* + name (`local_cpu_fp64`), folding precision into the name; a compile record has a + raw `hardware` string plus a **separate** `mixed_precision` bool. Reusing + `TIER_CONFIGS` would have mis-bucketed every row. +- **Records must be placed by their own fields, not their path.** Results are filed + under `/`, which drops class and instrument entirely. +- **`export_probe.py` / `trace_profile.py` share the results tree** with a different + schema. The first implementation reported their 4 records as "malformed", which + would have sent someone to fix two files that work correctly. Missing the whole + `(hardware, dataset_class, instrument)` identity triple means *sibling + instrument*; missing only some key fields is corruption. +- Profiling was the **only conductor without a test file**, so "runtime axis + unchanged" had nothing to assert against. `tests/test_profiling_conductor.py` was + part of the work, not a bonus. + +## Original prompt + +# Profiling Agent phase 1 — `campaign --axis compile`: what compile coverage do we actually have? + +Type: feature +Target: PyAutoBrain +Repos: +- PyAutoBrain +Difficulty: small +Autonomy: supervised +Priority: high +Status: formalised + +Phase 1 of `profiling_agent_jax_compile_time_scope.md` (read that first — it +carries the re-scope, the evidence and the comparability constraint). + +## Why this leg first + +It is the cheapest of the three, it is pure-read, and it produces the number that +justifies the other two. Right now nobody can say how much of the science grid the +compile corpus covers, because nothing cross-references the two trees. + +Measured by hand 2026-08-10 — this is what the mode should compute automatically: + +- **93 records**, all under `scripts/misc/jax_compile/results//.json`. +- Distinct `(dataset_class, model_type, instrument)` touched: **`imaging/mge/hst`, + `imaging/pixelization/hst`, `imaging/delaunay_matern/hst`, `imaging/knn/hst`**, + plus two synthetic multi-band classes (`datacube_img`, `datacube_img_hetero`). +- The runtime grid (`scripts/misc/likelihood_runtime/sweep.py::CELLS`) is **21** + `(class, model, instrument)` cells across `imaging` / `interferometer` / + `datacube` × `hst,jwst,ao` / `sma,alma,alma_high,jvla`. +- So: **no interferometer row, no datacube row, no `jwst`, no `ao`** — one + instrument out of seven, and `delaunay_matern` / `knn` are mesh variants that are + not grid cells at all. +- Hardware: `local_cpu` (82 rows), A100 (9, all `pixelization`/`mge` × `jit`/`vag`), + RTX 2060 (2). Plus **4 malformed rows** with `hardware`/`dataset_class`/ + `instrument` all `null`. + +## Scope + +Add `--axis compile` to the existing `campaign` mode in +`agents/conductors/profiling/_profiling.py`. The runtime axis stays the default and +is untouched. + +1. **Read the compile corpus.** Load `scripts/misc/jax_compile/results/*/*.json` + (append-only lists of flat records). Stdlib `json` only — the same + never-import-the-workspace rule the runtime path already follows via `ast`. +2. **Resolve each record to a grid cell** using its in-record + `(dataset_class, model_type, instrument)`, not its file path — the results tree + is filed by `/` and drops class/instrument from the path. +3. **Report coverage** over cell × transform × hardware-tier. The transform axis is + `probe.py`'s seven: `jit`, `grad`, `vag`, `vmap`, `vmap_vag`, `laxmap_vag`, + `pyloop_vag`. Cells not in `CELLS` are reported in their own **off-grid** bucket + rather than silently counted or silently dropped — `knn` and `delaunay_matern` + are real measurements from the Prodigy census and must not read as noise. +4. **Report malformed records** (missing `hardware` / `dataset_class` / + `instrument`) as their own bucket with file + index, so they can be fixed or + deleted deliberately. +5. **Emit a dispatch plan** of concrete `probe.py` invocations for what is missing, + mirroring how the runtime campaign emits `sweep.py` lines: + `python jax_compile/probe.py --model-type --transforms ` for local, + and the `hpc/batch_gpu/submit_*` form for `--tier a100`. + +## Explicitly not in scope + +- Any judgement about whether a compile time is *good* — that is phase 3. +- Any pin, baseline or dashboard — that is phase 2. +- Running `probe.py`. The agent reasons and emits a plan; it never executes the + workspace (`ProfilingDecision` contract). + +## Acceptance + +- `pyauto-brain profiling campaign --axis compile` and + `... --axis compile --tier a100` both run against a real `autolens_profiling` + checkout and report done / missing / off-grid / malformed counts. +- The reported on-grid coverage matches the hand count above on the corpus as it + stands. MEASURED once built: **11 of 147** cell×transform runs on the `local` + tier and **3 of 147** on `a100`, i.e. only `imaging/mge/hst` and + `imaging/pixelization/hst` are on-grid. (The pre-build estimate said "3–4 cells + of 24" — the grid is 21 cells, and `knn`/`delaunay_matern` are off-grid rather + than partial coverage.) +- `--json` emits the same structure, consistent with the existing modes. +- The runtime axis output is byte-identical to before the change (regression test). +- No file in `autolens_profiling` is written or executed by the agent. + + diff --git a/complete/2026/08/compile-axis-triage-drift.md b/complete/2026/08/compile-axis-triage-drift.md new file mode 100644 index 00000000..acddfeb7 --- /dev/null +++ b/complete/2026/08/compile-axis-triage-drift.md @@ -0,0 +1,100 @@ +# Compile-axis phase 3 — `triage --axis compile`, and the arc closed + +- shipped: 2026-08-10 +- issue: https://github.com/PyAutoLabs/PyAutoBrain/issues/221 +- pr: https://github.com/PyAutoLabs/PyAutoBrain/pull/222 (squash a50efc3) +- repos: PyAutoBrain +- arc: phase 3 of 3 — closes it + +## Summary + +`ingest` says a warm compile moved; that alone is useless, because a dead cache, a +flag that stopped reaching XLA, a busy laptop and a real library regression are the +same number. `triage --axis compile` separates them into seven classifications, +three actionable, and names the owner. + +`AGENTS.md` moves compile-time profiling out of **Future modes** into the Modes +table; all three modes now serve `--axis compile`. + +## Traps and findings + +- **The cold-scale comparison makes `cache-regression` a measurement, not a + guess.** 25 of 32 cell/transform keys carry both a warm and a cold row, so the + yardstick is real data from the same machine. Verified by injecting a synthetic + regression into a copy of the real workspace (warm `vag` 1.622s → its own + 34.592s cold cost). +- **Two of the prompt's five categories cannot reach triage by construction.** A + `jax_version` bump or a changed host is a *different comparability key*, so + `ingest` reports it as unpinned, never drifted. Classified as bookkeeping so + nothing vanishes; never regressions. +- **`host-load` was added in their place** — not in the prompt, but host load alone + has produced 7x errors here, and a classifier that cannot say "your laptop was + busy" sends people chasing phantoms. +- Boundaries now records that **release-validation script cost stayed with the + hygiene conductor**. It had been moved out of this agent once already; the note + exists to stop a third round-trip. + +## Original prompt + +# Profiling Agent phase 3 — `triage --axis compile`: classify compile drift, route the real ones + +Type: feature +Target: PyAutoBrain +Repos: +- PyAutoBrain +Difficulty: small +Autonomy: supervised +Priority: high +Status: formalised + +> Depends on phase 2, `draft/feature/profiling/compile_warm_baseline_dashboard.md` +> (its pins). Not declared as `Blocked-by:` yet because that key grades **issue** +> refs and phase 2 has no issue — add the real ref at `/start_dev` time. + +Phase 3, and the point of the arc: phases 1–2 make compile drift *visible*; this +makes it *actionable*. Read `profiling_agent_jax_compile_time_scope.md` first. + +## Scope + +Extend `triage` with `--axis compile`, classifying each drifted pin from phase 2 +into one of four outcomes — the same shape as the runtime triage's per-finding +classification. + +| classification | signal | action | +|---|---|---| +| **cache regression** | a `warm` row's compile reverts toward its `cold` scale on an unchanged key | **the alarm this arc exists for** — the persistent cache is not being hit; route to config/stack, not to the library | +| **autotune regression** | GPU compile jumps ~an order of magnitude on the pathological shapes | `--xla_gpu_autotune_level=0` is not reaching XLA — the exact PyAutoNerves#127 `XLA_FLAGS`-clobbering failure, which went undetected for two months | +| **expected recompile** | `jax_version` moved | **not drift** — cache keys include jax version, so one recompile is by design. Re-pin, do not report | +| **stale pin** | measurement conditions changed (host, core count, `mixed_precision`) | re-pin here | +| **library regression** | compile grows on an unchanged key with no config explanation | route to `bug/` via intake — *classify and route only* | + +## The two constraints inherited from the agent's existing contract + +- **Profiling records and flags; it never adjudicates library correctness.** A + suspected library regression is routed to `bug/` via intake; the debug is never + planned inside the profiling repo. This is the existing `triage` boundary and it + applies unchanged. +- **Never compare across the comparability key.** `(hardware, jax_version, + mixed_precision, cache_state)`. Compile timings are host-load-sensitive — the + 7×-wrong measurements (851 s vs 117 s, same compile) are the standing reminder. + +## Close the arc + +- Move "JAX compilation-time profiling of likelihood functions" from **Future + modes** to the **Modes** table in `agents/conductors/profiling/AGENTS.md`. +- Record in **Boundaries** that release-validation script cost stayed with the + hygiene conductor's `perf` mode, so the question does not get re-opened a third + time. + +## Acceptance + +- `pyauto-brain profiling triage --axis compile` classifies every phase-2 drift row + into exactly one of the five outcomes, with the evidence that drove it. +- A synthetic warm-reverting-to-cold row classifies as **cache regression**. +- A drift explained solely by a `jax_version` bump classifies as **expected + recompile** and is not reported as a regression. +- Nothing is written to `autolens_profiling`; library findings are emitted as + intake-routable `bug/` candidates only. +- `AGENTS.md` Modes/Boundaries updated as above. + + diff --git a/complete/2026/08/compile-warm-baseline-dashboard.md b/complete/2026/08/compile-warm-baseline-dashboard.md new file mode 100644 index 00000000..0f2f3f48 --- /dev/null +++ b/complete/2026/08/compile-warm-baseline-dashboard.md @@ -0,0 +1,144 @@ +# Compile-axis phase 2 — warm compile made identifiable, pinned, dashboarded + +- shipped: 2026-08-10 +- issues: https://github.com/PyAutoLabs/autolens_profiling/issues/103 +- prs: autolens_profiling#104 (squash 355d555) · PyAutoBrain#220 (squash 74d7b1b) +- repos: autolens_profiling, PyAutoBrain +- arc: phase 2 of 3 + +## Summary + +Tracking **warm** compile is the point of the arc — the persistent cache turned +117.0s into 2.3s (CPU MGE `vag`) and 5517.8s into 937.1s (A100 end-to-end), and +nothing watched for that reverting. Both wins are *settings*. + +`cache_state` is now derived from what the compile DID (cache entries counted +either side of `lowered.compile()`), 25 warm pins are committed, the dashboard +renders through the existing sentinel-block mechanism, and +`ingest --axis compile` reports drift. + +## Traps and findings + +- **`cache_dir` is non-empty on COLD rows too** — the cold run is the one that + populates the cache. Any truthiness check on it, or substring match on the tag, + gets warmness exactly backwards. The corpus carries ~40 ad-hoc tag spellings. +- **`hardware` alone pools different machines.** It is only ever `local_cpu` / + `local_gpu_`, so one `local_cpu` label spanned a laptop (66 records) and a + 32-core RAL node (12). The first rendered dashboard put pins from both side by + side under one heading — the 7x host-load hazard, live. `hostname` joined the + comparability key. +- **Pins must be STICKY.** "Most recent warm row wins" meant re-deriving pins after + a cache regression would move the pin *onto* the regressed value and report + all-clear forever — exactly inverted from the purpose. `--repin` is now required. +- **Rows predating a pin are not drift.** The first `ingest` run flagged four, all + measurements the pin had been *chosen over*. +- Backfill used **end-anchored** tag matching: `mb_homo_cold_laxmap_gpu` contains + "cold" mid-tag and is left `unknown` rather than mislabelled. +- `scripts/misc/test/` **ran in no workflow at all**; a pytest step was added to + `lint.yml` (47 tests, ~2s). Its apparent failure outside CI was a missing + `matplotlib` (imported at module scope by `aggregate.py`), not a defect. + +## Original prompt + +# Profiling Agent phase 2 — make warm compile machine-identifiable, pinnable, and dashboarded + +Type: feature +Target: autolens_profiling +Repos: +- autolens_profiling +- PyAutoBrain +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +> Depends on phase 1, `draft/feature/profiling/compile_axis_campaign_coverage.md` +> (its cell-resolution logic). Not declared as `Blocked-by:` yet because that key +> grades **issue** refs and phase 1 has no issue — add the real ref at +> `/start_dev` time, once phase 1 is issued. + +Phase 2 of `profiling_agent_jax_compile_time_scope.md`. **Absorbs** +`draft/feature/autolens_profiling/jax_compile_time_profiling.md`, whose surviving +scope (2026-07-28 re-scope: "the recurring cell-grid compile **dashboard** — track +warm compile per cell so cache regressions are caught") is this prompt's workspace +leg. + +## The blocker this prompt exists to remove + +The whole arc's purpose is to catch the persistent-compilation-cache win +(117.0 s → 2.3 s CPU; 5517.8 s → 937.1 s A100 end-to-end) silently reverting. +That requires tracking **warm** compile per cell. + +**Warm rows are not machine-identifiable today.** Warmness is encoded only in a +free-text `tag`. Across the 93 committed records there are ~40 distinct ad-hoc +values: `census-warm`, `census-warm2`, `cache-warm`, `a100-census-warm`, +`prodigy-census-warm`, `prodigy-census-warm-retry`, `prodigy-census-ral32-warm`, +`mb_homo_warm`, `mb_hetero_warm`, against colds spelled a dozen other ways, plus +`idle-check`, `smoke`, `matrix`, `pix-first`, `a100-at0`, `flag-parallel-codegen`. +No parser should be asked to guess at that, and a substring match on `"warm"` +would be a trap: `cache_dir` is non-empty on rows tagged `*-cold` too (the cache is +*configured*, and the cold row is the one that populates it). + +`cache_dir` is recorded but as a machine-specific path +(`/tmp/jax_cache` vs the RAL path), so it answers "a cache was configured", never +"this row hit it". + +## Workspace leg — `autolens_profiling` + +1. **Add an explicit `cache_state` field to `probe.py`'s record**: `cold` | `warm` | + `none` (no cache configured). Derive it from what the probe actually did, not + from the tag. `--cache-dir` with a pre-existing populated entry for the shape is + `warm`; a fresh/empty cache dir is `cold`; no `--cache-dir` is `none`. +2. **Add `host_state`** (or equivalent) capturing the idle/loaded provenance the + README currently carries in prose. This is load-bearing, not bookkeeping: the + first measurements were wrong by **7×** (851 s vs 117 s for the same compile) + purely from host load, and `prodigy-census-ral32-*` rows sit in the same tree at + a different core count. Record at least core count and load average; leave `tag` + as the free-text human note it already is. +3. **Backfill** the existing 93 records where the mapping is unambiguous from tag + + README provenance, and leave the rest `unknown` rather than guessing. Fix or + delete the **4 malformed records** (`hardware`/`dataset_class`/`instrument` all + `null`) as a deliberate call. +4. **Pin warm compile per cell** — the compile-axis equivalent of the runtime + results' `pinned_expected`, which is the hook `ingest`/`triage` are already + built around. A pin is only meaningful within one comparability key + `(hardware, jax_version, mixed_precision, cache_state)`, so the pin store must be + keyed on it. +5. **Dashboard rows** through the existing `build_baseline.py` / `build_readme.py` + pattern, so warm compile per cell appears alongside the runtime tables. + +## Brain leg — `PyAutoBrain` + +6. **`ingest --axis compile`**: report compile records that are unpinned, and + records whose warm compile has moved away from its pin — *only ever within one + comparability key*. Cross-key pairs are not a regression and must not be + reported as one. Mirror the runtime `ingest`'s freshness discipline (it skips + probes older than the table file for a documented reason). +7. Emit the same shape as the runtime axis: rows to apply, plus `steps` and + `next_action`. + +## The trap + +Do **not** let the dashboard compare a `cold` row against a `warm` pin, or an +A100 row against a CPU pin, or rows across a `jax_version` bump. Cache keys include +jax version and shapes, so a version bump recompiles once **by design** — that is +expected behaviour, not drift, and phase 3 classifies it as such. A tool that flags +it is a tool people learn to ignore. + +## Acceptance + +- `probe.py` records carry `cache_state` derived from probe behaviour; a fresh + cold/warm pair on one cell produces exactly one `cold` and one `warm` record with + no tag parsing anywhere in the pipeline. +- Warm compile per cell is pinned and rendered in the workspace dashboard. +- `pyauto-brain profiling ingest --axis compile` reports unpinned and drifted rows, + and **never** pairs rows across `(hardware, jax_version, mixed_precision, + cache_state)`. +- A synthetic warm row at cold-scale timing is reported as drift; the same row + under a bumped `jax_version` is not. +- The 4 malformed records are resolved. +- Runtime-axis behaviour unchanged. + + diff --git a/complete/index.md b/complete/index.md index 85da2500..d918722a 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -963 records across 7 buckets. +966 records across 7 buckets. ## Highlights @@ -28,6 +28,9 @@ _(curate hard-won records here — survives regeneration.)_ - [autohands-firewall-allowlist](2026/08/autohands-firewall-allowlist.md) — auto-closed by the merge - [autolens-multiplane-redshift-warning](2026/08/autolens-multiplane-redshift-warning.md) - [autolens-tracer-validation-guards](2026/08/autolens-tracer-validation-guards.md) +- [compile-axis-campaign-coverage](2026/08/compile-axis-campaign-coverage.md) +- [compile-axis-triage-drift](2026/08/compile-axis-triage-drift.md) +- [compile-warm-baseline-dashboard](2026/08/compile-warm-baseline-dashboard.md) - [conductor-discovery-lifecycle-split](2026/08/conductor-discovery-lifecycle-split.md) — closed on merge - [covariance-interpolator-rng-seed](2026/08/covariance-interpolator-rng-seed.md) — auto-closed by the merge - [database-guide-sample-weight-threshold](2026/08/database-guide-sample-weight-threshold.md) — `guides/results/database/start_here.py` ran its own Nautilus fits capped at n_like_max=300 then indexed sample… diff --git a/draft/feature/autolens_profiling/jax_compile_time_profiling.md b/draft/feature/autolens_profiling/jax_compile_time_profiling.md index 6485ddc4..e403733b 100644 --- a/draft/feature/autolens_profiling/jax_compile_time_profiling.md +++ b/draft/feature/autolens_profiling/jax_compile_time_profiling.md @@ -1,5 +1,14 @@ # Profile and speed up JAX likelihood-function compile times (all use cases) +> **ABSORBED 2026-08-10 — do not start from this file.** The surviving scope (the +> recurring cell-grid compile dashboard, per the 2026-07-28 note below) is now the +> workspace leg of +> `draft/feature/profiling/compile_warm_baseline_dashboard.md`, phase 2 of the +> compile-axis arc `draft/feature/profiling/profiling_agent_jax_compile_time_scope.md`. +> Merged there because the dashboard and the Profiling Agent that reads it are two +> halves of one thing, and building them from opposite ends risks incompatible +> schemas. This file is kept for its measurements and its overturned-claim trail. + > **Status note (2026-07-28):** largely superseded. The compile-time arc > (#71→#77) shipped cache-by-default + autotune-off and closed the speedup > question ("settings suffice"); this prompt's "autotuning ruled out" section diff --git a/draft/feature/profiling/profiling_agent_jax_compile_time_scope.md b/draft/feature/profiling/profiling_agent_jax_compile_time_scope.md index 5f896b8b..edc4ec69 100644 --- a/draft/feature/profiling/profiling_agent_jax_compile_time_scope.md +++ b/draft/feature/profiling/profiling_agent_jax_compile_time_scope.md @@ -1,4 +1,4 @@ -# Extend the Profiling Agent scope to track JAX compile/eval times of release-validation heavy scripts +# Give the Profiling Agent a compile-time axis — the arc Type: feature Target: PyAutoBrain @@ -8,7 +8,149 @@ Repos: Difficulty: large Autonomy: supervised Priority: high -Status: formalised +Status: planned (arc — execute via the three child prompts below) + +> **RE-SCOPED 2026-08-10.** This prompt was filed 2026-07-14 asking the Profiling +> Agent to *measure JAX compile/eval time of the release-validation heavy scripts +> so we can speed them up*. Both halves of that framing are now spent: the +> speed-up shipped, and the release-script half belongs to a different conductor. +> What survives — and is still unbuilt — is **regression surveillance**. The +> original text is preserved under § What the prompt originally said. + +## Why the original framing no longer holds + +**The speed-up already shipped.** The `#71 → #74 → #77` arc +(`autolens_profiling/scripts/misc/jax_compile/README.md`) settled the question +this prompt opened, with controlled A/Bs: + +| lever | effect | status | +|---|---|---| +| persistent compilation cache | 117.0 s → **2.3 s** (CPU MGE `vag`, 51×); 5517.8 s → **937.1 s** (A100 pixelized Nautilus end-to-end) | shipped | +| `--xla_gpu_autotune_level=0` | FD probe 498 s → **29 s** (17×); full 300×16 Adam fit −40 %; likelihoods bit-identical | shipped | +| source jit-boundaries | ruled out by evidence — "**Do not restructure**" | closed | + +The prompt's own candidate root causes are likewise resolved: `lax.map` is +innocent (every batched-gradient structure compiles in the same ~105–125 s band), +the >30-min repeated-fusion observations were host-load contention plus a +re-firing alarm banner, and the tracing floor is 58 % jax-internal with no PyAuto +lever. + +**So the durable risk is no longer slowness — it is silent regression.** Both wins +are *settings*. A config drift, a `jax` version bump, or a wrapper that clobbers +`XLA_FLAGS` (exactly what PyAutoNerves#127 fixed, and what invalidated the +2026-07-15 "autotuning ruled out" A/B for two months) puts the 70-minute +worst case straight back without anything failing. Nothing watches for that today. + +This is precisely the handoff the research left open — `jax_compile/README.md` +Verdict item 4: *"should track **warm** compile times per cell so cache +regressions are caught"*, and `probe.py`'s own docstring: *"industrializes this +across the full cell grid once the research settles the method."* The method has +settled. + +## Why the release-validation-scripts leg is dropped + +The original prompt asked profiling to cover the release-validation heavy scripts +(`jax_grad/imaging_pixelization.py` et al. blowing the 300 s cap). That crosses a +boundary both conductors already document: + +- `agents/conductors/profiling/AGENTS.md` — profiling owns "the product's + modelling / compute speed (likelihood on the science grid, GPU tiers, A100)". +- `agents/conductors/hygiene/AGENTS.md` — hygiene owns "the *developer loop's* + cost (unit tests, `PYAUTO_TEST_MODE` / `PYAUTO_SMALL_DATASETS` integration + scripts, import time)", and "Hunting generally-slow functions flagged by + integration tests is hygiene's `perf` mode (**moved here from profiling's staged + future modes**)". + +Script-suite cost was deliberately moved *out* of this agent once already. Pulling +it back in would re-open a settled boundary to serve a motivation (the 300 s cap) +that PyAutoHeart#72 has since handled. Decision 2026-08-10: **out of scope here**; +if release-script cost needs an owner, it is a `/hygiene` `perf` prompt, filed +separately. + +## The gap, confirmed against current main + +- `agents/conductors/profiling/AGENTS.md` still lists *"JAX compilation-time + profiling of likelihood functions"* under **Future modes**. +- `agents/conductors/profiling/_profiling.py` (353 lines) contains **zero** + occurrences of "compile". `campaign` / `ingest` / `triage` read only + `results/runtime/`. +- The workspace instrument exists and is already producing data — 93 records under + `scripts/misc/jax_compile/results//.json`. + +So this is a **wiring** task, not an instrument-building one. The agent cannot see +a tree that is already full. + +## The three blockers the child prompts exist to clear + +Measured against the 93 committed records, 2026-08-10: + +1. **Coverage is 2 of 21 grid cells.** Every record is `hst`; there is **no + interferometer, no datacube, no `jwst`, no `ao`** row, and only `local_cpu` is + meaningfully populated (A100 has 9 rows, all `pixelization`/`mge` × `jit`/`vag`). + Two model types present (`knn`, `delaunay_matern`) are mesh variants that are + not in the runtime `CELLS` grid at all. → **child prompt 1**. + + *(Corrected 2026-08-10 once phase 1 computed it: the grid is 21 cells, not the + 24 first written here, and only `imaging/mge/hst` + `imaging/pixelization/hst` + are on-grid — 11 of 147 cell×transform runs on the local tier, 3 on A100.)* +2. **Warm rows are not machine-identifiable.** "Warm" is encoded only in a + free-text `tag` — ~40 distinct ad-hoc values across the corpus + (`census-warm`, `census-warm2`, `prodigy-census-warm-retry`, `cache-warm`, + `a100-census-warm`, `mb_homo_cold`, `idle-check`, `smoke`, `matrix`, …). + A dashboard that must track *warm* compile literally cannot tell which rows are + warm. `cache_dir` is recorded but as a machine-specific path, so it answers + "a cache was configured", not "this row hit it". → **child prompt 2**. +3. **Compile rows have no pin.** Runtime results carry `pinned_expected`, which is + what `ingest` and `triage` are built around. Nothing equivalent exists for + compile, so there is no baseline to regress against. → **child prompt 2**, used + by **child prompt 3**. + +Also present and needing a decision rather than a fix: **4 malformed records** with +`hardware`, `dataset_class` and `instrument` all `null`. + +## The constraint any design must carry + +**Compile timings are host-load-sensitive and cross-host comparison is invalid.** +The README records the first measurements being wrong by up to **7×** (851 s vs +117 s for the same compile) purely from host load, and those rows are retained +"only with their original tags for provenance". The corpus already contains +`prodigy-census-ral32-*` rows taken on a different core count from the +`local_cpu` rows filed beside them. + +So comparability is a **key**, not a nicety: +`(hardware, jax_version, mixed_precision, cache_state)` — plus an explicit +idle/loaded provenance signal. An agent that compares across that key will report +regressions that are only a busy laptop, which is worse than no surveillance at +all. + +## Phasing + +| # | Prompt | Repos | Size | +|---|---|---|---| +| 1 | `compile_axis_campaign_coverage.md` | PyAutoBrain | small | +| 2 | `compile_warm_baseline_dashboard.md` | autolens_profiling + PyAutoBrain | medium | +| 3 | `compile_axis_triage_drift.md` | PyAutoBrain | small–medium | + +Strictly sequential: 2 needs 1's cell-resolution logic, 3 needs 2's pins. + +`draft/feature/autolens_profiling/jax_compile_time_profiling.md` (already +re-scoped 2026-07-28 to "the recurring cell-grid compile dashboard") is the +workspace half of child prompt 2 and is **absorbed** into it — see its header. + +## Acceptance (arc) + +- `pyauto-brain profiling --axis compile` answers all three modes. +- The compile axis never compares rows across the comparability key. +- `AGENTS.md` moves compile-time profiling from **Future modes** to **Modes**. +- A cache regression (warm compile reverting toward cold) is *detected*, which is + the surveillance function the whole arc exists for. + +--- + +## What the prompt originally said + +
+Original text, filed 2026-07-14 (superseded — kept for provenance) Extend the **PyAutoBrain Profiling Agent** (`agents/conductors/profiling/`, workspace `autolens_profiling`) scope to track **JAX compile-time and eval-time** of the @@ -36,4 +178,10 @@ release-validation JAX scripts) + the ranked report driving a hygiene/perf backl Large — expect to phase at start_dev time. Cross-ref the mode=release timeout/scope policy question on PyAutoHeart#72. +
+ +