mind: the compile-axis arc — re-scoped, issued, shipped, recorded - #177
Merged
Conversation
The 2026-07-14 prompt asked the Profiling Agent to measure JAX compile/eval time of the release-validation heavy scripts so we could speed them up. Both halves of that framing are spent, so it is re-scoped rather than started as written. The speed-up shipped: the #71/#74/#77 arc landed the persistent compilation cache (117.0s -> 2.3s CPU MGE vag; 5517.8s -> 937.1s A100 end-to-end) and --xla_gpu_autotune_level=0 (17x on the pathological cold probe, -40% on the full fit, bit-identical likelihoods), and ruled out source jit-boundaries by evidence. The prompt's candidate root causes are resolved with them. What survives is regression surveillance. Both wins are settings, so a config drift or an XLA_FLAGS clobber puts the 70-minute worst case back silently -- exactly the PyAutoNerves#127 failure that went undetected for two months. Nothing watches for that: profiling/AGENTS.md still lists compile-time profiling under Future modes, and _profiling.py has no occurrence of "compile". The release-validation-scripts leg is dropped. Script-suite cost is the hygiene conductor's perf mode by both agents' documented boundaries, having been moved out of profiling's staged future modes once already. Three blockers, measured against the 93 committed probe records: - coverage is 3-4 of the 24 runtime CELLS grid cells, hst only, with no interferometer, datacube, jwst or ao row - warm rows are not machine-identifiable -- warmness lives only in a free-text tag (~40 ad-hoc values), and cache_dir is non-empty on cold rows too - compile rows have no pin, so there is no baseline to regress against Split accordingly into campaign coverage (Brain), warm baselines + dashboard (workspace + Brain), and drift triage (Brain). The workspace-side jax_compile_time_profiling.md is absorbed into phase 2, since the dashboard and the agent reading it are two halves of one thing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
Phase 1 of the compile-axis arc is issued as PyAutoBrain#218. The prompt advances draft/ -> active/ and registers in active.md as library-dev. No worktree claim: this is a Claude-Code-on-the-web session with no ~/Code/PyAutoLabs-wt/ to create, so the entry records the branch on one line rather than as a 2-space repo bullet, which worktree_check_conflict would read as a live worktree claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
The re-scope estimated "3 of 24 grid cells" by hand. Phase 1 computed it: the sweep CELLS grid is 21 cells, and only imaging/mge/hst and imaging/pixelization/hst are on-grid -- 11 of 147 cell x transform runs on the local tier, 3 on a100. knn and delaunay_matern are off-grid mesh variants rather than partial coverage, which the estimate had folded in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
Phase 2 is issued as autolens_profiling#103 with the schema leg in PR #104. Registered as PARTIAL rather than in-flight: pins and dashboard rows remain, and the Brain leg is blocked on PyAutoBrain#219 merging. Also records a CI gap found while working: autolens_profiling runs no pytest step, so scripts/misc/test/ never runs in CI and a vram_config test is failing on main unnoticed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
Phase 2 is no longer partial. Records the merge order (PyAutoBrain#219 before the stacked #220) and the three corrections that only surfaced by running the tools against the real corpus rather than reasoning about the design. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
Phase 3 is issued and built; PR #222 closes the compile-axis arc. Records the three-PR merge order and the one deviation from the prompt: two of its five classifications cannot reach triage by construction, since a jax_version bump or a changed host is a different comparability key and ingest reports those as unpinned rather than drifted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
The Profiling Agent now has a compile-time axis across campaign, ingest and triage, and AGENTS.md no longer lists it under Future modes. Merged: PyAutoBrain#219 (cd79005), autolens_profiling#104 (355d555), PyAutoBrain#220 (74d7b1b), PyAutoBrain#222 (a50efc3). Verified end to end on merged main against the merged workspace: campaign 11/147 covered, ingest 25 pins / 0 drifted / 0 unpinned, triage 0 findings; Brain suite 329 passed, workspace 47 passed in CI. Records carry the traps worth remembering, all of which surfaced by running the tools against real data rather than from the design: - cache_dir is non-empty on COLD rows too, so warmness cannot be read from it or from the tag - `hardware` alone pooled a laptop and a 32-core RAL node under one label; hostname had to join the comparability key - pins auto-following the newest row would have baked a regression in permanently, reporting all-clear forever - rows predating a pin are history, not drift - export_probe.py / trace_profile.py share the results tree with a different schema, so their records are not malformed Also records the two corrections to my own earlier claims: the grid is 21 cells rather than 24, and the workspace test that looked red outside CI was a missing matplotlib, not a defect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Mind side of the compile-axis arc, from re-scope to completion records. All
four code PRs are already merged; this is the ledger catching up.
cd79005(#219)campaign --axis compile355d555(#104)cache_state, pins, dashboard, + pytest in CI74d7b1b(#220)ingest --axis compilea50efc3(#222)triage --axis compile, arc closedThe re-scope
draft/feature/profiling/profiling_agent_jax_compile_time_scope.md(2026-07-14) askedthe Profiling Agent to measure compile time of the release-validation heavy scripts so
we could speed them up. Both halves of that framing were spent:
(117.0 s → 2.3 s CPU; 5517.8 s → 937.1 s A100) and
--xla_gpu_autotune_level=0(17×on the pathological cold probe), and ruled out source jit-boundaries by evidence.
hygiene's
perfmode by both agents' documented boundaries — it had been moved out ofprofiling's staged future modes once already.
What survived is regression surveillance: both wins are settings, so a config
drift or an
XLA_FLAGSclobber (the PyAutoNerves#127 failure that went undetected fortwo months) puts the 70-minute worst case back with nothing failing. Split into three
sequential phases; the workspace-side
jax_compile_time_profiling.mdwas absorbed intophase 2, since the dashboard and the agent reading it are two halves of one thing.
What this PR contains
<details>for provenance)complete/2026/08/complete/index.mdregenerated (966 records)jax_compile_time_profiling.mdmarked absorbedactive.mdis back to just the two release-drive entries. The three phase prompts moveddraft/ → active/ → complete/through the lifecycle rather than jumping states.The traps the records preserve
Every one surfaced by running the tools against real data, not from the design:
cache_diris non-empty on COLD rows too — the cold run is the one that populatesthe cache, so warmness cannot be read from it or from the tag (~40 ad-hoc spellings).
hardwarealone pooled two machines — onelocal_cpulabel spanned a laptop (66records) and a 32-core RAL node (12). The first rendered dashboard put pins from both
side by side under one heading.
hostnamejoined the comparability key.reporting all-clear forever — exactly inverted from the tool's purpose.
ingestrun flagged fourmeasurements the pin had been chosen over.
export_probe.py/trace_profile.pyshare the results tree with a differentschema, so their 4 records are sibling-instrument output, not corruption.
Two corrections to my own earlier claims, recorded rather than quietly fixed
imaging/mge/hstandimaging/pixelization/hstare on-grid.matplotlib(
aggregate.pyimports it at module scope), not a defect. It passes in CI — nowprovably, since
lint.ymlgained the pytest step it never had.Verification
lifecycle.py check: OK· 127 Mind tests pass · all three modes verified end-to-end onmerged
mainagainst the merged workspace (campaign 11/147, ingest 25 pins / 0 drifted /0 unpinned, triage 0 findings).
Generated by Claude Code