jax-compile-stall epic: shipped phase 1, closed the rest as partial

[Jammy2211]

The Mind side of the jax-compile-stall epic: the epic registration, four prompt files, the phase-1 completion record, the epic close-out record, and the follow-up task that inherits the unfinished work.

Closed as partial by design (James, 2026-08-23): the stall is instrumented and characterised but not root-caused, and nothing was un-quarantined. The remaining breadth moves to a dedicated smoke timing/profiling task rather than being ground out one dispatch at a time.

What merged elsewhere

PR
PyAutoFit#1517	heartbeat + faulthandler watchdog + compile/execute timing split
PyAutoFit#1518	fixes #1517 — the dump threshold now sits below the runner's cap
PyAutoHeart#161	runner / runner-args / script-timeout inputs on the reusable smoke workflow
autolens_workspace_test#272, autogalaxy_workspace_test#110	the retime.py harness + workflow_dispatch caller

Issues PyAutoFit#1516 and autolens_workspace_test#271 both closed.

What this PR adds

• epics.md — the jax-compile-stall entry, status CLOSED AS PARTIAL
• draft/bug/ci/jax_vmap_jit_compile_stall.md — the campaign ledger, carrying the original filing plus the phase map and outcome banner. It arrived on an unmerged branch (claude/backport-per-script-timeout-r3w1sv) and is brought onto main here
• Three phase prompts under draft/bug/ci/ — phase 1 folded into its record; phases 2 and 3 retained
• complete/2026/08/jax-compile-stall-evidence.md — phase 1's record
• complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md — the epic close-out: 60 measured executions, both live leads, the traps, and what was deliberately left undone
• draft/research/ci/smoke_timing_and_profiling.md — the follow-up task, carrying the four measured verdicts forward so nothing is re-measured
• Supersede banner on draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md — the same defect, filed 2026-08-22, never issued

The findings worth keeping

• A SLOW marker is not evidence of slowness. Every 2026-07-14 marker reads "flakes at the 1800s cap" and records no timing at all. The first one measured — interferometer/datacube/shared_preloads.py — completed 10/10 runs at a worst case of 34.0s, 1.9% of the cap it supposedly flakes at.
• The stall is a >100× bimodality inside one step. A healthy compile of rectangular_mge.py is 3.1s; a stalled one exceeds 300s, same commit and runner image.
• vmap(jit) is contributory, not causal. A/B: 8/10 stalls on main vs 3/10 swapped, Fisher exact p = 0.070. The stall survives the swap, so the ordering is not necessary for it.
• The compile-cache hypothesis was never tested — the obvious next experiment.
• A watchdog whose threshold equals the cap never fires. #1517's 300s default against a 300s cap produced 20 stalled runs with heartbeats and zero stacks.

Verification

lifecycle.py check: OK. lifecycle.py index --check: OK. dashboard.md / dashboard.html regenerated. active.md is back to its single pre-existing HOLD entry.

One caveat

Two 1800s re-timing runs were still in flight at close-out (ag_test 32668061785, al_test 32668067325) and should carry the first faulthandler stack. The record names them as the first thing to read when this is resumed.

---

Generated by Claude Code