diff --git a/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md b/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md index 23e85561..b52b7b21 100644 --- a/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md +++ b/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md @@ -175,3 +175,68 @@ imply the API one is resolved. chasing speedups on scripts that are hanging. - No script is un-quarantined by this phase — restoring coverage is phase 3, and depends on the fix. + +## The stack, captured after close-out — it is NOT a compile stall + +The two 1800s runs named above finished at 22:38 and both dumped a +`faulthandler` traceback at 1440s, exactly as PyAutoFit#1518 intended. **The +stack contradicts the premise this whole epic was filed under.** + +`autogalaxy_workspace_test` run 32668061785, `imaging/jax_likelihood/mge_group.py`: + +``` +Thread 0x00007fbf1f0e4b80 (most recent call first): + File ".../jax/_src/api.py", line 2764 in try_to_block + File ".../jax/_src/api.py", line 2781 in block_until_ready + File ".../autofit/non_linear/jax_compile.py", line 264 in wrapper + File ".../scripts/imaging/jax_likelihood/mge_group.py", line 175 in +``` + +`autolens_workspace_test` run 32668067325, `multi_dataset/jax_likelihood/mge.py` +— different repo, different script, **same stack**: + +``` + File ".../jax/_src/api.py", line 2781 in block_until_ready + File ".../autofit/non_linear/jax_compile.py", line 264 in wrapper + File ".../scripts/multi_dataset/jax_likelihood/mge.py", line 150 in +``` + +`jax_compile.py:264` is the `jax.block_until_ready(result)` call — the **second** +half of the wrapped first call. The process is not tracing, lowering or +compiling. `func(*args, **kwargs)` **returned**; what never returns is the wait +for the result to materialize. + +### What this overturns + +- Every marker, issue and prompt in this epic calls it an *"intermittent XLA + compile stall"*. On this evidence that name is **wrong**, and it has been + wrong since the first quarantine on 2026-08-01. Anyone resuming should treat + "compile stall" as a label inherited from a guess, not a finding. +- The heartbeat line reads `JAX jit still compiling ... 1770s elapsed` while the + process sits in `block_until_ready`. That wording is a defect in + `log_on_first_compile` — the wrapper cannot tell which half it is in and says + "compiling" regardless. **Fix the message to name the phase before anyone + reads another one of these logs.** +- Both entries are `AMBIGUOUS` even at the 1800s cap: 2/2 runs capped on both + legs, no completion. Combined with the 300s round, that is **20 consecutive + cap hits** for `mge_group.py` and `multi_dataset/mge.py` with zero + completions — while `mge.py`'s own marker records it finishing in 32s + standalone. + +### Where this points instead + +An execution/materialization hang, not a compiler one. The hypotheses worth +testing are now different from the ones this epic queued up: + +1. **Device transfer or the async dispatch queue deadlocking** on the runner — + `try_to_block` is where a never-arriving buffer would park. +2. **The persistent compilation cache** is still live but for a different + reason than assumed: a cache *read* satisfying the compile instantly and + leaving execution to hang is consistent with what is seen here. +3. **The `vmap(jit)` ordering** result (80% -> 30%, p=0.070) needs reinterpreting + in this light: the ordering changes the *shape of the executed computation*, + not just the compile graph, which is a more plausible route to an execution + hang than to a compile one. + +Do not restart from "why is XLA slow to compile". Start from "why does +`block_until_ready` never return". diff --git a/draft/bug/ci/jax_vmap_jit_compile_stall.md b/draft/bug/ci/jax_vmap_jit_compile_stall.md index 8b99dc02..05856777 100644 --- a/draft/bug/ci/jax_vmap_jit_compile_stall.md +++ b/draft/bug/ci/jax_vmap_jit_compile_stall.md @@ -137,6 +137,15 @@ what made this diagnosable at all. - The SLOW-vs-stall question in step 1 answered in writing, so the Profiling Agent is not chasing speedups on scripts that are hanging. +## CORRECTION — the stack says this is NOT a compile stall + +Captured after close-out, from both test workspaces independently: the stalled +process is parked in `jax.block_until_ready` / `try_to_block`, i.e. the +**execution** half of the first call, not tracing/lowering/compiling. The title +of this file, and every marker calling this an "intermittent XLA compile +stall", inherit a guess made before there was any evidence. See the record's +final section. + ## CLOSED AS PARTIAL — 2026-08-23 Phase 1 shipped. Phases 2 and 3 were taken to a deliberate stopping point and the diff --git a/draft/research/ci/smoke_timing_and_profiling.md b/draft/research/ci/smoke_timing_and_profiling.md index 66be9353..b64d07e6 100644 --- a/draft/research/ci/smoke_timing_and_profiling.md +++ b/draft/research/ci/smoke_timing_and_profiling.md @@ -37,6 +37,14 @@ The headline for this task: **a SLOW marker is not evidence of slowness.** Every 2026-07-14 marker reads "flakes at the 1800s cap" and records no timing at all; the one entry measured so far was wrong by a factor of ~50. +## Before you start: the "compile stall" is a misnomer + +The stalled entries above were captured mid-hang on 2026-08-23 and the stack +puts them in `jax.block_until_ready`, not in compilation. If any of this work +touches those entries, do not carry the "XLA compile stall" framing over — it +predates the evidence. Detail in +[`../../../complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md`](../../../complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md). + ## The harness exists `.github/workflows/retime.yml` + `.github/scripts/retime.py` in both test diff --git a/epics.md b/epics.md index b23d86f6..8897aaff 100644 --- a/epics.md +++ b/epics.md @@ -32,6 +32,9 @@ epic, never picked standalone. - title: Intermittent XLA compile stall in the JAX vmap likelihood path - ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md - status: CLOSED AS PARTIAL 2026-08-23 — record complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md +- CORRECTION (post-close-out): the captured stack shows the hang is in jax.block_until_ready, NOT in + compilation. The epic's name and every marker calling this an "XLA compile stall" are wrong. Resume + from "why does block_until_ready never return", not from compiler behaviour. - notes: phase 1 (watchdog) shipped in full; phases 2/3 stopped deliberately at a measured-but-not-root-caused state. The stall is instrumented and characterised (>100x bimodality inside one compile step; vmap-of-jit contributory at p=0.070 but NOT causal; the compile-cache hypothesis never tested) and NOTHING was un-quarantined. Resume via draft/research/ci/smoke_timing_and_profiling.md, which is where this gets dug up. Superseded draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. ## graphical-ep