Skip to content

Correction: the jax-compile-stall is not a compile stall - #279

Merged
Jammy2211 merged 2 commits into
mainfrom
claude/jax-vmap-jit-stall-swz2tc
Aug 23, 2026
Merged

Correction: the jax-compile-stall is not a compile stall#279
Jammy2211 merged 2 commits into
mainfrom
claude/jax-vmap-jit-stall-swz2tc

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Follow-up to #278. The two 1800s re-timing runs that were still in flight at close-out finished, and the faulthandler stack they captured contradicts the premise the whole epic was filed under.

The evidence

Both runs dumped at 1440s, exactly as PyAutoFit#1518 intended. Two different repos, two different scripts, the same stack:

Thread 0x00007fbf1f0e4b80 (most recent call first):
  File ".../jax/_src/api.py", line 2764 in try_to_block
  File ".../jax/_src/api.py", line 2781 in block_until_ready
  File ".../autofit/non_linear/jax_compile.py", line 264 in wrapper
  File ".../scripts/imaging/jax_likelihood/mge_group.py", line 175 in <module>

jax_compile.py:264 is the jax.block_until_ready(result) call — the second half of the wrapped first call. The process is not tracing, lowering or compiling. func(*args, **kwargs) returned. What never returns is the wait for the result to materialize.

What this changes

  • Every marker, issue and prompt in this epic calls it an "intermittent XLA compile stall". That name is wrong, and has been since the first quarantine on 2026-08-01 — it was a guess made before any evidence existed. Corrected in the record, epics.md, the campaign ledger, and the follow-up smoke-timing task, so the speed-up work doesn't restart from the wrong question.
  • A defect in log_on_first_compile, worth fixing before anyone reads another of these logs: the heartbeat prints JAX jit still compiling ... 1770s elapsed while the process sits in block_until_ready. The wrapper can't tell which half it's in and says "compiling" regardless. That wording actively reinforced the wrong diagnosis for the full 1770 seconds.
  • Both entries are AMBIGUOUS even at 1800s — 2/2 capped on both legs. With the 300s round that's 20 consecutive cap hits and zero completions for each, while multi_dataset/mge.py's own marker records it finishing in 32s standalone.

Where a resumption should start

Not "why is XLA slow to compile" but "why does block_until_ready never return". The record lists three candidates: a device-transfer / async-dispatch hang (try_to_block is exactly where a never-arriving buffer parks); the compilation cache, still live but for a different reason (a cache read satisfying the compile instantly and leaving execution to hang fits what's seen); and the vmap(jit) ordering result reinterpreted — the ordering changes the shape of the executed computation, which is a more plausible route to an execution hang than to a compile one.

Note

The epic stays closed as partial — this sharpens the diagnosis, it does not root-cause it, and nothing is un-quarantined. Branch restarted from main after #278 merged, per the merged-PR rule.

lifecycle.py check: OK. Index and dashboard regenerated.


Generated by Claude Code

James Nightingale and others added 2 commits August 23, 2026 23:09
The two 1800s runs dumped faulthandler tracebacks at 1440s. Both repos, both
scripts, the same stack: parked in jax.block_until_ready / try_to_block — the
EXECUTION half of the first call. func(*args, **kwargs) returned; what never
returns is the wait for the result to materialize.

So the epic's name, and every marker calling this an intermittent XLA compile
stall, inherit a guess made before there was evidence. Corrected in the record,
the epic entry, the ledger and the follow-up task so the smoke speed-up work
does not restart from the wrong question.

Also recorded: the heartbeat says 'still compiling' while the process sits in
block_until_ready — log_on_first_compile cannot tell which half it is in. Fix
the wording before anyone reads another of these logs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K
The refresh check on PR #279 was right: my PyAutoBrain checkout was one commit
behind main (#260, which changed the board logo markup), so my render differed
from the one CI produces. Regenerated against main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K
@Jammy2211
Jammy2211 merged commit 9c1aafa into main Aug 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant