Correction: the jax-compile-stall is not a compile stall - #279
Merged
Conversation
The two 1800s runs dumped faulthandler tracebacks at 1440s. Both repos, both scripts, the same stack: parked in jax.block_until_ready / try_to_block — the EXECUTION half of the first call. func(*args, **kwargs) returned; what never returns is the wait for the result to materialize. So the epic's name, and every marker calling this an intermittent XLA compile stall, inherit a guess made before there was evidence. Corrected in the record, the epic entry, the ledger and the follow-up task so the smoke speed-up work does not restart from the wrong question. Also recorded: the heartbeat says 'still compiling' while the process sits in block_until_ready — log_on_first_compile cannot tell which half it is in. Fix the wording before anyone reads another of these logs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K
The refresh check on PR #279 was right: my PyAutoBrain checkout was one commit behind main (#260, which changed the board logo markup), so my render differed from the one CI produces. Regenerated against main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #278. The two 1800s re-timing runs that were still in flight at close-out finished, and the
faulthandlerstack they captured contradicts the premise the whole epic was filed under.The evidence
Both runs dumped at 1440s, exactly as PyAutoFit#1518 intended. Two different repos, two different scripts, the same stack:
jax_compile.py:264is thejax.block_until_ready(result)call — the second half of the wrapped first call. The process is not tracing, lowering or compiling.func(*args, **kwargs)returned. What never returns is the wait for the result to materialize.What this changes
epics.md, the campaign ledger, and the follow-up smoke-timing task, so the speed-up work doesn't restart from the wrong question.log_on_first_compile, worth fixing before anyone reads another of these logs: the heartbeat printsJAX jit still compiling ... 1770s elapsedwhile the process sits inblock_until_ready. The wrapper can't tell which half it's in and says "compiling" regardless. That wording actively reinforced the wrong diagnosis for the full 1770 seconds.AMBIGUOUSeven at 1800s — 2/2 capped on both legs. With the 300s round that's 20 consecutive cap hits and zero completions for each, whilemulti_dataset/mge.py's own marker records it finishing in 32s standalone.Where a resumption should start
Not "why is XLA slow to compile" but "why does
block_until_readynever return". The record lists three candidates: a device-transfer / async-dispatch hang (try_to_blockis exactly where a never-arriving buffer parks); the compilation cache, still live but for a different reason (a cache read satisfying the compile instantly and leaving execution to hang fits what's seen); and thevmap(jit)ordering result reinterpreted — the ordering changes the shape of the executed computation, which is a more plausible route to an execution hang than to a compile one.Note
The epic stays closed as partial — this sharpens the diagnosis, it does not root-cause it, and nothing is un-quarantined. Branch restarted from
mainafter #278 merged, per the merged-PR rule.lifecycle.py check: OK. Index and dashboard regenerated.Generated by Claude Code