Dump the compile traceback before the runner kills the process

[Jammy2211]

Fixes a defect in #1517, found by the measurement #1517 existed to enable. Part of the jax-compile-stall epic; evidence in autolens_workspace_test#271.

The bug

The faulthandler watchdog shipped in #1517 never fired. Its CI default was a flat 300.0 seconds — and the workspace smoke cap is also 300s. The dump timer and the runner's kill timer therefore raced, and the kill won every time. A SIGKILLed process writes no traceback.

This is not theoretical. The first CI use of the watchdog was a re-timing sweep over four quarantined JAX scripts, 5 repeats × 2 Python versions across two repos. 20 of those runs stalled to the cap. All 20 produced heartbeats. None produced a stack:

21:13:50 ... JAX jit compiling vectorized (vmap) likelihood function...
21:14:20 ... JAX jit still compiling ... 30s elapsed...
   ⋮
21:18:20 ... JAX jit still compiling ... 270s elapsed...
##[error]TIMEOUT after 300s — killed the process group.

So #1517 delivered the half that proves a stalled run is alive, and silently dropped the half that says where it is stuck — which is the half phase 3 of the epic actually needs.

The fix

Derive the CI default from BUILD_SCRIPT_TIMEOUT, the per-script cap that the workspace runners and PyAutoHands both enforce, at 80% of it:

BUILD_SCRIPT_TIMEOUT	Dump at
300 (smoke)	240s
1800 (release)	1440s
unset / 0 / malformed	240s flat fallback

The fallback matters: computing a fraction of an absent cap would yield 0, which disables the dump — the same silent no-op in a new disguise. An explicit PYAUTOFIT_JAX_COMPILE_DUMP_SECS still overrides everything.

Verification

Full suite 2011 passed, 34 skipped. test_jax_compile.py 19 passed, 4 new — including one that pins the invariant directly rather than testing today's numbers:

for cap in ("300", "1800", "60"):
    monkeypatch.setenv("BUILD_SCRIPT_TIMEOUT", cap)
    assert jax_compile.dump_traceback_seconds() < float(cap)

End-to-end, a simulated stall with a 10s cap — the dump now lands at 8s, before the kill at 12s:

21:31:37 ... JAX jit still compiling ... 3s elapsed...
21:31:40 ... JAX jit still compiling ... 6s elapsed...
Timeout (0:00:08)!
  File ".../autofit/non_linear/jax_compile.py", line 254 in wrapper
[exit 137 — SIGKILL]

Same scenario before this change wrote no traceback at all.

Note on the original review

Nothing in #1517's tests could have caught this. They monkeypatched faulthandler and asserted the timer was armed and cancelled — which it was, correctly. The failure lives in the relationship between two independently-correct timeouts owned by different repos, and only a real run under a real cap exposes it. Hence the new test asserting the relationship rather than the value.

---

Generated by Claude Code