From 6159e9b1d1eca44174ff73b9ff57637ff0e3be99 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 19:49:59 +0000 Subject: [PATCH 01/11] prompt: file issue for jax-compile-stall-evidence (#1516) Split the intermittent XLA vmap compile stall into the jax-compile-stall epic: phase 1 (PyAutoFit compile watchdog) is issued as PyAutoFit#1516, phases 2 (SLOW-vs-stall audit) and 3 (root cause + un-quarantine) stay in draft and are issued one at a time. The 2026-08-23 ci filing becomes the campaign ledger. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md (same defect, filed 2026-08-22, never issued); its two unique asks are folded into phases 1 and 3. Records two source-level findings from the start_dev read of PyAutoFit that phase 3 starts from: Fitness._vmap builds jax.vmap(jax.jit(...)), the inverted ordering, on exactly the path that stalls; and log_on_first_compile covers both the compile wait and the block_until_ready execution wait with one log line and no heartbeat, which is why three quarantines produced no diagnosis. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 13 + active/jax_compile_stall_1_evidence.md | 97 ++++++ dashboard.html | 90 ++--- dashboard.md | 325 ++++++++++-------- .../multi_dataset_jax_likelihood_xla_stall.md | 21 +- ...jax_compile_stall_2_slow_vs_stall_audit.md | 71 ++++ .../bug/ci/jax_compile_stall_3_root_cause.md | 82 +++++ draft/bug/ci/jax_vmap_jit_compile_stall.md | 194 +++++++++++ epics.md | 5 + 9 files changed, 716 insertions(+), 182 deletions(-) create mode 100644 active/jax_compile_stall_1_evidence.md create mode 100644 draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md create mode 100644 draft/bug/ci/jax_compile_stall_3_root_cause.md create mode 100644 draft/bug/ci/jax_vmap_jit_compile_stall.md diff --git a/active.md b/active.md index fb25a165..02549f7e 100644 --- a/active.md +++ b/active.md @@ -1,5 +1,18 @@ # Active Tasks +## jax-compile-stall-evidence +- issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1516 (issued 2026-08-23) +- issued: 2026-08-23 +- prompt: active/jax_compile_stall_1_evidence.md +- status: library-dev +- epic: jax-compile-stall (phase 1 of 3; ledger draft/bug/ci/jax_vmap_jit_compile_stall.md) +- worktree: ~/Code/PyAutoLabs-wt/jax-compile-stall-evidence +- worktree-note: started in a web-github session with no local tree — the branch was + cut against a direct clone at /home/user/pyautofit, not the worktree root above. + A local-dev session resuming this should create the worktree via worktree_create. +- repos: + - PyAutoFit: feature/jax-compile-stall-evidence + ## transformed-message-factor-gradient-unpack - issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1501 (issued 2026-08-19) - issued: 2026-08-19 diff --git a/active/jax_compile_stall_1_evidence.md b/active/jax_compile_stall_1_evidence.md new file mode 100644 index 00000000..d2aa0788 --- /dev/null +++ b/active/jax_compile_stall_1_evidence.md @@ -0,0 +1,97 @@ +# Phase 1: make a stalled JAX compile report itself (heartbeat + faulthandler + compile/execute split) + +Type: bug +Target: ci +Repos: +- @PyAutoFit +Difficulty: small +Autonomy: supervised +Priority: high +Status: formalised +Epic: jax-compile-stall +Phase: 1 +Campaign: bug/ci/jax_vmap_jit_compile_stall.md (Phase 1 — the enabler; phases 2 and 3 are blocked on this) +Filed: 2026-08-23 +Issued: 2026-08-23 + +## Why this is phase 1 + +The stall's whole cost is that it produces **no evidence**. The last line any +killed run emits is + +``` +autofit.non_linear.jax_compile - INFO - JAX jit compiling vectorized (vmap) + likelihood function, could take seconds or minutes... +``` + +and then silence until the cap kills it. Three separate quarantines +(`autolens_workspace_test` delaunay #245, `autogalaxy_workspace_test` +`multi_dataset/.../rectangular.py` 2026-08-01, `imaging/.../mge_group.py` +2026-08-23) produced no diagnosis between them, because there was nothing to +diagnose *from*. Phases 2 and 3 of this campaign both consume evidence this +phase creates. + +## What is wrong with the current instrumentation + +`log_on_first_compile(func, description)` in +`autofit/non_linear/jax_compile.py` wraps the `jax.jit` / `jax.vmap` / +`jax.grad` callables so the "this is compiling" line lands where the user +actually waits — on the first call. Inside that first call it does two very +different things under one log line: + +1. `result = func(*args, **kwargs)` — tracing, lowering and XLA compilation; +2. `jax.block_until_ready(result)` — execution, because JAX dispatches + asynchronously. + +Then it logs one `complete in {n} seconds` summary. So a hang anywhere in +either half looks identical from the outside, and a compile that is merely +*slow* looks identical to one that has stopped. Nothing reports liveness in +between. + +## Task + +All of this is library-side in @PyAutoFit. **Do not touch the workspace +scripts** — they are user-facing documentation, and a per-script workaround is +the quarantine pattern this campaign exists to stop. + +1. **Heartbeat.** While the first call is in flight, log + `still compiling {description}, {n}s elapsed` on an interval. Daemon thread, + stopped in the existing `finally` so it can never hold the process open. + Interval from `PYAUTOFIT_JAX_COMPILE_HEARTBEAT_SECS`, default `30`, `0` + disables. +2. **Watchdog.** Arm `faulthandler.dump_traceback_later(secs, repeat=True, + exit=False)` before the first call and `cancel_dump_traceback_later()` in the + `finally`, so a compile that overruns dumps its own traceback to stderr + before anything kills it. Threshold from `PYAUTOFIT_JAX_COMPILE_DUMP_SECS`. +3. **Default it on under CI.** Default the threshold to `300` when the `CI` + environment variable is set and `0` (off) otherwise, both overridable. This + is what makes the *next* CI stall self-diagnosing with no workspace edit and + no runner edit — the alternative, wiring an env var into each workspace's + `config/build/env_vars_*.yaml`, is a second repo touch for the same effect. +4. **Split the timing.** Time `func(...)` and `jax.block_until_ready(result)` + separately and log both, so the record says which half is stuck. Keep the + existing single `complete in {n} seconds` summary line unchanged. + +Applies automatically to all four call sites: `Fitness._vmap`, `Fitness._jit`, +`Fitness._grad` (`autofit/non_linear/fitness.py`) and the batched latent +computation in `autofit/non_linear/analysis/latent.py`. + +## Known limitation, to be stated in the PR + +A Python traceback taken during XLA compilation parks at the pybind boundary — +it will not show XLA internals. It still separates *in compile* from *in +execution* from *blocked on a Python-level lock* (for instance the persistent +compilation cache, `JAX_COMPILATION_CACHE_DIR`, on by default since +PyAutoConf#128). That three-way split is exactly the fork phase 3 needs, so the +limitation does not undermine the phase. + +## Acceptance + +- A stalled first compile emits periodic liveness lines with elapsed time. +- A stalled first compile leaves a traceback behind in CI without any workspace + or runner change. +- The log distinguishes the compile wait from the execution wait. +- Covered by tests in `test_autofit/non_linear/test_jax_compile.py` that need no + JAX import: heartbeat fires, watchdog is armed and cancelled, env defaults + including the `CI` branch. +- No workspace script and no CI runner is modified by this phase. diff --git a/dashboard.html b/dashboard.html index c0316600..f024773e 100644 --- a/dashboard.html +++ b/dashboard.html @@ -51,7 +51,7 @@

📋 PyAutoMind Dashboard

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-

In flight 1 · Parked 3 · Planned 6 · Backlog 153 · markdown version

+

In flight 2 · Parked 3 · Planned 6 · Backlog 156 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

TRIAGE: needs manual review before routing — medium · safe · high

@@ -72,6 +72,7 @@

Quick wins (small enough, and safe enough to run unatte

In flight markdown version

Issued — each has an open GitHub issue and usually a branch.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

+

Phase 1: make a stalled JAX compile report itself (heartbeat — issue #1516 — issued 2026-08-23 — library-dev

Parked markdown version

3 task(s) @@ -90,39 +91,7 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

153 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.

-
-feature — 29 -

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high

-

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high

-

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high

-

Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high

-

Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high

-

Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium

-

Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium

-

Brain board follow-ups: what real mornings surface — pyautobrain · small · supervised · normal

-

Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal

-

Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal

-

Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal

-

The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal

-

The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal

-

Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal

-

EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal

-

Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal

-

Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal

-

Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal

-

Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal

-

PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal

-

autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal

-

Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal

-

Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal

-

Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal

-

dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low

-

Release board: local run_logs enrichment — pyautohands · small · supervised · low

-

Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low

-

Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low

-

Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low

-
+

156 filed prompts, not started — sorted most-pickable first (priority, then size). 26 of them belong to an epic and are listed only under Epics below.

bug — 34

Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high

@@ -161,6 +130,38 @@

Backlog

interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens

+feature — 29 +

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high

+

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high

+

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high

+

Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high

+

Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high

+

Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium

+

Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium

+

Brain board follow-ups: what real mornings surface — pyautobrain · small · supervised · normal

+

Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal

+

Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal

+

Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal

+

The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal

+

The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal

+

Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal

+

EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal

+

Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal

+

Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal

+

Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal

+

Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal

+

PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal

+

autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal

+

Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal

+

Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal

+

Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal

+

dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low

+

Release board: local run_logs enrichment — pyautohands · small · supervised · low

+

Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low

+

Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low

+

Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low

+
+
research — 17

Deep research: Can we speed up Delaunay in PyAutoArray? — autoarray · too-large · supervised · high

Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high

@@ -253,6 +254,12 @@

Backlog 2026-08-23 +issued +Phase 1: make a stalled JAX compile report itself (heartbeat + + + +2026-08-23 filed Brain board follow-ups: what real mornings surface @@ -305,7 +312,7 @@

Backlog Defer the eager scipy.sparse import in derivative_util (~0.10 s of - + 2026-08-22 filed Bug: fix the tracer.fits existence guard in autolens_workspace… @@ -545,12 +552,6 @@

Backlog Nightly release has been blocked 8 nights running — triage - -2026-08-04 -filed -HowToLens ch4 tutorial 3: mask overlay is never actually drawn - -

Epics markdown version

@@ -575,6 +576,13 @@

Epics

Source & Cluster arc — magnification science, PointSolver trust, cluster — autolens · too-large · supervised · high

+Intermittent XLA compile stall in the JAX vmap likelihood path — 3 queued prompt(s), in order +

Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md

+

Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — ci · medium · supervised · high

+

Phase 3: root-cause the XLA vmap compile stall and clear — ci · large · supervised · high

+

Intermittent XLA compile stall in JAX vmap likelihood scripts — — ci · too-large · supervised · high

+
+
Expectation propagation (EP) campaign — 9 queued prompt(s), in order

Expectation propagation (EP) campaign — ledger: draft/research/graphical_ep/ep_campaign.md

Analytic Gaussian benchmark: closed-form validation of graphical + EP — graphical_ep · medium · supervised · high

diff --git a/dashboard.md b/dashboard.md index 57704fe0..18e865bb 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,10 +8,10 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 1 | +| [In flight](#in-flight) (`active/`) | 2 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | -| [Backlog](#backlog) (`draft/`) | 153 | +| [Backlog](#backlog) (`draft/`) | 156 | ## Start here @@ -143,6 +143,14 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
+
📋 Phase 1: make a stalled JAX compile report itself (heartbeat — issue #1516 — issued 2026-08-23 — library-dev + +``` +/start_dev active/jax_compile_stall_1_evidence.md +``` + +
+ ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -235,516 +243,516 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**153** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **23** of them belong to an epic and are listed only under [Epics](#epics) below. +**156** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **26** of them belong to an epic and are listed only under [Epics](#epics) below.
-feature — 29 +bug — 34 -
📋 Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high +
📋 Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high ``` -/start_dev draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md +/start_dev draft/bug/health_fixes/jax_runtime_and_parity.md ```
-
📋 Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high +
📋 Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high ``` -/start_dev draft/feature/autoarray/rectangular_bilinear_rtu_mesh_split.md +/start_dev draft/bug/health_fixes/jit_visualization_outputs.md ```
-
📋 Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high +
📋 Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high ``` -/start_dev draft/feature/autoarray/numba_cpu_likelihood_kernel_cdf_fast_path.md +/start_dev draft/bug/health_fixes/samples_parameter_paths.md ```
-
📋 Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high +
📋 pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium ``` -/start_dev draft/feature/autolens_profiling/jax_compile_time_profiling.md +/start_dev draft/bug/autoarray/pixel_scales_int_not_widened_to_tuple.md ```
-
📋 Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high +
📋 jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium ``` -/start_dev draft/feature/profiling/profiling_agent_jax_compile_time_scope.md +/start_dev draft/bug/autofit/jax_011_message_log_partition_tuple_shape.md ```
-
📋 Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium +
📋 Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium ``` -/start_dev draft/feature/autofit/prior_exit_sampler_coverage.md +/start_dev draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md ```
-
📋 Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium +
📋 Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium ``` -/start_dev draft/feature/autofit/search_seed_reproducibility.md +/start_dev draft/bug/autoarray/numba_kernel_shift_axes_swapped.md ```
-
📋 Brain board follow-ups: what real mornings surface — pyautobrain · small · supervised · normal +
📋 PyNUFFT dev extra is incompatible with current SciPy on Python — autoarray · small · supervised · normal ``` -/start_dev draft/feature/pyautobrain/brain_board_follow_ups.md +/start_dev draft/bug/autoarray/pynufft_scipy_pinv2_dev_extra.md ```
-
📋 Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal +
📋 LogGaussianPrior misreports its own support as (-inf, inf) — autofit · small · supervised · normal ``` -/start_dev draft/feature/autoarray/multiwavelength_inversion.md +/start_dev draft/bug/autofit/loggaussian_prior_declares_own_support.md ```
-
📋 Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal +
📋 autofit.plot functions accept **kwargs and silently discard them — autofit · small · supervised · normal ``` -/start_dev draft/feature/autolens_workspace/joss_cluster_benchmark_tuning.md +/start_dev draft/bug/autofit/plot_functions_discard_kwargs.md ```
-
📋 Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal +
📋 TEST_MODE bypass crashes on ordered-parameter assertion ties — autofit · small · supervised · normal ``` -/start_dev draft/feature/pyautomind/complete_archive_wiki.md +/start_dev draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md ```
-
📋 The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal +
📋 point.py JAX-vmap parity assert is non-deterministic under the smoke env — autolens · small · supervised · normal ``` -/start_dev draft/feature/workspaces/group_los_halos.md +/start_dev draft/bug/autolens/point_jax_vmap_parity_nondeterministic.md ```
-
📋 The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal +
📋 Scripts derive geometry from a hardcoded pixel_scale while the dataset — autolens_workspace · small · supervised · normal ``` -/start_dev draft/feature/workspaces/group_subhalo_sensitivity.md +/start_dev draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md ```
-
📋 Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal +
📋 HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX) — howtogalaxy · small · supervised · normal ``` -/start_dev draft/feature/autoarray/arcsecond_to_decimal.md +/start_dev draft/bug/howtogalaxy/small_api_drift_ellipse_and_plot_grid_lines.md ```
-
📋 EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal +
📋 PyAutoConf rename leftovers in Brain functional surfaces — pyautobrain · small · supervised · normal ``` -/start_dev draft/feature/autofit/ep_analytic_updates.md +/start_dev draft/bug/pyautobrain/pyautoconf_rename_functional_leftovers.md ```
-
📋 Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal +
📋 generate.py deletes notebooks/ before rejecting an unknown project — pyautohands · small · supervised · normal ``` -/start_dev draft/feature/autofit_assistant/remote_mcp_deployment_tiers.md +/start_dev draft/bug/pyautohands/generate_rejects_autocti_after_deleting_notebooks.md ```
-
📋 Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal +
📋 autoreduce 0.9 on PyPI never got the Python 3.12 floor — pyautoreduce · small · supervised · normal ``` -/start_dev draft/feature/autolens_profiling/search_settings_estimation_infrastructure.md +/start_dev draft/bug/pyautoreduce/published_autoreduce_09_missing_312_floor.md ```
-
📋 Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal +
📋 aplt.Output stale-API drift in the remaining workspace repos — workspaces · small · supervised · normal ``` -/start_dev draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md +/start_dev draft/bug/workspaces/aplt_output_drift_remaining_repos.md ```
-
📋 Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal +
📋 Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py — workspaces · small · safe · normal ``` -/start_dev draft/feature/autoarray/rectangular_multi_submesh.md +/start_dev draft/bug/workspaces/bug_fix_the_tracer_fits_existence_guard.md ```
-
📋 PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal +
📋 Three jax_likelihood pins are stale by ~1.24e-4 and fail the — workspaces · small · supervised · normal ``` -/start_dev draft/feature/autogalaxy/piemass_potential.md +/start_dev draft/bug/workspaces/jax_likelihood_pins_stale_by_1e4.md ```
-
📋 autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal +
📋 PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional? — autoarray · medium · supervised · normal ``` -/start_dev draft/feature/autolens_jax_joss/autolens_jax_joss_benchmark_repo.md +/start_dev draft/bug/autoarray/PROBE_adapt_double_square_coefficient.md ```
-
📋 Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal +
📋 interferometer Delaunay pixelization — non-PD FitException in test-mode bypass — autolens · medium · supervised · normal ``` -/start_dev draft/feature/jax_substructure/5_prng_key_vmap_noise.md +/start_dev draft/bug/autolens/interferometer_delaunay_nonpd_fitexception.md ```
-
📋 Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal +
📋 JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38 — autolens · medium · supervised · normal ``` -/start_dev draft/feature/jax_substructure/6_deflection_equivalence_test.md +/start_dev draft/bug/autolens/jax_point_source_point_smoke_sentinel.md ```
-
📋 Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal +
📋 JIT cache not hit in modeling_visualization delaunay/rectangular scripts — autolens · medium · supervised · normal ``` -/start_dev draft/feature/workspaces/restore_multiple_sources_lensing_of_lens.md +/start_dev draft/bug/autolens/jit_cache_not_hit_modeling_visualization.md ```
-
📋 dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low +
📋 multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall) — autolens_workspace_test · medium · supervised · normal ``` -/start_dev draft/feature/autogalaxy/dpie_sigma0_parameterization.md +/start_dev draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md ```
-
📋 Release board: local run_logs enrichment — pyautohands · small · supervised · low +
📋 @PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian — priors · medium · supervised · normal ``` -/start_dev draft/feature/pyautohands/release_board_run_logs_enrichment.md +/start_dev draft/bug/priors/15_transformed_message_logpdf_jacobian.md ```
-
📋 Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low +
📋 Resolve release-profile timeout scripts deliberately — health_fixes · too-large · supervised · normal ``` -/start_dev draft/feature/workspaces/gallery_runner_missing_tiers.md +/start_dev draft/bug/health_fixes/release_timeout_policy.md ```
-
📋 Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low +
📋 @PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms — priors · too-large · supervised · normal ``` -/start_dev draft/feature/autonomy/10_scheduled_runs.md +/start_dev draft/bug/priors/14_replace_transform_stack_with_bijectors.md ```
-
📋 Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low +
📋 Priors & Messages cleanup — tracker — priors · too-large · supervised · normal ``` -/start_dev draft/feature/pyautomind/repos-sync-config-stamper.md +/start_dev draft/bug/priors/z_features.md ```
-
- -
-bug — 34 - -
📋 Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high +
📋 autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main — autolens · low · supervised · normal ``` -/start_dev draft/bug/health_fixes/jax_runtime_and_parity.md +/start_dev draft/bug/autolens/jax_likelihood_smoke_pins_stale.md ```
-
📋 Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high +
📋 status.sh --repos sources a file that no longer exists — pyautomind · small · supervised · low ``` -/start_dev draft/bug/health_fixes/jit_visualization_outputs.md +/start_dev draft/bug/pyautomind/status_sh_repos_missing_source.md ```
-
📋 Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high +
📋 The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · low ``` -/start_dev draft/bug/health_fixes/samples_parameter_paths.md +/start_dev draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md ```
-
📋 pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium +
📋 Point-source JSON datasets record no resolution regime — pyautolens · medium · supervised · low ``` -/start_dev draft/bug/autoarray/pixel_scales_int_not_widened_to_tuple.md +/start_dev draft/bug/pyautolens/point_source_json_datasets_record_no_regime.md ```
-
📋 jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium +
📋 interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens ``` -/start_dev draft/bug/autofit/jax_011_message_log_partition_tuple_shape.md +/start_dev draft/bug/autolens/interferometer_release_leg_oom.md ```
-
📋 Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium +
+ +
+feature — 29 + +
📋 Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high ``` -/start_dev draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md +/start_dev draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md ```
-
📋 Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium +
📋 Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high ``` -/start_dev draft/bug/autoarray/numba_kernel_shift_axes_swapped.md +/start_dev draft/feature/autoarray/rectangular_bilinear_rtu_mesh_split.md ```
-
📋 PyNUFFT dev extra is incompatible with current SciPy on Python — autoarray · small · supervised · normal +
📋 Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high ``` -/start_dev draft/bug/autoarray/pynufft_scipy_pinv2_dev_extra.md +/start_dev draft/feature/autoarray/numba_cpu_likelihood_kernel_cdf_fast_path.md ```
-
📋 LogGaussianPrior misreports its own support as (-inf, inf) — autofit · small · supervised · normal +
📋 Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high ``` -/start_dev draft/bug/autofit/loggaussian_prior_declares_own_support.md +/start_dev draft/feature/autolens_profiling/jax_compile_time_profiling.md ```
-
📋 autofit.plot functions accept **kwargs and silently discard them — autofit · small · supervised · normal +
📋 Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high ``` -/start_dev draft/bug/autofit/plot_functions_discard_kwargs.md +/start_dev draft/feature/profiling/profiling_agent_jax_compile_time_scope.md ```
-
📋 TEST_MODE bypass crashes on ordered-parameter assertion ties — autofit · small · supervised · normal +
📋 Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium ``` -/start_dev draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md +/start_dev draft/feature/autofit/prior_exit_sampler_coverage.md ```
-
📋 point.py JAX-vmap parity assert is non-deterministic under the smoke env — autolens · small · supervised · normal +
📋 Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium ``` -/start_dev draft/bug/autolens/point_jax_vmap_parity_nondeterministic.md +/start_dev draft/feature/autofit/search_seed_reproducibility.md ```
-
📋 Scripts derive geometry from a hardcoded pixel_scale while the dataset — autolens_workspace · small · supervised · normal +
📋 Brain board follow-ups: what real mornings surface — pyautobrain · small · supervised · normal ``` -/start_dev draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md +/start_dev draft/feature/pyautobrain/brain_board_follow_ups.md ```
-
📋 HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX) — howtogalaxy · small · supervised · normal +
📋 Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal ``` -/start_dev draft/bug/howtogalaxy/small_api_drift_ellipse_and_plot_grid_lines.md +/start_dev draft/feature/autoarray/multiwavelength_inversion.md ```
-
📋 PyAutoConf rename leftovers in Brain functional surfaces — pyautobrain · small · supervised · normal +
📋 Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal ``` -/start_dev draft/bug/pyautobrain/pyautoconf_rename_functional_leftovers.md +/start_dev draft/feature/autolens_workspace/joss_cluster_benchmark_tuning.md ```
-
📋 generate.py deletes notebooks/ before rejecting an unknown project — pyautohands · small · supervised · normal +
📋 Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal ``` -/start_dev draft/bug/pyautohands/generate_rejects_autocti_after_deleting_notebooks.md +/start_dev draft/feature/pyautomind/complete_archive_wiki.md ```
-
📋 autoreduce 0.9 on PyPI never got the Python 3.12 floor — pyautoreduce · small · supervised · normal +
📋 The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal ``` -/start_dev draft/bug/pyautoreduce/published_autoreduce_09_missing_312_floor.md +/start_dev draft/feature/workspaces/group_los_halos.md ```
-
📋 aplt.Output stale-API drift in the remaining workspace repos — workspaces · small · supervised · normal +
📋 The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal ``` -/start_dev draft/bug/workspaces/aplt_output_drift_remaining_repos.md +/start_dev draft/feature/workspaces/group_subhalo_sensitivity.md ```
-
📋 Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py — workspaces · small · safe · normal +
📋 Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal ``` -/start_dev draft/bug/workspaces/bug_fix_the_tracer_fits_existence_guard.md +/start_dev draft/feature/autoarray/arcsecond_to_decimal.md ```
-
📋 Three jax_likelihood pins are stale by ~1.24e-4 and fail the — workspaces · small · supervised · normal +
📋 EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal ``` -/start_dev draft/bug/workspaces/jax_likelihood_pins_stale_by_1e4.md +/start_dev draft/feature/autofit/ep_analytic_updates.md ```
-
📋 PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional? — autoarray · medium · supervised · normal +
📋 Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal ``` -/start_dev draft/bug/autoarray/PROBE_adapt_double_square_coefficient.md +/start_dev draft/feature/autofit_assistant/remote_mcp_deployment_tiers.md ```
-
📋 interferometer Delaunay pixelization — non-PD FitException in test-mode bypass — autolens · medium · supervised · normal +
📋 Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal ``` -/start_dev draft/bug/autolens/interferometer_delaunay_nonpd_fitexception.md +/start_dev draft/feature/autolens_profiling/search_settings_estimation_infrastructure.md ```
-
📋 JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38 — autolens · medium · supervised · normal +
📋 Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal ``` -/start_dev draft/bug/autolens/jax_point_source_point_smoke_sentinel.md +/start_dev draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md ```
-
📋 JIT cache not hit in modeling_visualization delaunay/rectangular scripts — autolens · medium · supervised · normal +
📋 Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal ``` -/start_dev draft/bug/autolens/jit_cache_not_hit_modeling_visualization.md +/start_dev draft/feature/autoarray/rectangular_multi_submesh.md ```
-
📋 multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall) — autolens_workspace_test · medium · supervised · normal +
📋 PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal ``` -/start_dev draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md +/start_dev draft/feature/autogalaxy/piemass_potential.md ```
-
📋 @PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian — priors · medium · supervised · normal +
📋 autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal ``` -/start_dev draft/bug/priors/15_transformed_message_logpdf_jacobian.md +/start_dev draft/feature/autolens_jax_joss/autolens_jax_joss_benchmark_repo.md ```
-
📋 Resolve release-profile timeout scripts deliberately — health_fixes · too-large · supervised · normal +
📋 Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal ``` -/start_dev draft/bug/health_fixes/release_timeout_policy.md +/start_dev draft/feature/jax_substructure/5_prng_key_vmap_noise.md ```
-
📋 @PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms — priors · too-large · supervised · normal +
📋 Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal ``` -/start_dev draft/bug/priors/14_replace_transform_stack_with_bijectors.md +/start_dev draft/feature/jax_substructure/6_deflection_equivalence_test.md ```
-
📋 Priors & Messages cleanup — tracker — priors · too-large · supervised · normal +
📋 Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal ``` -/start_dev draft/bug/priors/z_features.md +/start_dev draft/feature/workspaces/restore_multiple_sources_lensing_of_lens.md ```
-
📋 autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main — autolens · low · supervised · normal +
📋 dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low ``` -/start_dev draft/bug/autolens/jax_likelihood_smoke_pins_stale.md +/start_dev draft/feature/autogalaxy/dpie_sigma0_parameterization.md ```
-
📋 status.sh --repos sources a file that no longer exists — pyautomind · small · supervised · low +
📋 Release board: local run_logs enrichment — pyautohands · small · supervised · low ``` -/start_dev draft/bug/pyautomind/status_sh_repos_missing_source.md +/start_dev draft/feature/pyautohands/release_board_run_logs_enrichment.md ```
-
📋 The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · low +
📋 Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low ``` -/start_dev draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md +/start_dev draft/feature/workspaces/gallery_runner_missing_tiers.md ```
-
📋 Point-source JSON datasets record no resolution regime — pyautolens · medium · supervised · low +
📋 Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low ``` -/start_dev draft/bug/pyautolens/point_source_json_datasets_record_no_regime.md +/start_dev draft/feature/autonomy/10_scheduled_runs.md ```
-
📋 interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens +
📋 Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low ``` -/start_dev draft/bug/autolens/interferometer_release_leg_oom.md +/start_dev draft/feature/pyautomind/repos-sync-config-stamper.md ```
@@ -1328,6 +1336,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| +| 2026-08-23 | issued | Phase 1: make a stalled JAX compile report itself (heartbeat | | 2026-08-23 | filed | Brain board follow-ups: what real mornings surface | | 2026-08-22 | filed | smoke_install.sh's stale jax<0.7 pin — CI is on the right | | 2026-08-22 | filed | multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA… | @@ -1337,12 +1346,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | | 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the | | 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of | -| 2026-08-22 | filed | Bug: fix the tracer.fits existence guard in autolens_workspace… |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| +| 2026-08-22 | filed | Bug: fix the tracer.fits existence guard in autolens_workspace… | | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-21 | filed | Numba PSF gathers derive the y/x kernel shifts from the | | 2026-08-20 | filed | Numba CPU likelihood phase 2: kernel-CDF numba fast path (the | @@ -1352,12 +1361,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | | 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor | -| 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| +| 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | | 2026-08-19 | filed | Release board: local run_logs enrichment | | 2026-08-19 | filed | Refactor Agent witness map lacks PyAutoNerves test suite | | 2026-08-19 | filed | RTD organism docs currency: Nerves page, organ-count drift, hands.md… | @@ -1367,12 +1376,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | | 2026-08-18 | parked | prior-message-collapse-design | -| 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| +| 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | | 2026-08-17 | filed | Teach repos_sync --write to stamp organ config surfaces | | 2026-08-16 | filed | LogGaussianPrior misreports its own support as (-inf, inf) | @@ -1382,12 +1391,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | | 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them | -| 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| +| 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | | 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are | | 2026-08-06 | filed | Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two… | @@ -1397,7 +1406,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | | 2026-08-04 | filed | aplt.Output stale-API drift in the remaining workspace repos | | 2026-08-04 | filed | Nightly release has been blocked 8 nights running — triage | -| 2026-08-04 | filed | HowToLens ch4 tutorial 3: mask overlay is never actually drawn |
@@ -1546,6 +1554,43 @@ Continue the 'Cluster strong lensing — Source & Cluster arc' epic. Its canonic
+
+Intermittent XLA compile stall in the JAX vmap likelihood path — 3 queued prompt(s), in order + +
📋 Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: `draft/bug/ci/jax_vmap_jit_compile_stall.md` + +``` +Continue the 'Intermittent XLA compile stall in the JAX vmap likelihood path' epic. Its canonical state lives in draft/bug/ci/jax_vmap_jit_compile_stall.md — read that ledger (and any DECISIONS/RESULTS files beside it) first. Cross-check this epic's entry in PyAutoMind/epics.md, any related rows in PyAutoMind/active.md, and the referenced repos' open issues and PRs, to work out the last completed phase and what is currently in flight. Then pick the next logical step and continue it through the normal workflow (/start_dev — filing the phase's prompt first if none exists), updating the ledger as the work advances. Note: three phases under draft/bug/ci/ — 1 evidence (PyAutoFit watchdog), 2 SLOW-vs-stall audit, 3 root cause + un-quarantine. Phase 3 is blocked on phase 1 shipping and a CI stall actually dumping a traceback; issue phases ONE at a time. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. +``` + +
+ +
📋 Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — ci · medium · supervised · high + +``` +/start_dev draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md +``` + +
+ +
📋 Phase 3: root-cause the XLA vmap compile stall and clear — ci · large · supervised · high + +``` +/start_dev draft/bug/ci/jax_compile_stall_3_root_cause.md +``` + +
+ +
📋 Intermittent XLA compile stall in JAX vmap likelihood scripts — — ci · too-large · supervised · high + +``` +/start_dev draft/bug/ci/jax_vmap_jit_compile_stall.md +``` + +
+ +
+
Expectation propagation (EP) campaign — 9 queued prompt(s), in order diff --git a/draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md b/draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md index 9ad2d63b..40d222e4 100644 --- a/draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md +++ b/draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md @@ -8,9 +8,28 @@ Repos: Difficulty: medium Autonomy: supervised Priority: normal -Status: draft +Status: superseded by epic jax-compile-stall (see banner below) Filed: 2026-08-22 (backfilled from git) +## SUPERSEDED 2026-08-23 — do not start this + +This defect is now the `jax-compile-stall` epic, whose ledger is +[`../ci/jax_vmap_jit_compile_stall.md`](../ci/jax_vmap_jit_compile_stall.md). +That filing carries the same stall with the cross-repo evidence +(`autogalaxy_workspace_test#109`, the two matrix legs disagreeing on the same +commit, the four 6-hour cancelled runs) and splits the work into three phases. +One root cause is one task, so this file is kept for its evidence and is **not** +issued. + +The two things this file asked for that the campaign filing did not already +carry are folded in, so nothing is lost: + +- *"make the runner dump a stack trace on timeout instead of just reporting + TIMEOUT"* → phase 1, solved library-side instead of in the runner + ([`../ci/jax_compile_stall_1_evidence.md`](../ci/jax_compile_stall_1_evidence.md)); +- *"re-enable `mge.py` and `shared_preloads.py` in `smoke_tests.txt`"* → phase 3 + ([`../ci/jax_compile_stall_3_root_cause.md`](../ci/jax_compile_stall_3_root_cause.md)). + ## Why this is filed now Filed 2026-08-22 after `multi_dataset/jax_likelihood/mge.py` was parked out of diff --git a/draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md b/draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md new file mode 100644 index 00000000..427b7e0c --- /dev/null +++ b/draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md @@ -0,0 +1,71 @@ +# Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is this stall wearing a different label? + +Type: bug +Target: ci +Repos: +- @autogalaxy_workspace_test +- @autolens_workspace_test +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised +Epic: jax-compile-stall +Phase: 2 +Campaign: bug/ci/jax_vmap_jit_compile_stall.md (Phase 2 — the classification; consumes phase 1's evidence) +Filed: 2026-08-23 + +## The question + +Step 1 of the campaign, on its own. Eight entries — six +`interferometer/jax_likelihood/*` and two `jax_grad/*` — were SLOW-skipped on +2026-07-14 for "flaking at the 1800s cap" (PyAutoHeart#74). A **SLOW** marker +says *make it faster*. A **stall** says *it never finishes*. Those route to +completely different places, and the Profiling Agent has been handed the first +description for a set that may partly belong to the second. + +Nothing in the current markers distinguishes them, because nothing measured the +distribution — a single observation at a cap looks the same either way. + +## Method + +A slow script has a **tight timing distribution**. A stalling one is +**bimodal**: tens of seconds when it completes, the full cap when it does not. +That is the discriminator, and it needs repeats, not one more run. + +1. Re-time each entry against the cap that actually applies to it (300s smoke, + 1800s release), N repeats each, both Python versions in the matrix. The + `rectangular_mge.py` result that passed on 3.13 and stalled on 3.12 **on the + same commit** shows one run per entry settles nothing. +2. Where phase 1's watchdog fires, keep the traceback — a stalled entry + identifies itself directly and does not need the distribution argument. +3. Classify each entry: genuinely slow, this stall mislabelled, or still + ambiguous after N runs. + +## Entries in scope + +The eight SLOW entries above, plus the three already carrying the stall +signature, so the whole set is classified by one consistent method: + +| Entry | Repo | Current marker | +|---|---|---| +| `multi_dataset/jax_likelihood/rectangular.py` | autogalaxy_workspace_test | NEEDS_FIX 2026-08-01 | +| `imaging/jax_likelihood/mge_group.py` | autogalaxy_workspace_test | NEEDS_FIX 2026-08-23 | +| `multi_dataset/jax_likelihood/delaunay.py` | autolens_workspace_test | NEEDS_FIX 2026-08-01 (#245) | +| six `interferometer/jax_likelihood/*` | both | SLOW 2026-07-14 | +| two `jax_grad/*` | both | SLOW 2026-07-14 | + +`imaging/jax_likelihood/rectangular_mge.py` (stalled on 3.12, passed on 3.13, +same commit) and `delaunay_mge.py` (disabled outright — `jax 0.7` removed +`jax.interpreters.xla.pytype_aval_mappings`, a genuinely different cause) are +reference points, not entries to re-time. + +## Acceptance + +- Every entry above classified, with the timing distribution that supports the + classification recorded — not asserted. +- Every marker in the `no_run.yaml` files and `smoke_tests.txt` rewritten to + carry its real reason, so a SLOW marker means slow and nothing else. +- The answer written down somewhere the Profiling Agent reads, so it stops + chasing speedups on scripts that are hanging. +- No script is un-quarantined by this phase — restoring coverage is phase 3, + and depends on the fix. diff --git a/draft/bug/ci/jax_compile_stall_3_root_cause.md b/draft/bug/ci/jax_compile_stall_3_root_cause.md new file mode 100644 index 00000000..fbcb1de4 --- /dev/null +++ b/draft/bug/ci/jax_compile_stall_3_root_cause.md @@ -0,0 +1,82 @@ +# Phase 3: root-cause the XLA vmap compile stall and clear every NEEDS_FIX it caused + +Type: bug +Target: ci +Repos: +- @PyAutoFit +- @autogalaxy_workspace_test +- @autolens_workspace_test +Difficulty: large +Autonomy: supervised +Priority: high +Status: formalised +Epic: jax-compile-stall +Phase: 3 +Campaign: bug/ci/jax_vmap_jit_compile_stall.md (Phase 3 — the fix; blocked on phases 1 and 2) +Filed: 2026-08-23 + +## Blocked on phase 1 + +Do not start this before phase 1's watchdog has shipped and a CI stall has +actually dumped a traceback. Diagnosing a hang that leaves no evidence is what +produced three quarantines and no root cause; repeating it without the +instrumentation would be a fourth. + +## Reproduce deliberately + +Loop `imaging/jax_likelihood/mge_group.py` under its declared CI env profile +until it hangs, rather than waiting for CI to hit it. Attach `py-spy dump` to +the hung process as well as reading phase 1's own `faulthandler` output — the +two see different things, and `py-spy` can read native frames the in-process +dump cannot. + +## Hypotheses, in the order they are cheapest to test + +1. **`vmap` of `jit`, the inverted ordering.** `Fitness._vmap` + (`autofit/non_linear/fitness.py`) builds `jax.vmap(jax.jit(self.call))`; + `autofit/non_linear/analysis/latent.py` builds + `jax.jit(jax.vmap(compute_latent_for_model))`, the conventional order. The + stalling path is exactly the `vmap` path, and the `_jit`-only scripts in the + same directories do not stall. One-line A/B — try it first. +2. **Persistent compilation cache contention.** `JAX_COMPILATION_CACHE_DIR` has + defaulted on since PyAutoConf#128 (merged 2026-07-17). Both NEEDS_FIX stalls + post-date it; the eight SLOW entries predate it. A/B with the cache dir set + to empty (which disables it) and see whether the stall probability moves. +3. **JAX/XLA version interaction.** This repo has form: `delaunay_mge.py` is + disabled outright because `jax 0.7` removed + `jax.interpreters.xla.pytype_aval_mappings`, and the smoke installer once + clobbered a working `tfp-nightly`. Pin-bisect jax/jaxlib across a run set. +4. **Runner CPU contention.** `complete/2026/07/jax-compile-time-research.md` + records that XLA compiles on **host** CPUs and that compile timing is + load-sensitive by up to 7×, which is why a hosted runner is the place this + reproduces and a workstation is not. +5. **Graph size in the vmap trace.** The affected set has a shape: plain `mge.py` + passes in 9.4s, while the *composite* variants — group, rectangular-MGE, + delaunay-MGE — stall or are already out. Complexity-driven compile blowup was + argued against by the autolens_profiling#71 research ("compile cost is + op-pattern-driven, not complexity-driven"), so treat this as the hypothesis + of last resort, not the first. + +## Then restore the coverage + +The point of the campaign. Quarantining removes exactly the heaviest JAX paths, +which are the ones most worth testing. + +1. Clear the NEEDS_FIX markers this campaign inherits — including the + 2026-08-01 `multi_dataset/jax_likelihood/rectangular.py` one and the + `autolens_workspace_test` `delaunay.py` entry citing #245. +2. Re-enable `multi_dataset/jax_likelihood/mge.py` and `shared_preloads.py` in + `autolens_workspace_test`'s `smoke_tests.txt` (folded in from the superseded + 2026-08-22 filing). +3. Anything that stays out after the fix stays out with a **recorded deliberate + reason**, not as an accumulated one-off. + +## Acceptance + +- A stated root cause, or an explicit recorded decision that it is an + infrastructure limit to be worked around rather than fixed. Not another + quarantine. +- Every entry marked NEEDS_FIX for this signature restored to its suite, or + re-marked with the real reason phase 2 established. +- The `multi_dataset/jax_likelihood/` family back under CI coverage in both test + workspaces, or its absence recorded as a deliberate choice. diff --git a/draft/bug/ci/jax_vmap_jit_compile_stall.md b/draft/bug/ci/jax_vmap_jit_compile_stall.md new file mode 100644 index 00000000..ce490091 --- /dev/null +++ b/draft/bug/ci/jax_vmap_jit_compile_stall.md @@ -0,0 +1,194 @@ +# Intermittent XLA compile stall in JAX vmap likelihood scripts — third repo, still unfixed + +Type: bug +Target: ci +Repos: +- @autogalaxy_workspace_test +- @autolens_workspace_test +- @PyAutoFit +Difficulty: too-large +Autonomy: supervised +Priority: high +Status: campaign map — phases route through /start_dev one at a time; this +Epic: jax-compile-stall + file is never issued itself and nothing here is bulk-issued +Filed: 2026-08-23 + +Surfaced 2026-08-23 by the per-script timeout backport +([`backport_per_script_timeout.md`](../../maintenance/ci/backport_per_script_timeout.md)), +which put a 300s cap on `autogalaxy_workspace_test`'s previously uncapped smoke +gate. The cap fired on its very first run — not on the change under test, but on +a latent hang the uncapped runner had been absorbing. + +## What was observed + +`autogalaxy_workspace_test#109`, `smoke (3.13)`, 36/37 passed: + +``` +TIMEOUT (300s) imaging/jax_likelihood/mge_group.py +``` + +The captured tail — preserved precisely because the cap keeps the killed child's +output — names the stall: + +``` +18:47:53 autoarray...dataset - INFO - IMAGING - Data masked, 1264 image-pixels +18:47:55 jax._src.xla_bridge - INFO - Unable to initialize backend 'tpu' ... +18:47:55 autofit.non_linear.jax_compile - INFO - JAX jit compiling vectorized + (vmap) likelihood function, could take seconds or minutes... + [silence for the full 300s] +``` + +It never emitted another line. This is a stall inside JAX vmap/JIT compilation, +not a slow script: + +- its sibling `imaging/jax_likelihood/mge.py` passes in **9.4s**, reporting + `JAX Time To VMAP + JIT Function: 2.50s`; +- `multi_dataset/jax_likelihood/mge_group.py` — same basename, same tier — + passes in **28.8s** in the same run; +- the script passes on `main`. + +So the same code path completes in seconds normally and occasionally never +completes at all. + +## It is not one script — the second leg named a second one + +The two matrix legs of the **same commit** disagreed, and the slower leg found +more: + +| Leg | Result | Timed out | +|---|---|---| +| `smoke (3.13)` | 36/37 | `imaging/jax_likelihood/mge_group.py` | +| `smoke (3.12)` | **35/37** | `imaging/jax_likelihood/mge_group.py` **and `imaging/jax_likelihood/rectangular_mge.py`** | + +`rectangular_mge.py` **passed on 3.13 and stalled on 3.12, on the same commit**. +So the stall is not a property of one script, and not deterministic per Python +version either — it is a per-compile probability that two runs of the same code +sample differently. + +The affected set has a shape. Of `imaging/jax_likelihood/`: + +| Script | State | +|---|---| +| `mge.py` | passes, 9.4s (2.5s vmap+JIT) | +| `mge_group.py` | stalled on both legs | +| `rectangular_mge.py` | stalled on 3.12, passed on 3.13 | +| `delaunay_mge.py` | already disabled (jax 0.7 removed `jax.interpreters.xla.pytype_aval_mappings`) | + +The plain `mge` is fine; the **composite** MGE variants — group, rectangular-MGE, +delaunay-MGE — are the ones that stall or are already out. That points at compile +graph size/complexity in the vmap trace, not at any one script's logic, and it +predicts which other entries are at risk. + +**This is why quarantining is the wrong end state.** Parking scripts as they +stall is whack-a-mole against a probability: each park removes coverage of +exactly the heaviest JAX paths, which are the ones most worth testing. + +## Why this is worth a task rather than a marker + +This is the **third** repo/script to hit the same signature, and the pattern in +the `no_run.yaml` files is that each occurrence gets quarantined and the +underlying stall is never diagnosed: + +| Entry | Marker | Note | +|---|---|---| +| `multi_dataset/jax_likelihood/rectangular.py` (autogalaxy_workspace_test) | NEEDS_FIX 2026-08-01 | "hung to the 1800s release cap … passes ~19s otherwise; intermittent XLA compile stall, same family as autolens_workspace_test delaunay" | +| autolens_workspace_test delaunay | — | `autolens_workspace_test#245` | +| `imaging/jax_likelihood/mge_group.py` (autogalaxy_workspace_test) | NEEDS_FIX 2026-08-23 | this one | + +Plus six `interferometer/jax_likelihood/*` and two `jax_grad/*` entries +SLOW-skipped on 2026-07-14 for "flaking at the 1800s cap" (PyAutoHeart#74) — +which, given this evidence, may not be slowness at all but the same stall +wearing a different label. **That distinction matters**: a SLOW marker says +"make it faster", a stall says "it never finishes", and the Profiling Agent has +been handed a set of scripts under the first description when some may belong to +the second. + +The cost was invisible until now because nothing was capped. +`autogalaxy_workspace_test` has **four** runs cancelled at the 6-hour Actions +ceiling (runs 30088422799, 30051301212, 30289779988, 31319423047) — roughly 24 +hours of runner time, with no diagnostic output, because the uncapped runner sat +on a held stdout pipe. Those runs are now impossible: the cap turns a 6-hour +silent hang into a 300s TIMEOUT with the compiling-step tail attached. That is +what made this diagnosable at all. + +## Task + +1. Establish whether the SLOW-marked `jax_likelihood`/`jax_grad` entries are + genuinely slow or are this stall mislabelled. Re-time them against the real + caps; a slow script has a tight timing distribution, a stalling one is + bimodal. +2. Reproduce the stall deliberately — loop `imaging/jax_likelihood/mge_group.py` + until it hangs, then attach `py-spy dump` / `faulthandler` to see where inside + the XLA compile it is parked. +3. Determine whether the trigger is a JAX/XLA version interaction (this repo has + prior form: `delaunay_mge.py` is disabled outright because `jax 0.7` removed + `jax.interpreters.xla.pytype_aval_mappings`, and the smoke installer once + clobbered a working `tfp-nightly`), a `vmap` shape/donation issue, or + contention on the runner. +4. Fix, then remove the NEEDS_FIX markers — including the 2026-08-01 one, which + this task inherits. + +## Acceptance + +- A stated root cause for the stall, not another quarantine. +- Every entry currently marked NEEDS_FIX for this signature is either restored to + its suite or re-marked with the real reason. +- The SLOW-vs-stall question in step 1 answered in writing, so the Profiling + Agent is not chasing speedups on scripts that are hanging. + +## Phase map (added 2026-08-23 at /start_dev) + +The Bug Agent sizes this `too-large` and returns `split-into-phases`. It is one +root cause but three separable deliverables, and phases 2 and 3 are both blocked +on evidence that does not exist today. Each phase is its own prompt, issued one +at a time — nothing here is bulk-issued. + +| Phase | Prompt | Deliverable | Repos | +|---|---|---|---| +| 1 | [`jax_compile_stall_1_evidence.md`](jax_compile_stall_1_evidence.md) | A stalled compile reports itself: heartbeat, `faulthandler` dump, compile-vs-execute split | PyAutoFit | +| 2 | [`jax_compile_stall_2_slow_vs_stall_audit.md`](jax_compile_stall_2_slow_vs_stall_audit.md) | The SLOW-vs-stall question (task step 1) answered in writing; every marker carries its real reason | autogalaxy_workspace_test, autolens_workspace_test | +| 3 | [`jax_compile_stall_3_root_cause.md`](jax_compile_stall_3_root_cause.md) | Root cause + fix; every NEEDS_FIX for this signature cleared (task steps 2–4) | PyAutoFit, both `_workspace_test` | + +Acceptance for the campaign as a whole is the § Acceptance section above; each +phase carries only its own slice of it. + +## Supersedes an earlier filing of the same defect + +[`../autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md`](../autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md) +(filed 2026-08-22, never issued) describes this same stall from the +`autolens_workspace_test` side. It is superseded by this campaign rather than +run alongside it — one root cause is one task. Its two asks that this filing did +not already carry are folded in: the runner leaving diagnostic evidence behind +(phase 1) and re-enabling `multi_dataset/jax_likelihood/mge.py` + +`shared_preloads.py` in `smoke_tests.txt` (phase 3). + +## Two source-level findings from the /start_dev read of PyAutoFit + +Recorded here because they are the leads phase 3 starts from, and neither was +known when this prompt was filed. + +**1. The stalling wrapper is `vmap` *of* `jit`, the inverted ordering.** +`Fitness._vmap` (`autofit/non_linear/fitness.py`) builds +`jax.vmap(jax.jit(self.call))`, while `latent.py`'s batched latent computation +builds `jax.jit(jax.vmap(compute_latent_for_model))` — the conventional order. +The path that stalls is exactly the `vmap` path; the `_jit`-only scripts in the +same directories do not stall. Whether the ordering is causal is unproven, but +it is a one-line A/B and it is the first thing phase 3 should try. + +**2. The silence spans two different waits, and nothing distinguishes them.** +`log_on_first_compile` (`autofit/non_linear/jax_compile.py`) logs +`JAX jit compiling {description}...`, calls the wrapped function, then calls +`jax.block_until_ready(result)` — trace/lower/compile and execution, with one +log line covering both and no heartbeat in between. That is why three +quarantines produced no diagnosis: the captured tail cannot say whether the +process is in XLA, in execution, or blocked on a lock. Phase 1 fixes exactly +this. + +**Timeline worth testing in phase 3.** Both NEEDS_FIX stalls (2026-08-01, +2026-08-23) post-date the persistent-compilation-cache default (PyAutoConf#128, +merged 2026-07-17); the eight SLOW entries predate it. `complete/2026/07/jax-compile-time-research.md` +also records that XLA compiles on host CPUs and that compile timing is +load-sensitive by up to 7×. Cache-lock contention and runner CPU contention are +therefore both live hypotheses alongside the version-interaction one already in +§ Task step 3. diff --git a/epics.md b/epics.md index 974a7fee..24773c73 100644 --- a/epics.md +++ b/epics.md @@ -28,6 +28,11 @@ epic, never picked standalone. - ledger: draft/feature/autolens/source_cluster_arc.md - notes: 12 phased prompts under draft/; issue phases ONE at a time as predecessors near shipping — no bulk issue queues. +## jax-compile-stall +- title: Intermittent XLA compile stall in the JAX vmap likelihood path +- ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md +- notes: three phases under draft/bug/ci/ — 1 evidence (PyAutoFit watchdog), 2 SLOW-vs-stall audit, 3 root cause + un-quarantine. Phase 3 is blocked on phase 1 shipping and a CI stall actually dumping a traceback; issue phases ONE at a time. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. + ## graphical-ep - title: Expectation propagation (EP) campaign - ledger: draft/research/graphical_ep/ep_campaign.md From c1a27615cfa30c94e9061cb9035ceb886b903181 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 19:58:56 +0000 Subject: [PATCH 02/11] prompt: jax-compile-stall-evidence PR#1517 open (#1516) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/active.md b/active.md index 02549f7e..70aa0bdd 100644 --- a/active.md +++ b/active.md @@ -10,6 +10,9 @@ - worktree-note: started in a web-github session with no local tree — the branch was cut against a direct clone at /home/user/pyautofit, not the worktree root above. A local-dev session resuming this should create the worktree via worktree_create. +- pr: https://github.com/PyAutoLabs/PyAutoFit/pull/1517 (opened 2026-08-23, awaiting review) +- heart: NOT consulted — pyauto-heart unreachable from the web session that opened the PR; + run the vitals check before merge. - repos: - PyAutoFit: feature/jax-compile-stall-evidence From c011830564254dc739312f405c13705801d5204c Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 20:13:19 +0000 Subject: [PATCH 03/11] prompt: file issue for jax-compile-stall-slow-vs-stall-audit (#271) Phase 2 of the jax-compile-stall epic, issued as autolens_workspace_test#271. Corrects the census from the actual marker files: 21 SLOW-marked JAX entries across the two test workspaces, not 8, and 5 stall quarantines, not 3 -- imaging/jax_likelihood/rectangular_mge.py was quarantined in ag_test on 2026-08-23, and multi_dataset/jax_likelihood/mge.py is disabled in al_test's smoke_tests.txt. 26 entries in scope, not 11. Records three things the marker text settles before any re-timing: a 27.8s script SLOW-marked for flaking at a 1800s cap; the same script path SLOW in one repo and a stall in the other; and every 2026-07-14 SLOW marker reading 'flakes' -- bimodal language -- with no timing recorded behind it. Parked blocked: the measurement leg needs a re-timing harness, which does not exist, and where it lives is an organ-boundary call between a Heart-owned runner input and a per-workspace duplicate. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 15 ++ ...jax_compile_stall_2_slow_vs_stall_audit.md | 132 ++++++++++++++++++ dashboard.html | 22 +-- dashboard.md | 34 ++--- ...jax_compile_stall_2_slow_vs_stall_audit.md | 71 ---------- 5 files changed, 175 insertions(+), 99 deletions(-) create mode 100644 active/jax_compile_stall_2_slow_vs_stall_audit.md delete mode 100644 draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md diff --git a/active.md b/active.md index 70aa0bdd..6cae62b8 100644 --- a/active.md +++ b/active.md @@ -1,5 +1,20 @@ # Active Tasks +## jax-compile-stall-slow-vs-stall-audit +- issue: https://github.com/PyAutoLabs/autolens_workspace_test/issues/271 (issued 2026-08-23) +- issued: 2026-08-23 +- prompt: active/jax_compile_stall_2_slow_vs_stall_audit.md +- status: workspace-dev +- epic: jax-compile-stall (phase 2 of 3; ledger draft/bug/ci/jax_vmap_jit_compile_stall.md) +- worktree: ~/Code/PyAutoLabs-wt/jax-compile-stall-slow-vs-stall-audit +- worktree-note: started in a web-github session with no local tree; no branch cut yet. +- blocked-on: a human decision about where the re-timing harness lives — a `runner` input + + workflow_dispatch on PyAutoHeart's reusable smoke-tests.yml (Heart-owned, serves every + workspace) vs a duplicated workflow_dispatch workflow per test workspace. See issue #271 + "Blocked on a design decision". The census and the three marker-text findings are done and + are on the issue; the measurement leg needs this answered first. +- repos: + ## jax-compile-stall-evidence - issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1516 (issued 2026-08-23) - issued: 2026-08-23 diff --git a/active/jax_compile_stall_2_slow_vs_stall_audit.md b/active/jax_compile_stall_2_slow_vs_stall_audit.md new file mode 100644 index 00000000..73feb768 --- /dev/null +++ b/active/jax_compile_stall_2_slow_vs_stall_audit.md @@ -0,0 +1,132 @@ +# Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is this stall wearing a different label? + +Type: bug +Target: ci +Repos: +- @autogalaxy_workspace_test +- @autolens_workspace_test +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised +Epic: jax-compile-stall +Phase: 2 +Campaign: bug/ci/jax_vmap_jit_compile_stall.md (Phase 2 — the classification; consumes phase 1's evidence) +Filed: 2026-08-23 +Issued: 2026-08-23 + +## The question + +Step 1 of the campaign, on its own. Eight entries — six +`interferometer/jax_likelihood/*` and two `jax_grad/*` — were SLOW-skipped on +2026-07-14 for "flaking at the 1800s cap" (PyAutoHeart#74). A **SLOW** marker +says *make it faster*. A **stall** says *it never finishes*. Those route to +completely different places, and the Profiling Agent has been handed the first +description for a set that may partly belong to the second. + +Nothing in the current markers distinguishes them, because nothing measured the +distribution — a single observation at a cap looks the same either way. + +## Method + +A slow script has a **tight timing distribution**. A stalling one is +**bimodal**: tens of seconds when it completes, the full cap when it does not. +That is the discriminator, and it needs repeats, not one more run. + +1. Re-time each entry against the cap that actually applies to it (300s smoke, + 1800s release), N repeats each, both Python versions in the matrix. The + `rectangular_mge.py` result that passed on 3.13 and stalled on 3.12 **on the + same commit** shows one run per entry settles nothing. +2. Where phase 1's watchdog fires, keep the traceback — a stalled entry + identifies itself directly and does not need the distribution argument. +3. Classify each entry: genuinely slow, this stall mislabelled, or still + ambiguous after N runs. + +## Corrected census (read off the marker files 2026-08-23, at /start_dev) + +The parent prompt's "eight entries SLOW-skipped on 2026-07-14" undercounts by +more than 2x, and it missed a fifth stall quarantine filed the same day. Read +from `config/build/no_run.yaml` in both repos plus `autolens_workspace_test/smoke_tests.txt`: + +**SLOW-marked JAX entries — 21, not 8.** + +| autogalaxy_workspace_test (9) | autolens_workspace_test (12) | +|---|---| +| `multi_dataset/jax_likelihood/delaunay_mge` (#72) | `imaging/jax_likelihood/delaunay_mge` | +| `imaging/jax_likelihood/delaunay_mge.py` | `imaging/jax_likelihood/mge_group` | +| `interferometer/jax_likelihood/mge.py` | `multi_dataset/jax_likelihood/delaunay_mge` (#72) | +| `interferometer/jax_likelihood/mge_group.py` | `multi_dataset/jax_likelihood/shared_preloads.py` | +| `interferometer/jax_likelihood/delaunay.py` | `interferometer/datacube/delaunay.py` | +| `interferometer/jax_likelihood/delaunay_mge.py` | `interferometer/datacube/shared_preloads.py` | +| `interferometer/jax_likelihood/rectangular_mge.py` | `interferometer/jax_likelihood/mge.py` | +| `interferometer/jax_grad/mge.py` | `interferometer/jax_likelihood/mge_group.py` | +| `multi_dataset/jax_grad/mge.py` | `interferometer/jax_likelihood/delaunay.py` | +| | `interferometer/jax_likelihood/delaunay_mge.py` | +| | `interferometer/jax_likelihood/rectangular_mge.py` | +| | `interferometer/jax_grad/gradient.py` | + +**Quarantined for the stall signature — 5, not 3.** + +| Entry | Repo | Marker | +|---|---|---| +| `multi_dataset/jax_likelihood/rectangular.py` | ag_test | NEEDS_FIX 2026-08-01 | +| `imaging/jax_likelihood/mge_group.py` | ag_test | NEEDS_FIX 2026-08-23 | +| `imaging/jax_likelihood/rectangular_mge.py` | ag_test | NEEDS_FIX 2026-08-23 | +| `multi_dataset/jax_likelihood/delaunay.py` | al_test | NEEDS_FIX 2026-08-01 (#245) | +| `multi_dataset/jax_likelihood/mge.py` | al_test | disabled in `smoke_tests.txt` 2026-08-22 | + +`imaging/jax_likelihood/rectangular_mge.py` was quarantined in ag_test on +2026-08-23, so the parent prompt's "passed on 3.13, stalled on 3.12" reference +point is now a quarantine in its own right. 26 entries in scope, not 11. + +## Three things the marker text already establishes, before any re-timing + +**1. A 27.8s script is SLOW-marked for "flaking" at a 1800s cap.** +`interferometer/datacube/shared_preloads.py` is SLOW-marked in +`autolens_workspace_test`'s `no_run.yaml` — *"flakes at the 1800s cap +(PyAutoHeart#74)"* — while the `smoke_tests.txt` comment on a sibling records it +running the PR gate in **27.8s**, and it is still enabled there. A script that +completes in 27.8s does not intermittently exceed 1800s by being slow: that is a +65x gap. Whatever is happening to it, slowness is not it. + +**2. The same script path is SLOW in one repo and a stall in the other.** +`imaging/jax_likelihood/mge_group` is SLOW-marked in `autolens_workspace_test` +(2026-07-21, *"flakes at the 1800s cap"*) and NEEDS_FIX-quarantined for the +stall in `autogalaxy_workspace_test` (2026-08-23, *"silence for the full 300s"*). +Same tier, same path, two labels. The label recorded which repo noticed it, not +what it was doing. + +**3. "Flakes" is bimodal language wearing a unimodal label.** +Every 2026-07-14 SLOW marker reads *"flakes at the 1800s cap"* and records **no +timing at all**. A genuinely slow script does not flake — it exceeds the cap +consistently. Compare the markers this repo writes when it has actually +measured something: `misc/database/scrape/*` say *"re-measured: times out at the +real 300s cap"*, and the stall quarantines say *"18s when it passes"*, +*"passes ~19s otherwise"*, *"runs green in 32s standalone"*. The batch that +carries the SLOW label is the batch with no measurement behind it. + +None of this is proof — it is three strong priors that make the re-timing a +confirmation rather than an exploration, and it means the burden now sits on +"these are slow", not on "these are stalls". + +## Entries in scope + +All 26 above. The method applies uniformly: an entry is classified from its own +distribution, not from which marker it happens to carry today. + +`imaging/jax_likelihood/delaunay_mge.py` (al_test) carries **two** unrelated +markers — SLOW in `no_run.yaml` and disabled in `smoke_tests.txt` for `jax 0.7` +removing `jax.interpreters.xla.pytype_aval_mappings`. The API removal is a real, +different cause; re-time it anyway, but do not let the SLOW marker's removal +imply the API one is resolved. + +## Acceptance + +- Every entry above classified, with the timing distribution that supports the + classification recorded — not asserted. +- Every marker in the `no_run.yaml` files and `smoke_tests.txt` rewritten to + carry its real reason, so a SLOW marker means slow and nothing else. +- The answer written down somewhere the Profiling Agent reads, so it stops + chasing speedups on scripts that are hanging. +- No script is un-quarantined by this phase — restoring coverage is phase 3, + and depends on the fix. diff --git a/dashboard.html b/dashboard.html index f024773e..421c6283 100644 --- a/dashboard.html +++ b/dashboard.html @@ -51,7 +51,7 @@

📋 PyAutoMind Dashboard

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-

In flight 2 · Parked 3 · Planned 6 · Backlog 156 · markdown version

+

In flight 3 · Parked 3 · Planned 6 · Backlog 155 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

TRIAGE: needs manual review before routing — medium · safe · high

@@ -73,6 +73,7 @@

In flight Issued — each has an open GitHub issue and usually a branch.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

Phase 1: make a stalled JAX compile report itself (heartbeat — issue #1516 — issued 2026-08-23 — library-dev

+

Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — issue #271 — issued 2026-08-23 — workspace-dev

Parked markdown version

3 task(s) @@ -91,7 +92,7 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

156 filed prompts, not started — sorted most-pickable first (priority, then size). 26 of them belong to an epic and are listed only under Epics below.

+

155 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

bug — 34

Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high

@@ -255,6 +256,12 @@

Backlog 2026-08-23 issued +Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or… + + + +2026-08-23 +issued Phase 1: make a stalled JAX compile report itself (heartbeat @@ -306,7 +313,7 @@

Backlog Is Intel macOS a supported platform, and what is the - + 2026-08-22 filed Defer the eager scipy.sparse import in derivative_util (~0.10 s of @@ -546,12 +553,6 @@

Backlog aplt.Output stale-API drift in the remaining workspace repos - -2026-08-04 -filed -Nightly release has been blocked 8 nights running — triage - -

Epics markdown version

@@ -576,9 +577,8 @@

Epics

Source & Cluster arc — magnification science, PointSolver trust, cluster — autolens · too-large · supervised · high

-Intermittent XLA compile stall in the JAX vmap likelihood path — 3 queued prompt(s), in order +Intermittent XLA compile stall in the JAX vmap likelihood path — 2 queued prompt(s), in order

Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md

-

Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — ci · medium · supervised · high

Phase 3: root-cause the XLA vmap compile stall and clear — ci · large · supervised · high

Intermittent XLA compile stall in JAX vmap likelihood scripts — — ci · too-large · supervised · high

diff --git a/dashboard.md b/dashboard.md index 18e865bb..0c157ded 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,10 +8,10 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 2 | +| [In flight](#in-flight) (`active/`) | 3 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | -| [Backlog](#backlog) (`draft/`) | 156 | +| [Backlog](#backlog) (`draft/`) | 155 | ## Start here @@ -151,6 +151,14 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
+
📋 Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — issue #271 — issued 2026-08-23 — workspace-dev + +``` +/start_dev active/jax_compile_stall_2_slow_vs_stall_audit.md +``` + +
+ ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -243,7 +251,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**156** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **26** of them belong to an epic and are listed only under [Epics](#epics) below. +**155** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below.
bug — 34 @@ -1336,6 +1344,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| +| 2026-08-23 | issued | Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or… | | 2026-08-23 | issued | Phase 1: make a stalled JAX compile report itself (heartbeat | | 2026-08-23 | filed | Brain board follow-ups: what real mornings surface | | 2026-08-22 | filed | smoke_install.sh's stale jax<0.7 pin — CI is on the right | @@ -1345,12 +1354,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | Remove pynufft + legacy TransformerNUFFTPyNUFFT | | 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | | 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the | -| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| +| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of | | 2026-08-22 | filed | Bug: fix the tracer.fits existence guard in autolens_workspace… | | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-21 | filed | Numba PSF gathers derive the y/x kernel shifts from the | @@ -1360,12 +1369,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit ('tuple'… | | 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | -| 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| +| 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor | | 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | | 2026-08-19 | filed | Release board: local run_logs enrichment | | 2026-08-19 | filed | Refactor Agent witness map lacks PyAutoNerves test suite | @@ -1375,12 +1384,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | -| 2026-08-18 | parked | prior-message-collapse-design |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| +| 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | | 2026-08-17 | filed | Teach repos_sync --write to stamp organ config surfaces | @@ -1390,12 +1399,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-09 | filed | pixel_scales given as an int (or np.float64) is never widened | | 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | -| 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| +| 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them | | 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | | 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are | @@ -1405,7 +1414,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of | | 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | | 2026-08-04 | filed | aplt.Output stale-API drift in the remaining workspace repos | -| 2026-08-04 | filed | Nightly release has been blocked 8 nights running — triage |
@@ -1555,7 +1563,7 @@ Continue the 'Cluster strong lensing — Source & Cluster arc' epic. Its canonic
-Intermittent XLA compile stall in the JAX vmap likelihood path — 3 queued prompt(s), in order +Intermittent XLA compile stall in the JAX vmap likelihood path — 2 queued prompt(s), in order
📋 Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: `draft/bug/ci/jax_vmap_jit_compile_stall.md` @@ -1565,14 +1573,6 @@ Continue the 'Intermittent XLA compile stall in the JAX vmap likelihood path' ep
-
📋 Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — ci · medium · supervised · high - -``` -/start_dev draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md -``` - -
-
📋 Phase 3: root-cause the XLA vmap compile stall and clear — ci · large · supervised · high ``` diff --git a/draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md b/draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md deleted file mode 100644 index 427b7e0c..00000000 --- a/draft/bug/ci/jax_compile_stall_2_slow_vs_stall_audit.md +++ /dev/null @@ -1,71 +0,0 @@ -# Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is this stall wearing a different label? - -Type: bug -Target: ci -Repos: -- @autogalaxy_workspace_test -- @autolens_workspace_test -Difficulty: medium -Autonomy: supervised -Priority: high -Status: formalised -Epic: jax-compile-stall -Phase: 2 -Campaign: bug/ci/jax_vmap_jit_compile_stall.md (Phase 2 — the classification; consumes phase 1's evidence) -Filed: 2026-08-23 - -## The question - -Step 1 of the campaign, on its own. Eight entries — six -`interferometer/jax_likelihood/*` and two `jax_grad/*` — were SLOW-skipped on -2026-07-14 for "flaking at the 1800s cap" (PyAutoHeart#74). A **SLOW** marker -says *make it faster*. A **stall** says *it never finishes*. Those route to -completely different places, and the Profiling Agent has been handed the first -description for a set that may partly belong to the second. - -Nothing in the current markers distinguishes them, because nothing measured the -distribution — a single observation at a cap looks the same either way. - -## Method - -A slow script has a **tight timing distribution**. A stalling one is -**bimodal**: tens of seconds when it completes, the full cap when it does not. -That is the discriminator, and it needs repeats, not one more run. - -1. Re-time each entry against the cap that actually applies to it (300s smoke, - 1800s release), N repeats each, both Python versions in the matrix. The - `rectangular_mge.py` result that passed on 3.13 and stalled on 3.12 **on the - same commit** shows one run per entry settles nothing. -2. Where phase 1's watchdog fires, keep the traceback — a stalled entry - identifies itself directly and does not need the distribution argument. -3. Classify each entry: genuinely slow, this stall mislabelled, or still - ambiguous after N runs. - -## Entries in scope - -The eight SLOW entries above, plus the three already carrying the stall -signature, so the whole set is classified by one consistent method: - -| Entry | Repo | Current marker | -|---|---|---| -| `multi_dataset/jax_likelihood/rectangular.py` | autogalaxy_workspace_test | NEEDS_FIX 2026-08-01 | -| `imaging/jax_likelihood/mge_group.py` | autogalaxy_workspace_test | NEEDS_FIX 2026-08-23 | -| `multi_dataset/jax_likelihood/delaunay.py` | autolens_workspace_test | NEEDS_FIX 2026-08-01 (#245) | -| six `interferometer/jax_likelihood/*` | both | SLOW 2026-07-14 | -| two `jax_grad/*` | both | SLOW 2026-07-14 | - -`imaging/jax_likelihood/rectangular_mge.py` (stalled on 3.12, passed on 3.13, -same commit) and `delaunay_mge.py` (disabled outright — `jax 0.7` removed -`jax.interpreters.xla.pytype_aval_mappings`, a genuinely different cause) are -reference points, not entries to re-time. - -## Acceptance - -- Every entry above classified, with the timing distribution that supports the - classification recorded — not asserted. -- Every marker in the `no_run.yaml` files and `smoke_tests.txt` rewritten to - carry its real reason, so a SLOW marker means slow and nothing else. -- The answer written down somewhere the Profiling Agent reads, so it stops - chasing speedups on scripts that are hanging. -- No script is un-quarantined by this phase — restoring coverage is phase 3, - and depends on the fix. From 9e56bb20e2f112989565175a9e90d2227a4c20c1 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 20:20:06 +0000 Subject: [PATCH 04/11] prompt: jax-compile-stall-slow-vs-stall-audit harness PRs open (#271) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Heart-side harness chosen: PyAutoHeart#161 gives the reusable smoke workflow a runner input, and autolens_workspace_test#272 / autogalaxy_workspace_test#110 add retime.py + a workflow_dispatch caller to each test workspace. #161 merges first — a caller cannot pass an input that does not exist. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/active.md b/active.md index 6cae62b8..538acaa5 100644 --- a/active.md +++ b/active.md @@ -8,11 +8,17 @@ - epic: jax-compile-stall (phase 2 of 3; ledger draft/bug/ci/jax_vmap_jit_compile_stall.md) - worktree: ~/Code/PyAutoLabs-wt/jax-compile-stall-slow-vs-stall-audit - worktree-note: started in a web-github session with no local tree; no branch cut yet. -- blocked-on: a human decision about where the re-timing harness lives — a `runner` input + - workflow_dispatch on PyAutoHeart's reusable smoke-tests.yml (Heart-owned, serves every - workspace) vs a duplicated workflow_dispatch workflow per test workspace. See issue #271 - "Blocked on a design decision". The census and the three marker-text findings are done and - are on the issue; the measurement leg needs this answered first. +- decision: harness lives Heart-side (human choice 2026-08-23) — a `runner` input on the + reusable smoke-tests.yml rather than a per-workspace duplicate of the ceremony. +- prs: + - https://github.com/PyAutoLabs/PyAutoHeart/pull/161 (runner input; MERGES FIRST) + - https://github.com/PyAutoLabs/autolens_workspace_test/pull/272 (retime harness) + - https://github.com/PyAutoLabs/autogalaxy_workspace_test/pull/110 (retime harness) +- order: #161 first — a caller cannot pass a `runner` input that does not exist yet, so a + dispatch of either retime.yml fails at workflow resolution until Heart merges. +- next: once merged, dispatch retime.yml over the 26 entries (300s smoke / 1800s release, + both matrix legs) and classify from the distributions; then rewrite the markers. +- heart: NOT consulted — pyauto-heart unreachable from this web session. - repos: ## jax-compile-stall-evidence From 986c956aa8fcdddce8e8547c54cf0ee4104f65e7 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 20:24:13 +0000 Subject: [PATCH 05/11] prompt: ship jax-compile-stall-evidence (#1516, PR#1517 merged) Phase 1 of the jax-compile-stall epic is shipped: PyAutoFit's compile wrapper now emits a heartbeat, arms a faulthandler watchdog (defaulting on under CI) and times the compile wait separately from the block_until_ready execution wait. The next CI stall diagnoses itself with no workspace or runner change. Record folds in the two phase-3 leads: Fitness._vmap is jax.vmap(jax.jit(...)), the inverted ordering, on exactly the path that stalls; and both NEEDS_FIX stalls post-date the persistent compile-cache default. Heart NOT consulted (web session, pyauto-heart unreachable); merge was a human instruction. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 16 ------------- .../2026/08/jax-compile-stall-evidence.md | 19 +++++++++++++++ complete/index.md | 3 ++- dashboard.html | 19 +++++++-------- dashboard.md | 24 +++++++------------ draft/bug/ci/jax_vmap_jit_compile_stall.md | 2 +- epics.md | 3 ++- 7 files changed, 41 insertions(+), 45 deletions(-) rename active/jax_compile_stall_1_evidence.md => complete/2026/08/jax-compile-stall-evidence.md (52%) diff --git a/active.md b/active.md index 538acaa5..9352380f 100644 --- a/active.md +++ b/active.md @@ -21,22 +21,6 @@ - heart: NOT consulted — pyauto-heart unreachable from this web session. - repos: -## jax-compile-stall-evidence -- issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1516 (issued 2026-08-23) -- issued: 2026-08-23 -- prompt: active/jax_compile_stall_1_evidence.md -- status: library-dev -- epic: jax-compile-stall (phase 1 of 3; ledger draft/bug/ci/jax_vmap_jit_compile_stall.md) -- worktree: ~/Code/PyAutoLabs-wt/jax-compile-stall-evidence -- worktree-note: started in a web-github session with no local tree — the branch was - cut against a direct clone at /home/user/pyautofit, not the worktree root above. - A local-dev session resuming this should create the worktree via worktree_create. -- pr: https://github.com/PyAutoLabs/PyAutoFit/pull/1517 (opened 2026-08-23, awaiting review) -- heart: NOT consulted — pyauto-heart unreachable from the web session that opened the PR; - run the vitals check before merge. -- repos: - - PyAutoFit: feature/jax-compile-stall-evidence - ## transformed-message-factor-gradient-unpack - issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1501 (issued 2026-08-19) - issued: 2026-08-19 diff --git a/active/jax_compile_stall_1_evidence.md b/complete/2026/08/jax-compile-stall-evidence.md similarity index 52% rename from active/jax_compile_stall_1_evidence.md rename to complete/2026/08/jax-compile-stall-evidence.md index d2aa0788..aa891f15 100644 --- a/active/jax_compile_stall_1_evidence.md +++ b/complete/2026/08/jax-compile-stall-evidence.md @@ -1,3 +1,22 @@ +# Make a stalled JAX compile report itself (jax-compile-stall phase 1) + +- **Issue:** PyAutoFit#1516 (closed) · **PR:** PyAutoFit#1517 (merged 2026-08-23) +- **Repos:** PyAutoFit (`autofit/non_linear/jax_compile.py`, `test_autofit/non_linear/test_jax_compile.py`) +- **Epic:** `jax-compile-stall` phase 1 of 3 — ledger `draft/bug/ci/jax_vmap_jit_compile_stall.md`. Phase 2 is autolens_workspace_test#271. +- **What:** `log_on_first_compile` gained a heartbeat (`still compiling , Ns elapsed`; `PYAUTOFIT_JAX_COMPILE_HEARTBEAT_SECS`, default 30, 0 disables), a `faulthandler` watchdog dumping the process's own traceback on overrun (`PYAUTOFIT_JAX_COMPILE_DUMP_SECS`, defaulting to 300 **when `CI` is set** and 0 otherwise), and separate timings for the trace/lower/compile wait vs the `jax.block_until_ready` execution wait. Existing `complete in {n} seconds` summary line unchanged. All four call sites inherit it (`Fitness._vmap/_jit/_grad`, `analysis/latent.py`). No workspace script and no CI runner touched. +- **Why:** the same intermittent XLA compile stall had been quarantined three times (autolens_workspace_test#245; ag_test `multi_dataset/.../rectangular.py` 2026-08-01; ag_test `imaging/.../mge_group.py` 2026-08-23) and diagnosed zero times, because a stalled run emitted the "compiling..." line and then nothing until the cap killed it. +- **Key traps recorded:** + - **The silence spanned two different waits.** The wrapper called `func(...)` (trace/lower/XLA compile) and then `jax.block_until_ready(result)` (execution) under **one** log line, so a captured tail could not say which half was stuck — or whether the process was alive at all. That ambiguity, not the stall itself, is what defeated three investigations. + - **`Fitness._vmap` is `jax.vmap(jax.jit(self.call))`** — `vmap` *of* `jit`, the inverted ordering — while `analysis/latent.py` uses `jax.jit(jax.vmap(...))`. The stalling path is exactly the `vmap` path; `_jit`-only scripts in the same directories do not stall. **Deliberately not changed here**: altering the transform while trying to observe the stall would destroy the thing being observed. First A/B for phase 3. + - **The CI-conditional default avoids a second repo touch.** Defaulting the dump on when `CI` is set makes the next stall self-diagnosing without threading an env var through every workspace's `config/build/env_vars_*.yaml`. + - **A traceback during XLA compile parks at the pybind boundary** and shows no XLA internals. It still separates *in compile* / *in execution* / *blocked on a Python-level lock* (e.g. the persistent compile cache, on by default since PyAutoConf#128) — which is the three-way fork phase 3 needs. + - **Diagnostics must never break a fit.** An unstartable heartbeat thread (`RuntimeError` from `Thread.start`), an unarmable dump (a capturing harness can leave stderr without a real fd) and a malformed or negative interval all fall back and continue. The heartbeat-start guard was added on an adversarial re-read before pushing — it was originally outside the `try`, where it could have killed a fit. +- **Tests/verification:** full PyAutoFit suite 2008 passed / 34 skipped (3.12); `test_jax_compile.py` 16 passed, 12 new. CI green on 3.12, 3.13 and **unittest-nojax** — the new tests need no JAX import. End-to-end: a simulated stall in a fresh process (`CI=true`, heartbeat 2s, dump 5s) emitted heartbeats and two repeating tracebacks before SIGKILL at 12s, exactly as a CI cap would kill it. +- **Heart:** **NOT consulted** — `pyauto-heart` was unreachable from the `web-github` session that opened and merged this. Flagged in the PR body and in `active.md`; merge was a human instruction ("merge when green"). +- **Provenance:** started, implemented, shipped and merged by one `web-github` session 2026-08-23 (`claude/jax-vmap-jit-stall-swz2tc`), against a direct PyAutoFit clone rather than a worktree. Filed prompt lived on an unmerged branch (`claude/backport-per-script-timeout-r3w1sv`) and was brought onto the task branch at `/start_dev`. + +## Original prompt + # Phase 1: make a stalled JAX compile report itself (heartbeat + faulthandler + compile/execute split) Type: bug diff --git a/complete/index.md b/complete/index.md index 81d14416..f6072563 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1086 records across 7 buckets. +1087 records across 7 buckets. ## Highlights @@ -97,6 +97,7 @@ _(curate hard-won records here — survives regeneration.)_ - [interferometer-delaunay-flaky-fitexception](2026/08/interferometer-delaunay-flaky-fitexception.md) - [interferometer-start-here-integrate-oom](2026/08/interferometer-start-here-integrate-oom.md) - [intra-family-dep-floors](2026/08/intra-family-dep-floors.md) +- [jax-compile-stall-evidence](2026/08/jax-compile-stall-evidence.md) — jax-compile-stall phase 1 - [jax-default-dependency](2026/08/jax-default-dependency.md) - [jax-grad-local-vs-ci-assertions](2026/08/jax-grad-local-vs-ci-assertions.md) - [jax-grad-smoke-timeout-budget](2026/08/jax-grad-smoke-timeout-budget.md) diff --git a/dashboard.html b/dashboard.html index 421c6283..d9d3afca 100644 --- a/dashboard.html +++ b/dashboard.html @@ -51,7 +51,7 @@

📋 PyAutoMind Dashboard

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-

In flight 3 · Parked 3 · Planned 6 · Backlog 155 · markdown version

+

In flight 2 · Parked 3 · Planned 6 · Backlog 155 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

TRIAGE: needs manual review before routing — medium · safe · high

@@ -72,7 +72,6 @@

Quick wins (small enough, and safe enough to run unatte

In flight markdown version

Issued — each has an open GitHub issue and usually a branch.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

-

Phase 1: make a stalled JAX compile report itself (heartbeat — issue #1516 — issued 2026-08-23 — library-dev

Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — issue #271 — issued 2026-08-23 — workspace-dev

Parked markdown version

@@ -261,12 +260,6 @@

Backlog 2026-08-23 -issued -Phase 1: make a stalled JAX compile report itself (heartbeat - - - -2026-08-23 filed Brain board follow-ups: what real mornings surface @@ -313,7 +306,7 @@

Backlog Is Intel macOS a supported platform, and what is the - + 2026-08-22 filed Defer the eager scipy.sparse import in derivative_util (~0.10 s of @@ -553,6 +546,12 @@

Backlog aplt.Output stale-API drift in the remaining workspace repos + +2026-08-04 +filed +Nightly release has been blocked 8 nights running — triage + +

Epics markdown version

@@ -578,7 +577,7 @@

Epics

Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md

+

Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md — phase 1 shipped 2026-08-23 (PyAutoFit#1517); phase 2 in flight (autolens_workspace_test#271)

Phase 3: root-cause the XLA vmap compile stall and clear — ci · large · supervised · high

Intermittent XLA compile stall in JAX vmap likelihood scripts — — ci · too-large · supervised · high

diff --git a/dashboard.md b/dashboard.md index 0c157ded..4b7af02a 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,7 +8,7 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 3 | +| [In flight](#in-flight) (`active/`) | 2 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | | [Backlog](#backlog) (`draft/`) | 155 | @@ -143,14 +143,6 @@ Issued — each has an open GitHub issue and usually a branch. The full record f

-
📋 Phase 1: make a stalled JAX compile report itself (heartbeat — issue #1516 — issued 2026-08-23 — library-dev - -``` -/start_dev active/jax_compile_stall_1_evidence.md -``` - -
-
📋 Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — issue #271 — issued 2026-08-23 — workspace-dev ``` @@ -1345,7 +1337,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| | 2026-08-23 | issued | Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or… | -| 2026-08-23 | issued | Phase 1: make a stalled JAX compile report itself (heartbeat | | 2026-08-23 | filed | Brain board follow-ups: what real mornings surface | | 2026-08-22 | filed | smoke_install.sh's stale jax<0.7 pin — CI is on the right | | 2026-08-22 | filed | multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA… | @@ -1354,12 +1345,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | Remove pynufft + legacy TransformerNUFFTPyNUFFT | | 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | | 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the | +| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| -| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of | | 2026-08-22 | filed | Bug: fix the tracer.fits existence guard in autolens_workspace… | | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-21 | filed | Numba PSF gathers derive the y/x kernel shifts from the | @@ -1369,12 +1360,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit ('tuple'… | | 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | +| 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| -| 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor | | 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | | 2026-08-19 | filed | Release board: local run_logs enrichment | | 2026-08-19 | filed | Refactor Agent witness map lacks PyAutoNerves test suite | @@ -1384,12 +1375,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | +| 2026-08-18 | parked | prior-message-collapse-design |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| -| 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | | 2026-08-17 | filed | Teach repos_sync --write to stamp organ config surfaces | @@ -1399,12 +1390,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-09 | filed | pixel_scales given as an int (or np.float64) is never widened | | 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | +| 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| -| 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them | | 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | | 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are | @@ -1414,6 +1405,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of | | 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | | 2026-08-04 | filed | aplt.Output stale-API drift in the remaining workspace repos | +| 2026-08-04 | filed | Nightly release has been blocked 8 nights running — triage |
@@ -1565,10 +1557,10 @@ Continue the 'Cluster strong lensing — Source & Cluster arc' epic. Its canonic
Intermittent XLA compile stall in the JAX vmap likelihood path — 2 queued prompt(s), in order -
📋 Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: `draft/bug/ci/jax_vmap_jit_compile_stall.md` +
📋 Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: `draft/bug/ci/jax_vmap_jit_compile_stall.md` — phase 1 shipped 2026-08-23 (PyAutoFit#1517); phase 2 in flight (autolens_workspace_test#271) ``` -Continue the 'Intermittent XLA compile stall in the JAX vmap likelihood path' epic. Its canonical state lives in draft/bug/ci/jax_vmap_jit_compile_stall.md — read that ledger (and any DECISIONS/RESULTS files beside it) first. Cross-check this epic's entry in PyAutoMind/epics.md, any related rows in PyAutoMind/active.md, and the referenced repos' open issues and PRs, to work out the last completed phase and what is currently in flight. Then pick the next logical step and continue it through the normal workflow (/start_dev — filing the phase's prompt first if none exists), updating the ledger as the work advances. Note: three phases under draft/bug/ci/ — 1 evidence (PyAutoFit watchdog), 2 SLOW-vs-stall audit, 3 root cause + un-quarantine. Phase 3 is blocked on phase 1 shipping and a CI stall actually dumping a traceback; issue phases ONE at a time. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. +Continue the 'Intermittent XLA compile stall in the JAX vmap likelihood path' epic. Its canonical state lives in draft/bug/ci/jax_vmap_jit_compile_stall.md — read that ledger (and any DECISIONS/RESULTS files beside it) first. Cross-check this epic's entry in PyAutoMind/epics.md, any related rows in PyAutoMind/active.md, and the referenced repos' open issues and PRs, to work out the last completed phase and what is currently in flight. Then pick the next logical step and continue it through the normal workflow (/start_dev — filing the phase's prompt first if none exists), updating the ledger as the work advances. Note: three phases under draft/bug/ci/ — 1 evidence (PyAutoFit watchdog, SHIPPED), 2 SLOW-vs-stall audit, 3 root cause + un-quarantine. Phase 3 is blocked on phase 1 shipping and a CI stall actually dumping a traceback; issue phases ONE at a time. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. ```
diff --git a/draft/bug/ci/jax_vmap_jit_compile_stall.md b/draft/bug/ci/jax_vmap_jit_compile_stall.md index ce490091..47a4eb0a 100644 --- a/draft/bug/ci/jax_vmap_jit_compile_stall.md +++ b/draft/bug/ci/jax_vmap_jit_compile_stall.md @@ -146,7 +146,7 @@ at a time — nothing here is bulk-issued. | Phase | Prompt | Deliverable | Repos | |---|---|---|---| -| 1 | [`jax_compile_stall_1_evidence.md`](jax_compile_stall_1_evidence.md) | A stalled compile reports itself: heartbeat, `faulthandler` dump, compile-vs-execute split | PyAutoFit | +| 1 | ~~`jax_compile_stall_1_evidence.md`~~ **SHIPPED 2026-08-23** — record [`complete/2026/08/jax-compile-stall-evidence.md`](../../../complete/2026/08/jax-compile-stall-evidence.md) (PyAutoFit#1516, PR#1517 merged) | A stalled compile reports itself: heartbeat, `faulthandler` dump, compile-vs-execute split | PyAutoFit | | 2 | [`jax_compile_stall_2_slow_vs_stall_audit.md`](jax_compile_stall_2_slow_vs_stall_audit.md) | The SLOW-vs-stall question (task step 1) answered in writing; every marker carries its real reason | autogalaxy_workspace_test, autolens_workspace_test | | 3 | [`jax_compile_stall_3_root_cause.md`](jax_compile_stall_3_root_cause.md) | Root cause + fix; every NEEDS_FIX for this signature cleared (task steps 2–4) | PyAutoFit, both `_workspace_test` | diff --git a/epics.md b/epics.md index 24773c73..fc287b31 100644 --- a/epics.md +++ b/epics.md @@ -31,7 +31,8 @@ epic, never picked standalone. ## jax-compile-stall - title: Intermittent XLA compile stall in the JAX vmap likelihood path - ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md -- notes: three phases under draft/bug/ci/ — 1 evidence (PyAutoFit watchdog), 2 SLOW-vs-stall audit, 3 root cause + un-quarantine. Phase 3 is blocked on phase 1 shipping and a CI stall actually dumping a traceback; issue phases ONE at a time. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. +- status: phase 1 shipped 2026-08-23 (PyAutoFit#1517); phase 2 in flight (autolens_workspace_test#271) +- notes: three phases under draft/bug/ci/ — 1 evidence (PyAutoFit watchdog, SHIPPED), 2 SLOW-vs-stall audit, 3 root cause + un-quarantine. Phase 3 is blocked on phase 1 shipping and a CI stall actually dumping a traceback; issue phases ONE at a time. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. ## graphical-ep - title: Expectation propagation (EP) campaign From a94f526ae2e8d1bbb2112dade62fafa04aac7de2 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 20:31:16 +0000 Subject: [PATCH 06/11] prompt: jax-compile-stall retime harness merged in all three repos (#271) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PyAutoHeart#161, autolens_workspace_test#272 and autogalaxy_workspace_test#110 all merged. The measurement leg is now unblocked; the full 26-entry sweep is left undispatched on purpose — many runner-hours at the 1800s cap. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/active.md b/active.md index 9352380f..50d6081a 100644 --- a/active.md +++ b/active.md @@ -10,14 +10,16 @@ - worktree-note: started in a web-github session with no local tree; no branch cut yet. - decision: harness lives Heart-side (human choice 2026-08-23) — a `runner` input on the reusable smoke-tests.yml rather than a per-workspace duplicate of the ceremony. -- prs: - - https://github.com/PyAutoLabs/PyAutoHeart/pull/161 (runner input; MERGES FIRST) - - https://github.com/PyAutoLabs/autolens_workspace_test/pull/272 (retime harness) - - https://github.com/PyAutoLabs/autogalaxy_workspace_test/pull/110 (retime harness) -- order: #161 first — a caller cannot pass a `runner` input that does not exist yet, so a - dispatch of either retime.yml fails at workflow resolution until Heart merges. -- next: once merged, dispatch retime.yml over the 26 entries (300s smoke / 1800s release, - both matrix legs) and classify from the distributions; then rewrite the markers. +- prs: ALL MERGED 2026-08-23 + - https://github.com/PyAutoLabs/PyAutoHeart/pull/161 (runner input) — merged + - https://github.com/PyAutoLabs/autolens_workspace_test/pull/272 (retime harness) — merged + - https://github.com/PyAutoLabs/autogalaxy_workspace_test/pull/110 (retime harness) — merged +- harness: `retime.yml` (workflow_dispatch) + `.github/scripts/retime.py` now live in both test + workspaces. Inputs: scripts (comma-separated, relative to scripts/), repeats, script-timeout + (300 = smoke, 1800 = release). Verdicts STALL/SLOW/NEITHER/AMBIGUOUS/ERROR + retime_results.json. +- next: the measurement. A full 26-entry x N sweep at the 1800s cap is many runner-hours, so it + is deliberately NOT dispatched wholesale — start bounded, on the entries that decide the most, + and widen with a human's say-so. - heart: NOT consulted — pyauto-heart unreachable from this web session. - repos: From c07b29ce586fba169d1d5778e51209fd87d87448 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 21:32:53 +0000 Subject: [PATCH 07/11] prompt: first retime measurement lands; phase-1 watchdog defect found (#271) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 40 script executions across both test workspaces settle three things: the datacube shared_preloads SLOW marker is refuted (10/10 completions, 34s worst against the 1800s cap it supposedly flakes at); rectangular_mge is a measured STALL, bimodal on both Python legs at ~80% per-compile probability; and mge_group + multi_dataset mge need the 1800s pass. The run also exposed a defect in phase 1's own watchdog: its 300s CI default collided with the 300s smoke cap, so the runner's SIGKILL beat the dump in all 20 stalled runs — heartbeats but no stacks. PyAutoFit#1518 derives the threshold from BUILD_SCRIPT_TIMEOUT instead. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/active.md b/active.md index 50d6081a..d1342eab 100644 --- a/active.md +++ b/active.md @@ -17,9 +17,23 @@ - harness: `retime.yml` (workflow_dispatch) + `.github/scripts/retime.py` now live in both test workspaces. Inputs: scripts (comma-separated, relative to scripts/), repeats, script-timeout (300 = smoke, 1800 = release). Verdicts STALL/SLOW/NEITHER/AMBIGUOUS/ERROR + retime_results.json. -- next: the measurement. A full 26-entry x N sweep at the 1800s cap is many runner-hours, so it - is deliberately NOT dispatched wholesale — start bounded, on the entries that decide the most, - and widen with a human's say-so. +- measured 2026-08-23, first dispatch (4 entries x 5 repeats x 2 legs = 40 executions, 300s cap; + ag_test run 32664679042, al_test run 32664682689) — findings on issue #271: + - `interferometer/datacube/shared_preloads.py` (al_test) NEITHER — 10/10 completed, slowest + 34.0s = 1.9% of the 1800s cap its SLOW marker says it "flakes at". Marker REFUTED; it should + be removed, not rewritten. This is finding 1 of the marker-text analysis, now measured. + - `imaging/jax_likelihood/rectangular_mge.py` (ag_test) STALL — 4/5 capped + one ~22s + completion (7% of cap), the SAME split independently on 3.12 and 3.13. Bimodality measured; + ~80% stall probability per compile, indifferent to Python version. + - `imaging/jax_likelihood/mge_group.py` (ag_test) and `multi_dataset/jax_likelihood/mge.py` + (al_test) AMBIGUOUS — 20 consecutive cap hits, zero completions. Need the 1800s pass. +- phase-1 defect found by this run: the faulthandler dump NEVER fired — its CI default was a flat + 300s and the smoke cap is also 300s, so the runner's SIGKILL beat the dump in all 20 stalled + runs. Heartbeats worked; stacks did not. Fix: PyAutoFit#1518 derives the default from + BUILD_SCRIPT_TIMEOUT at 80% (300s cap -> 240s). Phase 3 needs that stack, so this gates it. +- next: (1) merge #1518; (2) re-dispatch the two AMBIGUOUS entries at the 1800s cap so they stall + WITH a stack; (3) the remaining 17 SLOW entries at 300s — expect several NEITHER; (4) marker + rewrites once every entry has a verdict. No un-quarantining — that is phase 3. - heart: NOT consulted — pyauto-heart unreachable from this web session. - repos: From b075e00002665fdd408d3407607b4047b82df950 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 21:37:22 +0000 Subject: [PATCH 08/11] prompt: #1518 merged; 1800s pass dispatched on the two AMBIGUOUS entries (#271) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/active.md b/active.md index d1342eab..2313430e 100644 --- a/active.md +++ b/active.md @@ -31,8 +31,14 @@ 300s and the smoke cap is also 300s, so the runner's SIGKILL beat the dump in all 20 stalled runs. Heartbeats worked; stacks did not. Fix: PyAutoFit#1518 derives the default from BUILD_SCRIPT_TIMEOUT at 80% (300s cap -> 240s). Phase 3 needs that stack, so this gates it. -- next: (1) merge #1518; (2) re-dispatch the two AMBIGUOUS entries at the 1800s cap so they stall - WITH a stack; (3) the remaining 17 SLOW entries at 300s — expect several NEITHER; (4) marker +- PyAutoFit#1518 MERGED 2026-08-23 — the dump now fires at 80% of BUILD_SCRIPT_TIMEOUT. +- dispatched 2026-08-23 21:37, the 1800s pass on the two AMBIGUOUS entries, repeats=2 not 5: + ag_test imaging/jax_likelihood/mge_group.py and al_test multi_dataset/jax_likelihood/mge.py. + Two repeats because the deliverable here is the STACK, not more distribution — both entries + already hit the cap 10/10 at 300s, so one completion-or-cap settles the AMBIGUOUS question and + any single stall yields the 1440s traceback. Caps the bill at ~1h/job instead of ~2.5h. +- next: (1) read those two runs — the traceback is what phase 3 starts from; (2) the remaining + 17 SLOW entries at 300s — expect several NEITHER given the shared_preloads result; (3) marker rewrites once every entry has a verdict. No un-quarantining — that is phase 3. - heart: NOT consulted — pyauto-heart unreachable from this web session. - repos: From 5c76211c1a285aaeaad78d78abcc0ae31ea5dcbd Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 21:40:23 +0000 Subject: [PATCH 09/11] prompt: cut phase 2 short, open phase 3 root-cause experiments (#271) James's call: stop widening the timing measurement. Four entries was enough to prove the SLOW markers unreliable; the remaining 17 entries and the marker rewrites move out of the epic into draft/research/ci/smoke_timing_and_profiling.md, a follow-up task for proper smoke timing and profiling that carries the phase-2 results forward so nothing is re-measured. Phase 3 starts with the vmap-of-jit ordering A/B: experiment/jax-vmap-jit-ordering in PyAutoFit swaps jax.vmap(jax.jit(...)) for jax.jit(jax.vmap(...)), and a same-named branch in autogalaxy_workspace_test makes the workspace CI's matching-branch clone pick it up. Control is the measured 4/5 stall rate on both legs for imaging/jax_likelihood/rectangular_mge.py. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 37 +++++++++- dashboard.html | 21 +++--- dashboard.md | 24 ++++--- .../research/ci/smoke_timing_and_profiling.md | 70 +++++++++++++++++++ 4 files changed, 131 insertions(+), 21 deletions(-) create mode 100644 draft/research/ci/smoke_timing_and_profiling.md diff --git a/active.md b/active.md index 2313430e..8d3b40d1 100644 --- a/active.md +++ b/active.md @@ -1,5 +1,33 @@ # Active Tasks +## jax-compile-stall-root-cause +- epic: jax-compile-stall (phase 3 of 3; ledger draft/bug/ci/jax_vmap_jit_compile_stall.md) +- phase-prompt (still in draft/, deliberately not advanced to active/): the phase 3 file + draft/bug/ci/jax_compile_stall_3_root_cause.md has no GitHub issue of its own — the experiments + run under autolens_workspace_test#271 while they are exploratory. Issue it if phase 3 turns into + a shippable fix rather than a hunt. +- status: workspace-dev (experiment in flight) +- started: 2026-08-23 +- experiment 1 — the vmap-of-jit ordering: + - Fitness._vmap builds jax.vmap(jax.jit(self.call)); analysis/latent.py builds the conventional + jax.jit(jax.vmap(...)). The stalling scripts are exactly the vmap-path ones. + - branch `experiment/jax-vmap-jit-ordering` in PyAutoFit (swapped, NOT for merge) and a + same-named branch in autogalaxy_workspace_test so the workspace CI's matching-branch clone + picks the swapped library up. + - dispatched retime.yml from that branch on imaging/jax_likelihood/rectangular_mge.py, + repeats=5, 300s cap. Control: the same entry on main measured 4/5 stalls on BOTH legs + (~80% per compile). A clear drop in stall rate implicates the ordering; no change exonerates it. +- experiment 2 — the persistent compile cache (not yet run): JAX_COMPILATION_CACHE_DIR has been on + by default since PyAutoConf#128 (2026-07-17); both NEEDS_FIX stalls post-date it, the SLOW batch + predates it. A/B with the cache disabled. +- in flight from phase 2, kept because they feed phase 3: the 1800s runs on + imaging/jax_likelihood/mge_group.py (ag_test) and multi_dataset/jax_likelihood/mge.py (al_test), + repeats=2 — with PyAutoFit#1518 merged these should dump a faulthandler stack at 1440s, which is + the first look at WHERE the stall sits. +- repos: + - PyAutoFit: experiment/jax-vmap-jit-ordering + - autogalaxy_workspace_test: experiment/jax-vmap-jit-ordering + ## jax-compile-stall-slow-vs-stall-audit - issue: https://github.com/PyAutoLabs/autolens_workspace_test/issues/271 (issued 2026-08-23) - issued: 2026-08-23 @@ -37,9 +65,12 @@ Two repeats because the deliverable here is the STACK, not more distribution — both entries already hit the cap 10/10 at 300s, so one completion-or-cap settles the AMBIGUOUS question and any single stall yields the 1440s traceback. Caps the bill at ~1h/job instead of ~2.5h. -- next: (1) read those two runs — the traceback is what phase 3 starts from; (2) the remaining - 17 SLOW entries at 300s — expect several NEITHER given the shared_preloads result; (3) marker - rewrites once every entry has a verdict. No un-quarantining — that is phase 3. +- SCOPE CUT 2026-08-23 (James): stop widening the measurement — the remaining 17 SLOW entries and + the marker rewrites move OUT of this epic into `draft/research/ci/smoke_timing_and_profiling.md`, + a follow-up task for proper smoke timing/profiling. Four entries measured was enough to prove the + SLOW markers unreliable; sweeping the rest one dispatch at a time was not worth the runner budget. + Phase 2 is therefore DONE for the epic's purposes — its question is answered. +- next: phase 3, the root cause. - heart: NOT consulted — pyauto-heart unreachable from this web session. - repos: diff --git a/dashboard.html b/dashboard.html index d9d3afca..97302281 100644 --- a/dashboard.html +++ b/dashboard.html @@ -51,7 +51,7 @@

📋 PyAutoMind Dashboard

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-

In flight 2 · Parked 3 · Planned 6 · Backlog 155 · markdown version

+

In flight 2 · Parked 3 · Planned 6 · Backlog 156 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

TRIAGE: needs manual review before routing — medium · safe · high

@@ -91,7 +91,7 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

155 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

+

156 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

bug — 34

Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high

@@ -162,7 +162,7 @@

Backlog

Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low

-research — 17 +research — 18

Deep research: Can we speed up Delaunay in PyAutoArray? — autoarray · too-large · supervised · high

Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high

Delaunay-family JAX modules never hit the persistent compilation cache — autoarray · medium · supervised · medium

@@ -172,6 +172,7 @@

Backlog

Re-baseline the slacs0008 acceptance parity after the HAP-dedupe fix — pyautoreduce · small · supervised · normal

Kernel-CDF bandwidth defaults — config-dependent quality, investigate adaptivity — autoarray · medium · supervised · normal

We have lots of examples which profile how long JAX — autolens_workspace_developer · medium · supervised · normal

+

Properly time and profile the smoke/release script surface — ci · medium · supervised · normal

Is Intel macOS a supported platform, and what is the — libraries · medium · supervised · normal

Checkerboard PSF-mismatch residual diagnostic — research + document + ingest — pyautomemory · medium · supervised · normal

autofit_profiling: bootstrap the repo + general PyAutoFit profiling epic — autofit · large · supervised · normal

@@ -254,6 +255,12 @@

Backlog 2026-08-23 +filed +Properly time and profile the smoke/release script surface + + + +2026-08-23 issued Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or… @@ -306,7 +313,7 @@

Backlog Is Intel macOS a supported platform, and what is the - + 2026-08-22 filed Defer the eager scipy.sparse import in derivative_util (~0.10 s of @@ -546,12 +553,6 @@

Backlog aplt.Output stale-API drift in the remaining workspace repos - -2026-08-04 -filed -Nightly release has been blocked 8 nights running — triage - -

Epics markdown version

diff --git a/dashboard.md b/dashboard.md index 4b7af02a..06d8a2e1 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | [In flight](#in-flight) (`active/`) | 2 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | -| [Backlog](#backlog) (`draft/`) | 155 | +| [Backlog](#backlog) (`draft/`) | 156 | ## Start here @@ -243,7 +243,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**155** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below. +**156** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below.
bug — 34 @@ -760,7 +760,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-research — 17 +research — 18
📋 Deep research: Can we speed up Delaunay in PyAutoArray? — autoarray · too-large · supervised · high @@ -834,6 +834,14 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
+
📋 Properly time and profile the smoke/release script surface — ci · medium · supervised · normal + +``` +/start_dev draft/research/ci/smoke_timing_and_profiling.md +``` + +
+
📋 Is Intel macOS a supported platform, and what is the — libraries · medium · supervised · normal ``` @@ -1336,6 +1344,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| +| 2026-08-23 | filed | Properly time and profile the smoke/release script surface | | 2026-08-23 | issued | Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or… | | 2026-08-23 | filed | Brain board follow-ups: what real mornings surface | | 2026-08-22 | filed | smoke_install.sh's stale jax<0.7 pin — CI is on the right | @@ -1345,12 +1354,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | Remove pynufft + legacy TransformerNUFFTPyNUFFT | | 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | | 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the | -| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| +| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of | | 2026-08-22 | filed | Bug: fix the tracer.fits existence guard in autolens_workspace… | | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-21 | filed | Numba PSF gathers derive the y/x kernel shifts from the | @@ -1360,12 +1369,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit ('tuple'… | | 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | -| 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| +| 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor | | 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | | 2026-08-19 | filed | Release board: local run_logs enrichment | | 2026-08-19 | filed | Refactor Agent witness map lacks PyAutoNerves test suite | @@ -1375,12 +1384,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | -| 2026-08-18 | parked | prior-message-collapse-design |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| +| 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | | 2026-08-17 | filed | Teach repos_sync --write to stamp organ config surfaces | @@ -1390,12 +1399,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-09 | filed | pixel_scales given as an int (or np.float64) is never widened | | 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | -| 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| +| 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them | | 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | | 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are | @@ -1405,7 +1414,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of | | 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | | 2026-08-04 | filed | aplt.Output stale-API drift in the remaining workspace repos | -| 2026-08-04 | filed | Nightly release has been blocked 8 nights running — triage |
diff --git a/draft/research/ci/smoke_timing_and_profiling.md b/draft/research/ci/smoke_timing_and_profiling.md new file mode 100644 index 00000000..66be9353 --- /dev/null +++ b/draft/research/ci/smoke_timing_and_profiling.md @@ -0,0 +1,70 @@ +# Properly time and profile the smoke/release script surface + +Type: research +Target: ci +Repos: +- @autolens_workspace_test +- @autogalaxy_workspace_test +- @PyAutoHeart +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: formalised +Filed: 2026-08-23 + +Filed 2026-08-23 at James's direction, to hold the timing/profiling work that +the `jax-compile-stall` epic's phase 2 kept pulling in. Phase 2 asked one narrow +question — *is this entry slow, or is it stalling?* — and answering it for four +entries was enough to prove the SLOW markers are unreliable. Sweeping the +remaining seventeen one at a time was not the best use of the runner budget, and +the real want is broader than the epic: **a proper timing and profiling picture +of the whole smoke surface**, not an entry-by-entry audit driven by a bug hunt. + +## What is already known (do not re-measure) + +From the phase-2 dispatch, 40 executions at the 300s cap over both Python legs +(`autogalaxy_workspace_test` run 32664679042, `autolens_workspace_test` run +32664682689) — full detail on autolens_workspace_test#271: + +| Entry | Verdict | Evidence | +|---|---|---| +| `interferometer/datacube/shared_preloads.py` | NEITHER | 10/10 completed, worst 34.0s — **1.9%** of the 1800s cap its SLOW marker claims it flakes at | +| `imaging/jax_likelihood/rectangular_mge.py` | STALL | 4/5 capped + one ~22s completion, same split on 3.12 and 3.13 | +| `imaging/jax_likelihood/mge_group.py` | AMBIGUOUS | 5/5 capped, both legs | +| `multi_dataset/jax_likelihood/mge.py` | AMBIGUOUS | 5/5 capped, both legs | + +The headline for this task: **a SLOW marker is not evidence of slowness.** Every +2026-07-14 marker reads "flakes at the 1800s cap" and records no timing at all; +the one entry measured so far was wrong by a factor of ~50. + +## The harness exists + +`.github/workflows/retime.yml` + `.github/scripts/retime.py` in both test +workspaces (merged 2026-08-23), reached through PyAutoHeart's reusable +`smoke-tests.yml` `runner` input. Inputs: `scripts`, `repeats`, +`script-timeout`. Emits per-run timings, a per-script verdict +(STALL/SLOW/NEITHER/AMBIGUOUS/ERROR) and `retime_results.json`. Reuses +`run_smoke.py`'s `run_one`, so it cannot disagree with the PR gate or the +release runner about a script's environment. + +It is a *classifier*, deliberately narrow. This task is where a real profile +belongs. + +## Task + +1. Decide what the useful picture actually is — per-script wall clock across the + whole surface, where the time goes inside a script (import, dataset + simulation, compile, sample), or which entries dominate the mega-run. Those + want different instrumentation; pick before spending runner hours. +2. Re-time the 17 SLOW-marked JAX entries not yet covered, in one batched sweep + rather than one dispatch at a time. +3. Rewrite every marker to carry its measured reason, and delete the ones the + measurement refutes. +4. Consider whether the runner should record per-script timings routinely, so + this is a standing dataset rather than a periodic archaeology exercise. + +## Acceptance + +- Every SLOW marker in both test workspaces carries a measurement, or is gone. +- A stated view on whether routine per-script timing should be collected. +- The Profiling Agent has real numbers instead of the word "flakes". From 68ea3540a5bf875d0c8e82d43cc7fdd970788c70 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 22:07:09 +0000 Subject: [PATCH 10/11] =?UTF-8?q?prompt:=20ordering=20A/B=20result=20?= =?UTF-8?q?=E2=80=94=20contributory,=20not=20causal=20(#271)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Control 8/10 stalls, experiment 3/10, Fisher p=0.070. The stall survives the swap, so jax.vmap(jax.jit(...)) raises the probability rather than causing it. Recorded so nobody reads 80->30 as a fix. Healthy compile of rectangular_mge measured at 3.1s against a >300s stall — 100x bimodality inside one step. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/active.md b/active.md index 8d3b40d1..4551f006 100644 --- a/active.md +++ b/active.md @@ -14,9 +14,17 @@ - branch `experiment/jax-vmap-jit-ordering` in PyAutoFit (swapped, NOT for merge) and a same-named branch in autogalaxy_workspace_test so the workspace CI's matching-branch clone picks the swapped library up. - - dispatched retime.yml from that branch on imaging/jax_likelihood/rectangular_mge.py, - repeats=5, 300s cap. Control: the same entry on main measured 4/5 stalls on BOTH legs - (~80% per compile). A clear drop in stall rate implicates the ordering; no change exonerates it. + - RESULT 2026-08-23 (run 32668184161): control 8/10 stalls (80%), experiment 3/10 (30%), + Fisher exact two-tailed p=0.070. CONTRIBUTORY, NOT CAUSAL — the stall SURVIVES the swap at + 30%, so vmap-of-jit is not necessary for it, and p=0.070 at n=10/arm is suggestive rather + than significant. Do not merge the swap as "the fix"; it may deserve merging on its own + merits (jit(vmap) is conventional and latent.py already uses it) but that is a different + claim needing more samples. + - side-observation worth keeping: on the runs that complete, the #1517 split reports + "traced, lowered and compiled in 3.1s, result materialized in 0.5s". A healthy compile of + this script is ~3.1s and a stalled one exceeds 300s — >100x bimodality inside one step on the + same commit and runner image. The split does NOT localise the stall (it only prints after + both halves finish); only the faulthandler stack can. - experiment 2 — the persistent compile cache (not yet run): JAX_COMPILATION_CACHE_DIR has been on by default since PyAutoConf#128 (2026-07-17); both NEEDS_FIX stalls post-date it, the SLOW batch predates it. A/B with the cache disabled. From 659b988392b4e90f09ca73b46b9e99783a391d54 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 22:29:18 +0000 Subject: [PATCH 11/11] prompt: close the jax-compile-stall epic as partial (#271) Phase 1 shipped in full. Phases 2 and 3 stop at a deliberate point (James's call): the stall is instrumented and characterised but NOT root-caused, and nothing was un-quarantined. The remaining breadth lives in draft/research/ci/smoke_timing_and_profiling.md, where this record is meant to be dug up. Record complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md carries the 60 measured executions, the two live leads (vmap-of-jit contributory at p=0.070 but not causal; the compile-cache A/B never run), the traps, and what was deliberately left undone. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- active.md | 82 ------------------- .../jax-compile-stall-slow-vs-stall-audit.md | 45 ++++++++++ complete/index.md | 3 +- dashboard.html | 19 ++--- dashboard.md | 24 ++---- draft/bug/ci/jax_vmap_jit_compile_stall.md | 9 ++ epics.md | 4 +- 7 files changed, 75 insertions(+), 111 deletions(-) rename active/jax_compile_stall_2_slow_vs_stall_audit.md => complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md (52%) diff --git a/active.md b/active.md index 4551f006..fb25a165 100644 --- a/active.md +++ b/active.md @@ -1,87 +1,5 @@ # Active Tasks -## jax-compile-stall-root-cause -- epic: jax-compile-stall (phase 3 of 3; ledger draft/bug/ci/jax_vmap_jit_compile_stall.md) -- phase-prompt (still in draft/, deliberately not advanced to active/): the phase 3 file - draft/bug/ci/jax_compile_stall_3_root_cause.md has no GitHub issue of its own — the experiments - run under autolens_workspace_test#271 while they are exploratory. Issue it if phase 3 turns into - a shippable fix rather than a hunt. -- status: workspace-dev (experiment in flight) -- started: 2026-08-23 -- experiment 1 — the vmap-of-jit ordering: - - Fitness._vmap builds jax.vmap(jax.jit(self.call)); analysis/latent.py builds the conventional - jax.jit(jax.vmap(...)). The stalling scripts are exactly the vmap-path ones. - - branch `experiment/jax-vmap-jit-ordering` in PyAutoFit (swapped, NOT for merge) and a - same-named branch in autogalaxy_workspace_test so the workspace CI's matching-branch clone - picks the swapped library up. - - RESULT 2026-08-23 (run 32668184161): control 8/10 stalls (80%), experiment 3/10 (30%), - Fisher exact two-tailed p=0.070. CONTRIBUTORY, NOT CAUSAL — the stall SURVIVES the swap at - 30%, so vmap-of-jit is not necessary for it, and p=0.070 at n=10/arm is suggestive rather - than significant. Do not merge the swap as "the fix"; it may deserve merging on its own - merits (jit(vmap) is conventional and latent.py already uses it) but that is a different - claim needing more samples. - - side-observation worth keeping: on the runs that complete, the #1517 split reports - "traced, lowered and compiled in 3.1s, result materialized in 0.5s". A healthy compile of - this script is ~3.1s and a stalled one exceeds 300s — >100x bimodality inside one step on the - same commit and runner image. The split does NOT localise the stall (it only prints after - both halves finish); only the faulthandler stack can. -- experiment 2 — the persistent compile cache (not yet run): JAX_COMPILATION_CACHE_DIR has been on - by default since PyAutoConf#128 (2026-07-17); both NEEDS_FIX stalls post-date it, the SLOW batch - predates it. A/B with the cache disabled. -- in flight from phase 2, kept because they feed phase 3: the 1800s runs on - imaging/jax_likelihood/mge_group.py (ag_test) and multi_dataset/jax_likelihood/mge.py (al_test), - repeats=2 — with PyAutoFit#1518 merged these should dump a faulthandler stack at 1440s, which is - the first look at WHERE the stall sits. -- repos: - - PyAutoFit: experiment/jax-vmap-jit-ordering - - autogalaxy_workspace_test: experiment/jax-vmap-jit-ordering - -## jax-compile-stall-slow-vs-stall-audit -- issue: https://github.com/PyAutoLabs/autolens_workspace_test/issues/271 (issued 2026-08-23) -- issued: 2026-08-23 -- prompt: active/jax_compile_stall_2_slow_vs_stall_audit.md -- status: workspace-dev -- epic: jax-compile-stall (phase 2 of 3; ledger draft/bug/ci/jax_vmap_jit_compile_stall.md) -- worktree: ~/Code/PyAutoLabs-wt/jax-compile-stall-slow-vs-stall-audit -- worktree-note: started in a web-github session with no local tree; no branch cut yet. -- decision: harness lives Heart-side (human choice 2026-08-23) — a `runner` input on the - reusable smoke-tests.yml rather than a per-workspace duplicate of the ceremony. -- prs: ALL MERGED 2026-08-23 - - https://github.com/PyAutoLabs/PyAutoHeart/pull/161 (runner input) — merged - - https://github.com/PyAutoLabs/autolens_workspace_test/pull/272 (retime harness) — merged - - https://github.com/PyAutoLabs/autogalaxy_workspace_test/pull/110 (retime harness) — merged -- harness: `retime.yml` (workflow_dispatch) + `.github/scripts/retime.py` now live in both test - workspaces. Inputs: scripts (comma-separated, relative to scripts/), repeats, script-timeout - (300 = smoke, 1800 = release). Verdicts STALL/SLOW/NEITHER/AMBIGUOUS/ERROR + retime_results.json. -- measured 2026-08-23, first dispatch (4 entries x 5 repeats x 2 legs = 40 executions, 300s cap; - ag_test run 32664679042, al_test run 32664682689) — findings on issue #271: - - `interferometer/datacube/shared_preloads.py` (al_test) NEITHER — 10/10 completed, slowest - 34.0s = 1.9% of the 1800s cap its SLOW marker says it "flakes at". Marker REFUTED; it should - be removed, not rewritten. This is finding 1 of the marker-text analysis, now measured. - - `imaging/jax_likelihood/rectangular_mge.py` (ag_test) STALL — 4/5 capped + one ~22s - completion (7% of cap), the SAME split independently on 3.12 and 3.13. Bimodality measured; - ~80% stall probability per compile, indifferent to Python version. - - `imaging/jax_likelihood/mge_group.py` (ag_test) and `multi_dataset/jax_likelihood/mge.py` - (al_test) AMBIGUOUS — 20 consecutive cap hits, zero completions. Need the 1800s pass. -- phase-1 defect found by this run: the faulthandler dump NEVER fired — its CI default was a flat - 300s and the smoke cap is also 300s, so the runner's SIGKILL beat the dump in all 20 stalled - runs. Heartbeats worked; stacks did not. Fix: PyAutoFit#1518 derives the default from - BUILD_SCRIPT_TIMEOUT at 80% (300s cap -> 240s). Phase 3 needs that stack, so this gates it. -- PyAutoFit#1518 MERGED 2026-08-23 — the dump now fires at 80% of BUILD_SCRIPT_TIMEOUT. -- dispatched 2026-08-23 21:37, the 1800s pass on the two AMBIGUOUS entries, repeats=2 not 5: - ag_test imaging/jax_likelihood/mge_group.py and al_test multi_dataset/jax_likelihood/mge.py. - Two repeats because the deliverable here is the STACK, not more distribution — both entries - already hit the cap 10/10 at 300s, so one completion-or-cap settles the AMBIGUOUS question and - any single stall yields the 1440s traceback. Caps the bill at ~1h/job instead of ~2.5h. -- SCOPE CUT 2026-08-23 (James): stop widening the measurement — the remaining 17 SLOW entries and - the marker rewrites move OUT of this epic into `draft/research/ci/smoke_timing_and_profiling.md`, - a follow-up task for proper smoke timing/profiling. Four entries measured was enough to prove the - SLOW markers unreliable; sweeping the rest one dispatch at a time was not worth the runner budget. - Phase 2 is therefore DONE for the epic's purposes — its question is answered. -- next: phase 3, the root cause. -- heart: NOT consulted — pyauto-heart unreachable from this web session. -- repos: - ## transformed-message-factor-gradient-unpack - issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1501 (issued 2026-08-19) - issued: 2026-08-19 diff --git a/active/jax_compile_stall_2_slow_vs_stall_audit.md b/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md similarity index 52% rename from active/jax_compile_stall_2_slow_vs_stall_audit.md rename to complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md index 73feb768..23e85561 100644 --- a/active/jax_compile_stall_2_slow_vs_stall_audit.md +++ b/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md @@ -1,3 +1,48 @@ +# JAX vmap compile stall — instrumented, measured, partially explained (jax-compile-stall epic) + +- **Issues:** PyAutoFit#1516 (closed) · autolens_workspace_test#271 (closed) · **PRs:** PyAutoFit#1517, PyAutoFit#1518, PyAutoHeart#161, autolens_workspace_test#272, autogalaxy_workspace_test#110 — all merged 2026-08-23 +- **Repos:** PyAutoFit (`non_linear/jax_compile.py`, tests), PyAutoHeart (`.github/workflows/smoke-tests.yml`), autolens_workspace_test + autogalaxy_workspace_test (`.github/scripts/retime.py`, `.github/workflows/retime.yml`) +- **Epic:** `jax-compile-stall`, 3 phases. Phase 1 shipped in full; phases 2 and 3 taken to a deliberate stopping point (James, 2026-08-23) — **the remaining breadth moves to `draft/research/ci/smoke_timing_and_profiling.md`**, where this record is meant to be dug up. +- **Status: CLOSED AS PARTIAL.** The stall is instrumented and characterised but **not root-caused**. Nothing was un-quarantined. + +## What shipped + +1. **A stalled JAX compile now reports itself** (#1517, #1518). `log_on_first_compile` gained a heartbeat (`still compiling , Ns elapsed`), a `faulthandler` watchdog, and separate timings for the trace/lower/compile wait vs the `block_until_ready` execution wait. All four call sites inherit it. No workspace script or CI runner touched. +2. **A re-timing harness** (Heart#161 + the two workspace PRs). `retime.yml` (`workflow_dispatch`: scripts, repeats, script-timeout) → `retime.py`, which reuses `run_smoke.py`'s `run_one` so it cannot disagree with the PR gate about a script's environment. Heart's reusable workflow gained `runner` / `runner-args` / `script-timeout` inputs, all defaulting to prior behaviour, so the ceremony is shared rather than copied. + +## What was measured (60 script executions, all on hosted runners) + +| Entry | Verdict | Evidence | +|---|---|---| +| `interferometer/datacube/shared_preloads.py` (al) | **NEITHER** | 10/10 completed, worst 34.0s = **1.9%** of the 1800s cap its SLOW marker claims it flakes at | +| `imaging/jax_likelihood/rectangular_mge.py` (ag) | **STALL** | 4/5 capped on *both* legs, completions ~22s (7% of cap) | +| `imaging/jax_likelihood/mge_group.py` (ag) | **AMBIGUOUS** | 5/5 capped both legs, no completion | +| `multi_dataset/jax_likelihood/mge.py` (al) | **AMBIGUOUS** | 5/5 capped both legs, though its own marker records 32s standalone | + +## Key findings — the part worth digging up + +- **A SLOW marker is not evidence of slowness.** Every 2026-07-14 marker reads "flakes at the 1800s cap" and records *no timing at all*. The first one measured was wrong by ~50x. Do not trust the remaining 17 without re-measuring. +- **The stall is a >100x bimodality inside one step.** A healthy compile of `rectangular_mge.py` is **3.1s** (trace/lower/compile) + 0.5s (materialize); a stalled one exceeds 300s. Same commit, same runner image. +- **`vmap(jit)` ordering is contributory, not causal.** `Fitness._vmap` builds `jax.vmap(jax.jit(self.call))` while `analysis/latent.py` builds the conventional `jax.jit(jax.vmap(...))`. A/B on `rectangular_mge.py`: control **8/10 stalls (80%)** vs experiment **3/10 (30%)**, Fisher exact two-tailed **p = 0.070**. The stall SURVIVES the swap, so the ordering is not necessary for it, and n=10/arm is not significant. Branch `experiment/jax-vmap-jit-ordering` exists in PyAutoFit and autogalaxy_workspace_test, unmerged, for whoever resumes. +- **Untested hypothesis, still live:** the persistent compilation cache (`JAX_COMPILATION_CACHE_DIR`, default-on since PyAutoConf#128, 2026-07-17). Both NEEDS_FIX stalls post-date it; the SLOW batch predates it. A/B with the cache disabled is the obvious next experiment and was never run. + +## Traps recorded + +- **A watchdog whose threshold equals the cap never fires.** #1517 defaulted the `faulthandler` dump to 300s under CI; the smoke cap is also 300s, so the runner's SIGKILL beat the dump in all 20 stalled runs — heartbeats, no stacks. #1518 derives it from `BUILD_SCRIPT_TIMEOUT` at 80%. Its tests assert the *relationship* (`dump < cap` for every cap), not today's numbers, because nothing in #1517's own tests could have caught a collision between two independently-correct timeouts owned by different repos. +- **A Python traceback during XLA compile parks at the pybind boundary** — it separates *in compile* / *in execution* / *blocked on a Python lock*, but shows no XLA internals. +- **The compile/execute split does not localise a stall.** It only prints once *both* halves finish, so it characterises the healthy case only. Only the stack does. +- **Testing a library change through workspace CI:** the reusable workflow clones the dependency chain at the **matching branch name**, so an identically-named branch in the workspace repo makes its CI pick up an experimental library branch. That is how the ordering A/B was run. +- Diagnostics must never break a fit: unstartable heartbeat thread, unarmable dump, malformed interval all fall back and continue. + +## Left undone, deliberately + +- No root cause. No marker rewritten. **Nothing un-quarantined** — all five stall quarantines and all 21 SLOW markers stand as they were. +- The two 1800s runs dispatched at 21:37 (ag_test run 32668061785, al_test run 32668067325; `mge_group.py` and `multi_dataset/jax_likelihood/mge.py`, 2 repeats) were still in flight at close-out and should carry the first `faulthandler` stack at 1440s. **Read those two runs first when resuming** — they are the only pending evidence. +- **Heart:** never consulted this session — `pyauto-heart` unreachable from the `web-github` environment. Every merge was on an explicit human instruction ("merge when green"). +- Merged branches could not be deleted: this session's git proxy refuses ref deletions. `feature/jax-compile-stall-evidence`, `feature/jax-compile-dump-below-cap` (PyAutoFit), `feature/reusable-smoke-runner-input` (PyAutoHeart), `feature/jax-stall-retime-harness` (both workspaces), plus the two `experiment/jax-vmap-jit-ordering` branches. All need a local `/repo_cleanup`. + +## Original prompt + # Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is this stall wearing a different label? Type: bug diff --git a/complete/index.md b/complete/index.md index f6072563..93cd53ad 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1087 records across 7 buckets. +1088 records across 7 buckets. ## Highlights @@ -98,6 +98,7 @@ _(curate hard-won records here — survives regeneration.)_ - [interferometer-start-here-integrate-oom](2026/08/interferometer-start-here-integrate-oom.md) - [intra-family-dep-floors](2026/08/intra-family-dep-floors.md) - [jax-compile-stall-evidence](2026/08/jax-compile-stall-evidence.md) — jax-compile-stall phase 1 +- [jax-compile-stall-slow-vs-stall-audit](2026/08/jax-compile-stall-slow-vs-stall-audit.md) — jax-compile-stall epic - [jax-default-dependency](2026/08/jax-default-dependency.md) - [jax-grad-local-vs-ci-assertions](2026/08/jax-grad-local-vs-ci-assertions.md) - [jax-grad-smoke-timeout-budget](2026/08/jax-grad-smoke-timeout-budget.md) diff --git a/dashboard.html b/dashboard.html index 97302281..4eb045ae 100644 --- a/dashboard.html +++ b/dashboard.html @@ -51,7 +51,7 @@

📋 PyAutoMind Dashboard

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-

In flight 2 · Parked 3 · Planned 6 · Backlog 156 · markdown version

+

In flight 1 · Parked 3 · Planned 6 · Backlog 156 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

TRIAGE: needs manual review before routing — medium · safe · high

@@ -72,7 +72,6 @@

Quick wins (small enough, and safe enough to run unatte

In flight markdown version

Issued — each has an open GitHub issue and usually a branch.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

-

Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — issue #271 — issued 2026-08-23 — workspace-dev

Parked markdown version

3 task(s) @@ -261,12 +260,6 @@

Backlog 2026-08-23 -issued -Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or… - - - -2026-08-23 filed Brain board follow-ups: what real mornings surface @@ -313,7 +306,7 @@

Backlog Is Intel macOS a supported platform, and what is the - + 2026-08-22 filed Defer the eager scipy.sparse import in derivative_util (~0.10 s of @@ -553,6 +546,12 @@

Backlog aplt.Output stale-API drift in the remaining workspace repos + +2026-08-04 +filed +Nightly release has been blocked 8 nights running — triage + +

Epics markdown version

@@ -578,7 +577,7 @@

Epics

Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md — phase 1 shipped 2026-08-23 (PyAutoFit#1517); phase 2 in flight (autolens_workspace_test#271)

+

Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md — CLOSED AS PARTIAL 2026-08-23 — record complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md

Phase 3: root-cause the XLA vmap compile stall and clear — ci · large · supervised · high

Intermittent XLA compile stall in JAX vmap likelihood scripts — — ci · too-large · supervised · high

diff --git a/dashboard.md b/dashboard.md index 06d8a2e1..fb171b89 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,7 +8,7 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 2 | +| [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | | [Backlog](#backlog) (`draft/`) | 156 | @@ -143,14 +143,6 @@ Issued — each has an open GitHub issue and usually a branch. The full record f

-
📋 Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is — issue #271 — issued 2026-08-23 — workspace-dev - -``` -/start_dev active/jax_compile_stall_2_slow_vs_stall_audit.md -``` - -
- ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -1345,7 +1337,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| | 2026-08-23 | filed | Properly time and profile the smoke/release script surface | -| 2026-08-23 | issued | Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or… | | 2026-08-23 | filed | Brain board follow-ups: what real mornings surface | | 2026-08-22 | filed | smoke_install.sh's stale jax<0.7 pin — CI is on the right | | 2026-08-22 | filed | multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA… | @@ -1354,12 +1345,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | Remove pynufft + legacy TransformerNUFFTPyNUFFT | | 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | | 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the | +| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| -| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of | | 2026-08-22 | filed | Bug: fix the tracer.fits existence guard in autolens_workspace… | | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-21 | filed | Numba PSF gathers derive the y/x kernel shifts from the | @@ -1369,12 +1360,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit ('tuple'… | | 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | | 2026-08-19 | filed | autofit_profiling: bootstrap the repo + general PyAutoFit profiling… | +| 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| -| 2026-08-19 | filed | autoreduce 0.9 on PyPI never got the Python 3.12 floor | | 2026-08-19 | issued | @PyAutoFit TransformedMessage.factor_gradient crashes on first… | | 2026-08-19 | filed | Release board: local run_logs enrichment | | 2026-08-19 | filed | Refactor Agent witness map lacks PyAutoNerves test suite | @@ -1384,12 +1375,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | | 2026-08-18 | parked | single-source-density-design | +| 2026-08-18 | parked | prior-message-collapse-design |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| -| 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | | 2026-08-17 | filed | Teach repos_sync --write to stamp organ config surfaces | @@ -1399,12 +1390,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-09 | filed | pixel_scales given as an int (or np.float64) is never widened | | 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | +| 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| -| 2026-08-07 | filed | autofit.plot functions accept **kwargs and silently discard them | | 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | | 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are | @@ -1414,6 +1405,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of | | 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | | 2026-08-04 | filed | aplt.Output stale-API drift in the remaining workspace repos | +| 2026-08-04 | filed | Nightly release has been blocked 8 nights running — triage |
@@ -1565,10 +1557,10 @@ Continue the 'Cluster strong lensing — Source & Cluster arc' epic. Its canonic
Intermittent XLA compile stall in the JAX vmap likelihood path — 2 queued prompt(s), in order -
📋 Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: `draft/bug/ci/jax_vmap_jit_compile_stall.md` — phase 1 shipped 2026-08-23 (PyAutoFit#1517); phase 2 in flight (autolens_workspace_test#271) +
📋 Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: `draft/bug/ci/jax_vmap_jit_compile_stall.md` — CLOSED AS PARTIAL 2026-08-23 — record complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md ``` -Continue the 'Intermittent XLA compile stall in the JAX vmap likelihood path' epic. Its canonical state lives in draft/bug/ci/jax_vmap_jit_compile_stall.md — read that ledger (and any DECISIONS/RESULTS files beside it) first. Cross-check this epic's entry in PyAutoMind/epics.md, any related rows in PyAutoMind/active.md, and the referenced repos' open issues and PRs, to work out the last completed phase and what is currently in flight. Then pick the next logical step and continue it through the normal workflow (/start_dev — filing the phase's prompt first if none exists), updating the ledger as the work advances. Note: three phases under draft/bug/ci/ — 1 evidence (PyAutoFit watchdog, SHIPPED), 2 SLOW-vs-stall audit, 3 root cause + un-quarantine. Phase 3 is blocked on phase 1 shipping and a CI stall actually dumping a traceback; issue phases ONE at a time. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. +Continue the 'Intermittent XLA compile stall in the JAX vmap likelihood path' epic. Its canonical state lives in draft/bug/ci/jax_vmap_jit_compile_stall.md — read that ledger (and any DECISIONS/RESULTS files beside it) first. Cross-check this epic's entry in PyAutoMind/epics.md, any related rows in PyAutoMind/active.md, and the referenced repos' open issues and PRs, to work out the last completed phase and what is currently in flight. Then pick the next logical step and continue it through the normal workflow (/start_dev — filing the phase's prompt first if none exists), updating the ledger as the work advances. Note: phase 1 (watchdog) shipped in full; phases 2/3 stopped deliberately at a measured-but-not-root-caused state. The stall is instrumented and characterised (>100x bimodality inside one compile step; vmap-of-jit contributory at p=0.070 but NOT causal; the compile-cache hypothesis never tested) and NOTHING was un-quarantined. Resume via draft/research/ci/smoke_timing_and_profiling.md, which is where this gets dug up. Superseded draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. ```
diff --git a/draft/bug/ci/jax_vmap_jit_compile_stall.md b/draft/bug/ci/jax_vmap_jit_compile_stall.md index 47a4eb0a..8b99dc02 100644 --- a/draft/bug/ci/jax_vmap_jit_compile_stall.md +++ b/draft/bug/ci/jax_vmap_jit_compile_stall.md @@ -137,6 +137,15 @@ what made this diagnosable at all. - The SLOW-vs-stall question in step 1 answered in writing, so the Profiling Agent is not chasing speedups on scripts that are hanging. +## CLOSED AS PARTIAL — 2026-08-23 + +Phase 1 shipped. Phases 2 and 3 were taken to a deliberate stopping point and the +epic closed: the stall is **instrumented and characterised but not root-caused**, +and nothing was un-quarantined. The full account, including the two live leads and +every trap, is the record +[`complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md`](../../../complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md). +Resume through [`../../research/ci/smoke_timing_and_profiling.md`](../../research/ci/smoke_timing_and_profiling.md). + ## Phase map (added 2026-08-23 at /start_dev) The Bug Agent sizes this `too-large` and returns `split-into-phases`. It is one diff --git a/epics.md b/epics.md index fc287b31..b23d86f6 100644 --- a/epics.md +++ b/epics.md @@ -31,8 +31,8 @@ epic, never picked standalone. ## jax-compile-stall - title: Intermittent XLA compile stall in the JAX vmap likelihood path - ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md -- status: phase 1 shipped 2026-08-23 (PyAutoFit#1517); phase 2 in flight (autolens_workspace_test#271) -- notes: three phases under draft/bug/ci/ — 1 evidence (PyAutoFit watchdog, SHIPPED), 2 SLOW-vs-stall audit, 3 root cause + un-quarantine. Phase 3 is blocked on phase 1 shipping and a CI stall actually dumping a traceback; issue phases ONE at a time. Supersedes draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. +- status: CLOSED AS PARTIAL 2026-08-23 — record complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md +- notes: phase 1 (watchdog) shipped in full; phases 2/3 stopped deliberately at a measured-but-not-root-caused state. The stall is instrumented and characterised (>100x bimodality inside one compile step; vmap-of-jit contributory at p=0.070 but NOT causal; the compile-cache hypothesis never tested) and NOTHING was un-quarantined. Resume via draft/research/ci/smoke_timing_and_profiling.md, which is where this gets dug up. Superseded draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md. ## graphical-ep - title: Expectation propagation (EP) campaign