From cbc2cb11fc91b3449e9a61f20ca634156b3ee1e0 Mon Sep 17 00:00:00 2001 From: James Nightingale Date: Sun, 23 Aug 2026 23:09:32 +0000 Subject: [PATCH 1/2] prompt: the captured stack says this is not a compile stall MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two 1800s runs dumped faulthandler tracebacks at 1440s. Both repos, both scripts, the same stack: parked in jax.block_until_ready / try_to_block — the EXECUTION half of the first call. func(*args, **kwargs) returned; what never returns is the wait for the result to materialize. So the epic's name, and every marker calling this an intermittent XLA compile stall, inherit a guess made before there was evidence. Corrected in the record, the epic entry, the ledger and the follow-up task so the smoke speed-up work does not restart from the wrong question. Also recorded: the heartbeat says 'still compiling' while the process sits in block_until_ready — log_on_first_compile cannot tell which half it is in. Fix the wording before anyone reads another of these logs. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K --- .../jax-compile-stall-slow-vs-stall-audit.md | 65 +++ dashboard.html | 500 ++++++++---------- dashboard.md | 102 ++-- draft/bug/ci/jax_vmap_jit_compile_stall.md | 9 + .../research/ci/smoke_timing_and_profiling.md | 8 + epics.md | 3 + 6 files changed, 349 insertions(+), 338 deletions(-) diff --git a/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md b/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md index 23e85561..b52b7b21 100644 --- a/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md +++ b/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md @@ -175,3 +175,68 @@ imply the API one is resolved. chasing speedups on scripts that are hanging. - No script is un-quarantined by this phase — restoring coverage is phase 3, and depends on the fix. + +## The stack, captured after close-out — it is NOT a compile stall + +The two 1800s runs named above finished at 22:38 and both dumped a +`faulthandler` traceback at 1440s, exactly as PyAutoFit#1518 intended. **The +stack contradicts the premise this whole epic was filed under.** + +`autogalaxy_workspace_test` run 32668061785, `imaging/jax_likelihood/mge_group.py`: + +``` +Thread 0x00007fbf1f0e4b80 (most recent call first): + File ".../jax/_src/api.py", line 2764 in try_to_block + File ".../jax/_src/api.py", line 2781 in block_until_ready + File ".../autofit/non_linear/jax_compile.py", line 264 in wrapper + File ".../scripts/imaging/jax_likelihood/mge_group.py", line 175 in +``` + +`autolens_workspace_test` run 32668067325, `multi_dataset/jax_likelihood/mge.py` +— different repo, different script, **same stack**: + +``` + File ".../jax/_src/api.py", line 2781 in block_until_ready + File ".../autofit/non_linear/jax_compile.py", line 264 in wrapper + File ".../scripts/multi_dataset/jax_likelihood/mge.py", line 150 in +``` + +`jax_compile.py:264` is the `jax.block_until_ready(result)` call — the **second** +half of the wrapped first call. The process is not tracing, lowering or +compiling. `func(*args, **kwargs)` **returned**; what never returns is the wait +for the result to materialize. + +### What this overturns + +- Every marker, issue and prompt in this epic calls it an *"intermittent XLA + compile stall"*. On this evidence that name is **wrong**, and it has been + wrong since the first quarantine on 2026-08-01. Anyone resuming should treat + "compile stall" as a label inherited from a guess, not a finding. +- The heartbeat line reads `JAX jit still compiling ... 1770s elapsed` while the + process sits in `block_until_ready`. That wording is a defect in + `log_on_first_compile` — the wrapper cannot tell which half it is in and says + "compiling" regardless. **Fix the message to name the phase before anyone + reads another one of these logs.** +- Both entries are `AMBIGUOUS` even at the 1800s cap: 2/2 runs capped on both + legs, no completion. Combined with the 300s round, that is **20 consecutive + cap hits** for `mge_group.py` and `multi_dataset/mge.py` with zero + completions — while `mge.py`'s own marker records it finishing in 32s + standalone. + +### Where this points instead + +An execution/materialization hang, not a compiler one. The hypotheses worth +testing are now different from the ones this epic queued up: + +1. **Device transfer or the async dispatch queue deadlocking** on the runner — + `try_to_block` is where a never-arriving buffer would park. +2. **The persistent compilation cache** is still live but for a different + reason than assumed: a cache *read* satisfying the compile instantly and + leaving execution to hang is consistent with what is seen here. +3. **The `vmap(jit)` ordering** result (80% -> 30%, p=0.070) needs reinterpreting + in this light: the ordering changes the *shape of the executed computation*, + not just the compile graph, which is a more plausible route to an execution + hang than to a compile one. + +Do not restart from "why is XLA slow to compile". Start from "why does +`block_until_ready` never return". diff --git a/dashboard.html b/dashboard.html index 82316565..92468288 100644 --- a/dashboard.html +++ b/dashboard.html @@ -6,142 +6,69 @@ PyAutoMind Dashboard -
📋

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-
  • 1In flight
  • 3Parked
  • 6Planned
  • 156Backlog
-

markdown version

+

📋 PyAutoMind Dashboard

+

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

+

In flight 1 · Parked 3 · Planned 6 · Backlog 156 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

-

TRIAGE: needs manual review before routing❓ triagemediumsafehigh

-

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

-

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU)✨ featureautoarraymediumsupervisedhigh

-

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88% lever)✨ featureautoarraylargesupervisedhigh

-

multi_galaxy package: new regime package in autolens_workspace📖 docsautolenslargesupervisedhigh

-

Profile and speed up JAX likelihood-function compile times (all use cases)✨ featureautolens_profilinglargesupervisedhigh

-

Give the Profiling Agent a compile-time axis — the arc✨ featureprofilinglargesupervisedhigh

-

Deep research: Can we speed up Delaunay in PyAutoArray?🔬 researchautoarraytoo-largesupervisedhigh

-

Census of priors and messages — confirmed bugs + redesign ideas🔬 researchautofittoo-largesupervisedhigh

-

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder♻️ refactorautogalaxytoo-largesupervisedhigh

-

Split lensing regimes: multi_galaxy / group / cluster (epic plan)📖 docsautolenstoo-largesupervisedhigh

-

Re-baseline the MGE imaging JIT profiling regression value🧪 testautolens_workspace_developertoo-largesupervisedhigh

+

TRIAGE: needs manual review before routing — medium · safe · high

+

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high

+

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high

+

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high

+

multi_galaxy package: new regime package in autolens_workspace — autolens · large · supervised · high

+

Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high

+

Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high

+

Deep research: Can we speed up Delaunay in PyAutoArray? — autoarray · too-large · supervised · high

+

Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high

+

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder — autogalaxy · too-large · supervised · high

+

Split lensing regimes: multi_galaxy / group / cluster (epic plan) — autolens · too-large · supervised · high

+

Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high

Quick wins (small enough, and safe enough to run unattended)

-

Defer the eager scipy.sparse import in derivative_util (~0.10 s of import)🧹 maintenancelibrariessmallsafenormal

-

Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py🐛 bugworkspacessmallsafenormal

+

Defer the eager scipy.sparse import in derivative_util (~0.10 s of — libraries · small · safe · normal

+

Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py — workspaces · small · safe · normal

In flight markdown version

Issued — each has an open GitHub issue and usually a branch.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

@@ -166,161 +93,161 @@

Backlog 156 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

feature — 29 -

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

-

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU)✨ featureautoarraymediumsupervisedhigh

-

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88% lever)✨ featureautoarraylargesupervisedhigh

-

Profile and speed up JAX likelihood-function compile times (all use cases)✨ featureautolens_profilinglargesupervisedhigh

-

Give the Profiling Agent a compile-time axis — the arc✨ featureprofilinglargesupervisedhigh

-

Which other searches need prior-support handling — coverage audit after Prodigy✨ featureautofitmediumsupervisedmedium

-

Give PyAutoFit searches a seed — today no search can be made…✨ featureautofitmediumsupervisedmedium

-

Brain board follow-ups: what real mornings surface✨ featurepyautobrainsmallsupervisednormal

-

Can create a list of InversionMatrix objects for each dataset✨ featureautoarraymediumsupervisednormal

-

Tune cluster-scale JOSS benchmarks toward their 5-minute targets✨ featureautolens_workspacemediumsupervisednormal

-

Token-light wiki index over the complete/ archive✨ featurepyautomindmediumsupervisednormal

-

The imaging features/advanced/los_halos example needs improving and padding out before✨ featureworkspacesmediumsafenormal

-

The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before✨ featureworkspacesmediumsafenormal

-

Claude Development Prompt: Arcsecond Tick Label Decimal Placement✨ featureautoarraylargesupervisednormal

-

EP analytic updates — implement the four planned work packages✨ featureautofitlargesupervisednormal

-

Remote-MCP deployment tiers (2 + 3) for the results-inspector server✨ featureautofit_assistantlargehuman-requirednormal

-

Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch)✨ featureautolens_profilinglargesupervisednormal

-

Adopt oversampled PSFs in the start-here dataset chain (option a)✨ featureautolens_workspacelargesupervisednormal

-

Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A✨ featureautoarraytoo-largesupervisednormal

-

PIEMass.potential_2d_from: implement the missing lensing potential✨ featureautogalaxytoo-largesupervisednormal

-

autolens_jax_joss benchmark repo + real-data start_here pairing✨ featureautolens_jax_josstoo-largesupervisednormal

-

Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the✨ featurejax_substructuretoo-largesupervisednormal

-

Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the✨ featurejax_substructuretoo-largesupervisednormal

-

Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver✨ featureworkspacestoo-largesupervisednormal

-

dPIE: optional central-dispersion (sigma_0) parameterization✨ featureautogalaxysmallsupervisedlow

-

Release board: local run_logs enrichment✨ featurepyautohandssmallsupervisedlow

-

Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier✨ featureworkspacessmallsupervisedlow

-

Scheduled runs — overnight queue passes with a morning report✨ featureautonomymediumsupervisedlow

-

Teach repos_sync --write to stamp organ config surfaces✨ featurepyautomindhardsupervisedlow

+

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high

+

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high

+

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high

+

Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high

+

Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high

+

Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium

+

Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium

+

Brain board follow-ups: what real mornings surface — pyautobrain · small · supervised · normal

+

Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal

+

Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal

+

Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal

+

The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal

+

The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal

+

Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal

+

EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal

+

Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal

+

Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal

+

Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal

+

Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal

+

PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal

+

autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal

+

Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal

+

Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal

+

Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal

+

dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low

+

Release board: local run_logs enrichment — pyautohands · small · supervised · low

+

Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low

+

Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low

+

Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low

bug — 32 -

Fix release JAX runtime compatibility and likelihood parity🐛 bughealth_fixestoo-largesupervisedhigh

-

Fix JIT quick-update visualization output regressions🐛 bughealth_fixestoo-largesupervisedhigh

-

Fix release result/sample parameter-path regressions🐛 bughealth_fixestoo-largesupervisedhigh

-

jax 0.11 breaks beta/gamma message log_partition under jit…🐛 bugautofitsmallsupervisedmedium

-

Heart script_timing baselines are orphaned by path moves and filled…🐛 bugpyautoheartsmallsupervisedmedium

-

Numba PSF gathers derive the y/x kernel shifts from the wrong kernel…🐛 bugautoarraylowsupervisedmedium

-

LogGaussianPrior misreports its own support as (-inf, inf)🐛 bugautofitsmallsupervisednormal

-

autofit.plot functions accept **kwargs and silently discard them🐛 bugautofitsmallsupervisednormal

-

TEST_MODE bypass crashes on ordered-parameter assertion ties🐛 bugautofitsmallsupervisednormal

-

point.py JAX-vmap parity assert is non-deterministic under the smoke env🐛 bugautolenssmallsupervisednormal

-

Scripts derive geometry from a hardcoded pixel_scale while the dataset carries…🐛 bugautolens_workspacesmallsupervisednormal

-

HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX)🐛 bughowtogalaxysmallsupervisednormal

-

PyAutoConf rename leftovers in Brain functional surfaces🐛 bugpyautobrainsmallsupervisednormal

-

generate.py deletes notebooks/ before rejecting an unknown project🐛 bugpyautohandssmallsupervisednormal

-

autoreduce 0.9 on PyPI never got the Python 3.12 floor🐛 bugpyautoreducesmallsupervisednormal

-

aplt.Output stale-API drift in the remaining workspace repos🐛 bugworkspacessmallsupervisednormal

-

Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py🐛 bugworkspacessmallsafenormal

-

Three jax_likelihood pins are stale by ~1.24e-4 and fail the smoke gate…🐛 bugworkspacessmallsupervisednormal

-

PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional?🐛 bugautoarraymediumsupervisednormal

-

interferometer Delaunay pixelization — non-PD FitException in test-mode bypass🐛 bugautolensmediumsupervisednormal

-

JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38🐛 bugautolensmediumsupervisednormal

-

JIT cache not hit in modeling_visualization delaunay/rectangular scripts🐛 bugautolensmediumsupervisednormal

-

multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall)🐛 bugautolens_workspace_testmediumsupervisednormal

-

@PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian🐛 bugpriorsmediumsupervisednormal

-

Resolve release-profile timeout scripts deliberately🐛 bughealth_fixestoo-largesupervisednormal

-

@PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms🐛 bugpriorstoo-largesupervisednormal

-

Priors & Messages cleanup — tracker🐛 bugpriorstoo-largesupervisednormal

-

autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main🐛 bugautolenslowsupervisednormal

-

status.sh --repos sources a file that no longer exists🐛 bugpyautomindsmallsupervisedlow

-

The reconstruction noise map describes a different estimator than the default solver🐛 bugautoarraymediumhuman-requiredlow

-

Point-source JSON datasets record no resolution regime🐛 bugpyautolensmediumsupervisedlow

-

interferometer/start_here.py OOM in nightly release-validation integrate leg🐛 bugautolens

+

Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high

+

Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high

+

Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high

+

jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium

+

Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium

+

Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium

+

LogGaussianPrior misreports its own support as (-inf, inf) — autofit · small · supervised · normal

+

autofit.plot functions accept **kwargs and silently discard them — autofit · small · supervised · normal

+

TEST_MODE bypass crashes on ordered-parameter assertion ties — autofit · small · supervised · normal

+

point.py JAX-vmap parity assert is non-deterministic under the smoke env — autolens · small · supervised · normal

+

Scripts derive geometry from a hardcoded pixel_scale while the dataset — autolens_workspace · small · supervised · normal

+

HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX) — howtogalaxy · small · supervised · normal

+

PyAutoConf rename leftovers in Brain functional surfaces — pyautobrain · small · supervised · normal

+

generate.py deletes notebooks/ before rejecting an unknown project — pyautohands · small · supervised · normal

+

autoreduce 0.9 on PyPI never got the Python 3.12 floor — pyautoreduce · small · supervised · normal

+

aplt.Output stale-API drift in the remaining workspace repos — workspaces · small · supervised · normal

+

Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py — workspaces · small · safe · normal

+

Three jax_likelihood pins are stale by ~1.24e-4 and fail the — workspaces · small · supervised · normal

+

PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional? — autoarray · medium · supervised · normal

+

interferometer Delaunay pixelization — non-PD FitException in test-mode bypass — autolens · medium · supervised · normal

+

JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38 — autolens · medium · supervised · normal

+

JIT cache not hit in modeling_visualization delaunay/rectangular scripts — autolens · medium · supervised · normal

+

multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall) — autolens_workspace_test · medium · supervised · normal

+

@PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian — priors · medium · supervised · normal

+

Resolve release-profile timeout scripts deliberately — health_fixes · too-large · supervised · normal

+

@PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms — priors · too-large · supervised · normal

+

Priors & Messages cleanup — tracker — priors · too-large · supervised · normal

+

autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main — autolens · low · supervised · normal

+

status.sh --repos sources a file that no longer exists — pyautomind · small · supervised · low

+

The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · low

+

Point-source JSON datasets record no resolution regime — pyautolens · medium · supervised · low

+

interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens

research — 18 -

Deep research: Can we speed up Delaunay in PyAutoArray?🔬 researchautoarraytoo-largesupervisedhigh

-

Census of priors and messages — confirmed bugs + redesign ideas🔬 researchautofittoo-largesupervisedhigh

-

Delaunay-family JAX modules never hit the persistent compilation cache🔬 researchautoarraymediumsupervisedmedium

-

Quick-update plotting cost — minutes per update, and it is not JAX…🔬 researchautolensmediumsupervisedmedium

-

Use readthedocs or migrate to GitHub docs🔬 researchautobuildsmallsupervisednormal

-

Explore: dashboardify the Brain's operational surfaces with pasteable conductor prompts🔬 researchpyautobrainsmallsupervisednormal

-

Re-baseline the slacs0008 acceptance parity after the HAP-dedupe fix🔬 researchpyautoreducesmallsupervisednormal

-

Kernel-CDF bandwidth defaults — config-dependent quality, investigate adaptivity🔬 researchautoarraymediumsupervisednormal

-

We have lots of examples which profile how long JAX🔬 researchautolens_workspace_developermediumsupervisednormal

-

Properly time and profile the smoke/release script surface🔬 researchcimediumsupervisednormal

-

Is Intel macOS a supported platform, and what is the numpy-only contract?🔬 researchlibrariesmediumsupervisednormal

-

Checkerboard PSF-mismatch residual diagnostic — research + document + ingest papers🔬 researchpyautomemorymediumsupervisednormal

-

autofit_profiling: bootstrap the repo + general PyAutoFit profiling epic🔬 researchautofitlargesupervisednormal

-

Multi-band compile census completion — A100/multi-core + hetero GPU rows🔬 researchautolens_profilingsmallsupervisedlow

-

Chase the ~6% flux scale between PyAutoReduce and legacy SLACS reductions🔬 researchpyautoreducemediumsupervisedlow

-

PyAutoArray Delaunay interpolator's pure_callback vs vmap — minor efficiency follow-up🔬 researchautoarraytoo-largesupervisedlow

-

Cluster-scale gradient-search benchmark (Prodigy vs Nautilus, point-source)🔬 researchautolens_profiling

-

Adopt Python 3.12 as the PyAuto ecosystem minimum🔬 researchlibraries

+

Deep research: Can we speed up Delaunay in PyAutoArray? — autoarray · too-large · supervised · high

+

Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high

+

Delaunay-family JAX modules never hit the persistent compilation cache — autoarray · medium · supervised · medium

+

Quick-update plotting cost — minutes per update, and it is — autolens · medium · supervised · medium

+

Use readthedocs or migrate to GitHub docs — autobuild · small · supervised · normal

+

Explore: dashboardify the Brain's operational surfaces with pasteable conductor prompts — pyautobrain · small · supervised · normal

+

Re-baseline the slacs0008 acceptance parity after the HAP-dedupe fix — pyautoreduce · small · supervised · normal

+

Kernel-CDF bandwidth defaults — config-dependent quality, investigate adaptivity — autoarray · medium · supervised · normal

+

We have lots of examples which profile how long JAX — autolens_workspace_developer · medium · supervised · normal

+

Properly time and profile the smoke/release script surface — ci · medium · supervised · normal

+

Is Intel macOS a supported platform, and what is the — libraries · medium · supervised · normal

+

Checkerboard PSF-mismatch residual diagnostic — research + document + ingest — pyautomemory · medium · supervised · normal

+

autofit_profiling: bootstrap the repo + general PyAutoFit profiling epic — autofit · large · supervised · normal

+

Multi-band compile census completion — A100/multi-core + hetero GPU rows — autolens_profiling · small · supervised · low

+

Chase the ~6% flux scale between PyAutoReduce and legacy SLACS — pyautoreduce · medium · supervised · low

+

PyAutoArray Delaunay interpolator's pure_callback vs vmap — minor efficiency follow-up — autoarray · too-large · supervised · low

+

Cluster-scale gradient-search benchmark (Prodigy vs Nautilus, point-source) — autolens_profiling

+

Adopt Python 3.12 as the PyAuto ecosystem minimum — libraries

maintenance — 23 -

Untrack the generated FITS test artifacts in autoarray🧹 maintenancelibrariessmallsupervisedmedium

-

smoke_install.sh's stale jax<0.7 pin — CI is on the right jax…🧹 maintenancecilowsupervisedmedium

-

autolens_workspace_developer rectangular experiments — Gut stash + rename🧹 maintenanceautolens_workspace_developersmallsupervisednormal

-

Defer the eager scipy.sparse import in derivative_util (~0.10 s of import)🧹 maintenancelibrariessmallsafenormal

-

Mirror drifted library config keys into the workspace configs🧹 maintenanceworkspacessmallsupervisednormal

-

Un-park imaging/features/scaling_relation/slam once PyAutoArray#431 merges🧹 maintenanceworkspacessmallsupervisednormal

-

autolens_workspace_developer: broad stale-API rot (56 symbols, no CI)🧹 maintenanceautolens_workspace_developermediumsupervisednormal

-

run_smoke.py: three runner variants across 10 repos, no sync mechanism🧹 maintenancecimediumsupervisednormal

-

Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two dead deps🧹 maintenancelibrariesmediumsupervisednormal

-

PyAutoMemory canonical-key TODO sweep🧹 maintenancepyautomemorymediumsupervisednormal

-

Single-source the "Never rewrite history" policy as a generated AGENTS.md block🧹 maintenancepyautomindmediumsupervisednormal

-

Capped smoke datasets were committed as if they were real🧹 maintenanceworkspacesmediumsupervisednormal

-

autolens_profiling is now a mature project, with a good separation🧹 maintenanceautolens_profilinglargesupervisednormal

-

Auto-request GitHub Copilot code review on every PR, org-wide🧹 maintenancecilargesupervisednormal

-

autolens_workspace🧹 maintenanceworkspacestoo-largesupervisednormal

-

Remove pynufft + legacy TransformerNUFFTPyNUFFT🧹 maintenancelibrarieslow-mediumsupervisednormal

-

pynufft removal: unswept downstream residue (1 hard break + stale docs/CI)🧹 maintenanceworkspaceslow-mediumsupervisednormal

-

Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI install docs🧹 maintenanceworkspaceslowsupervisednormal

-

dataset/imaging/jwst_lw is untracked because the gitignore was never extended for it🧹 maintenanceautolens_profilingsmallsupervisedlow

-

cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of the repo's…🧹 maintenanceautolens_workspacesmallsupervisedlow

-

LaTeX in non-raw docstrings emits SyntaxWarning: invalid escape sequence🧹 maintenanceautolens_workspacesmallsupervisedlow

-

Regenerate setup_notebook-drifted notebooks in autogalaxy/autofit/HowToFit workspaces🧹 maintenanceworkspacessmallsupervisedlow

-

Refactor Agent witness map lacks PyAutoNerves test suite🧹 maintenancepyautobrainlowsafelow

+

Untrack the generated FITS test artifacts in autoarray — libraries · small · supervised · medium

+

smoke_install.sh's stale jax<0.7 pin — CI is on the right — ci · low · supervised · medium

+

autolens_workspace_developer rectangular experiments — Gut stash + rename — autolens_workspace_developer · small · supervised · normal

+

Defer the eager scipy.sparse import in derivative_util (~0.10 s of — libraries · small · safe · normal

+

Mirror drifted library config keys into the workspace configs — workspaces · small · supervised · normal

+

Un-park imaging/features/scaling_relation/slam once PyAutoArray#431 merges — workspaces · small · supervised · normal

+

autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) — autolens_workspace_developer · medium · supervised · normal

+

run_smoke.py: three runner variants across 10 repos, no sync mechanism — ci · medium · supervised · normal

+

Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two dead — libraries · medium · supervised · normal

+

PyAutoMemory canonical-key TODO sweep — pyautomemory · medium · supervised · normal

+

Single-source the "Never rewrite history" policy as a generated AGENTS.md — pyautomind · medium · supervised · normal

+

Capped smoke datasets were committed as if they were real — workspaces · medium · supervised · normal

+

autolens_profiling is now a mature project, with a good separation — autolens_profiling · large · supervised · normal

+

Auto-request GitHub Copilot code review on every PR, org-wide — ci · large · supervised · normal

+

autolens_workspace — workspaces · too-large · supervised · normal

+

Remove pynufft + legacy TransformerNUFFTPyNUFFT — libraries · low-medium · supervised · normal

+

pynufft removal: unswept downstream residue (1 hard break + stale — workspaces · low-medium · supervised · normal

+

Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI — workspaces · low · supervised · normal

+

dataset/imaging/jwst_lw is untracked because the gitignore was never extended for — autolens_profiling · small · supervised · low

+

cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of — autolens_workspace · small · supervised · low

+

LaTeX in non-raw docstrings emits SyntaxWarning: invalid escape sequence — autolens_workspace · small · supervised · low

+

Regenerate setup_notebook-drifted notebooks in autogalaxy/autofit/HowToFit workspaces — workspaces · small · supervised · low

+

Refactor Agent witness map lacks PyAutoNerves test suite — pyautobrain · low · safe · low

docs — 14 -

multi_galaxy package: new regime package in autolens_workspace📖 docsautolenslargesupervisedhigh

-

Split lensing regimes: multi_galaxy / group / cluster (epic plan)📖 docsautolenstoo-largesupervisedhigh

-

Advanced workspace guide: Preloads (PyAutoArray)📖 docsworkspacestoo-largesupervisedhigh

-

Regenerate autolens_workspace markdown/ so the MGE pages show sigma_min📖 docsautolens_workspacesmallsupervisednormal

-

RTD organism docs currency: Nerves page, organ-count drift, hands.md rename📖 docspyautobrainsmallsupervisednormal

-

Propagate the shear_galaxy-at-(0,0) idiom to group/ and cluster/📖 docsworkspacessmallsupervisednormal

-

Rectangular mesh Enzi citation — user-workspace pixelization examples📖 docsworkspacessmallsupervisednormal

-

Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead📖 docsautoctimediumsupervisednormal

-

extra_galaxies feature parity: point_source + multi_galaxy (both workspaces)📖 docsworkspacesmediumsupervisednormal

-

HowToLens ch4 tutorial 3: mask overlay is never actually drawn📖 docshowtolenssmallsupervisedlow

-

Assistants: regime-aware routing for multi_galaxy / group / cluster (follow-up)📖 docsworkspacesmediumsupervisedlow

-

Phase 2 — drop the hand-written quick-update sentence from the workspace scripts📖 docsautolens_workspace

-

plot coverage — follow-ups deferred from plot-coverage-gaps📖 docsworkspaces

-

Phase 2: Make workspace READMEs assistant-first📖 docsworkspaces

+

multi_galaxy package: new regime package in autolens_workspace — autolens · large · supervised · high

+

Split lensing regimes: multi_galaxy / group / cluster (epic plan) — autolens · too-large · supervised · high

+

Advanced workspace guide: Preloads (PyAutoArray) — workspaces · too-large · supervised · high

+

Regenerate autolens_workspace markdown/ so the MGE pages show sigma_min — autolens_workspace · small · supervised · normal

+

RTD organism docs currency: Nerves page, organ-count drift, hands.md rename — pyautobrain · small · supervised · normal

+

Propagate the shear_galaxy-at-(0,0) idiom to group/ and cluster/ — workspaces · small · supervised · normal

+

Rectangular mesh Enzi citation — user-workspace pixelization examples — workspaces · small · supervised · normal

+

Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are — autocti · medium · supervised · normal

+

extra_galaxies feature parity: point_source + multi_galaxy (both workspaces) — workspaces · medium · supervised · normal

+

HowToLens ch4 tutorial 3: mask overlay is never actually drawn — howtolens · small · supervised · low

+

Assistants: regime-aware routing for multi_galaxy / group / cluster (follow-up) — workspaces · medium · supervised · low

+

Phase 2 — drop the hand-written quick-update sentence from the — autolens_workspace

+

plot coverage — follow-ups deferred from plot-coverage-gaps — workspaces

+

Phase 2: Make workspace READMEs assistant-first — workspaces

refactor — 5 -

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder♻️ refactorautogalaxytoo-largesupervisedhigh

-

Vendor bessel_kve into autoarray and drop the tensorflow-probability dependency♻️ refactorautoarraylargesupervisedmedium

-

Split Fitness.batch_size into lh_batch_size and latent_batch_size♻️ refactorautofitsmallsupervisednormal

-

Remove the dead EDEN packaging tooling from PyAutoFit♻️ refactorpyautofitmediumsupervisednormal

-

Deduplicate repos_sync.py's check/write pairs♻️ refactorpyautomindmediumsafelow

+

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder — autogalaxy · too-large · supervised · high

+

Vendor bessel_kve into autoarray and drop the tensorflow-probability dependency — autoarray · large · supervised · medium

+

Split Fitness.batch_size into lh_batch_size and latent_batch_size — autofit · small · supervised · normal

+

Remove the dead EDEN packaging tooling from PyAutoFit — pyautofit · medium · supervised · normal

+

Deduplicate repos_sync.py's check/write pairs — pyautomind · medium · safe · low

test — 5 -

Re-baseline the MGE imaging JIT profiling regression value🧪 testautolens_workspace_developertoo-largesupervisedhigh

-

Restore absolute NumPy likelihood regression baselines in the _workspace_test🧪 testworkspacestoo-largesupervisedhigh

-

Relevance-gate the reusable smoke workflow so a PR only runs…🧪 testpyautoheartmediumsupervisednormal

-

Speed up the three slowest autolens_workspace_test smoke-gate scripts🧪 testworkspacesmediumsupervisednormal

-

The new workspace smoke-test GitHub Actions (added via feature/smoke-test-ci) surfaced🧪 testworkspacestoo-largesupervisednormal

+

Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high

+

Restore absolute NumPy likelihood regression baselines in the _workspace_test — workspaces · too-large · supervised · high

+

Relevance-gate the reusable smoke workflow so a PR only runs — pyautoheart · medium · supervised · normal

+

Speed up the three slowest autolens_workspace_test smoke-gate scripts — workspaces · medium · supervised · normal

+

The new workspace smoke-test GitHub Actions (added via feature/smoke-test-ci) surfaced — workspaces · too-large · supervised · normal

triage — 4 -

TRIAGE: needs manual review before routing❓ triagemediumsafehigh

-

Triage: Convolver "No blurring_image provided" warning in canonical workspace scripts❓ triagesmallsupervisednormal

-

Nightly release has been blocked 8 nights running — triage the streak❓ triagemediumsupervisednormal

-

Bug in autocti_workspace: the dataset_1d results/database example scripts have drifted❓ triagelargesupervisednormal

+

TRIAGE: needs manual review before routing — medium · safe · high

+

Triage: Convolver "No blurring_image provided" warning in canonical workspace scripts — small · supervised · normal

+

Nightly release has been blocked 8 nights running — triage — medium · supervised · normal

+

Bug in autocti_workspace: the dataset_1d results/database example scripts have drifted — large · supervised · normal

release — 1 -

CTI release-train wiring — first modern autocti release🚀 releaseautoctimediumhuman-requirednormal

+

CTI release-train wiring — first modern autocti release — autocti · medium · human-required · normal

Recent markdown version

The 50 newest things to happen to the work in hand, newest first — issued, parked, filed. Every other section on this page is laid out by state, which is exactly why none of them can answer “what has been happening?”. Shipped work is not here: it is read from complete/index.md, and a thousand records deep it would crowd out everything anyone can still act on. Showing the newest 10; … opens the next 10.

@@ -328,7 +255,7 @@

Backlog 2026-08-23 filed -pynufft removal: unswept downstream residue (1 hard break + stale… +pynufft removal: unswept downstream residue (1 hard break + stale @@ -340,7 +267,7 @@

Backlog 2026-08-23 filed -Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI… +Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI @@ -352,7 +279,7 @@

Backlog 2026-08-22 filed -smoke_install.sh's stale jax<0.7 pin — CI is on the right jax… +smoke_install.sh's stale jax<0.7 pin — CI is on the right @@ -370,7 +297,7 @@

Backlog 2026-08-22 filed -The reconstruction noise map describes a different estimator than the… +The reconstruction noise map describes a different estimator than the @@ -388,13 +315,13 @@

Backlog 2026-08-22 filed -Is Intel macOS a supported platform, and what is the numpy-only… +Is Intel macOS a supported platform, and what is the 2026-08-22 filed -Defer the eager scipy.sparse import in derivative_util (~0.10 s of… +Defer the eager scipy.sparse import in derivative_util (~0.10 s of @@ -412,13 +339,13 @@

Backlog 2026-08-21 filed -Numba PSF gathers derive the y/x kernel shifts from the wrong kernel… +Numba PSF gathers derive the y/x kernel shifts from the 2026-08-20 filed -Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… +Numba CPU likelihood phase 2: kernel-CDF numba fast path (the @@ -436,7 +363,7 @@

Backlog 2026-08-19 filed -jax 0.11 breaks beta/gamma message log_partition under jit… +jax 0.11 breaks beta/gamma message log_partition under jit ('tuple'… @@ -544,7 +471,7 @@

Backlog 2026-08-14 filed -Three jax_likelihood pins are stale by ~1.24e-4 and fail the smoke… +Three jax_likelihood pins are stale by ~1.24e-4 and fail the @@ -586,7 +513,7 @@

Backlog 2026-08-06 filed -Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead +Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are @@ -598,7 +525,7 @@

Backlog 2026-08-05 filed -Give PyAutoFit searches a seed — today no search can be made… +Give PyAutoFit searches a seed — today no search can @@ -610,7 +537,7 @@

Backlog 2026-08-04 filed -cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of… +cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of @@ -633,50 +560,49 @@

Epics

Cluster strong lensing — Source & Cluster arc — ledger: draft/feature/autolens/source_cluster_arc.md

-

PointSolver error-behavior change: bisect the 2025-11→2026-05 candidates, then health-harden🐛 bugautolenstoo-largesupervisedhigh

-

PointSolver profiling cells: lensed quasar → cluster runtime tier →🔬 researchautolens_profilinglargesupervisednormal

-

Critical curves: implement context-aware dispatch, dedupe engines, make clusters honor♻️ refactorautogalaxytoo-largesupervisedhigh

-

Mesh magnification correctness: simulate-and-recover across every mesh variant🧪 testworkspaceslargesupervisednormal

-

Magnification at a point: surface the existing API in source_science✨ featureautolenslargesupervisedhigh

-

Area magnification (LEGGOS-style): per-pixel inversion sum as primary; ShapeSolver rehabilitate-or-retire✨ featureautolenslargesupervisedhigh

-

Magnification errors via posterior draws, standalone in source_science✨ featureautolenslargesupervisednormal

-

Magnification maps: image-plane contour maps, source-plane mesh maps, uncertainty maps✨ featureautolenslargesupervisedhigh

-

Cluster source_science.py: robust magnification science at cluster scale (no meshes✨ featureworkspaceslargesupervisedhigh

-

Cluster package: point-source-default narrative + extended-source follow-up feature📖 docsworkspacesmediumsupervisedhigh

-

Cluster pixelized-source refinement: per-source masks via AnalysisFactor✨ featureworkspaceslargesupervisednormal

-

Cluster extended-source inference: gradient-based fitting building on JAX knowledge🔬 researchautolenslargesupervisednormal

-

HowToLens cluster tutorial: show a pixelized source + fix the📖 docshowtolenslargesupervisednormal

-

Source & Cluster arc — magnification science, PointSolver trust, cluster extended sources✨ featureautolenstoo-largesupervisedhigh

+

PointSolver error-behavior change: bisect the 2025-11→2026-05 candidates, then health-harden — autolens · too-large · supervised · high

+

PointSolver profiling cells: lensed quasar → cluster runtime tier → — autolens_profiling · large · supervised · normal

+

Critical curves: implement context-aware dispatch, dedupe engines, make clusters honor — autogalaxy · too-large · supervised · high

+

Mesh magnification correctness: simulate-and-recover across every mesh variant — workspaces · large · supervised · normal

+

Magnification at a point: surface the existing API in source_science — autolens · large · supervised · high

+

Area magnification (LEGGOS-style): per-pixel inversion sum as primary; ShapeSolver rehabilitate-or-retire — autolens · large · supervised · high

+

Magnification errors via posterior draws, standalone in source_science — autolens · large · supervised · normal

+

Magnification maps: image-plane contour maps, source-plane mesh maps, uncertainty maps — autolens · large · supervised · high

+

Cluster source_science.py: robust magnification science at cluster scale (no meshes — workspaces · large · supervised · high

+

Cluster package: point-source-default narrative + extended-source follow-up feature — workspaces · medium · supervised · high

+

Cluster pixelized-source refinement: per-source masks via AnalysisFactor — workspaces · large · supervised · normal

+

Cluster extended-source inference: gradient-based fitting building on JAX knowledge — autolens · large · supervised · normal

+

HowToLens cluster tutorial: show a pixelized source + fix the — howtolens · large · supervised · normal

+

Source & Cluster arc — magnification science, PointSolver trust, cluster — autolens · too-large · supervised · high

Intermittent XLA compile stall in the JAX vmap likelihood path — 2 queued prompt(s), in order

Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md — CLOSED AS PARTIAL 2026-08-23 — record complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md

-

Phase 3: root-cause the XLA vmap compile stall and clear every NEEDS_FIX…🐛 bugcilargesupervisedhigh

-

Intermittent XLA compile stall in JAX vmap likelihood scripts — third repo…🐛 bugcitoo-largesupervisedhigh

+

Phase 3: root-cause the XLA vmap compile stall and clear — ci · large · supervised · high

+

Intermittent XLA compile stall in JAX vmap likelihood scripts — — ci · too-large · supervised · high

Expectation propagation (EP) campaign — 9 queued prompt(s), in order

Expectation propagation (EP) campaign — ledger: draft/research/graphical_ep/ep_campaign.md

-

Analytic Gaussian benchmark: closed-form validation of graphical + EP🔬 researchgraphical_epmediumsupervisedhigh

-

EP hierarchical parent-scale collapse: cure the basin, or document the caveat🐛 bugautofittoo-largehuman-requiredhigh

-

Graphical Model Scale-Up — Scoping🔬 researchgraphical_eptoo-largesupervisedhigh

-

slope_hierarchy: scale the hierarchical slope recovery to N=25–50🔬 researchgraphical_epmediumsupervisednormal

-

The project @z_projects/ic50_workspace is our IC50 use case which we✨ featureautofitmediumsafenormal

-

IC50 use case: EP end-to-end with existing derived-variable handling🔬 researchgraphical_eplargesupervisednormal

-

EP campaign — phase map for the 2026 Q3 graphical/EP push🔬 researchgraphical_eptoo-largesupervisedhigh

-

Expectation Propagation Scale-Up — Scoping🔬 researchgraphical_eptoo-largesupervisedhigh

-

slope_hierarchy: methods write-up (NUTS headline, EP cautionary)🔬 researchgraphical_epmediumsupervisednormal

+

Analytic Gaussian benchmark: closed-form validation of graphical + EP — graphical_ep · medium · supervised · high

+

EP hierarchical parent-scale collapse: cure the basin, or document the — autofit · too-large · human-required · high

+

Graphical Model Scale-Up — Scoping — graphical_ep · too-large · supervised · high

+

slope_hierarchy: scale the hierarchical slope recovery to N=25–50 — graphical_ep · medium · supervised · normal

+

The project @z_projects/ic50_workspace is our IC50 use case which we — autofit · medium · safe · normal

+

IC50 use case: EP end-to-end with existing derived-variable handling — graphical_ep · large · supervised · normal

+

EP campaign — phase map for the 2026 Q3 graphical/EP — graphical_ep · too-large · supervised · high

+

Expectation Propagation Scale-Up — Scoping — graphical_ep · too-large · supervised · high

+

slope_hierarchy: methods write-up (NUTS headline, EP cautionary) — graphical_ep · medium · supervised · normal

- +

Boards: brain · heart · hands · memory · organism