diff --git a/complete/2026/08/jax-compile-stall-evidence.md b/complete/2026/08/jax-compile-stall-evidence.md new file mode 100644 index 00000000..aa891f15 --- /dev/null +++ b/complete/2026/08/jax-compile-stall-evidence.md @@ -0,0 +1,116 @@ +# Make a stalled JAX compile report itself (jax-compile-stall phase 1) + +- **Issue:** PyAutoFit#1516 (closed) · **PR:** PyAutoFit#1517 (merged 2026-08-23) +- **Repos:** PyAutoFit (`autofit/non_linear/jax_compile.py`, `test_autofit/non_linear/test_jax_compile.py`) +- **Epic:** `jax-compile-stall` phase 1 of 3 — ledger `draft/bug/ci/jax_vmap_jit_compile_stall.md`. Phase 2 is autolens_workspace_test#271. +- **What:** `log_on_first_compile` gained a heartbeat (`still compiling , Ns elapsed`; `PYAUTOFIT_JAX_COMPILE_HEARTBEAT_SECS`, default 30, 0 disables), a `faulthandler` watchdog dumping the process's own traceback on overrun (`PYAUTOFIT_JAX_COMPILE_DUMP_SECS`, defaulting to 300 **when `CI` is set** and 0 otherwise), and separate timings for the trace/lower/compile wait vs the `jax.block_until_ready` execution wait. Existing `complete in {n} seconds` summary line unchanged. All four call sites inherit it (`Fitness._vmap/_jit/_grad`, `analysis/latent.py`). No workspace script and no CI runner touched. +- **Why:** the same intermittent XLA compile stall had been quarantined three times (autolens_workspace_test#245; ag_test `multi_dataset/.../rectangular.py` 2026-08-01; ag_test `imaging/.../mge_group.py` 2026-08-23) and diagnosed zero times, because a stalled run emitted the "compiling..." line and then nothing until the cap killed it. +- **Key traps recorded:** + - **The silence spanned two different waits.** The wrapper called `func(...)` (trace/lower/XLA compile) and then `jax.block_until_ready(result)` (execution) under **one** log line, so a captured tail could not say which half was stuck — or whether the process was alive at all. That ambiguity, not the stall itself, is what defeated three investigations. + - **`Fitness._vmap` is `jax.vmap(jax.jit(self.call))`** — `vmap` *of* `jit`, the inverted ordering — while `analysis/latent.py` uses `jax.jit(jax.vmap(...))`. The stalling path is exactly the `vmap` path; `_jit`-only scripts in the same directories do not stall. **Deliberately not changed here**: altering the transform while trying to observe the stall would destroy the thing being observed. First A/B for phase 3. + - **The CI-conditional default avoids a second repo touch.** Defaulting the dump on when `CI` is set makes the next stall self-diagnosing without threading an env var through every workspace's `config/build/env_vars_*.yaml`. + - **A traceback during XLA compile parks at the pybind boundary** and shows no XLA internals. It still separates *in compile* / *in execution* / *blocked on a Python-level lock* (e.g. the persistent compile cache, on by default since PyAutoConf#128) — which is the three-way fork phase 3 needs. + - **Diagnostics must never break a fit.** An unstartable heartbeat thread (`RuntimeError` from `Thread.start`), an unarmable dump (a capturing harness can leave stderr without a real fd) and a malformed or negative interval all fall back and continue. The heartbeat-start guard was added on an adversarial re-read before pushing — it was originally outside the `try`, where it could have killed a fit. +- **Tests/verification:** full PyAutoFit suite 2008 passed / 34 skipped (3.12); `test_jax_compile.py` 16 passed, 12 new. CI green on 3.12, 3.13 and **unittest-nojax** — the new tests need no JAX import. End-to-end: a simulated stall in a fresh process (`CI=true`, heartbeat 2s, dump 5s) emitted heartbeats and two repeating tracebacks before SIGKILL at 12s, exactly as a CI cap would kill it. +- **Heart:** **NOT consulted** — `pyauto-heart` was unreachable from the `web-github` session that opened and merged this. Flagged in the PR body and in `active.md`; merge was a human instruction ("merge when green"). +- **Provenance:** started, implemented, shipped and merged by one `web-github` session 2026-08-23 (`claude/jax-vmap-jit-stall-swz2tc`), against a direct PyAutoFit clone rather than a worktree. Filed prompt lived on an unmerged branch (`claude/backport-per-script-timeout-r3w1sv`) and was brought onto the task branch at `/start_dev`. + +## Original prompt + +# Phase 1: make a stalled JAX compile report itself (heartbeat + faulthandler + compile/execute split) + +Type: bug +Target: ci +Repos: +- @PyAutoFit +Difficulty: small +Autonomy: supervised +Priority: high +Status: formalised +Epic: jax-compile-stall +Phase: 1 +Campaign: bug/ci/jax_vmap_jit_compile_stall.md (Phase 1 — the enabler; phases 2 and 3 are blocked on this) +Filed: 2026-08-23 +Issued: 2026-08-23 + +## Why this is phase 1 + +The stall's whole cost is that it produces **no evidence**. The last line any +killed run emits is + +``` +autofit.non_linear.jax_compile - INFO - JAX jit compiling vectorized (vmap) + likelihood function, could take seconds or minutes... +``` + +and then silence until the cap kills it. Three separate quarantines +(`autolens_workspace_test` delaunay #245, `autogalaxy_workspace_test` +`multi_dataset/.../rectangular.py` 2026-08-01, `imaging/.../mge_group.py` +2026-08-23) produced no diagnosis between them, because there was nothing to +diagnose *from*. Phases 2 and 3 of this campaign both consume evidence this +phase creates. + +## What is wrong with the current instrumentation + +`log_on_first_compile(func, description)` in +`autofit/non_linear/jax_compile.py` wraps the `jax.jit` / `jax.vmap` / +`jax.grad` callables so the "this is compiling" line lands where the user +actually waits — on the first call. Inside that first call it does two very +different things under one log line: + +1. `result = func(*args, **kwargs)` — tracing, lowering and XLA compilation; +2. `jax.block_until_ready(result)` — execution, because JAX dispatches + asynchronously. + +Then it logs one `complete in {n} seconds` summary. So a hang anywhere in +either half looks identical from the outside, and a compile that is merely +*slow* looks identical to one that has stopped. Nothing reports liveness in +between. + +## Task + +All of this is library-side in @PyAutoFit. **Do not touch the workspace +scripts** — they are user-facing documentation, and a per-script workaround is +the quarantine pattern this campaign exists to stop. + +1. **Heartbeat.** While the first call is in flight, log + `still compiling {description}, {n}s elapsed` on an interval. Daemon thread, + stopped in the existing `finally` so it can never hold the process open. + Interval from `PYAUTOFIT_JAX_COMPILE_HEARTBEAT_SECS`, default `30`, `0` + disables. +2. **Watchdog.** Arm `faulthandler.dump_traceback_later(secs, repeat=True, + exit=False)` before the first call and `cancel_dump_traceback_later()` in the + `finally`, so a compile that overruns dumps its own traceback to stderr + before anything kills it. Threshold from `PYAUTOFIT_JAX_COMPILE_DUMP_SECS`. +3. **Default it on under CI.** Default the threshold to `300` when the `CI` + environment variable is set and `0` (off) otherwise, both overridable. This + is what makes the *next* CI stall self-diagnosing with no workspace edit and + no runner edit — the alternative, wiring an env var into each workspace's + `config/build/env_vars_*.yaml`, is a second repo touch for the same effect. +4. **Split the timing.** Time `func(...)` and `jax.block_until_ready(result)` + separately and log both, so the record says which half is stuck. Keep the + existing single `complete in {n} seconds` summary line unchanged. + +Applies automatically to all four call sites: `Fitness._vmap`, `Fitness._jit`, +`Fitness._grad` (`autofit/non_linear/fitness.py`) and the batched latent +computation in `autofit/non_linear/analysis/latent.py`. + +## Known limitation, to be stated in the PR + +A Python traceback taken during XLA compilation parks at the pybind boundary — +it will not show XLA internals. It still separates *in compile* from *in +execution* from *blocked on a Python-level lock* (for instance the persistent +compilation cache, `JAX_COMPILATION_CACHE_DIR`, on by default since +PyAutoConf#128). That three-way split is exactly the fork phase 3 needs, so the +limitation does not undermine the phase. + +## Acceptance + +- A stalled first compile emits periodic liveness lines with elapsed time. +- A stalled first compile leaves a traceback behind in CI without any workspace + or runner change. +- The log distinguishes the compile wait from the execution wait. +- Covered by tests in `test_autofit/non_linear/test_jax_compile.py` that need no + JAX import: heartbeat fires, watchdog is armed and cancelled, env defaults + including the `CI` branch. +- No workspace script and no CI runner is modified by this phase. diff --git a/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md b/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md new file mode 100644 index 00000000..23e85561 --- /dev/null +++ b/complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md @@ -0,0 +1,177 @@ +# JAX vmap compile stall — instrumented, measured, partially explained (jax-compile-stall epic) + +- **Issues:** PyAutoFit#1516 (closed) · autolens_workspace_test#271 (closed) · **PRs:** PyAutoFit#1517, PyAutoFit#1518, PyAutoHeart#161, autolens_workspace_test#272, autogalaxy_workspace_test#110 — all merged 2026-08-23 +- **Repos:** PyAutoFit (`non_linear/jax_compile.py`, tests), PyAutoHeart (`.github/workflows/smoke-tests.yml`), autolens_workspace_test + autogalaxy_workspace_test (`.github/scripts/retime.py`, `.github/workflows/retime.yml`) +- **Epic:** `jax-compile-stall`, 3 phases. Phase 1 shipped in full; phases 2 and 3 taken to a deliberate stopping point (James, 2026-08-23) — **the remaining breadth moves to `draft/research/ci/smoke_timing_and_profiling.md`**, where this record is meant to be dug up. +- **Status: CLOSED AS PARTIAL.** The stall is instrumented and characterised but **not root-caused**. Nothing was un-quarantined. + +## What shipped + +1. **A stalled JAX compile now reports itself** (#1517, #1518). `log_on_first_compile` gained a heartbeat (`still compiling , Ns elapsed`), a `faulthandler` watchdog, and separate timings for the trace/lower/compile wait vs the `block_until_ready` execution wait. All four call sites inherit it. No workspace script or CI runner touched. +2. **A re-timing harness** (Heart#161 + the two workspace PRs). `retime.yml` (`workflow_dispatch`: scripts, repeats, script-timeout) → `retime.py`, which reuses `run_smoke.py`'s `run_one` so it cannot disagree with the PR gate about a script's environment. Heart's reusable workflow gained `runner` / `runner-args` / `script-timeout` inputs, all defaulting to prior behaviour, so the ceremony is shared rather than copied. + +## What was measured (60 script executions, all on hosted runners) + +| Entry | Verdict | Evidence | +|---|---|---| +| `interferometer/datacube/shared_preloads.py` (al) | **NEITHER** | 10/10 completed, worst 34.0s = **1.9%** of the 1800s cap its SLOW marker claims it flakes at | +| `imaging/jax_likelihood/rectangular_mge.py` (ag) | **STALL** | 4/5 capped on *both* legs, completions ~22s (7% of cap) | +| `imaging/jax_likelihood/mge_group.py` (ag) | **AMBIGUOUS** | 5/5 capped both legs, no completion | +| `multi_dataset/jax_likelihood/mge.py` (al) | **AMBIGUOUS** | 5/5 capped both legs, though its own marker records 32s standalone | + +## Key findings — the part worth digging up + +- **A SLOW marker is not evidence of slowness.** Every 2026-07-14 marker reads "flakes at the 1800s cap" and records *no timing at all*. The first one measured was wrong by ~50x. Do not trust the remaining 17 without re-measuring. +- **The stall is a >100x bimodality inside one step.** A healthy compile of `rectangular_mge.py` is **3.1s** (trace/lower/compile) + 0.5s (materialize); a stalled one exceeds 300s. Same commit, same runner image. +- **`vmap(jit)` ordering is contributory, not causal.** `Fitness._vmap` builds `jax.vmap(jax.jit(self.call))` while `analysis/latent.py` builds the conventional `jax.jit(jax.vmap(...))`. A/B on `rectangular_mge.py`: control **8/10 stalls (80%)** vs experiment **3/10 (30%)**, Fisher exact two-tailed **p = 0.070**. The stall SURVIVES the swap, so the ordering is not necessary for it, and n=10/arm is not significant. Branch `experiment/jax-vmap-jit-ordering` exists in PyAutoFit and autogalaxy_workspace_test, unmerged, for whoever resumes. +- **Untested hypothesis, still live:** the persistent compilation cache (`JAX_COMPILATION_CACHE_DIR`, default-on since PyAutoConf#128, 2026-07-17). Both NEEDS_FIX stalls post-date it; the SLOW batch predates it. A/B with the cache disabled is the obvious next experiment and was never run. + +## Traps recorded + +- **A watchdog whose threshold equals the cap never fires.** #1517 defaulted the `faulthandler` dump to 300s under CI; the smoke cap is also 300s, so the runner's SIGKILL beat the dump in all 20 stalled runs — heartbeats, no stacks. #1518 derives it from `BUILD_SCRIPT_TIMEOUT` at 80%. Its tests assert the *relationship* (`dump < cap` for every cap), not today's numbers, because nothing in #1517's own tests could have caught a collision between two independently-correct timeouts owned by different repos. +- **A Python traceback during XLA compile parks at the pybind boundary** — it separates *in compile* / *in execution* / *blocked on a Python lock*, but shows no XLA internals. +- **The compile/execute split does not localise a stall.** It only prints once *both* halves finish, so it characterises the healthy case only. Only the stack does. +- **Testing a library change through workspace CI:** the reusable workflow clones the dependency chain at the **matching branch name**, so an identically-named branch in the workspace repo makes its CI pick up an experimental library branch. That is how the ordering A/B was run. +- Diagnostics must never break a fit: unstartable heartbeat thread, unarmable dump, malformed interval all fall back and continue. + +## Left undone, deliberately + +- No root cause. No marker rewritten. **Nothing un-quarantined** — all five stall quarantines and all 21 SLOW markers stand as they were. +- The two 1800s runs dispatched at 21:37 (ag_test run 32668061785, al_test run 32668067325; `mge_group.py` and `multi_dataset/jax_likelihood/mge.py`, 2 repeats) were still in flight at close-out and should carry the first `faulthandler` stack at 1440s. **Read those two runs first when resuming** — they are the only pending evidence. +- **Heart:** never consulted this session — `pyauto-heart` unreachable from the `web-github` environment. Every merge was on an explicit human instruction ("merge when green"). +- Merged branches could not be deleted: this session's git proxy refuses ref deletions. `feature/jax-compile-stall-evidence`, `feature/jax-compile-dump-below-cap` (PyAutoFit), `feature/reusable-smoke-runner-input` (PyAutoHeart), `feature/jax-stall-retime-harness` (both workspaces), plus the two `experiment/jax-vmap-jit-ordering` branches. All need a local `/repo_cleanup`. + +## Original prompt + +# Phase 2: are the SLOW-marked jax_likelihood/jax_grad entries slow, or is this stall wearing a different label? + +Type: bug +Target: ci +Repos: +- @autogalaxy_workspace_test +- @autolens_workspace_test +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised +Epic: jax-compile-stall +Phase: 2 +Campaign: bug/ci/jax_vmap_jit_compile_stall.md (Phase 2 — the classification; consumes phase 1's evidence) +Filed: 2026-08-23 +Issued: 2026-08-23 + +## The question + +Step 1 of the campaign, on its own. Eight entries — six +`interferometer/jax_likelihood/*` and two `jax_grad/*` — were SLOW-skipped on +2026-07-14 for "flaking at the 1800s cap" (PyAutoHeart#74). A **SLOW** marker +says *make it faster*. A **stall** says *it never finishes*. Those route to +completely different places, and the Profiling Agent has been handed the first +description for a set that may partly belong to the second. + +Nothing in the current markers distinguishes them, because nothing measured the +distribution — a single observation at a cap looks the same either way. + +## Method + +A slow script has a **tight timing distribution**. A stalling one is +**bimodal**: tens of seconds when it completes, the full cap when it does not. +That is the discriminator, and it needs repeats, not one more run. + +1. Re-time each entry against the cap that actually applies to it (300s smoke, + 1800s release), N repeats each, both Python versions in the matrix. The + `rectangular_mge.py` result that passed on 3.13 and stalled on 3.12 **on the + same commit** shows one run per entry settles nothing. +2. Where phase 1's watchdog fires, keep the traceback — a stalled entry + identifies itself directly and does not need the distribution argument. +3. Classify each entry: genuinely slow, this stall mislabelled, or still + ambiguous after N runs. + +## Corrected census (read off the marker files 2026-08-23, at /start_dev) + +The parent prompt's "eight entries SLOW-skipped on 2026-07-14" undercounts by +more than 2x, and it missed a fifth stall quarantine filed the same day. Read +from `config/build/no_run.yaml` in both repos plus `autolens_workspace_test/smoke_tests.txt`: + +**SLOW-marked JAX entries — 21, not 8.** + +| autogalaxy_workspace_test (9) | autolens_workspace_test (12) | +|---|---| +| `multi_dataset/jax_likelihood/delaunay_mge` (#72) | `imaging/jax_likelihood/delaunay_mge` | +| `imaging/jax_likelihood/delaunay_mge.py` | `imaging/jax_likelihood/mge_group` | +| `interferometer/jax_likelihood/mge.py` | `multi_dataset/jax_likelihood/delaunay_mge` (#72) | +| `interferometer/jax_likelihood/mge_group.py` | `multi_dataset/jax_likelihood/shared_preloads.py` | +| `interferometer/jax_likelihood/delaunay.py` | `interferometer/datacube/delaunay.py` | +| `interferometer/jax_likelihood/delaunay_mge.py` | `interferometer/datacube/shared_preloads.py` | +| `interferometer/jax_likelihood/rectangular_mge.py` | `interferometer/jax_likelihood/mge.py` | +| `interferometer/jax_grad/mge.py` | `interferometer/jax_likelihood/mge_group.py` | +| `multi_dataset/jax_grad/mge.py` | `interferometer/jax_likelihood/delaunay.py` | +| | `interferometer/jax_likelihood/delaunay_mge.py` | +| | `interferometer/jax_likelihood/rectangular_mge.py` | +| | `interferometer/jax_grad/gradient.py` | + +**Quarantined for the stall signature — 5, not 3.** + +| Entry | Repo | Marker | +|---|---|---| +| `multi_dataset/jax_likelihood/rectangular.py` | ag_test | NEEDS_FIX 2026-08-01 | +| `imaging/jax_likelihood/mge_group.py` | ag_test | NEEDS_FIX 2026-08-23 | +| `imaging/jax_likelihood/rectangular_mge.py` | ag_test | NEEDS_FIX 2026-08-23 | +| `multi_dataset/jax_likelihood/delaunay.py` | al_test | NEEDS_FIX 2026-08-01 (#245) | +| `multi_dataset/jax_likelihood/mge.py` | al_test | disabled in `smoke_tests.txt` 2026-08-22 | + +`imaging/jax_likelihood/rectangular_mge.py` was quarantined in ag_test on +2026-08-23, so the parent prompt's "passed on 3.13, stalled on 3.12" reference +point is now a quarantine in its own right. 26 entries in scope, not 11. + +## Three things the marker text already establishes, before any re-timing + +**1. A 27.8s script is SLOW-marked for "flaking" at a 1800s cap.** +`interferometer/datacube/shared_preloads.py` is SLOW-marked in +`autolens_workspace_test`'s `no_run.yaml` — *"flakes at the 1800s cap +(PyAutoHeart#74)"* — while the `smoke_tests.txt` comment on a sibling records it +running the PR gate in **27.8s**, and it is still enabled there. A script that +completes in 27.8s does not intermittently exceed 1800s by being slow: that is a +65x gap. Whatever is happening to it, slowness is not it. + +**2. The same script path is SLOW in one repo and a stall in the other.** +`imaging/jax_likelihood/mge_group` is SLOW-marked in `autolens_workspace_test` +(2026-07-21, *"flakes at the 1800s cap"*) and NEEDS_FIX-quarantined for the +stall in `autogalaxy_workspace_test` (2026-08-23, *"silence for the full 300s"*). +Same tier, same path, two labels. The label recorded which repo noticed it, not +what it was doing. + +**3. "Flakes" is bimodal language wearing a unimodal label.** +Every 2026-07-14 SLOW marker reads *"flakes at the 1800s cap"* and records **no +timing at all**. A genuinely slow script does not flake — it exceeds the cap +consistently. Compare the markers this repo writes when it has actually +measured something: `misc/database/scrape/*` say *"re-measured: times out at the +real 300s cap"*, and the stall quarantines say *"18s when it passes"*, +*"passes ~19s otherwise"*, *"runs green in 32s standalone"*. The batch that +carries the SLOW label is the batch with no measurement behind it. + +None of this is proof — it is three strong priors that make the re-timing a +confirmation rather than an exploration, and it means the burden now sits on +"these are slow", not on "these are stalls". + +## Entries in scope + +All 26 above. The method applies uniformly: an entry is classified from its own +distribution, not from which marker it happens to carry today. + +`imaging/jax_likelihood/delaunay_mge.py` (al_test) carries **two** unrelated +markers — SLOW in `no_run.yaml` and disabled in `smoke_tests.txt` for `jax 0.7` +removing `jax.interpreters.xla.pytype_aval_mappings`. The API removal is a real, +different cause; re-time it anyway, but do not let the SLOW marker's removal +imply the API one is resolved. + +## Acceptance + +- Every entry above classified, with the timing distribution that supports the + classification recorded — not asserted. +- Every marker in the `no_run.yaml` files and `smoke_tests.txt` rewritten to + carry its real reason, so a SLOW marker means slow and nothing else. +- The answer written down somewhere the Profiling Agent reads, so it stops + chasing speedups on scripts that are hanging. +- No script is un-quarantined by this phase — restoring coverage is phase 3, + and depends on the fix. diff --git a/complete/index.md b/complete/index.md index 5ffa9693..16f90c79 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1091 records across 7 buckets. +1093 records across 7 buckets. ## Highlights @@ -98,6 +98,8 @@ _(curate hard-won records here — survives regeneration.)_ - [interferometer-delaunay-flaky-fitexception](2026/08/interferometer-delaunay-flaky-fitexception.md) - [interferometer-start-here-integrate-oom](2026/08/interferometer-start-here-integrate-oom.md) - [intra-family-dep-floors](2026/08/intra-family-dep-floors.md) +- [jax-compile-stall-evidence](2026/08/jax-compile-stall-evidence.md) — jax-compile-stall phase 1 +- [jax-compile-stall-slow-vs-stall-audit](2026/08/jax-compile-stall-slow-vs-stall-audit.md) — jax-compile-stall epic - [jax-default-dependency](2026/08/jax-default-dependency.md) - [jax-grad-local-vs-ci-assertions](2026/08/jax-grad-local-vs-ci-assertions.md) - [jax-grad-smoke-timeout-budget](2026/08/jax-grad-smoke-timeout-budget.md) diff --git a/dashboard.html b/dashboard.html index e097db6f..92468288 100644 --- a/dashboard.html +++ b/dashboard.html @@ -6,142 +6,69 @@ PyAutoMind Dashboard -
📋

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

- -

markdown version

+

📋 PyAutoMind Dashboard

+

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

+

In flight 1 · Parked 3 · Planned 6 · Backlog 156 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

-

TRIAGE: needs manual review before routing❓ triagemediumsafehigh

-

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

-

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU)✨ featureautoarraymediumsupervisedhigh

-

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88% lever)✨ featureautoarraylargesupervisedhigh

-

multi_galaxy package: new regime package in autolens_workspace📖 docsautolenslargesupervisedhigh

-

Profile and speed up JAX likelihood-function compile times (all use cases)✨ featureautolens_profilinglargesupervisedhigh

-

Give the Profiling Agent a compile-time axis — the arc✨ featureprofilinglargesupervisedhigh

-

Deep research: Can we speed up Delaunay in PyAutoArray?🔬 researchautoarraytoo-largesupervisedhigh

-

Census of priors and messages — confirmed bugs + redesign ideas🔬 researchautofittoo-largesupervisedhigh

-

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder♻️ refactorautogalaxytoo-largesupervisedhigh

-

Split lensing regimes: multi_galaxy / group / cluster (epic plan)📖 docsautolenstoo-largesupervisedhigh

-

Re-baseline the MGE imaging JIT profiling regression value🧪 testautolens_workspace_developertoo-largesupervisedhigh

+

TRIAGE: needs manual review before routing — medium · safe · high

+

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high

+

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high

+

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high

+

multi_galaxy package: new regime package in autolens_workspace — autolens · large · supervised · high

+

Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high

+

Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high

+

Deep research: Can we speed up Delaunay in PyAutoArray? — autoarray · too-large · supervised · high

+

Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high

+

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder — autogalaxy · too-large · supervised · high

+

Split lensing regimes: multi_galaxy / group / cluster (epic plan) — autolens · too-large · supervised · high

+

Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high

Quick wins (small enough, and safe enough to run unattended)

-

Defer the eager scipy.sparse import in derivative_util (~0.10 s of import)🧹 maintenancelibrariessmallsafenormal

-

Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py🐛 bugworkspacessmallsafenormal

+

Defer the eager scipy.sparse import in derivative_util (~0.10 s of — libraries · small · safe · normal

+

Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py — workspaces · small · safe · normal

In flight markdown version

Issued — each has an open GitHub issue and usually a branch.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

@@ -163,163 +90,164 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

153 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.

+

156 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

feature — 29 -

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

-

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU)✨ featureautoarraymediumsupervisedhigh

-

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88% lever)✨ featureautoarraylargesupervisedhigh

-

Profile and speed up JAX likelihood-function compile times (all use cases)✨ featureautolens_profilinglargesupervisedhigh

-

Give the Profiling Agent a compile-time axis — the arc✨ featureprofilinglargesupervisedhigh

-

Which other searches need prior-support handling — coverage audit after Prodigy✨ featureautofitmediumsupervisedmedium

-

Give PyAutoFit searches a seed — today no search can be made…✨ featureautofitmediumsupervisedmedium

-

Brain board follow-ups: what real mornings surface✨ featurepyautobrainsmallsupervisednormal

-

Can create a list of InversionMatrix objects for each dataset✨ featureautoarraymediumsupervisednormal

-

Tune cluster-scale JOSS benchmarks toward their 5-minute targets✨ featureautolens_workspacemediumsupervisednormal

-

Token-light wiki index over the complete/ archive✨ featurepyautomindmediumsupervisednormal

-

The imaging features/advanced/los_halos example needs improving and padding out before✨ featureworkspacesmediumsafenormal

-

The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before✨ featureworkspacesmediumsafenormal

-

Claude Development Prompt: Arcsecond Tick Label Decimal Placement✨ featureautoarraylargesupervisednormal

-

EP analytic updates — implement the four planned work packages✨ featureautofitlargesupervisednormal

-

Remote-MCP deployment tiers (2 + 3) for the results-inspector server✨ featureautofit_assistantlargehuman-requirednormal

-

Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch)✨ featureautolens_profilinglargesupervisednormal

-

Adopt oversampled PSFs in the start-here dataset chain (option a)✨ featureautolens_workspacelargesupervisednormal

-

Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A✨ featureautoarraytoo-largesupervisednormal

-

PIEMass.potential_2d_from: implement the missing lensing potential✨ featureautogalaxytoo-largesupervisednormal

-

autolens_jax_joss benchmark repo + real-data start_here pairing✨ featureautolens_jax_josstoo-largesupervisednormal

-

Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the✨ featurejax_substructuretoo-largesupervisednormal

-

Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the✨ featurejax_substructuretoo-largesupervisednormal

-

Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver✨ featureworkspacestoo-largesupervisednormal

-

dPIE: optional central-dispersion (sigma_0) parameterization✨ featureautogalaxysmallsupervisedlow

-

Release board: local run_logs enrichment✨ featurepyautohandssmallsupervisedlow

-

Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier✨ featureworkspacessmallsupervisedlow

-

Scheduled runs — overnight queue passes with a morning report✨ featureautonomymediumsupervisedlow

-

Teach repos_sync --write to stamp organ config surfaces✨ featurepyautomindhardsupervisedlow

+

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high

+

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high

+

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high

+

Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high

+

Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high

+

Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium

+

Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium

+

Brain board follow-ups: what real mornings surface — pyautobrain · small · supervised · normal

+

Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal

+

Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal

+

Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal

+

The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal

+

The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal

+

Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal

+

EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal

+

Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal

+

Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal

+

Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal

+

Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal

+

PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal

+

autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal

+

Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal

+

Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal

+

Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal

+

dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low

+

Release board: local run_logs enrichment — pyautohands · small · supervised · low

+

Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low

+

Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low

+

Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low

bug — 32 -

Fix release JAX runtime compatibility and likelihood parity🐛 bughealth_fixestoo-largesupervisedhigh

-

Fix JIT quick-update visualization output regressions🐛 bughealth_fixestoo-largesupervisedhigh

-

Fix release result/sample parameter-path regressions🐛 bughealth_fixestoo-largesupervisedhigh

-

jax 0.11 breaks beta/gamma message log_partition under jit…🐛 bugautofitsmallsupervisedmedium

-

Heart script_timing baselines are orphaned by path moves and filled…🐛 bugpyautoheartsmallsupervisedmedium

-

Numba PSF gathers derive the y/x kernel shifts from the wrong kernel…🐛 bugautoarraylowsupervisedmedium

-

LogGaussianPrior misreports its own support as (-inf, inf)🐛 bugautofitsmallsupervisednormal

-

autofit.plot functions accept **kwargs and silently discard them🐛 bugautofitsmallsupervisednormal

-

TEST_MODE bypass crashes on ordered-parameter assertion ties🐛 bugautofitsmallsupervisednormal

-

point.py JAX-vmap parity assert is non-deterministic under the smoke env🐛 bugautolenssmallsupervisednormal

-

Scripts derive geometry from a hardcoded pixel_scale while the dataset carries…🐛 bugautolens_workspacesmallsupervisednormal

-

HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX)🐛 bughowtogalaxysmallsupervisednormal

-

PyAutoConf rename leftovers in Brain functional surfaces🐛 bugpyautobrainsmallsupervisednormal

-

generate.py deletes notebooks/ before rejecting an unknown project🐛 bugpyautohandssmallsupervisednormal

-

autoreduce 0.9 on PyPI never got the Python 3.12 floor🐛 bugpyautoreducesmallsupervisednormal

-

aplt.Output stale-API drift in the remaining workspace repos🐛 bugworkspacessmallsupervisednormal

-

Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py🐛 bugworkspacessmallsafenormal

-

Three jax_likelihood pins are stale by ~1.24e-4 and fail the smoke gate…🐛 bugworkspacessmallsupervisednormal

-

PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional?🐛 bugautoarraymediumsupervisednormal

-

interferometer Delaunay pixelization — non-PD FitException in test-mode bypass🐛 bugautolensmediumsupervisednormal

-

JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38🐛 bugautolensmediumsupervisednormal

-

JIT cache not hit in modeling_visualization delaunay/rectangular scripts🐛 bugautolensmediumsupervisednormal

-

multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall)🐛 bugautolens_workspace_testmediumsupervisednormal

-

@PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian🐛 bugpriorsmediumsupervisednormal

-

Resolve release-profile timeout scripts deliberately🐛 bughealth_fixestoo-largesupervisednormal

-

@PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms🐛 bugpriorstoo-largesupervisednormal

-

Priors & Messages cleanup — tracker🐛 bugpriorstoo-largesupervisednormal

-

autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main🐛 bugautolenslowsupervisednormal

-

status.sh --repos sources a file that no longer exists🐛 bugpyautomindsmallsupervisedlow

-

The reconstruction noise map describes a different estimator than the default solver🐛 bugautoarraymediumhuman-requiredlow

-

Point-source JSON datasets record no resolution regime🐛 bugpyautolensmediumsupervisedlow

-

interferometer/start_here.py OOM in nightly release-validation integrate leg🐛 bugautolens

+

Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high

+

Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high

+

Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high

+

jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium

+

Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium

+

Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium

+

LogGaussianPrior misreports its own support as (-inf, inf) — autofit · small · supervised · normal

+

autofit.plot functions accept **kwargs and silently discard them — autofit · small · supervised · normal

+

TEST_MODE bypass crashes on ordered-parameter assertion ties — autofit · small · supervised · normal

+

point.py JAX-vmap parity assert is non-deterministic under the smoke env — autolens · small · supervised · normal

+

Scripts derive geometry from a hardcoded pixel_scale while the dataset — autolens_workspace · small · supervised · normal

+

HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX) — howtogalaxy · small · supervised · normal

+

PyAutoConf rename leftovers in Brain functional surfaces — pyautobrain · small · supervised · normal

+

generate.py deletes notebooks/ before rejecting an unknown project — pyautohands · small · supervised · normal

+

autoreduce 0.9 on PyPI never got the Python 3.12 floor — pyautoreduce · small · supervised · normal

+

aplt.Output stale-API drift in the remaining workspace repos — workspaces · small · supervised · normal

+

Bug: fix the tracer.fits existence guard in autolens_workspace imaging modeling.py — workspaces · small · safe · normal

+

Three jax_likelihood pins are stale by ~1.24e-4 and fail the — workspaces · small · supervised · normal

+

PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional? — autoarray · medium · supervised · normal

+

interferometer Delaunay pixelization — non-PD FitException in test-mode bypass — autolens · medium · supervised · normal

+

JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38 — autolens · medium · supervised · normal

+

JIT cache not hit in modeling_visualization delaunay/rectangular scripts — autolens · medium · supervised · normal

+

multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall) — autolens_workspace_test · medium · supervised · normal

+

@PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian — priors · medium · supervised · normal

+

Resolve release-profile timeout scripts deliberately — health_fixes · too-large · supervised · normal

+

@PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms — priors · too-large · supervised · normal

+

Priors & Messages cleanup — tracker — priors · too-large · supervised · normal

+

autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main — autolens · low · supervised · normal

+

status.sh --repos sources a file that no longer exists — pyautomind · small · supervised · low

+

The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · low

+

Point-source JSON datasets record no resolution regime — pyautolens · medium · supervised · low

+

interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens

-research — 17 -

Deep research: Can we speed up Delaunay in PyAutoArray?🔬 researchautoarraytoo-largesupervisedhigh

-

Census of priors and messages — confirmed bugs + redesign ideas🔬 researchautofittoo-largesupervisedhigh

-

Delaunay-family JAX modules never hit the persistent compilation cache🔬 researchautoarraymediumsupervisedmedium

-

Quick-update plotting cost — minutes per update, and it is not JAX…🔬 researchautolensmediumsupervisedmedium

-

Use readthedocs or migrate to GitHub docs🔬 researchautobuildsmallsupervisednormal

-

Explore: dashboardify the Brain's operational surfaces with pasteable conductor prompts🔬 researchpyautobrainsmallsupervisednormal

-

Re-baseline the slacs0008 acceptance parity after the HAP-dedupe fix🔬 researchpyautoreducesmallsupervisednormal

-

Kernel-CDF bandwidth defaults — config-dependent quality, investigate adaptivity🔬 researchautoarraymediumsupervisednormal

-

We have lots of examples which profile how long JAX🔬 researchautolens_workspace_developermediumsupervisednormal

-

Is Intel macOS a supported platform, and what is the numpy-only contract?🔬 researchlibrariesmediumsupervisednormal

-

Checkerboard PSF-mismatch residual diagnostic — research + document + ingest papers🔬 researchpyautomemorymediumsupervisednormal

-

autofit_profiling: bootstrap the repo + general PyAutoFit profiling epic🔬 researchautofitlargesupervisednormal

-

Multi-band compile census completion — A100/multi-core + hetero GPU rows🔬 researchautolens_profilingsmallsupervisedlow

-

Chase the ~6% flux scale between PyAutoReduce and legacy SLACS reductions🔬 researchpyautoreducemediumsupervisedlow

-

PyAutoArray Delaunay interpolator's pure_callback vs vmap — minor efficiency follow-up🔬 researchautoarraytoo-largesupervisedlow

-

Cluster-scale gradient-search benchmark (Prodigy vs Nautilus, point-source)🔬 researchautolens_profiling

-

Adopt Python 3.12 as the PyAuto ecosystem minimum🔬 researchlibraries

+research — 18 +

Deep research: Can we speed up Delaunay in PyAutoArray? — autoarray · too-large · supervised · high

+

Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high

+

Delaunay-family JAX modules never hit the persistent compilation cache — autoarray · medium · supervised · medium

+

Quick-update plotting cost — minutes per update, and it is — autolens · medium · supervised · medium

+

Use readthedocs or migrate to GitHub docs — autobuild · small · supervised · normal

+

Explore: dashboardify the Brain's operational surfaces with pasteable conductor prompts — pyautobrain · small · supervised · normal

+

Re-baseline the slacs0008 acceptance parity after the HAP-dedupe fix — pyautoreduce · small · supervised · normal

+

Kernel-CDF bandwidth defaults — config-dependent quality, investigate adaptivity — autoarray · medium · supervised · normal

+

We have lots of examples which profile how long JAX — autolens_workspace_developer · medium · supervised · normal

+

Properly time and profile the smoke/release script surface — ci · medium · supervised · normal

+

Is Intel macOS a supported platform, and what is the — libraries · medium · supervised · normal

+

Checkerboard PSF-mismatch residual diagnostic — research + document + ingest — pyautomemory · medium · supervised · normal

+

autofit_profiling: bootstrap the repo + general PyAutoFit profiling epic — autofit · large · supervised · normal

+

Multi-band compile census completion — A100/multi-core + hetero GPU rows — autolens_profiling · small · supervised · low

+

Chase the ~6% flux scale between PyAutoReduce and legacy SLACS — pyautoreduce · medium · supervised · low

+

PyAutoArray Delaunay interpolator's pure_callback vs vmap — minor efficiency follow-up — autoarray · too-large · supervised · low

+

Cluster-scale gradient-search benchmark (Prodigy vs Nautilus, point-source) — autolens_profiling

+

Adopt Python 3.12 as the PyAuto ecosystem minimum — libraries

maintenance — 23 -

Untrack the generated FITS test artifacts in autoarray🧹 maintenancelibrariessmallsupervisedmedium

-

smoke_install.sh's stale jax<0.7 pin — CI is on the right jax…🧹 maintenancecilowsupervisedmedium

-

autolens_workspace_developer rectangular experiments — Gut stash + rename🧹 maintenanceautolens_workspace_developersmallsupervisednormal

-

Defer the eager scipy.sparse import in derivative_util (~0.10 s of import)🧹 maintenancelibrariessmallsafenormal

-

Mirror drifted library config keys into the workspace configs🧹 maintenanceworkspacessmallsupervisednormal

-

Un-park imaging/features/scaling_relation/slam once PyAutoArray#431 merges🧹 maintenanceworkspacessmallsupervisednormal

-

autolens_workspace_developer: broad stale-API rot (56 symbols, no CI)🧹 maintenanceautolens_workspace_developermediumsupervisednormal

-

run_smoke.py: three runner variants across 10 repos, no sync mechanism🧹 maintenancecimediumsupervisednormal

-

Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two dead deps🧹 maintenancelibrariesmediumsupervisednormal

-

PyAutoMemory canonical-key TODO sweep🧹 maintenancepyautomemorymediumsupervisednormal

-

Single-source the "Never rewrite history" policy as a generated AGENTS.md block🧹 maintenancepyautomindmediumsupervisednormal

-

Capped smoke datasets were committed as if they were real🧹 maintenanceworkspacesmediumsupervisednormal

-

autolens_profiling is now a mature project, with a good separation🧹 maintenanceautolens_profilinglargesupervisednormal

-

Auto-request GitHub Copilot code review on every PR, org-wide🧹 maintenancecilargesupervisednormal

-

autolens_workspace🧹 maintenanceworkspacestoo-largesupervisednormal

-

Remove pynufft + legacy TransformerNUFFTPyNUFFT🧹 maintenancelibrarieslow-mediumsupervisednormal

-

pynufft removal: unswept downstream residue (1 hard break + stale docs/CI)🧹 maintenanceworkspaceslow-mediumsupervisednormal

-

Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI install docs🧹 maintenanceworkspaceslowsupervisednormal

-

dataset/imaging/jwst_lw is untracked because the gitignore was never extended for it🧹 maintenanceautolens_profilingsmallsupervisedlow

-

cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of the repo's…🧹 maintenanceautolens_workspacesmallsupervisedlow

-

LaTeX in non-raw docstrings emits SyntaxWarning: invalid escape sequence🧹 maintenanceautolens_workspacesmallsupervisedlow

-

Regenerate setup_notebook-drifted notebooks in autogalaxy/autofit/HowToFit workspaces🧹 maintenanceworkspacessmallsupervisedlow

-

Refactor Agent witness map lacks PyAutoNerves test suite🧹 maintenancepyautobrainlowsafelow

+

Untrack the generated FITS test artifacts in autoarray — libraries · small · supervised · medium

+

smoke_install.sh's stale jax<0.7 pin — CI is on the right — ci · low · supervised · medium

+

autolens_workspace_developer rectangular experiments — Gut stash + rename — autolens_workspace_developer · small · supervised · normal

+

Defer the eager scipy.sparse import in derivative_util (~0.10 s of — libraries · small · safe · normal

+

Mirror drifted library config keys into the workspace configs — workspaces · small · supervised · normal

+

Un-park imaging/features/scaling_relation/slam once PyAutoArray#431 merges — workspaces · small · supervised · normal

+

autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) — autolens_workspace_developer · medium · supervised · normal

+

run_smoke.py: three runner variants across 10 repos, no sync mechanism — ci · medium · supervised · normal

+

Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two dead — libraries · medium · supervised · normal

+

PyAutoMemory canonical-key TODO sweep — pyautomemory · medium · supervised · normal

+

Single-source the "Never rewrite history" policy as a generated AGENTS.md — pyautomind · medium · supervised · normal

+

Capped smoke datasets were committed as if they were real — workspaces · medium · supervised · normal

+

autolens_profiling is now a mature project, with a good separation — autolens_profiling · large · supervised · normal

+

Auto-request GitHub Copilot code review on every PR, org-wide — ci · large · supervised · normal

+

autolens_workspace — workspaces · too-large · supervised · normal

+

Remove pynufft + legacy TransformerNUFFTPyNUFFT — libraries · low-medium · supervised · normal

+

pynufft removal: unswept downstream residue (1 hard break + stale — workspaces · low-medium · supervised · normal

+

Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI — workspaces · low · supervised · normal

+

dataset/imaging/jwst_lw is untracked because the gitignore was never extended for — autolens_profiling · small · supervised · low

+

cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of — autolens_workspace · small · supervised · low

+

LaTeX in non-raw docstrings emits SyntaxWarning: invalid escape sequence — autolens_workspace · small · supervised · low

+

Regenerate setup_notebook-drifted notebooks in autogalaxy/autofit/HowToFit workspaces — workspaces · small · supervised · low

+

Refactor Agent witness map lacks PyAutoNerves test suite — pyautobrain · low · safe · low

docs — 14 -

multi_galaxy package: new regime package in autolens_workspace📖 docsautolenslargesupervisedhigh

-

Split lensing regimes: multi_galaxy / group / cluster (epic plan)📖 docsautolenstoo-largesupervisedhigh

-

Advanced workspace guide: Preloads (PyAutoArray)📖 docsworkspacestoo-largesupervisedhigh

-

Regenerate autolens_workspace markdown/ so the MGE pages show sigma_min📖 docsautolens_workspacesmallsupervisednormal

-

RTD organism docs currency: Nerves page, organ-count drift, hands.md rename📖 docspyautobrainsmallsupervisednormal

-

Propagate the shear_galaxy-at-(0,0) idiom to group/ and cluster/📖 docsworkspacessmallsupervisednormal

-

Rectangular mesh Enzi citation — user-workspace pixelization examples📖 docsworkspacessmallsupervisednormal

-

Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead📖 docsautoctimediumsupervisednormal

-

extra_galaxies feature parity: point_source + multi_galaxy (both workspaces)📖 docsworkspacesmediumsupervisednormal

-

HowToLens ch4 tutorial 3: mask overlay is never actually drawn📖 docshowtolenssmallsupervisedlow

-

Assistants: regime-aware routing for multi_galaxy / group / cluster (follow-up)📖 docsworkspacesmediumsupervisedlow

-

Phase 2 — drop the hand-written quick-update sentence from the workspace scripts📖 docsautolens_workspace

-

plot coverage — follow-ups deferred from plot-coverage-gaps📖 docsworkspaces

-

Phase 2: Make workspace READMEs assistant-first📖 docsworkspaces

+

multi_galaxy package: new regime package in autolens_workspace — autolens · large · supervised · high

+

Split lensing regimes: multi_galaxy / group / cluster (epic plan) — autolens · too-large · supervised · high

+

Advanced workspace guide: Preloads (PyAutoArray) — workspaces · too-large · supervised · high

+

Regenerate autolens_workspace markdown/ so the MGE pages show sigma_min — autolens_workspace · small · supervised · normal

+

RTD organism docs currency: Nerves page, organ-count drift, hands.md rename — pyautobrain · small · supervised · normal

+

Propagate the shear_galaxy-at-(0,0) idiom to group/ and cluster/ — workspaces · small · supervised · normal

+

Rectangular mesh Enzi citation — user-workspace pixelization examples — workspaces · small · supervised · normal

+

Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are — autocti · medium · supervised · normal

+

extra_galaxies feature parity: point_source + multi_galaxy (both workspaces) — workspaces · medium · supervised · normal

+

HowToLens ch4 tutorial 3: mask overlay is never actually drawn — howtolens · small · supervised · low

+

Assistants: regime-aware routing for multi_galaxy / group / cluster (follow-up) — workspaces · medium · supervised · low

+

Phase 2 — drop the hand-written quick-update sentence from the — autolens_workspace

+

plot coverage — follow-ups deferred from plot-coverage-gaps — workspaces

+

Phase 2: Make workspace READMEs assistant-first — workspaces

refactor — 5 -

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder♻️ refactorautogalaxytoo-largesupervisedhigh

-

Vendor bessel_kve into autoarray and drop the tensorflow-probability dependency♻️ refactorautoarraylargesupervisedmedium

-

Split Fitness.batch_size into lh_batch_size and latent_batch_size♻️ refactorautofitsmallsupervisednormal

-

Remove the dead EDEN packaging tooling from PyAutoFit♻️ refactorpyautofitmediumsupervisednormal

-

Deduplicate repos_sync.py's check/write pairs♻️ refactorpyautomindmediumsafelow

+

einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder — autogalaxy · too-large · supervised · high

+

Vendor bessel_kve into autoarray and drop the tensorflow-probability dependency — autoarray · large · supervised · medium

+

Split Fitness.batch_size into lh_batch_size and latent_batch_size — autofit · small · supervised · normal

+

Remove the dead EDEN packaging tooling from PyAutoFit — pyautofit · medium · supervised · normal

+

Deduplicate repos_sync.py's check/write pairs — pyautomind · medium · safe · low

test — 5 -

Re-baseline the MGE imaging JIT profiling regression value🧪 testautolens_workspace_developertoo-largesupervisedhigh

-

Restore absolute NumPy likelihood regression baselines in the _workspace_test🧪 testworkspacestoo-largesupervisedhigh

-

Relevance-gate the reusable smoke workflow so a PR only runs…🧪 testpyautoheartmediumsupervisednormal

-

Speed up the three slowest autolens_workspace_test smoke-gate scripts🧪 testworkspacesmediumsupervisednormal

-

The new workspace smoke-test GitHub Actions (added via feature/smoke-test-ci) surfaced🧪 testworkspacestoo-largesupervisednormal

+

Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high

+

Restore absolute NumPy likelihood regression baselines in the _workspace_test — workspaces · too-large · supervised · high

+

Relevance-gate the reusable smoke workflow so a PR only runs — pyautoheart · medium · supervised · normal

+

Speed up the three slowest autolens_workspace_test smoke-gate scripts — workspaces · medium · supervised · normal

+

The new workspace smoke-test GitHub Actions (added via feature/smoke-test-ci) surfaced — workspaces · too-large · supervised · normal

triage — 4 -

TRIAGE: needs manual review before routing❓ triagemediumsafehigh

-

Triage: Convolver "No blurring_image provided" warning in canonical workspace scripts❓ triagesmallsupervisednormal

-

Nightly release has been blocked 8 nights running — triage the streak❓ triagemediumsupervisednormal

-

Bug in autocti_workspace: the dataset_1d results/database example scripts have drifted❓ triagelargesupervisednormal

+

TRIAGE: needs manual review before routing — medium · safe · high

+

Triage: Convolver "No blurring_image provided" warning in canonical workspace scripts — small · supervised · normal

+

Nightly release has been blocked 8 nights running — triage — medium · supervised · normal

+

Bug in autocti_workspace: the dataset_1d results/database example scripts have drifted — large · supervised · normal

release — 1 -

CTI release-train wiring — first modern autocti release🚀 releaseautoctimediumhuman-requirednormal

+

CTI release-train wiring — first modern autocti release — autocti · medium · human-required · normal

Recent markdown version

The 50 newest things to happen to the work in hand, newest first — issued, parked, filed. Every other section on this page is laid out by state, which is exactly why none of them can answer “what has been happening?”. Shipped work is not here: it is read from complete/index.md, and a thousand records deep it would crowd out everything anyone can still act on. Showing the newest 10; … opens the next 10.

@@ -327,13 +255,19 @@

Backlog 2026-08-23 filed -pynufft removal: unswept downstream residue (1 hard break + stale… +pynufft removal: unswept downstream residue (1 hard break + stale 2026-08-23 filed -Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI… +Properly time and profile the smoke/release script surface + + + +2026-08-23 +filed +Phase 3: stop installing pynufft in Hands/Heart CI and PyAutoCTI @@ -345,7 +279,7 @@

Backlog 2026-08-22 filed -smoke_install.sh's stale jax<0.7 pin — CI is on the right jax… +smoke_install.sh's stale jax<0.7 pin — CI is on the right @@ -363,7 +297,7 @@

Backlog 2026-08-22 filed -The reconstruction noise map describes a different estimator than the… +The reconstruction noise map describes a different estimator than the @@ -378,16 +312,16 @@

Backlog Point-source JSON datasets record no resolution regime - + 2026-08-22 filed -Is Intel macOS a supported platform, and what is the numpy-only… +Is Intel macOS a supported platform, and what is the 2026-08-22 filed -Defer the eager scipy.sparse import in derivative_util (~0.10 s of… +Defer the eager scipy.sparse import in derivative_util (~0.10 s of @@ -405,13 +339,13 @@

Backlog 2026-08-21 filed -Numba PSF gathers derive the y/x kernel shifts from the wrong kernel… +Numba PSF gathers derive the y/x kernel shifts from the 2026-08-20 filed -Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… +Numba CPU likelihood phase 2: kernel-CDF numba fast path (the @@ -429,7 +363,7 @@

Backlog 2026-08-19 filed -jax 0.11 breaks beta/gamma message log_partition under jit… +jax 0.11 breaks beta/gamma message log_partition under jit ('tuple'… @@ -537,7 +471,7 @@

Backlog 2026-08-14 filed -Three jax_likelihood pins are stale by ~1.24e-4 and fail the smoke… +Three jax_likelihood pins are stale by ~1.24e-4 and fail the @@ -579,7 +513,7 @@

Backlog 2026-08-06 filed -Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead +Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are @@ -591,7 +525,7 @@

Backlog 2026-08-05 filed -Give PyAutoFit searches a seed — today no search can be made… +Give PyAutoFit searches a seed — today no search can @@ -603,7 +537,7 @@

Backlog 2026-08-04 filed -cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of… +cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of @@ -618,12 +552,6 @@

Backlog aplt.Output stale-API drift in the remaining workspace repos - -2026-08-04 -filed -Nightly release has been blocked 8 nights running — triage the streak - -

Epics markdown version

@@ -632,44 +560,49 @@

Epics

Cluster strong lensing — Source & Cluster arc — ledger: draft/feature/autolens/source_cluster_arc.md

-

PointSolver error-behavior change: bisect the 2025-11→2026-05 candidates, then health-harden🐛 bugautolenstoo-largesupervisedhigh

-

PointSolver profiling cells: lensed quasar → cluster runtime tier →🔬 researchautolens_profilinglargesupervisednormal

-

Critical curves: implement context-aware dispatch, dedupe engines, make clusters honor♻️ refactorautogalaxytoo-largesupervisedhigh

-

Mesh magnification correctness: simulate-and-recover across every mesh variant🧪 testworkspaceslargesupervisednormal

-

Magnification at a point: surface the existing API in source_science✨ featureautolenslargesupervisedhigh

-

Area magnification (LEGGOS-style): per-pixel inversion sum as primary; ShapeSolver rehabilitate-or-retire✨ featureautolenslargesupervisedhigh

-

Magnification errors via posterior draws, standalone in source_science✨ featureautolenslargesupervisednormal

-

Magnification maps: image-plane contour maps, source-plane mesh maps, uncertainty maps✨ featureautolenslargesupervisedhigh

-

Cluster source_science.py: robust magnification science at cluster scale (no meshes✨ featureworkspaceslargesupervisedhigh

-

Cluster package: point-source-default narrative + extended-source follow-up feature📖 docsworkspacesmediumsupervisedhigh

-

Cluster pixelized-source refinement: per-source masks via AnalysisFactor✨ featureworkspaceslargesupervisednormal

-

Cluster extended-source inference: gradient-based fitting building on JAX knowledge🔬 researchautolenslargesupervisednormal

-

HowToLens cluster tutorial: show a pixelized source + fix the📖 docshowtolenslargesupervisednormal

-

Source & Cluster arc — magnification science, PointSolver trust, cluster extended sources✨ featureautolenstoo-largesupervisedhigh

+

PointSolver error-behavior change: bisect the 2025-11→2026-05 candidates, then health-harden — autolens · too-large · supervised · high

+

PointSolver profiling cells: lensed quasar → cluster runtime tier → — autolens_profiling · large · supervised · normal

+

Critical curves: implement context-aware dispatch, dedupe engines, make clusters honor — autogalaxy · too-large · supervised · high

+

Mesh magnification correctness: simulate-and-recover across every mesh variant — workspaces · large · supervised · normal

+

Magnification at a point: surface the existing API in source_science — autolens · large · supervised · high

+

Area magnification (LEGGOS-style): per-pixel inversion sum as primary; ShapeSolver rehabilitate-or-retire — autolens · large · supervised · high

+

Magnification errors via posterior draws, standalone in source_science — autolens · large · supervised · normal

+

Magnification maps: image-plane contour maps, source-plane mesh maps, uncertainty maps — autolens · large · supervised · high

+

Cluster source_science.py: robust magnification science at cluster scale (no meshes — workspaces · large · supervised · high

+

Cluster package: point-source-default narrative + extended-source follow-up feature — workspaces · medium · supervised · high

+

Cluster pixelized-source refinement: per-source masks via AnalysisFactor — workspaces · large · supervised · normal

+

Cluster extended-source inference: gradient-based fitting building on JAX knowledge — autolens · large · supervised · normal

+

HowToLens cluster tutorial: show a pixelized source + fix the — howtolens · large · supervised · normal

+

Source & Cluster arc — magnification science, PointSolver trust, cluster — autolens · too-large · supervised · high

+ +
+Intermittent XLA compile stall in the JAX vmap likelihood path — 2 queued prompt(s), in order +

Intermittent XLA compile stall in the JAX vmap likelihood path — ledger: draft/bug/ci/jax_vmap_jit_compile_stall.md — CLOSED AS PARTIAL 2026-08-23 — record complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md

+

Phase 3: root-cause the XLA vmap compile stall and clear — ci · large · supervised · high

+

Intermittent XLA compile stall in JAX vmap likelihood scripts — — ci · too-large · supervised · high

Expectation propagation (EP) campaign — 9 queued prompt(s), in order

Expectation propagation (EP) campaign — ledger: draft/research/graphical_ep/ep_campaign.md

-

Analytic Gaussian benchmark: closed-form validation of graphical + EP🔬 researchgraphical_epmediumsupervisedhigh

-

EP hierarchical parent-scale collapse: cure the basin, or document the caveat🐛 bugautofittoo-largehuman-requiredhigh

-

Graphical Model Scale-Up — Scoping🔬 researchgraphical_eptoo-largesupervisedhigh

-

slope_hierarchy: scale the hierarchical slope recovery to N=25–50🔬 researchgraphical_epmediumsupervisednormal

-

The project @z_projects/ic50_workspace is our IC50 use case which we✨ featureautofitmediumsafenormal

-

IC50 use case: EP end-to-end with existing derived-variable handling🔬 researchgraphical_eplargesupervisednormal

-

EP campaign — phase map for the 2026 Q3 graphical/EP push🔬 researchgraphical_eptoo-largesupervisedhigh

-

Expectation Propagation Scale-Up — Scoping🔬 researchgraphical_eptoo-largesupervisedhigh

-

slope_hierarchy: methods write-up (NUTS headline, EP cautionary)🔬 researchgraphical_epmediumsupervisednormal

+

Analytic Gaussian benchmark: closed-form validation of graphical + EP — graphical_ep · medium · supervised · high

+

EP hierarchical parent-scale collapse: cure the basin, or document the — autofit · too-large · human-required · high

+

Graphical Model Scale-Up — Scoping — graphical_ep · too-large · supervised · high

+

slope_hierarchy: scale the hierarchical slope recovery to N=25–50 — graphical_ep · medium · supervised · normal

+

The project @z_projects/ic50_workspace is our IC50 use case which we — autofit · medium · safe · normal

+

IC50 use case: EP end-to-end with existing derived-variable handling — graphical_ep · large · supervised · normal

+

EP campaign — phase map for the 2026 Q3 graphical/EP — graphical_ep · too-large · supervised · high

+

Expectation Propagation Scale-Up — Scoping — graphical_ep · too-large · supervised · high

+

slope_hierarchy: methods write-up (NUTS headline, EP cautionary) — graphical_ep · medium · supervised · normal

- +

Boards: brain · heart · hands · memory · organism