From d3368ed781013326eb01d03576eaf04c72ca3e12 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 13:02:30 +0000 Subject: [PATCH 1/7] mind: sweep draft/ for shipped work, starting with the 18 PyAutoArray prompts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-08-08 planned.md prune found 6 of 13 entries describing work that had already shipped. draft/ is 150 prompts graded by no check at all, so it was always the larger exposure. This is the first target-repo pass: every prompt under draft/{feature,bug,research,refactor}/autoarray graded by reading its acceptance criteria against PyAutoArray main (efaf3041), not by trusting the prompt's own text about its state. 3 of 18 described shipped work. 1 more was half-shipped, 1 unstartable. RECORDED AS COMPLETE oversampling_kxs_coupling.md is the source prompt for the whole k x s series, which shipped over 2026-07-08/09 and closed PyAutoArray#362 as "series complete". Six dated records for its individual phases already sat in complete/2026/07 (kxs-design/core/cache/refactor/workspace-tests/ surface-refactor); only the prompt they all came from never advanced. Graded against its own five-part Scope, all met on main: the divisibility rule in _validate_convolve_over_sample_size, whose docstring names "the k x s coupling" verbatim; the partial pre-bin as over_sample_util.binned_to_convolve_size_from with uniform-k and adaptive-k tests; PyAutoGalaxy callers via 486; the workspace_test adaptive + s=2 leg via 154. Its section 5 was deliberately re-scoped, not skipped — kxs-core records the fork resolved as option (c) with option (a) split out as oversampled_psf_dataset_adoption.md, which correctly remains a live draft. rectangular_adapt_constant_split_guard.md is a duplicate of phase 1 of the rhayes777 audit epic, shipped 2026-07-28 as PyAutoArray#417 + PyAutoLens#662. The guard is at pixelization.py:154 raising PixelizationException off two capability flags, with all 9 rectangular x split combinations covered by test_split_regularization_support.py. The Mind already held this completion record — inside the sibling prompt rhayes_audit_validation_and_crashes.md, which planned.md tracks. One leg is recorded as NOT CONFIRMED: the prompt also wanted the af.Model composition form to fail before Nautilus starts, and no pre-fit model-inspection guard exists in PyAutoLens. RE-SCOPED IN PLACE nufft_simulator_chunking.md is overtaken. Its whole option-1 implementation is on main from PyAutoArray#330, merged 2026-05-22 — seven weeks BEFORE the Intake Agent retroactively formalised this prompt on 2026-07-08. It landed to the letter: the chunk_size kwarg under the prompt's own suggested name and default, jax.lax.scan rather than a Python loop, and the image_from adjoint the prompt flagged as out of scope. Its named sibling blocker shipped too (#329). What remains is one wiring leg, already recorded in interferometer-jax-jit.md: SimulatorInterferometer never sets chunk_size, so the scan branch is unreachable from the simulator. Difficulty too-large drops to small; priority stays high because the profiling sweep is still blocked. regularization_jax_gradient_gaps.md leg 3 marked DONE — the same #417 guard covers it. Leg 2 is now its only open work, and the "merge at intake if so" question it raises about the split-guard prompt is moot. rectangular_multi_submesh.md marked STALE PREMISE. Path B is written to subclass RectangularRotatedAdaptImage and build RectangularSplineAdaptImage per mode; neither class exists on main after the #402/#403 consolidation. All four research artefacts it says to read first do survive in files/, so the science holds and only the code the plan attaches to moved. ARCHIVED reg_matrix_logdet_nonfinite_fix.md self-declares WITHDRAWN 2026-07-17, kept for provenance, do NOT start. Moved to complete/archive/shelved/, which exists for exactly that. METHOD NOTE A slug-similarity scan of all 148 drafts against all 934 records MISSED the k x s finding entirely — that stem against kxs-core scores 0.25 Jaccard. Same weakness as the ep-optimise-updater case. Both cheap scans (slug similarity, and drafts cited by name inside record bodies) are pre-filters, not graders. Two of the three hits were provable from PyAutoMind alone, since the records were already in complete/. That pass is cheaper than cloning and should come first on the remaining 132. lifecycle check / orphans / index --check all OK; pytest tests/ 118 passed. draft/ 150 to 147. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E3MuurHXi3xo9TLRpMLJA6 --- .../2026/07/oversampling-kxs-coupling.md | 71 +++++++++++++++++ .../rectangular-adapt-constant-split-guard.md | 77 +++++++++++++++++++ .../reg_matrix_logdet_nonfinite_fix.md | 0 complete/index.md | 4 +- .../autoarray/nufft_simulator_chunking.md | 46 ++++++++++- .../autoarray/rectangular_multi_submesh.md | 34 +++++++- .../regularization_jax_gradient_gaps.md | 16 +++- 7 files changed, 243 insertions(+), 5 deletions(-) rename draft/feature/autoarray/oversampling_kxs_coupling.md => complete/2026/07/oversampling-kxs-coupling.md (61%) rename draft/feature/autoarray/rectangular_adapt_constant_split_guard.md => complete/2026/07/rectangular-adapt-constant-split-guard.md (51%) rename {draft/bug/autoarray => complete/archive/shelved}/reg_matrix_logdet_nonfinite_fix.md (100%) diff --git a/draft/feature/autoarray/oversampling_kxs_coupling.md b/complete/2026/07/oversampling-kxs-coupling.md similarity index 61% rename from draft/feature/autoarray/oversampling_kxs_coupling.md rename to complete/2026/07/oversampling-kxs-coupling.md index 09862598..19c048b1 100644 --- a/draft/feature/autoarray/oversampling_kxs_coupling.md +++ b/complete/2026/07/oversampling-kxs-coupling.md @@ -1,3 +1,74 @@ +# oversampling-kxs-coupling + +- shipped: 2026-07-09 (series closed; the prompt never left `draft/`) +- issue: https://github.com/PyAutoLabs/PyAutoArray/issues/362 (CLOSED — "series complete") +- prs: PyAutoArray#363, #364, #365 · PyAutoGalaxy#486, #489 · autolens_workspace#236 · autolens_workspace_test#154 (all merged) +- repos: + - PyAutoArray, PyAutoGalaxy + - autolens_workspace, autolens_workspace_test +- phase-records: [[kxs-design]] [[kxs-core]] [[kxs-cache]] [[kxs-workspace-tests]] [[kxs-refactor]] [[kxs-surface-refactor]] + +## Summary + +This is the **source prompt** for the k×s coupling series — the five-phase plan +it lays out was executed in full over 2026-07-08/09 and the series tracker +PyAutoArray#362 was closed as "series complete". Six dated completion records +already existed for the individual phases; only the prompt they all came from +was never advanced out of `draft/feature/autoarray/`. + +Recorded 2026-08-09 by the draft/ sweep. No work is owed. + +## Verified against PyAutoArray main (`efaf3041`), 2026-08-09 + +Graded against the prompt's own § Scope, not inferred from the phase records: + +1. **§1 relax the coupling rule — SHIPPED.** + `_validate_convolve_over_sample_size` (`autoarray/dataset/imaging/dataset.py:23`) + accepts any `over_sample_size` (int or adaptive `Array2D`) whose every entry + is divisible by `convolve_over_sample_size`, and raises `DatasetException` + naming the rule otherwise. Its docstring states the mechanism in the prompt's + own words — "the k x s coupling, whereby values evaluated at per-pixel sizes + k_i * s are partially binned to the uniform s the convolution requires". + The pre-change equality rule is gone. + +2. **§2 partial pre-bin — SHIPPED**, as + `over_sample_util.binned_to_convolve_size_from` + (`autoarray/operators/over_sampling/over_sample_util.py:205`). The placement + fork the prompt required a design paragraph for was settled and is recorded in + [[kxs-design]]. Covered by `test_over_sample_util.py` on both legs the prompt + asked for — `binned_to_convolve_size_from__uniform_k__equals_manual_reshape_mean` + and `__adaptive_k__and_identity_fast_path`. + +3. **§3 PyAutoGalaxy callers — SHIPPED** via PyAutoGalaxy#486 ([[kxs-core]]). + +4. **§4 tests — SHIPPED.** Library numpy-only tests as above; the + `convolution_over_sampled.py` adaptive + s=2 `FitImaging` leg the prompt + specifies landed as autolens_workspace_test#154 ([[kxs-workspace-tests]]: + lp round trip chi2=0, adaptive pixelized 6.6e-5, divisibility guard, + adaptive simulate→fit 1.1e-29). + +5. **§5 simulator adoption — DELIBERATELY RE-SCOPED, not skipped.** + [[kxs-core]] records the fork resolved as option (c): executed simulators stay + at `s=1`, and option (a) was split out as + `draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md`. That + prompt is correctly still a live draft — it is the residue of this §5, and is + NOT covered by this record. + +6. **Phase-5 refactor exercise — SHIPPED** as PyAutoGalaxy#489 + ([[kxs-surface-refactor]]), plus the extra [[kxs-cache]] and [[kxs-refactor]] + legs the series added on top of the original plan. + +## Why it was missed + +Nothing upstream drifted — the trackers were right the whole time, exactly as in +the 2026-08-08 `planned.md` prune. The prompt is stale purely because `draft/` is +graded by no check, and slug-similarity is too weak to catch it: this file's stem +against `kxs-core` scores a Jaccard of 0.25, well under any workable threshold. +What found it was reading the prompt's acceptance criteria against the tree — and +it was provable from PyAutoMind alone, since the six phase records were already +sitting in `complete/2026/07/`. + +## Original prompt # Oversampled PSF: k×s evaluation/convolution coupling + simulator adoption Type: feature diff --git a/draft/feature/autoarray/rectangular_adapt_constant_split_guard.md b/complete/2026/07/rectangular-adapt-constant-split-guard.md similarity index 51% rename from draft/feature/autoarray/rectangular_adapt_constant_split_guard.md rename to complete/2026/07/rectangular-adapt-constant-split-guard.md index ecfb8712..3a0a70ae 100644 --- a/draft/feature/autoarray/rectangular_adapt_constant_split_guard.md +++ b/complete/2026/07/rectangular-adapt-constant-split-guard.md @@ -1,3 +1,80 @@ +# rectangular-adapt-constant-split-guard + +- shipped: 2026-07-28 (phase 1 of the @rhayes777 audit epic; the prompt never left `draft/`) +- epic: https://github.com/PyAutoLabs/PyAutoArray/issues/415 (open — phases 2-4 remain) +- issue: https://github.com/PyAutoLabs/PyAutoArray/issues/332 (the reporter's finding), tracker #416 closed +- prs: PyAutoArray#417 (`9411904d`) + PyAutoLens#662 (`2a3f1a63`), both merged +- repos: + - PyAutoArray +- see-also: `draft/bug/autoarray/rhayes_audit_validation_and_crashes.md` § "Phase 1 completion record — 2026-07-28" + +## Summary + +A duplicate prompt for work that shipped as phase 1 of the @rhayes777 API-audit +epic. The Mind already held the completion record — inside the *sibling* prompt +`rhayes_audit_validation_and_crashes.md`, which planned.md tracks — but this +second, independently-filed prompt for the same surface never learned about it. + +Recorded 2026-08-09 by the draft/ sweep. No work is owed on the library leg; +one leg of the prompt's § Verification is noted below as unconfirmed. + +## Verified against PyAutoArray main (`efaf3041`), 2026-08-09 + +The prompt asks for "an explicit validation guard which rejects this unsupported +configuration early, with an error message that tells users what to do instead". +That guard is on main: + +- **Guard**: `Pixelization.__init__` (`autoarray/inversion/pixelization.py:154`) + raises `exc.PixelizationException` when a split regularization meets a mesh + that does not support it. The message names both classes and tells the user + the two ways out — an adaptive mesh (`Delaunay`/`KNNBarycentric`) with the same + regularization, or a non-split scheme (`Constant` for `ConstantSplit`, `Adapt` + for `AdaptSplit`) with the same mesh. Substantively the prompt's suggested text. +- **Mechanism**: two capability flags rather than a type blacklist — + `AbstractMesh.supports_split_regularization` (default `True`, set `False` on + the rectangular family) × `AbstractRegularization.is_split_regularization` + (default `False`, set `True` on `ConstantSplit`/`AdaptSplit`/`AdaptSplitZeroth`). +- **The false pass-through is gone**: `InterpolatorRectangular`'s claim that split + "reuses the same mappings" — the source of the `IndexError: index 4 is out of + bounds for axis 0 with size 4` this prompt reproduces — no longer stands; + `interpolator/rectangular.py:466` now records that the combination is rejected + at construction instead. + +Against the prompt's § Verification: + +1. **Criterion 1 (concrete construction raises) — MET.** + `test_autoarray/inversion/pixelization/test_split_regularization_support.py` + parametrizes all **9** rectangular × split combinations (the prompt reported + 1) and asserts both class names appear in the message. +2. **Criterion 2 (`af.Model` composition form fails before Nautilus starts) — + NOT CONFIRMED.** The guard sits in `Pixelization.__init__`, so it fires + whenever the model is instantiated rather than at composition time. A search + for `supports_split_regularization` in PyAutoLens returns nothing, so no + separate pre-fit model-inspection guard was added. In practice the concrete + guard is reached on the first instantiation, which is early — but that this + precedes sampling was not verified here and would need a run to settle. The + prompt itself allowed this ordering ("add the concrete PyAutoArray guard + first and add a companion AutoLens / analysis guard where the prior model can + be inspected"), so this is a possible residue, not a regression. +3. **Criterion 3 (allowed combinations still work) — MET.** Rectangular + + `Constant`, adaptive + split, and rectangular + no regularization all have + passing parametrized tests. +4. **Criterion 4 (low-level regression test) — MET in the form the fix took.** + Because the capability is deliberately absent rather than repaired, the tests + assert the *clear failure*; the test module says so explicitly. The prompt's + "do not paper over this by clipping indices" instruction was honoured. + +## Why it was missed + +Two prompts described one surface. `rhayes_audit_validation_and_crashes.md` came +in through the audit and is tracked by `planned.md`, so it was updated when phase +1 shipped; this one came in separately from a user repro +(`z_help/jacob/HerBS-28…`) and, sitting in `draft/`, was graded by nothing. The +related `draft/feature/autoarray/regularization_jax_gradient_gaps.md` § 3 flagged +the same surface a third time and asked "merge at intake if so" — that merge is +now moot, and its leg 3 has been marked done in place. + +## Original prompt # Users keep combining `RectangularAdaptDensity` meshes with `ConstantSplit` Type: feature diff --git a/draft/bug/autoarray/reg_matrix_logdet_nonfinite_fix.md b/complete/archive/shelved/reg_matrix_logdet_nonfinite_fix.md similarity index 100% rename from draft/bug/autoarray/reg_matrix_logdet_nonfinite_fix.md rename to complete/archive/shelved/reg_matrix_logdet_nonfinite_fix.md diff --git a/complete/index.md b/complete/index.md index 5afcb39f..b75b83d5 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -947 records across 7 buckets. +949 records across 7 buckets. ## Highlights @@ -383,6 +383,7 @@ _(curate hard-won records here — survives regeneration.)_ - [optional-none-default-typos](2026/07/optional-none-default-typos.md) — Fixed three PyAutoLens sites using the typing construct - [opus-wrapup](2026/07/opus-wrapup.md) — executed on Fable - [over-sample-trailing-one-to-two](2026/07/over-sample-trailing-one-to-two.md) — Replaced every trailing sub_size 1 in adaptive over-sampling schemes across five repos (169 files, 186 sites: … +- [oversampling-kxs-coupling](2026/07/oversampling-kxs-coupling.md) - [parked-sweep](2026/07/parked-sweep.md) - [per-frame-psf](2026/07/per-frame-psf.md) — per-frame native ePSFs live (psf/frame_epsf.py: sky-subtracted, DQ local-median patch in ESTIMATOR input only … - [per-project-literature](2026/07/per-project-literature.md) — task 3 of the autolens_assistant batch — hybrid literature rule live: Create scaffolds wiki/project/bibliograp… @@ -465,6 +466,7 @@ _(curate hard-won records here — survives regeneration.)_ - [python_312_floor_phase_4d_euclid_assistant](2026/07/python_312_floor_phase_4d_euclid_assistant.md) - [raw-guard-migration](2026/07/raw-guard-migration.md) — Leg 3 of the dataset-bulk series. Migrated 116 (autolens, 113 scripts) + 61 (autogalaxy, 59 scripts) raw `if n… - [rect-adapt](2026/07/rect-adapt.md) — rectangular adaptive-mesh edges — MERGED +- [rectangular-adapt-constant-split-guard](2026/07/rectangular-adapt-constant-split-guard.md) - [rectangular-kernel-cdf-mesh](2026/07/rectangular-kernel-cdf-mesh.md) — kernel-density CDF meshes RectangularKernelAdapt{Density,Image} (Enzi RTU) shipped opt-in — strict FD certifie… - [rectangular-mesh-consolidation](2026/07/rectangular-mesh-consolidation.md) — Consolidated PyAutoArray's rectangular mesh family from 8 mesh classes / 5 interpolators / 2 geometries down t… - [refactor-conductor](2026/07/refactor-conductor.md) diff --git a/draft/feature/autoarray/nufft_simulator_chunking.md b/draft/feature/autoarray/nufft_simulator_chunking.md index e7abe793..bff52abf 100644 --- a/draft/feature/autoarray/nufft_simulator_chunking.md +++ b/draft/feature/autoarray/nufft_simulator_chunking.md @@ -2,10 +2,52 @@ Type: feature Target: PyAutoArray -Difficulty: too-large +Difficulty: small Autonomy: supervised Priority: high -Status: formalised +Status: OVERTAKEN — re-scoped 2026-08-09, see the block below before reading further + +## 2026-08-09 — the library work below is SHIPPED; only the wiring is left + +Found by the `draft/` sweep. Verified against PyAutoArray main (`efaf3041`). + +**Everything in § "The fix" option 1 is already on main**, delivered by +**PyAutoArray#330 ("TransformerNUFFT: add chunk_size knob to cap nufftax gather +buffer"), merged 2026-05-22** — roughly seven weeks BEFORE the Intake Agent +retroactively formalised this prompt on 2026-07-08. It landed to the letter of +the plumbing section below: + +- `TransformerNUFFT.__init__` takes `chunk_size: Optional[int] = None` — this + prompt's suggested name and its "no chunking by default" default, so the + small-N `sma` callers pay nothing. A non-positive value raises `ValueError`. +- `_forward_native` (`autoarray/operators/transformer.py:681`) splits the + visibility axis and iterates with **`jax.lax.scan`** on the JAX path (a Python + loop only on the numpy path) — exactly the "do NOT use a Python `for`, it + unrolls and blows up JIT compile time" requirement below. +- `image_from` — the adjoint via `nufft2d1`, which this prompt flagged as "out + of scope today, but flag it" — **is chunked too**, in the same shape. + +The sibling blocker this prompt names has also shipped: the `apply_sparse_operator` +alma-scale precompute OOM closed 2026-05-22 as PyAutoArray#329 +([[alma-apply-sparse-operator-oom]]). + +**What is actually left is one wiring leg.** `complete/2026/07/interferometer-jax-jit.md` +records it: "`chunk_size` is a `TransformerNUFFT.__init__` argument that +`SimulatorInterferometer` **NEVER sets**, so the `lax.scan` branch is unreachable +via the simulator." So the capability exists and is untested-in-anger from the +simulator side. The remaining task is to plumb `chunk_size` from +`SimulatorInterferometer` (a default chosen by the memory budget already derived +below, ~1M for nspread=14/complex64 on a 40 GB working budget), then run the +§ Verification below to confirm alma_high actually lands. + +`Difficulty:` accordingly drops `too-large` → `small`; `Priority:` stays high +because the profiling sweep it unblocks is still blocked. Do **not** re-implement +the chunking. + +Note also that `option 2` below (upstream `nufftax` `chunk_size`) was never the +chosen scope and remains untouched — still a legitimate follow-up, still optional. + +--- The `al.SimulatorInterferometer` path that uses `al.TransformerNUFFT` (nufftax-backed) can't scale to ALMA-realistic visibility counts. At ~5M visibilities on an 800×800 real-space grid it OOMs on an A100 (80 GB) with a single ~15.7 GB allocation; at 10M it's ~31 GB. The likelihood path scales fine to the same regime because `apply_sparse_operator` precomputes a small W-Tilde matrix bounded by `N_source_pixels` (~thousands), not by `N_visibilities`. The simulator has no equivalent escape valve — every forward call does one dense nufftax spread. diff --git a/draft/feature/autoarray/rectangular_multi_submesh.md b/draft/feature/autoarray/rectangular_multi_submesh.md index 18b9f271..de1f8be2 100644 --- a/draft/feature/autoarray/rectangular_multi_submesh.md +++ b/draft/feature/autoarray/rectangular_multi_submesh.md @@ -5,7 +5,39 @@ Target: PyAutoArray Difficulty: too-large Autonomy: supervised Priority: normal -Status: formalised +Status: STALE PREMISE — needs re-basing before it can be issued (2026-08-09) + +## 2026-08-09 — the implementation base named below no longer exists + +Found by the `draft/` sweep, verified against PyAutoArray main (`efaf3041`). + +Path B is written as "subclass or compose alongside" `RectangularRotatedAdaptImage` +(Path A's mesh class), building one `RectangularSplineAdaptImage` per detected +mode. **Neither class is on main.** The mesh package is now exactly: + +``` +autoarray/inversion/mesh/mesh/{delaunay,knn,rectangular_adapt_density, + rectangular_adapt_image,rectangular_uniform}.py +``` + +They were removed by the rectangular-mesh consolidation (#402/#403), which also +moved the kernel-CDF machinery into `mesh_geometry/rectangular.py` and +`interpolator/rectangular.py`. So the class names, the subclassing plan, and the +"each sub-mesh runs the existing single-mode CDF code unchanged" claim all need +re-deriving against the consolidated classes before this can be issued. + +**The research context survives intact** — all four artefacts this prompt tells +you to read first are still present in the repo: `files/cdf_audit.md`, +`files/ghost_peak_findings.md`, `files/ghost_peak.png`, `files/pca_rotation.png` +(plus `ghost_peak_experiment.py` / `pca_rotation_experiment.py`). The separability +problem and the A/B/C fork are unaffected; only the code the plan attaches to moved. + +Before issuing: re-read the consolidated mesh classes, decide which one now plays +Path A's role (the PCA-rotation behaviour may live as a parameter rather than a +class), and rewrite § "The approach in more detail" step 2 against it. Everything +below this line predates the consolidation. + +--- Follow-up to `rectangular_adapt_cdf.md` (issue #322) and Path A (`RectangularRotatedAdaptImage`). The PCA-rotation hack we shipped fully diff --git a/draft/feature/autoarray/regularization_jax_gradient_gaps.md b/draft/feature/autoarray/regularization_jax_gradient_gaps.md index 6a609dad..754dcfa7 100644 --- a/draft/feature/autoarray/regularization_jax_gradient_gaps.md +++ b/draft/feature/autoarray/regularization_jax_gradient_gaps.md @@ -56,7 +56,21 @@ Reformulation candidates (in the spirit of the opt-in slogdet, PyAutoArray#391): Gate any change on the `regularization.py` jax_grad script re-passing and on FoM parity on the numpy path. -## 3. Split-family shape guard on rectangular meshes (papercut) +## 3. Split-family shape guard on rectangular meshes (papercut) — DONE + +*Shipped 2026-07-28 as phase 1 of the @rhayes777 audit epic — +PyAutoArray#417 (`9411904d`), merged. The first of the two options below is what +landed: `Pixelization.__init__` (`autoarray/inversion/pixelization.py:154`) +raises `PixelizationException` on any split regularization × rectangular mesh, +driven by `AbstractMesh.supports_split_regularization` × +`AbstractRegularization.is_split_regularization`, with all 9 combinations +covered by `test_autoarray/inversion/pixelization/test_split_regularization_support.py`. +True pixel-centre crosses for the rectangular geometry were deliberately NOT +implemented — the capability is absent by design. The +`rectangular_adapt_constant_split_guard.md` merge question this section raises is +therefore moot; that prompt was recorded as complete on 2026-08-09 +([[rectangular-adapt-constant-split-guard]]). Verified by the draft/ sweep against +main `efaf3041`. **Leg 2 is now the only open work in this prompt.*** `ConstantSplit`/`AdaptSplit`/`AdaptSplitZeroth` on a rectangular mesh fail with a raw broadcasting `TypeError` ((784,784) vs (3808,3808)): the split From fe5aa35fe16d25db2c2c31d7cac7afa686aee96c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 13:08:47 +0000 Subject: [PATCH 2/7] =?UTF-8?q?mind:=20second=20draft/=20sweep=20target=20?= =?UTF-8?q?=E2=80=94=20the=2022=20PyAutoFit-domain=20prompts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows the PyAutoArray pass in the previous commit. Same method: read each prompt's acceptance criteria against PyAutoFit main (3b960609), never trust the prompt's own text about its state. Covers draft/{bug,feature,refactor,research}/ autofit, maintenance+refactor/pyautofit, bug/priors and research/graphical_ep. RECORDED AS COMPLETE latent_samples_none_on_resumed_fit.md shipped as PyAutoFit#1418, merged 2026-07-25 — the SAME DAY the prompt records the finding from the full health sweep. Option (b) of its own Task section is what landed: latent_samples_from now opens with an explicit samples-is-None guard raising SamplesException, and the message names the cause and both remedies. Its open sub-question is answered by the guard being unconditional rather than test-mode-gated. This is the class of drift the Mind cannot detect by itself. There is NO completion record for #1418 anywhere in complete/, so no amount of cross-referencing draft prompts against the ledger would surface it. Worth knowing, because both PyAutoArray hits last commit WERE provable from the Mind alone — the cheap pass is real but it is not sufficient. test_mode_representative_outputs_size_realistic.md is an umbrella whose own status block records phases 1 and 2 COMPLETE and phases 3 and 4 ABSORBED, then states "EPIC CLOSES when #70 ships its recipe leg". autolens_profiling#70 is closed, state_reason completed, closed 2026-07-17 — the same day that status block was written. The condition was met within hours of the note and nothing advanced the file. This one needed neither a clone nor a code read, just one issue-state lookup. SHARPENED — a fix that reads like the prompt's but is not test_mode_bypass_ordered_assertion_ties.md still reproduces, and main now looks like it does not. The bypass path DOES now catch FitException and continue with the sentinel, which reads exactly like this prompt's suggested fix. But the catch wraps only the likelihood call: model.instance_from_vector sits on the line BEFORE the try, and that is where check_assertions raises FitException on an ordering tie at the prior medians. ignore_assertions defaults False and the bypass never passes it, so the assertion escapes the guard entirely. Annotated in place with that trace, because a future session reading main would very plausibly mark this shipped. The upside is that the fix is now a one-liner rather than the catch-and-perturb design the prompt sketches — either move the instantiation inside the existing try, or pass ignore_assertions at the bypass. Recorded both with the semantic difference between them stated. HALF SHIPPED 11_transformed_message_semantics_doc.md — the EP review's Phase 2 (PyAutoFit 1334) explicitly carried "incl. bug/priors/11 doc half", so which half mattered. Edge 1, the asymmetric reversal convention, is DONE: _transform and _inverse_transform both carry direction-naming docstrings calling the asymmetry deliberate and load-bearing, and graphical/README.md points at it. Edge 2 is NOT: LinearShiftTransform still has no docstring and the reciprocal call sits bare. Difficulty large drops to small, scope narrows to edge 2 only. remove_eden_packaging_tooling.md — autofit/tools/edenise is gone and a repo-wide grep returns zero hits, which settles the prompt's own guardrail check. Root eden.yaml remains. Also flagged that its follow-up names PyAutoConf, since renamed PyAutoNerves. Difficulty medium drops to small. UNBLOCKED, NOT SHIPPED Prompts 12, 13 and 14 all sequence behind Phases 1-2 of the EP framework review ("do not open the design issue until Phases 1-2 land"). That review completed 2026-07-08, all 8 phases — Phase 1 complete with F1-F9 on PyAutoFit#1332, Phase 2 shipped as #1334. The design input they were waiting on exists. Annotated all three as ready to plan. ep_analytic_updates WP1 says "land after PyAutoFit#1334" — merged; annotated to rebase rather than wait. Same shape as the canonical_key_todo_sweep case: a closed prerequisite means newly unblocked, not finished. VERIFIED GENUINELY OPEN cli_noise_pyautofit_batch (items 1, 3 and 4 all still present verbatim — no handler.close, evidence() at nautilus search.py:535, disp/iprint still passed), plot_functions_discard_kwargs (all five still take kwargs and reference it nowhere but the def lines), search_seed_reproducibility (no seed on AbstractSearch; the covariance test still monkeypatches dynesty's get_random_generator), messages_xp_stack_jax_trace (10 xp.array-of-list sites across normal/truncated_normal/beta/gamma), split_fitness_batch_size, priors 09 plus the 12/13/14 anchors (composed_transform reversal, np.reciprocal at transform.py:175, Prior.__getattr__ delegation — line numbers drifted, structures intact), ep_hierarchical_scale_collapse (#1405 open, accurately tracked), the two slope_hierarchy residues and ep_lbfgs_jax (external checkouts, laptop-only), and the two graphical_ep scoping umbrellas. lifecycle check / orphans / index --check all OK; pytest tests/ 118 passed. draft/ 147 to 145. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E3MuurHXi3xo9TLRpMLJA6 --- .../07/latent-samples-none-on-resumed-fit.md | 87 +++++++++++++++++++ ...e-representative-outputs-size-realistic.md | 42 +++++++++ complete/index.md | 4 +- .../latent_samples_none_on_resumed_fit.md | 39 --------- ...test_mode_bypass_ordered_assertion_ties.md | 47 +++++++++- .../11_transformed_message_semantics_doc.md | 31 ++++++- .../12_single_source_density_refactor.md | 14 +++ .../priors/13_collapse_prior_and_message.md | 14 +++ ..._replace_transform_stack_with_bijectors.md | 14 +++ draft/feature/autofit/ep_analytic_updates.md | 3 + .../remove_eden_packaging_tooling.md | 23 ++++- 11 files changed, 275 insertions(+), 43 deletions(-) create mode 100644 complete/2026/07/latent-samples-none-on-resumed-fit.md rename draft/feature/autofit/test_mode_representative_outputs_size_realistic.md => complete/2026/07/test-mode-representative-outputs-size-realistic.md (68%) delete mode 100644 draft/bug/autofit/latent_samples_none_on_resumed_fit.md diff --git a/complete/2026/07/latent-samples-none-on-resumed-fit.md b/complete/2026/07/latent-samples-none-on-resumed-fit.md new file mode 100644 index 00000000..28d21fc9 --- /dev/null +++ b/complete/2026/07/latent-samples-none-on-resumed-fit.md @@ -0,0 +1,87 @@ +# latent-samples-none-on-resumed-fit + +- shipped: 2026-07-25 (the same day the prompt recorded the finding) +- library-pr: https://github.com/PyAutoLabs/PyAutoFit/pull/1418 (merged) — "fix: raise SamplesException from latent computation when samples is None (resumed fits)" +- repos: + - PyAutoFit + +## Summary + +The prompt was filed from the 2026-07-25 full health sweep and fixed upstream the +same day. Nothing in PyAutoMind recorded it, so the prompt sat in `draft/` for two +weeks looking like open work. + +Recorded 2026-08-09 by the draft/ sweep. No work is owed. + +## Verified against PyAutoFit main (`3b960609`), 2026-08-09 + +The prompt's § Task offers two options — (a) reload persisted samples on the +resume path, or (b) raise a clear guarded error. **Option (b) is what landed**, +and its § Acceptance is met exactly: "either computes latent samples or fails with +an intentional, documented message — never an `AttributeError` from inside +`latent_samples_from`." + +`autofit/non_linear/analysis/latent.py` now opens `latent_samples_from` with an +explicit `if samples is None:` guard raising `exc.SamplesException`, and the +message names the cause and both remedies — samples output disabled via +`output.yaml`'s `samples: false` or `general.yaml`'s `samples_to_csv: false`, so +`result.samples` comes back `None` on the reload; enable samples output and re-run +from cleared output, or pass a `Samples` object explicitly. It also states why +`samples_summary.json` is not a substitute (latents are per-posterior-sample). + +The function's `Raises` docstring carries the same diagnosis the prompt reached +independently — that a completed fit short-circuits to +`NonLinearSearch.result_via_completed_fit`, which reloads from `samples.csv`. + +The prompt's open sub-question ("May be test-mode-specific — check whether a real +non-bypass resumed fit also returns samples=None") was answered by the fix taking +the general path: the guard is unconditional, not gated on `PYAUTO_TEST_MODE`. + +## Why it was missed + +This is the one class of drift the Mind cannot see by itself. Unlike the +PyAutoArray k×s finding, there is **no completion record for #1418 anywhere in +`complete/`** — the fix went in upstream without a Mind entry, so no amount of +cross-referencing draft prompts against the ledger would surface it. Only reading +the prompt's acceptance criteria against the upstream tree finds this shape. + +## Original prompt +# compute_latent_samples crashes on a resumed completed fit (samples is None) + +Type: bug +Target: autofit +Repos: +- @PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: low +Status: draft + +## Finding (2026-07-25 full health sweep) + +Running `autolens_workspace_test` `misc/latent/latent_variables_smoke.py` (and +`latent_nan_robustness.py`) twice in the same output tree fails on the second +run: the search resumes ("Fit Already Completed: skipping non-linear search"), +`result.samples` comes back `None` on the resume path (PYAUTO_TEST_MODE=2 +bypass), and + + autofit/non_linear/analysis/latent.py:113 latent_samples_from + -> samples.model -> AttributeError: 'NoneType' object has no attribute 'model' + +A fresh run (output cleared) passes. So the latent pipeline works, but the +resume/load path hands `compute_latent_samples` a `None` samples object +instead of the persisted samples (or a clear error). + +## Task + +Determine whether the resume path should (a) reload persisted samples so +latent computation works on resumed results, or (b) raise a clear, guarded +error from `compute_latent_samples` when samples are unavailable. May be +test-mode-specific — check whether a real (non-bypass) resumed fit also +returns samples=None. + +## Acceptance + +Second invocation of the latent smoke scripts in an existing output tree +either computes latent samples or fails with an intentional, documented +message — never an AttributeError from inside latent_samples_from. diff --git a/draft/feature/autofit/test_mode_representative_outputs_size_realistic.md b/complete/2026/07/test-mode-representative-outputs-size-realistic.md similarity index 68% rename from draft/feature/autofit/test_mode_representative_outputs_size_realistic.md rename to complete/2026/07/test-mode-representative-outputs-size-realistic.md index 7270a849..75fbbdbc 100644 --- a/draft/feature/autofit/test_mode_representative_outputs_size_realistic.md +++ b/complete/2026/07/test-mode-representative-outputs-size-realistic.md @@ -1,3 +1,45 @@ +# test-mode-representative-outputs-size-realistic + +- shipped: 2026-07-17 (the epic's own closing condition was met and never actioned) +- issues: PyAutoFit#1378 (phase 1), PyAutoFit#1381 + PyAutoConf#126 (phase 2), autolens_profiling#70 (the closing gate — CLOSED as `completed` 2026-07-17) +- phase-records: [[test-mode-representative-samples-phase-1-design]] [[test-mode-representative-samples-phase-2-core-api]] +- repos: + - PyAutoFit, PyAutoConf (now PyAutoNerves), autolens_profiling + +## Summary + +An umbrella prompt whose four phases were all resolved, tracked in the prompt's +own status line, and whose stated exit condition then quietly came true. + +The prompt is explicit that it is the umbrella — *"issue the phase files, not this +one"* — and its 2026-07-17 status block already recorded: + +- phase 1 (design) **COMPLETE**, PyAutoFit#1378, record in `complete/2026/07/` +- phase 2 (core API) **COMPLETE + MERGED**, PyAutoConf#126 + PyAutoFit#1381 +- phase 3 **ABSORBED** into slam-resume-profiling (autolens_profiling#70), prompt retired +- phase 4 **ABSORBED** (survey found RTD has no test-mode surface), prompt retired +- *"EPIC CLOSES when #70 ships its recipe leg."* + +## Verified 2026-08-09 + +**autolens_profiling#70 is closed, `state_reason: completed`, closed 2026-07-17 by +Jammy2211** — the same day the prompt's status block was last written. So the only +outstanding condition the umbrella named was satisfied within hours of the note +being taken, and nothing advanced the file. + +Recorded by the draft/ sweep. No work is owed. + +## Why it was missed + +This one needed neither a clone nor a code read — the prompt states its own exit +condition, and one issue-state lookup settles it. That makes it the cheapest class +of draft/ drift to detect and a good argument for extending the advisory +`issues --drafts` pass: #70 is named in the prompt body but is not the kind of +citation that pass currently nets, because the prompt cites it as a *gate* rather +than as its own tracking issue. Worth considering whether an "epic closes when X" +idiom should be machine-readable. + +## Original prompt # Test-mode representative outputs: size-realistic samples for instant pipeline runs Type: feature diff --git a/complete/index.md b/complete/index.md index b75b83d5..ef6b68e7 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -949 records across 7 buckets. +951 records across 7 buckets. ## Highlights @@ -310,6 +310,7 @@ _(curate hard-won records here — survives regeneration.)_ - [kxs-refactor](2026/07/kxs-refactor.md) - [kxs-surface-refactor](2026/07/kxs-surface-refactor.md) - [kxs-workspace-tests](2026/07/kxs-workspace-tests.md) +- [latent-samples-none-on-resumed-fit](2026/07/latent-samples-none-on-resumed-fit.md) - [lenstool-example](2026/07/lenstool-example.md) - [lenstool-scaling-reference-magnitude](2026/07/lenstool-scaling-reference-magnitude.md) — PR1 cluster + PR2 group/imaging — complete; PR3 SLaM deferred - [lifecycle-drift-self-heal](2026/07/lifecycle-drift-self-heal.md) @@ -544,6 +545,7 @@ _(curate hard-won records here — survives regeneration.)_ - [tenant-firewall-drift](2026/07/tenant-firewall-drift.md) - [test-mirror-restructure](2026/07/test-mirror-restructure.md) - [test-mode-bypass-completed](2026/07/test-mode-bypass-completed.md) — One-line fix — _fit_bypass_test_mode now calls paths.completed() before returning (mirrors start_resume_fit), … +- [test-mode-representative-outputs-size-realistic](2026/07/test-mode-representative-outputs-size-realistic.md) - [test-mode-representative-samples-phase-1-design](2026/07/test-mode-representative-samples-phase-1-design.md) - [test-mode-representative-samples-phase-2-core-api](2026/07/test-mode-representative-samples-phase-2-core-api.md) - [test-results-relayout](2026/07/test-results-relayout.md) — Hands#193 + Heart#106, lockstep pair diff --git a/draft/bug/autofit/latent_samples_none_on_resumed_fit.md b/draft/bug/autofit/latent_samples_none_on_resumed_fit.md deleted file mode 100644 index 28d3bab6..00000000 --- a/draft/bug/autofit/latent_samples_none_on_resumed_fit.md +++ /dev/null @@ -1,39 +0,0 @@ -# compute_latent_samples crashes on a resumed completed fit (samples is None) - -Type: bug -Target: autofit -Repos: -- @PyAutoFit -Difficulty: medium -Autonomy: supervised -Priority: low -Status: draft - -## Finding (2026-07-25 full health sweep) - -Running `autolens_workspace_test` `misc/latent/latent_variables_smoke.py` (and -`latent_nan_robustness.py`) twice in the same output tree fails on the second -run: the search resumes ("Fit Already Completed: skipping non-linear search"), -`result.samples` comes back `None` on the resume path (PYAUTO_TEST_MODE=2 -bypass), and - - autofit/non_linear/analysis/latent.py:113 latent_samples_from - -> samples.model -> AttributeError: 'NoneType' object has no attribute 'model' - -A fresh run (output cleared) passes. So the latent pipeline works, but the -resume/load path hands `compute_latent_samples` a `None` samples object -instead of the persisted samples (or a clear error). - -## Task - -Determine whether the resume path should (a) reload persisted samples so -latent computation works on resumed results, or (b) raise a clear, guarded -error from `compute_latent_samples` when samples are unavailable. May be -test-mode-specific — check whether a real (non-bypass) resumed fit also -returns samples=None. - -## Acceptance - -Second invocation of the latent smoke scripts in an existing output tree -either computes latent samples or fails with an intentional, documented -message — never an AttributeError from inside latent_samples_from. diff --git a/draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md b/draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md index 9f0e6811..3149fe80 100644 --- a/draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md +++ b/draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md @@ -7,7 +7,52 @@ Repos: Difficulty: small Autonomy: supervised Priority: normal -Status: formalised +Status: formalised — STILL REPRODUCES; see the 2026-08-09 note before grading this against main + +## 2026-08-09 — do NOT mistake the adjacent FitException catch for this fix + +Checked by the draft/ sweep against PyAutoFit main (`3b960609`). The bypass path +in `abstract_search.py` **now catches `exc.FitException`** and continues with the +`-1e99` sentinel, logging "TEST MODE 2: likelihood verification raised +FitException … treating as a resample-rejected instance". That reads exactly like +this prompt's suggested fix. **It is not.** The bug below still reproduces. + +The catch wraps only the likelihood call. The model instantiation is on the line +*before* the `try`: + +```python +if call_likelihood: + instance = model.instance_from_vector(vector=parameter_vector) # <-- outside + try: + log_likelihood = float(analysis.log_likelihood_function(instance)) + except exc.FitException as e: + ... +``` + +and `instance_from_vector` → `instance_for_arguments` → `check_assertions` +(`autofit/mapper/prior_model/abstract.py:193`) is precisely what raises +`exc.FitException("N assertions failed!")` when an ordering assertion ties at the +prior medians. `ignore_assertions` defaults to `False` and the bypass does not +pass it. So the assertion exception escapes the guard entirely and still +hard-fails the run. + +The upside: the fix is now a one-liner rather than the "catch and retry with a +perturbation" design sketched below. Two options, both cheap and both +deterministic: + +- move the `instance_from_vector` call inside the existing `try` — the sentinel + path already does the right thing for a rejected instance; or +- pass `ignore_assertions=True` at the bypass instantiation, on the grounds that + a verification eval at the medians is not a sampled point and assertions exist + to steer sampling. + +The second is probably the better semantics (a tied median is not a pathological +model), but it changes what the verification eval attests to — pick deliberately. +Prefer either over adding perturbation logic. + +`Difficulty:` stays small. The § Blocks note below still holds. + +--- Found during the CTI resurrection epic (Phase 4, 2026-07-17). `PYAUTO_TEST_MODE=2/3` bypass evaluates the model at the **prior medians**. A model whose components have diff --git a/draft/bug/priors/11_transformed_message_semantics_doc.md b/draft/bug/priors/11_transformed_message_semantics_doc.md index 7feb6353..211693a7 100644 --- a/draft/bug/priors/11_transformed_message_semantics_doc.md +++ b/draft/bug/priors/11_transformed_message_semantics_doc.md @@ -5,7 +5,36 @@ Target: priors Difficulty: large Autonomy: supervised Priority: normal -Status: formalised +Status: HALF SHIPPED — edge 1 is documented, edge 2 is not (2026-08-09) + +## 2026-08-09 — the reversal-convention half landed; the reciprocal half did not + +Checked by the draft/ sweep against PyAutoFit main (`3b960609`). The EP framework +review's Phase 2 (PyAutoFit#1334, shipped 2026-07-08) explicitly carried +"transform composition convention **incl. bug/priors/11 doc half**" — see +[[ep-framework-review]]. Verified which half: + +**Edge 1 (asymmetric reversal convention) — DONE.** `_transform` and +`_inverse_transform` in `messages/composed_transform.py` now both carry +docstrings naming the direction each maps ("physical → base" / "base → physical") +and stating that "the asymmetry with `_transform` is deliberate and load-bearing +(module docstring)". `autofit/graphical/README.md` § 2 also points readers at the +module docstring for the composition-order convention. That is the foot-gun this +prompt's § 1 exists to defuse. + +**Edge 2 (`LinearShiftTransform` stores the inverse of the intuitive scale) — NOT +DONE.** `messages/transform.py:171` is unchanged: the class still has **no +docstring at all**, and `super().__init__(DiagonalMatrix(np.reciprocal(self.scale)))` +sits bare — not even the `# Jacobian = 1/scale` inline comment this prompt writes +out. The physical-vs-base kwarg confusion that compounded the LogUniform sign bug +is still undocumented. + +**Remaining work is § 2 only.** Add the class docstring stating that `shift`/`scale` +describe physical space while the stored parent Jacobian is `1/scale` because the +transform runs physical → base, and keep the `log_det` sign explanation with it. +`Difficulty:` drops `large` → `small` accordingly; skip § 1 entirely. + +--- Found during the priors/messages audit (see `PyAutoPrompt/autofit/priors_and_messages_math_audit.md`, finding C6). diff --git a/draft/bug/priors/12_single_source_density_refactor.md b/draft/bug/priors/12_single_source_density_refactor.md index ed5ff590..9d58599b 100644 --- a/draft/bug/priors/12_single_source_density_refactor.md +++ b/draft/bug/priors/12_single_source_density_refactor.md @@ -160,6 +160,20 @@ kwarg as today; everything else inherits. into one issue. +## 2026-08-09 — PREREQUISITE HAS LANDED; this is unblocked + +Checked by the draft/ sweep. The Fable verdict below sequences this behind +Phases 1-2 of `research/graphical_ep/ep_framework_review.md`. That review +**completed 2026-07-08**, all 8 phases — record [[ep-framework-review]]. Phase 1 +(EP statistics correctness review) is complete with findings F1-F9 on +PyAutoFit#1332; Phase 2 (formal documentation of the graphical package) **shipped** +as PyAutoFit#1334, adding `autofit/graphical/README.md` (16 numbered +code-anchored equations) plus the statistical docstrings. + +So the design input this work package was waiting on — the EP review's inventory +of which message operations the factor graph actually needs — exists. The gate is +open; this is ready to plan, not blocked. Nothing here has shipped. + ## Fable verdict (2026-07-08, PyAutoFit main @ 0f26ff2d8; PyAutoFit#1330) **Verdict: still valid — design discussion; sequence behind the EP review.** diff --git a/draft/bug/priors/13_collapse_prior_and_message.md b/draft/bug/priors/13_collapse_prior_and_message.md index 2e59136b..1dcbbb52 100644 --- a/draft/bug/priors/13_collapse_prior_and_message.md +++ b/draft/bug/priors/13_collapse_prior_and_message.md @@ -176,6 +176,20 @@ the codebase ~half as legible as it could be. 7. **Stop. No code changes until a design is approved.** +## 2026-08-09 — PREREQUISITE HAS LANDED; this is unblocked + +Checked by the draft/ sweep. The Fable verdict below sequences this behind +Phases 1-2 of `research/graphical_ep/ep_framework_review.md`. That review +**completed 2026-07-08**, all 8 phases — record [[ep-framework-review]]. Phase 1 +(EP statistics correctness review) is complete with findings F1-F9 on +PyAutoFit#1332; Phase 2 (formal documentation of the graphical package) **shipped** +as PyAutoFit#1334, adding `autofit/graphical/README.md` (16 numbered +code-anchored equations) plus the statistical docstrings. + +So the design input this work package was waiting on — the EP review's inventory +of which message operations the factor graph actually needs — exists. The gate is +open; this is ready to plan, not blocked. Nothing here has shipped. + ## Fable verdict (2026-07-08, PyAutoFit main @ 0f26ff2d8; PyAutoFit#1330) **Verdict: still valid — bundle with 12 behind the EP review.** diff --git a/draft/bug/priors/14_replace_transform_stack_with_bijectors.md b/draft/bug/priors/14_replace_transform_stack_with_bijectors.md index 99280f5d..07cc25f3 100644 --- a/draft/bug/priors/14_replace_transform_stack_with_bijectors.md +++ b/draft/bug/priors/14_replace_transform_stack_with_bijectors.md @@ -198,6 +198,20 @@ one release cycle. the transform stack underneath `TransformedMessage`. +## 2026-08-09 — PREREQUISITE HAS LANDED; this is unblocked + +Checked by the draft/ sweep. The Fable verdict below sequences this behind +Phases 1-2 of `research/graphical_ep/ep_framework_review.md`. That review +**completed 2026-07-08**, all 8 phases — record [[ep-framework-review]]. Phase 1 +(EP statistics correctness review) is complete with findings F1-F9 on +PyAutoFit#1332; Phase 2 (formal documentation of the graphical package) **shipped** +as PyAutoFit#1334, adding `autofit/graphical/README.md` (16 numbered +code-anchored equations) plus the statistical docstrings. + +So the design input this work package was waiting on — the EP review's inventory +of which message operations the factor graph actually needs — exists. The gate is +open; this is ready to plan, not blocked. Nothing here has shipped. + ## Fable verdict (2026-07-08, PyAutoFit main @ 0f26ff2d8; PyAutoFit#1330) **Verdict: still valid — go/no-go after 12/13.** diff --git a/draft/feature/autofit/ep_analytic_updates.md b/draft/feature/autofit/ep_analytic_updates.md index 6f8f11f9..f610866d 100644 --- a/draft/feature/autofit/ep_analytic_updates.md +++ b/draft/feature/autofit/ep_analytic_updates.md @@ -18,6 +18,9 @@ do not bulk-issue): 1. **WP1** — exact PriorFactor updates in the declarative path + document the analytic-projection contract (~2–3 days). Start here. Note: rebase over / land after PyAutoFit#1334 (owns README.md). + **UNBLOCKED 2026-08-09 (draft/ sweep): PyAutoFit#1334 is MERGED** — it shipped + `autofit/graphical/README.md` as Phase 2 of the EP framework review on + 2026-07-08 ([[ep-framework-review]]). Rebase, don't wait. 2. **WP2** — first-class linear-Gaussian factor; validate against an IC50-shaped integration script (~1 week). 3. **WP3** — Gamma–Poisson / Beta–Bernoulli conjugate factors. diff --git a/draft/refactor/pyautofit/remove_eden_packaging_tooling.md b/draft/refactor/pyautofit/remove_eden_packaging_tooling.md index b1376d35..2003f8b9 100644 --- a/draft/refactor/pyautofit/remove_eden_packaging_tooling.md +++ b/draft/refactor/pyautofit/remove_eden_packaging_tooling.md @@ -8,7 +8,28 @@ Repos: Difficulty: medium Autonomy: supervised Priority: normal -Status: formalised +Status: PARTIALLY SHIPPED — the module is gone, `eden.yaml` is not (2026-08-09) + +## 2026-08-09 — half the primary scope has already landed + +Checked by the draft/ sweep against PyAutoFit main (`3b960609`): + +- **`autofit/tools/edenise` — GONE.** `autofit/tools/` now holds only + `__init__.py`, `add_notebook_quotes.py`, `namer.py`, `util.py`. A repo-wide + grep for `edenise` returns zero Python hits, so the § Guardrails + "confirm nothing imports it" check is settled by the removal itself. +- **Root `eden.yaml` — STILL PRESENT.** The second half of the § Scope + PyAutoFit bullet is outstanding. + +Also note the § Scope follow-up names **PyAutoConf**, which has since been renamed +**PyAutoNerves** (the `autonerves` package). Re-target `scripts/edenise.py` there +before acting, and confirm it survived the rename rather than assuming the path. + +What is left is deleting one dead config file plus the PyAutoNerves driver, so +`Difficulty:` drops `medium` → `small`. The guardrail about re-scoping if a live +consumer surfaces no longer applies to the PyAutoFit leg. + +--- ## Why From a37d04425d2277b7d0bc81083db8389072173ba2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 13:17:50 +0000 Subject: [PATCH 3/7] mind: measure `intake reconcile` against the sweep's findings, file the gap Two sweeps in, the obvious question is why the existing reconcile pass had not already caught this. Answer: it flags too much and misses the big ones. MEASURED, not asserted. Ran `pyauto-brain intake reconcile` against the pre-sweep tree (f25e154e) so its ranking could be scored against findings later confirmed by reading upstream source: 96 suspects of 148 scanned -- 65% flag rate (52 high / 20 medium / 24 low) of the 5 confirmed findings it flagged 2: the test-mode umbrella (high) and the latent-samples bug (low, i.e. buried) it MISSED the three largest -- oversampling_kxs_coupling (a whole shipped five-phase series), rectangular_adapt_constant_split_guard (PyAutoArray#417), nufft_simulator_chunking (PyAutoArray#330) So ~40% recall, and the one true positive at high sits among 51 other highs. This is a precision problem, not a missing-tool problem: the current matchers fire on prompts that merely reference each other, which is most of them. The read-only contract is right and should stay -- retiring a prompt is human. Filed draft/feature/pyautomind/draft_staleness_detection_signals.md with the three signals that actually found things, each grounded in a specific finding rather than speculated: 1. Machine-readable gates. The test-mode umbrella stated its own exit condition in prose ("EPIC CLOSES when #70") and #70 closed completed the same day that line was written. Proposes Closes-when: / Blocked-by: header keys, since the two readings are opposite -- gate closed means DONE, blocker closed means newly UNBLOCKED -- which is exactly the ambiguity that keeps `issues --drafts` advisory today. This sweep hand-annotated four unblocked cases (priors 12/13/14, ep_analytic_updates WP1); the key would have found them. 2. Prompt names an identifier that now exists upstream. The k x s prompt named _validate_convolve_over_sample_size, which is on main with the prompt's own phrase in its docstring; the nufft prompt asked for a chunk_size kwarg using lax.scan and got exactly that under its suggested name. Extract backticked identifiers, grep the target repo, rank on hits the prompt says should not exist. A treeless clone makes this cheap. 3. A completion record already names the deliverable. interferometer-jax-jit.md says outright that chunk_size is a TransformerNUFFT argument SimulatorInterferometer never sets -- that one sentence resolves the nufft prompt. Search record BODIES for the prompt's identifiers, not record slugs for the prompt's slug: slug similarity was measured here and is useless, scoring kxs_coupling against kxs-core at 0.25 Jaccard. Records the hard limit too, because it bounds what any such tool can promise: one of the five findings had NO signal in PyAutoMind at all. #1418 fixed the latent bug the same day it was filed and left no completion record anywhere, so the upstream read stays load-bearing and the Mind-only passes are a pre-filter. The deeper fix is that work shipping without a Mind record is the root cause; flagged as a separate question for the ship skills. And records the trap: test_mode_bypass_ordered_assertion_ties reads as shipped and is not, so an identifier-presence matcher would confidently mis-rank it. Acceptance criteria therefore include that it is NOT reported shipped, using f25e154e as the labelled set. Also regenerates dashboard.md, stale since the four prompts recorded complete in the previous two commits (148 to 144, still 6 hygiene flags). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E3MuurHXi3xo9TLRpMLJA6 --- dashboard.md | 20 +-- .../draft_staleness_detection_signals.md | 159 ++++++++++++++++++ 2 files changed, 167 insertions(+), 12 deletions(-) create mode 100644 draft/feature/pyautomind/draft_staleness_detection_signals.md diff --git a/dashboard.md b/dashboard.md index 86d4ee39..21c06e44 100644 --- a/dashboard.md +++ b/dashboard.md @@ -1,13 +1,13 @@ # PyAutoMind backlog dashboard - + -**148** filed prompts in the backlog · **6** already dispatched to issues (`active/`). Backlog view only — organism health lives with the Heart (`/health`), not here. +**144** filed prompts in the backlog · **6** already dispatched to issues (`active/`). Backlog view only — organism health lives with the Heart (`/health`), not here. | Work-type | Prompts | |-----------|--------:| -| bug | 42 | -| feature | 36 | +| bug | 40 | +| feature | 34 | | maintenance | 22 | | research | 18 | | docs | 17 | @@ -17,17 +17,15 @@ | experiment | 1 | | release | 1 | -## bug (42) +## bug (40) | Prompt | Target | Difficulty | Autonomy | Priority | |--------|--------|------------|----------|----------| | [PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional?](draft/bug/autoarray/PROBE_adapt_double_square_coefficient.md) | autoarray | medium | supervised | normal | | [ConstantZeroth regularization is broken twice over — dead code presenting](draft/bug/autoarray/constant_zeroth_broken_dead_code.md) | autoarray | small | supervised | normal | | [PyNUFFT dev extra is incompatible with current SciPy on Python](draft/bug/autoarray/pynufft_scipy_pinv2_dev_extra.md) | autoarray | small | supervised | normal | -| [[WITHDRAWN 2026-07-17] superseded by draft/feature/autoarray/gradient_safe_logdet_settings_option.md — the verdict ruled out](draft/bug/autoarray/reg_matrix_logdet_nonfinite_fix.md) | autoarray | medium | supervised | high | | [@rhayes777's 2026-05-23 API audit — all 16 findings re-verified, still](draft/bug/autoarray/rhayes_audit_validation_and_crashes.md) | autoarray | medium | supervised | high | | [EP: cure the hierarchical parent-scale collapse basin (and make F10](draft/bug/autofit/ep_hierarchical_scale_collapse_moment_match.md) | autofit | medium | supervised | high | -| [compute_latent_samples crashes on a resumed completed fit (samples is None)](draft/bug/autofit/latent_samples_none_on_resumed_fit.md) | autofit | medium | supervised | low | | [Make autofit.messages safe under JAX jit trace (xp.array → xp.stack)](draft/bug/autofit/messages_xp_stack_jax_trace.md) | autofit | large | supervised | normal | | [`autofit.plot` functions accept `**kwargs` and silently discard them](draft/bug/autofit/plot_functions_discard_kwargs.md) | autofit | small | supervised | normal | | [TEST_MODE bypass crashes on ordered-parameter assertion ties](draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md) | autofit | small | supervised | normal | @@ -64,22 +62,19 @@ | [Tenant firewall: release_run.py carries an unlisted 'PyAutoLabs' instance fact](draft/bug/pyautoheart/tenant_firewall_release_run_instance_fact.md) | pyautoheart | small | safe | normal | | [`aplt.Output` stale-API drift in the remaining workspace repos](draft/bug/workspaces/aplt_output_drift_remaining_repos.md) | workspaces | small | supervised | normal | -## feature (36) +## feature (34) | Prompt | Target | Difficulty | Autonomy | Priority | |--------|--------|------------|----------|----------| | [Claude Development Prompt: Arcsecond Tick Label Decimal Placement](draft/feature/autoarray/arcsecond_to_decimal.md) | autoarray | large | supervised | normal | | [Can create a list of InversionMatrix objects for each dataset](draft/feature/autoarray/multiwavelength_inversion.md) | autoarray | medium | supervised | normal | | [Column-chunk the interferometer inversion mapping-matrix NUFFT so alma_high fits on](draft/feature/autoarray/nufft_mapping_matrix_column_chunking.md) | autoarray | large | supervised | normal | -| [The `al.SimulatorInterferometer` path that uses `al.TransformerNUFFT` (nufftax-backed) can't scale to](draft/feature/autoarray/nufft_simulator_chunking.md) | autoarray | too-large | supervised | high | -| [Oversampled PSF: k×s evaluation/convolution coupling + simulator adoption](draft/feature/autoarray/oversampling_kxs_coupling.md) | autoarray | large | supervised | normal | -| [Users keep combining `RectangularAdaptDensity` meshes with `ConstantSplit`](draft/feature/autoarray/rectangular_adapt_constant_split_guard.md) | autoarray | too-large | supervised | high | +| [The `al.SimulatorInterferometer` path that uses `al.TransformerNUFFT` (nufftax-backed) can't scale to](draft/feature/autoarray/nufft_simulator_chunking.md) | autoarray | small | supervised | high | | [Follow-up to `rectangular_adapt_cdf.md` (issue #322) and Path A](draft/feature/autoarray/rectangular_multi_submesh.md) | autoarray | too-large | supervised | normal | | [Regularization JAX gradient gaps — xp-ports + kernel-scheme linear algebra](draft/feature/autoarray/regularization_jax_gradient_gaps.md) | autoarray | medium | supervised | normal | | [EP analytic updates — implement the four planned work packages](draft/feature/autofit/ep_analytic_updates.md) | autofit | large | supervised | normal | | [The project @z_projects/ic50_workspace is our IC50 use case which we](draft/feature/autofit/ep_lbfgs_jax.md) | autofit | medium | safe | normal | | [Give PyAutoFit searches a `seed` — today no search can](draft/feature/autofit/search_seed_reproducibility.md) | autofit | medium | supervised | medium | -| [Test-mode representative outputs: size-realistic samples for instant pipeline runs](draft/feature/autofit/test_mode_representative_outputs_size_realistic.md) | autofit | medium | supervised | normal | | [Remote-MCP deployment tiers (2 + 3) for the results-inspector server](draft/feature/autofit_assistant/remote_mcp_deployment_tiers.md) | autofit_assistant | large | human-required | normal | | [dPIE: optional central-dispersion (sigma_0) parameterization](draft/feature/autogalaxy/dpie_sigma0_parameterization.md) | autogalaxy | small | supervised | low | | [`PIEMass.potential_2d_from`: implement the missing lensing potential](draft/feature/autogalaxy/piemass_potential.md) | autogalaxy | too-large | supervised | normal | @@ -99,6 +94,7 @@ | [Extend the Profiling Agent scope to track JAX compile/eval times](draft/feature/profiling/profiling_agent_jax_compile_time_scope.md) | profiling | large | supervised | high | | [SamplerSurface: scan the autolens-side findings-lane tiers](draft/feature/pyautobrain/samplers_surface_autolens_tiers.md) | pyautobrain | small | safe | normal | | [Token-light wiki index over the complete/ archive](draft/feature/pyautomind/complete_archive_wiki.md) | pyautomind | medium | supervised | normal | +| [Make draft/ staleness detectable — `intake reconcile` measured, and the](draft/feature/pyautomind/draft_staleness_detection_signals.md) | pyautomind | medium | supervised | high | | [LACosmic per-frame CR masking option + decouple PSF-star pass from](draft/feature/pyautoreduce/lacosmic_cr_option_and_star_pass_decoupling.md) | pyautoreduce | medium | supervised | high | | [Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier](draft/feature/workspaces/gallery_runner_missing_tiers.md) | workspaces | small | supervised | low | | [The imaging `features/advanced/los_halos` example needs improving and padding out before](draft/feature/workspaces/group_los_halos.md) | workspaces | medium | safe | normal | diff --git a/draft/feature/pyautomind/draft_staleness_detection_signals.md b/draft/feature/pyautomind/draft_staleness_detection_signals.md new file mode 100644 index 00000000..7475f12c --- /dev/null +++ b/draft/feature/pyautomind/draft_staleness_detection_signals.md @@ -0,0 +1,159 @@ +# Make draft/ staleness detectable — `intake reconcile` measured, and the three signals that actually worked + +Type: feature +Target: PyAutoMind +Repos: +- PyAutoMind +- PyAutoBrain +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +## Why + +`draft/` is ~145 prompts graded by **no check at all**. The 2026-08-09 sweep read +acceptance criteria against upstream `main` for two target clusters — 18 +PyAutoArray prompts, then 22 PyAutoFit-domain prompts — and found roughly a third +carrying stale state: + +| outcome | count (of 40) | +|---|---| +| shipped, recorded to `complete/` | 4 | +| half-shipped (scope narrowed in place) | 2 | +| unblocked by a since-closed prerequisite | 4 | +| unstartable (premise removed upstream) | 1 | +| withdrawn, archived | 1 | +| **mis-gradeable** (an adjacent upstream fix reads like the prompt's) | 1 | + +At that rate the remaining ~105 prompts hold real drift. Doing it by hand is +expensive; the question is what can be mechanised. + +## The measurement — `intake reconcile` as it stands + +A reconcile pass already exists (`pyauto-brain intake reconcile` — "rank backlog +prompts that look already-shipped … always read-only"). It was run against the +pre-sweep tree (PyAutoMind `f25e154e`) so its output could be scored against +findings that were later confirmed by reading upstream source. Result: + +- **96 suspects of 148 scanned** — a 65% flag rate (52 `high`, 20 `medium`, 24 `low`). +- Of the 5 confirmed findings, it flagged **2**: the test-mode umbrella (`high`) + and the latent-samples bug (`low`, i.e. buried). +- It **missed the three largest** — `oversampling_kxs_coupling` (a whole shipped + 5-phase series), `rectangular_adapt_constant_split_guard` (shipped as + PyAutoArray#417), `nufft_simulator_chunking` (shipped as PyAutoArray#330). + +So ~40% recall, and the one true positive at `high` is indistinguishable from 51 +other `high`s. **This is a precision problem, not a missing-tool problem.** The +existing matchers — cross-file references and shared topic words — fire on +prompts that merely *mention* each other, which is most of them. Do not rewrite +reconcile from scratch; make its ranking discriminative. + +Not a criticism of the tool's existence: it is read-only by design and retiring a +prompt is meant to stay human. The goal here is a signal a human can act on. + +## The three signals that actually found things + +Each is what surfaced a specific confirmed finding, so each is grounded rather +than speculative. + +### 1. Machine-readable "epic closes when X" gates (cheapest; highest precision) + +`test_mode_representative_outputs_size_realistic.md` stated its own exit +condition in prose — *"EPIC CLOSES when #70 ships its recipe leg"* — and +autolens_profiling#70 closed as `completed` on 2026-07-17, **the same day that +status line was written**. One issue-state lookup settles the prompt; no clone, +no code read. + +`lifecycle.py issues --drafts` does not net this, because it treats a cited issue +as *context* and this one is a *gate*. Proposal: a header key, e.g. + +``` +Closes-when: https://github.com/PyAutoLabs/autolens_profiling/issues/70 +Blocked-by: https://github.com/PyAutoLabs/PyAutoFit/issues/1331 +``` + +`Closes-when` closed → the prompt is **done**; `Blocked-by` closed → the prompt is +**newly unblocked**. Both are actionable and the two readings are opposite, which +is exactly the ambiguity that makes today's `--drafts` advisory-only. Backfill the +key on the prompts that already say it in prose (this sweep annotated four +unblocked-by-a-closed-gate cases by hand: `bug/priors/12`, `13`, `14`, and +`ep_analytic_updates` WP1). + +### 2. Prompt names an identifier that now exists upstream + +`oversampling_kxs_coupling.md` § Scope named `_validate_convolve_over_sample_size` +and a partial pre-bin util. Both are on PyAutoArray `main`, and the validator's +**docstring uses the prompt's own phrase** ("the k x s coupling"). Likewise +`nufft_simulator_chunking.md` asked for a `chunk_size` kwarg using `jax.lax.scan`; +`TransformerNUFFT.__init__` has exactly that, under the prompt's suggested name. + +Mechanisable: extract backticked `snake_case` / `CamelCase` identifiers from a +prompt's scope/acceptance sections, grep the target repo's `main` (an anonymous +treeless clone is seconds — `GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1 +--filter=blob:none`), and rank on **hits for identifiers the prompt says do not +exist yet**. Far more discriminative than shared topic words, and it is what a +human grader is doing anyway. + +### 3. A completion record already names the prompt's deliverable + +`complete/2026/07/interferometer-jax-jit.md` says outright: "`chunk_size` is a +`TransformerNUFFT.__init__` argument that `SimulatorInterferometer` NEVER sets". +That sentence resolves `nufft_simulator_chunking.md` — library shipped, wiring +not. Similarly the split-guard prompt's twin, `rhayes_audit_validation_and_crashes.md`, +carried a full "Phase 1 completion record" for the same surface. + +So: search record **bodies** for the prompt's identifiers, not record **slugs** for +the prompt's slug. Slug similarity was measured on this sweep and is useless here +— `oversampling_kxs_coupling` against `kxs-core` scores a Jaccard of **0.25**, +under any workable threshold, and the whole scan missed that finding. + +## Hard limit — worth stating so nobody over-promises + +**One of the five findings had no signal in PyAutoMind at all.** The +`latent_samples_none_on_resumed_fit` bug was fixed by PyAutoFit#1418 the same day +it was filed, and **no completion record for #1418 exists anywhere in `complete/`**. +No amount of ledger cross-referencing surfaces that. Any design must accept that +the upstream read is load-bearing and the Mind-only passes are a cheap pre-filter, +not a substitute. + +Related, and arguably the deeper fix: work that ships without a Mind record is the +root cause here. Worth asking separately whether the ship skills can fail louder. + +## The trap any such tool must not create + +`test_mode_bypass_ordered_assertion_ties.md` reads as shipped and is not. Main now +catches `exc.FitException` in the TEST_MODE bypass — which looks exactly like the +prompt's requested fix — but the catch wraps only the likelihood call, while +`model.instance_from_vector` (where `check_assertions` actually raises on an +ordering tie) sits on the line *before* the `try`. An identifier-presence matcher +would score this "shipped" with high confidence and be wrong. + +**So the tool must rank for human review and never retire a prompt itself** — which +is already reconcile's stated contract. Keep it. + +## Scope + +1. Add `Closes-when:` / `Blocked-by:` header keys (README "Prompt file format"), + backfill from prose where prompts already state a gate, and grade them in + `lifecycle.py issues --drafts` with the two readings reported separately. + This leg alone is small and worth landing first. +2. Re-rank `intake reconcile`: demote bare cross-references and shared topic + words; promote identifier-presence and record-body-names-deliverable. Target a + flag rate that a human can actually work through — under ~15% of the backlog, + against the 65% measured today. +3. Optional: a `--repo ` mode that does the treeless clone and runs leg 2 + for one target cluster, which is the shape the manual sweep took. + +## Acceptance + +- Re-running the ranker against PyAutoMind `f25e154e` (the pre-sweep tree, which + is the labelled set this prompt establishes) puts **all five** confirmed + findings in the top band, and flags materially fewer than 96 prompts. +- `test_mode_bypass_ordered_assertion_ties.md` is NOT reported as shipped. +- No prompt is moved or retired by the tool. + + From 99264b19201a9fe2b5d564063b2e4a221ac13d1f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 13:22:27 +0000 Subject: [PATCH 4/7] =?UTF-8?q?mind:=20third=20draft/=20sweep=20target=20?= =?UTF-8?q?=E2=80=94=20the=20autolens=5Fworkspace=20prompts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six of the eight prompts filed against autolens_workspace graded by reading acceptance criteria against workspace main (9974f891). Two not re-graded: markdown_regeneration_sigma_min (known laptop-only, needs real fits) and joss_cluster_benchmark_tuning. RECORDED AS COMPLETE normalise_auto_simulate_guard_idiom.md — all four hand-rolled guard sites now call al.util.dataset.should_simulate, and a repo-wide sweep finds zero remaining hand-rolled data.fits-exists simulator guards against 492 files on the standard idiom. Its § Proposed work item 2 warned "do not silently drop" the mass.csv check in cluster/likelihood_function.py. The shipped form keeps it AND fixes an ordering subtlety the prompt did not raise: should_simulate is evaluated first, with a comment saying so, because written the other way round a present mass.csv would short-circuit past the PYAUTO_SMALL_DATASETS rebuild — exactly the failure the conversion exists to prevent. Item 4 (an optional required_files= argument on should_simulate) was not taken; it was explicitly optional and would have made this library+workspace rather than workspace-only. Two if-not-data_fits_path-exists sites survive and are correctly untouched: they guard one-off urllib downloads of real HST data (RXJ1131, Abell 2744), not simulator runs, so should_simulate's capped-rebuild semantics would delete a downloaded file in order to re-download it. Converting them would regress. Notable for the detection prompt filed last commit: this is the ONE finding across three sweeps that intake reconcile ranked high for the right reason — the auto-simulate-guard-targets record names the prompt's path in its body, which is signal 3 working as intended. It still arrived among 51 other highs. UNBLOCKED sampler_cli_output_workspace_sweep.md says "Do not start until #1436 has merged". PyAutoFit#1436 merged 2026-07-30. Ready to start, nothing shipped. Refreshed its counts while there, since it estimated "14+": 19 .py scripts still print the line, 16 still carry the "cell with progress" typo, and 54 notebooks carry it. Flagged that the notebook count far exceeds the script count, so whether all 54 regenerate from those 19 needs checking before assuming a regeneration pass covers them. That makes FIVE gate-closed-but-prompt-unaware cases across three sweeps, which is the strongest evidence yet for the Closes-when:/Blocked-by: header keys proposed in draft/feature/pyautomind/draft_staleness_detection_signals.md. MEASURED, STILL OPEN latex_docstrings_invalid_escape_warnings.md asks for a count before anyone proposes a fix, so the count is now in the file: 80 warnings across 17 files in autolens_workspace scripts/ alone — six times the "roughly a dozen per script across the four" it was filed on, confirming its own instinct that this is not confined to potential_correction. Recorded the per-escape breakdown and the worst files, which are all likelihood_function.py, i.e. the LaTeX-heavy derivations. Sibling workspaces and HowTo* repos remain uncounted. Recorded a trap in the same block, because it bit during this sweep: the prompt's own suggested command uses -W always::SyntaxWarning, and these escapes are only a SyntaxWarning on Python 3.12+. On 3.11 they are a DeprecationWarning, so that exact command reports ZERO and looks precisely like "already fixed". Reading the source directly is what caught the mis-grade. Also needs -f, or __pycache__ suppresses recompilation and the count silently drops. VERIFIED GENUINELY OPEN script_local_pixel_scale_vs_dataset_pixel_scales — the cited line still reads image_half_width = 0.5 * min(dataset_full.shape_native) * pixel_scale with the literal, and config/build/no_run.yaml's NEEDS_FIX reason still names only the 0.0-luminosity cause, exactly as the prompt says needs updating. cosmos_web_ring_mask_dtype — all five mask_extra_galaxies.fits still BITPIX -64 (float64), dataset still 12 MB. oversampled_psf_dataset_adoption is the correct live residue of the k x s series § 5, established two commits ago. Also regenerates dashboard.md (144 to 143). lifecycle check / orphans / index --check all OK; pytest tests/ 118 passed. draft/ 145 to 144. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E3MuurHXi3xo9TLRpMLJA6 --- .../08/normalise-auto-simulate-guard-idiom.md | 68 +++++++++++++++++++ complete/index.md | 3 +- dashboard.md | 7 +- .../sampler_cli_output_workspace_sweep.md | 25 +++++++ ...atex_docstrings_invalid_escape_warnings.md | 59 +++++++++++++++- 5 files changed, 156 insertions(+), 6 deletions(-) rename draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md => complete/2026/08/normalise-auto-simulate-guard-idiom.md (50%) diff --git a/draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md b/complete/2026/08/normalise-auto-simulate-guard-idiom.md similarity index 50% rename from draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md rename to complete/2026/08/normalise-auto-simulate-guard-idiom.md index d4429732..0a48ebf1 100644 --- a/draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md +++ b/complete/2026/08/normalise-auto-simulate-guard-idiom.md @@ -1,3 +1,71 @@ +# normalise-auto-simulate-guard-idiom + +- shipped: verified on autolens_workspace main `9974f891` (the prompt never left `draft/`) +- follows: [[auto-simulate-guard-targets]] (autolens_workspace#359 → #364, autogalaxy_workspace#175) and the `should_simulate` migration (autolens_workspace#354) +- repos: + - autolens_workspace + +## Summary + +The prompt asks for four hand-rolled auto-simulate guards to be converted to +`al.util.dataset.should_simulate`, with one specific caution about not dropping a +stricter check while doing it. All four are converted on main, and the caution was +honoured — more carefully than the prompt asked. + +Recorded 2026-08-09 by the draft/ sweep. No work is owed. + +## Verified against autolens_workspace main (`9974f891`), 2026-08-09 + +Against the prompt's § Proposed work: + +1. **All four idiom-B sites converted — DONE.** `cluster/likelihood_function.py`, + `interferometer/features/pixelization/many_visibilities_preparation.py`, + `imaging/features/advanced/subhalo/sensitivity/slam_source_parametric.py` and + `…/slam_source_pixelized.py` all call `should_simulate` now. A repo-wide sweep + finds **zero** remaining hand-rolled `data.fits … .exists()` simulator guards + (492 files use the standard idiom). + +2. **The `mass.csv` check was preserved — DONE, and better than specified.** The + prompt warned "do not silently drop it" and offered two ways out. The shipped + form takes the first *and* fixes an ordering subtlety the prompt did not raise: + + ```python + if ( + al.util.dataset.should_simulate(str(dataset_path)) + or not (dataset_path / "mass.csv").exists() + ): + ``` + + with a comment recording that `should_simulate` is evaluated **first** so its + `PYAUTO_SMALL_DATASETS` rebuild always runs. Written the other way round, a + present `mass.csv` would short-circuit past the capped-rebuild side effect — + which is precisely the failure the conversion exists to prevent. + +3. **Re-running under the capped profile** — not independently re-run here; that + needs a real execution environment. + +4. **`required_files=[...]` on `should_simulate` — NOT taken.** PyAutoArray's + `should_simulate(dataset_path)` still has the single argument. This leg was + explicitly optional in the prompt ("consider whether…", and it would have made + the task library+workspace rather than workspace-only). The `mass.csv` clause + living in the script is the alternative the prompt allowed. + +**Not the same class, correctly left alone:** two `if not data_fits_path.exists():` +sites survive, in `multi_dataset/features/imaging_and_point_source/modeling.py:75` +and `cluster/start_here.py:153`. Both guard a one-off `urllib` download of real HST +data (RXJ1131, Abell 2744), not a simulator invocation, so `should_simulate`'s +capped-rebuild semantics would be actively wrong there — it would delete a +downloaded file to re-download it. Converting these would be a regression. + +## Note on detection + +This is the one finding across three sweeps that `pyauto-brain intake reconcile` +ranked `high` for the right reason: [[auto-simulate-guard-targets]] names the +prompt's path directly in its body. That is signal 3 in +`draft/feature/pyautomind/draft_staleness_detection_signals.md` working as +intended — though it still arrived among 51 other `high`s. + +## Original prompt # Normalise the two auto-simulate guard idioms Type: maintenance diff --git a/complete/index.md b/complete/index.md index ef6b68e7..466f4942 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -951 records across 7 buckets. +952 records across 7 buckets. ## Highlights @@ -50,6 +50,7 @@ _(curate hard-won records here — survives regeneration.)_ - [missing-auto-simulate-guards](2026/08/missing-auto-simulate-guards.md) - [multi-start-auto-convergence-real-search](2026/08/multi-start-auto-convergence-real-search.md) — `scripts/jax_assertions/multi_start_gradient_auto_convergence.py` failed - [nautilus-1core-serial-pool](2026/08/nautilus-1core-serial-pool.md) — corrective for the Heart RED "release validation FAILED (stage +- [normalise-auto-simulate-guard-idiom](2026/08/normalise-auto-simulate-guard-idiom.md) - [notebook-setup-notebook-regen-drift](2026/08/notebook-setup-notebook-regen-drift.md) - [nufft-parity-full-datasets](2026/08/nufft-parity-full-datasets.md) - [plot-array-stale-kwargs](2026/08/plot-array-stale-kwargs.md) diff --git a/dashboard.md b/dashboard.md index 21c06e44..28309f58 100644 --- a/dashboard.md +++ b/dashboard.md @@ -2,13 +2,13 @@ -**144** filed prompts in the backlog · **6** already dispatched to issues (`active/`). Backlog view only — organism health lives with the Heart (`/health`), not here. +**143** filed prompts in the backlog · **6** already dispatched to issues (`active/`). Backlog view only — organism health lives with the Heart (`/health`), not here. | Work-type | Prompts | |-----------|--------:| | bug | 40 | | feature | 34 | -| maintenance | 22 | +| maintenance | 21 | | research | 18 | | docs | 17 | | refactor | 5 | @@ -101,7 +101,7 @@ | [The imaging `features/advanced/subhalo/sensitivity` example needs improving and padding out before](draft/feature/workspaces/group_subhalo_sensitivity.md) | workspaces | medium | safe | normal | | [Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver](draft/feature/workspaces/restore_multiple_sources_lensing_of_lens.md) | workspaces | too-large | supervised | normal | -## maintenance (22) +## maintenance (21) | Prompt | Target | Difficulty | Autonomy | Priority | |--------|--------|------------|----------|----------| @@ -109,7 +109,6 @@ | [autolens_profiling is now a mature project, with a good separation](draft/maintenance/autolens_profiling/polish.md) | autolens_profiling | large | supervised | normal | | [cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of](draft/maintenance/autolens_workspace/cosmos_web_ring_mask_dtype.md) | autolens_workspace | small | supervised | low | | [LaTeX in non-raw docstrings emits SyntaxWarning: invalid escape sequence](draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md) | autolens_workspace | small | supervised | low | -| [Normalise the two auto-simulate guard idioms](draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md) | autolens_workspace | small | supervised | low | | [autolens_workspace_developer rectangular experiments — Gut stash + rename](draft/maintenance/autolens_workspace_developer/rectangular_experiments_gut_stash.md) | autolens_workspace_developer | small | supervised | normal | | [autolens_workspace_developer: broad stale-API rot (56 symbols, no CI)](draft/maintenance/autolens_workspace_developer/stale_api_rot_audit.md) | autolens_workspace_developer | medium | supervised | normal | | [Auto-request GitHub Copilot code review on every PR, org-wide](draft/maintenance/ci/copilot_auto_review.md) | ci | large | supervised | normal | diff --git a/draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md b/draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md index 4d666569..77f7cf44 100644 --- a/draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md +++ b/draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md @@ -3,6 +3,31 @@ Follow-up to **PyAutoFit#1434 / PR#1436**, which moved the on-the-fly update cadence message into the library. Do not start until #1436 has merged. +## 2026-08-09 — UNBLOCKED, and the counts refreshed + +Checked by the draft/ sweep. **PyAutoFit#1436 merged 2026-07-30T21:49:25Z** +(`daa0dbcb`, closes #1434), so the gate above is satisfied — this is ready to +start, not waiting. Nothing here has shipped. + +Counts re-measured against autolens_workspace main (`9974f891`), which sharpens +the table below (it estimated "14+"): + +| surface | count | +|---|--:| +| `.py` scripts still printing `On-the-fly updates every iterations_per_quick_update` | **19** | +| of those, still carrying the "notebook cell **with** progress" typo | **16** | +| `.ipynb` notebooks carrying the line | **54** | + +The notebook count is much larger than the script count because the line also +appears in generated notebooks across sibling directories — confirm whether those +are all regenerated from the 19 scripts, or whether some notebooks are authored +directly, before assuming a regeneration pass covers them. + +For what the library now emits in its place, see PyAutoFit#1436's own summary: the +message has two branches, because the packaged default cadence is the inf-like +`1e99` never-sentinel, so the replacement text is either a real integer cadence or +a statement that updates are disabled naming the config key. + ## Problem 23 workspace scripts print this block before `search.fit(...)`: diff --git a/draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md b/draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md index 19ba9e30..b1c011fc 100644 --- a/draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md +++ b/draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md @@ -7,7 +7,64 @@ Repos: Difficulty: small Autonomy: supervised Priority: low -Status: draft +Status: draft — the § "Scope to establish first" measurement is DONE for autolens_workspace (2026-08-09) + +## 2026-08-09 — the sweep this prompt asks for, run + +Still open; nothing has been fixed. But § "Scope to establish first" says to report +the count and affected repos **before** proposing a change, so here is that number +for autolens_workspace main (`9974f891`): + +**80 warnings across 17 files** in `scripts/` — six times the "roughly a dozen +per script across the four" this prompt was filed on, and well beyond +potential_correction. So the prompt's own instinct ("do NOT assume this is +confined to potential_correction") was right. + +By escape sequence: + +| seq | n | | seq | n | | seq | n | +|---|--:|---|---|--:|---|---|--:| +| `\d` | 15 | | `\e` | 7 | | `\l` | 4 | +| `\c` | 15 | | `\k` | 5 | | `\*` | 3 | +| `\s` | 10 | | `\,` | 5 | | `\p` | 2 | +| `\m` | 9 | | `\o` | 4 | | `\h` | 1 | + +Worst files (all `likelihood_function.py`, i.e. the maths-heavy prose): + +``` +12 scripts/imaging/features/advanced/potential_correction/likelihood_function.py +10 scripts/interferometer/features/advanced/potential_correction/likelihood_function.py + 8 scripts/interferometer/features/pixelization/likelihood_function.py + 8 scripts/imaging/features/pixelization/likelihood_function.py + 7 scripts/interferometer/likelihood_function.py + 7 scripts/imaging/likelihood_function.py +``` + +The concentration in `likelihood_function.py` files is a useful shape: these are +the scripts that carry LaTeX-heavy derivations, so option 1 (raw docstrings) would +touch mostly files whose rendered output is already equation-dense — check the +`r"""` prefix's appearance there specifically before choosing. + +**The sibling workspaces and HowTo* repos are still uncounted** — this measurement +covers autolens_workspace only. + +### TRAP — the command in § "Scope to establish first" silently reports zero + +Both snippets below use `SyntaxWarning`. These escapes are only a `SyntaxWarning` +on **Python 3.12+**; on 3.11 and earlier they are a `DeprecationWarning`, so + +```bash +python3 -W always::SyntaxWarning -m compileall -q scripts/ # → 0 hits on 3.11 +python3 -W always::DeprecationWarning -m compileall -q scripts/ # → 80 hits on 3.11 +``` + +The first form reports a clean sweep on an older interpreter and looks exactly +like "already fixed". This cost a mis-grade during the sweep before the source was +read directly and found unchanged. Use `-f` too, or `__pycache__` suppresses +recompilation and the counts silently drop. Anyone picking this up on a 3.11 +environment should run both. + +--- ## Origin From 89a0a7b94416850446ba3ba09097483239d7d89b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 13:33:48 +0000 Subject: [PATCH 5/7] =?UTF-8?q?mind:=20fourth=20draft/=20sweep=20target=20?= =?UTF-8?q?=E2=80=94=20the=20health=5Ffixes=20cluster?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six prompts from PyAutoHeart#27 release run 28784914443. Nothing graded shipped this time, but the cluster was misleading in three ways that would have cost a future session real time. All findings are offline-verifiable; the reproduction legs need runs a cloud session cannot do. THE README WAS WRONG BY ONE WHOLE PROMPT It lists seven prompts. aggregator_output_contracts SHIPPED 2026-07-07 -- PyAutoFit#1324, with autogalaxy_workspace#122, autolens_workspace#229 and autolens_workspace_test#146 all merged, record in complete/2026/07. Its file has not been in the folder for a month; the table still advertised it as live work covering 7 scripts. Struck through with the record cited. EVERY SCRIPT PATH IN TWO PROMPTS IS STALE -- AND THAT NEARLY FOOLED ME All 4 scripts in jit_visualization_outputs and all 6 in jax_runtime_and_parity return 404 at the paths written in the prompts. Ten 404s out of ten is exactly what "this was fixed by deleting the scripts" looks like. It is not: all ten still exist under a renamed layout. Two systematic renames: scripts/jax_likelihood_functions//X.py -> scripts//jax_likelihood/X.py scripts//modeling_visualization_jit.py -> scripts//visualization/... scripts/multi/... -> scripts/multi_dataset/... Verified in autolens_workspace_test (4cea3f8c, cloned) and autogalaxy_workspace_test (raw; all six resolve 200). This is the SAME rename family that cost a previous session time on the point-smoke prompt, so a 404 in this cluster should be treated as drift until proven otherwise. Corrected path tables written into both prompts. Also worth recording: autogalaxy_workspace_test has no README.md, so the handoff's "sanity-check with a path you know exists before trusting a 404" initially suggested the whole repo was unreachable. AGENTS.md and config/build/no_run.yaml both return 200. Pick the probe path per repo. A GREEN RELEASE RUN IS NOT EVIDENCE THESE WERE FIXED Many named scripts are now parked in their workspace's config/build/no_run.yaml with dates AFTER these prompts were filed -- 4 of the 6 in jax_runtime_and_parity (all SLOW 2026-07-14, citing PyAutoHeart#74 and the 1800s cap), 1 of the 4 in jit_visualization_outputs (SLOW 2026-07-08, 300s cap). That matters for how the 2026-08-07 release drive reads. Its Stage 3 integrate reports 51/51 jobs green, 657p/0f/101s/0t -- and those 101 SKIPS are where this cluster went. A parked script cannot fail validation, so zero failures says nothing about whether these defects survive. The parkings also cite a DIFFERENT failure (cap timeout) from the defects the prompts describe, which means unparking is a precondition for reproducing any of them. release_timeout_policy: 1 of its 5 has taken the prompt's own option 3b -- autolens_workspace cluster/start_here parked SLOW 2026-07-22 citing PyAutoHeart run 29912642195. The other four are in neither no_run.yaml, which does NOT mean they were optimized; that inference needs a benchmark and is the one the prompt's item 4 warns against. Flagged that the caps themselves moved too: the prompt is written against 300s, the 2026-07-22 parking cites 1800s mode=release. VERIFIED PATHS STILL CORRECT numerical_inversion_failures (2 scripts) and autofit_sampler_database (9) name paths that all resolve 200 unchanged, and neither prompt's scripts are parked. lifecycle check / orphans / index --check all OK; pytest tests/ 118 passed. draft/ unchanged at 144 -- no prompt retired, four annotated. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E3MuurHXi3xo9TLRpMLJA6 --- draft/bug/health_fixes/README.md | 26 ++++++++- .../health_fixes/jax_runtime_and_parity.md | 58 +++++++++++++++++++ .../health_fixes/jit_visualization_outputs.md | 54 +++++++++++++++++ .../health_fixes/release_timeout_policy.md | 30 ++++++++++ 4 files changed, 167 insertions(+), 1 deletion(-) diff --git a/draft/bug/health_fixes/README.md b/draft/bug/health_fixes/README.md index ed3f5592..26636959 100644 --- a/draft/bug/health_fixes/README.md +++ b/draft/bug/health_fixes/README.md @@ -15,7 +15,7 @@ Each failing script is assigned to exactly one prompt: |---|---:|---| | [samples_parameter_paths.md](samples_parameter_paths.md) — ⚠️ parked, does not reproduce on current `main` ([PyAutoFit#1327](https://github.com/PyAutoLabs/PyAutoFit/issues/1327), blocked on clean-CI re-validation) | 9 | PyAutoFit result/sample path resolution | | [autofit_sampler_database.md](autofit_sampler_database.md) | 9 | Emcee NaNs and database output discovery | -| [aggregator_output_contracts.md](aggregator_output_contracts.md) | 7 | Result/aggregator prerequisites and generated paths | +| ~~aggregator_output_contracts.md~~ — ✅ **SHIPPED 2026-07-07**, record `complete/2026/07/aggregator-output-contracts.md` (PyAutoFit#1324; autogalaxy_workspace#122, autolens_workspace#229, autolens_workspace_test#146 all merged) | 7 | Result/aggregator prerequisites and generated paths | | [jax_runtime_and_parity.md](jax_runtime_and_parity.md) | 6 | JAX/TFP compatibility and likelihood parity | | [jit_visualization_outputs.md](jit_visualization_outputs.md) | 4 | Quick-update visualizations not producing images | | [numerical_inversion_failures.md](numerical_inversion_failures.md) | 2 | Non-positive-definite inversion matrices | @@ -24,3 +24,27 @@ Each failing script is assigned to exactly one prompt: Total: **42 scripts**. Scripts that pass on current `main` remain listed because they still require a clean-worktree, directory-order reproduction before being declared fixed. Do not rebaseline assertions or edit tutorials to conceal a library regression. + +## 2026-08-09 sweep — three things to know before working this folder + +1. **One of the seven has shipped** (aggregator_output_contracts, struck through above); + its file is gone from this folder and lives in `complete/2026/07/`. The table said + nothing about that for a month. + +2. **Script paths in `jit_visualization_outputs.md` and `jax_runtime_and_parity.md` are + all stale** — every one 404s, and every one still exists under a renamed layout + (`scripts/jax_likelihood_functions//` → `scripts//jax_likelihood/`; + `/modeling_visualization_jit.py` → `/visualization/…`; `multi/` → + `multi_dataset/`). Corrected tables are in each prompt. A 404 in this cluster means + drift, not deletion. + +3. **Many of the named scripts are now parked** in their workspace's + `config/build/no_run.yaml`, mostly `SLOW` for cap timeouts, with dates *after* these + prompts were filed. A parked script cannot fail release validation, so a green + release run is not evidence of a fix — the 2026-08-07 Stage 3 integrate reports + `657p/0f/101s/0t`, and those **101 skips** are where this cluster's scripts went. The + parkings also cite a **different failure** (timeout) from the defects these prompts + describe, so unparking is a precondition for reproducing any of them. + +None of the five remaining prompts was graded shipped. What this sweep could establish +offline is recorded in each file; the reproduction legs all need real runs. diff --git a/draft/bug/health_fixes/jax_runtime_and_parity.md b/draft/bug/health_fixes/jax_runtime_and_parity.md index c493f12c..e164a6f8 100644 --- a/draft/bug/health_fixes/jax_runtime_and_parity.md +++ b/draft/bug/health_fixes/jax_runtime_and_parity.md @@ -7,6 +7,64 @@ Autonomy: supervised Priority: high Status: formalised +## 2026-08-09 — EVERY SCRIPT PATH BELOW IS STALE (they moved, they were not deleted) + +Checked by the draft/ sweep. All 6 scripts named in § Scripts return **404** at the +paths written below, and **all 6 still exist** under a renamed layout. Do not read the +404s as "the scripts were deleted" or "this was fixed by removal". + +Two systematic renames landed in both test workspaces since this prompt was filed: + +``` +scripts/jax_likelihood_functions//X.py -> scripts//jax_likelihood/X.py +scripts//modeling_visualization_jit.py -> scripts//visualization/modeling_visualization_jit.py +scripts/multi/... -> scripts/multi_dataset/... +``` + +Verified in `autolens_workspace_test` (`4cea3f8c`, cloned) and +`autogalaxy_workspace_test` (raw, every path resolves 200). This is the **same rename +family** that cost a previous session time on +`draft/bug/autolens/jax_point_source_point_smoke_sentinel.md` — treat a 404 in this +cluster as path drift until proven otherwise. + +Corrected paths: + +| named in § Scripts | actual path on main | repo | +|---|---|---| +| `…/jax_likelihood_functions/imaging/delaunay_mge.py` | `scripts/imaging/jax_likelihood/delaunay_mge.py` | autogalaxy_workspace_test | +| `…/jax_likelihood_functions/interferometer/delaunay_mge.py` | `scripts/interferometer/jax_likelihood/delaunay_mge.py` | autogalaxy_workspace_test | +| `…/jax_likelihood_functions/multi/delaunay_mge.py` | `scripts/multi_dataset/jax_likelihood/delaunay_mge.py` | autogalaxy_workspace_test | +| `…/jax_likelihood_functions/interferometer/delaunay_mge.py` | `scripts/interferometer/jax_likelihood/delaunay_mge.py` | autolens_workspace_test | +| `…/jax_likelihood_functions/multi/rectangular.py` | `scripts/multi_dataset/jax_likelihood/rectangular.py` | autolens_workspace_test | +| `…/jax_likelihood_functions/multi/rectangular_mge.py` | `scripts/multi_dataset/jax_likelihood/rectangular_mge.py` | autolens_workspace_test | + +### Parking state — read before assuming CI still exercises these + +**4 of the 6 are parked SLOW**, all dated 2026-07-14 and all citing PyAutoHeart#74 +(flaking at the 1800s `mode=release` cap): + +- `autogalaxy_workspace_test` `no_run.yaml`: `imaging/jax_likelihood/delaunay_mge.py`, + `interferometer/jax_likelihood/delaunay_mge.py`, `multi_dataset/jax_likelihood/delaunay_mge` +- `autolens_workspace_test` `no_run.yaml`: `interferometer/jax_likelihood/delaunay_mge.py` + +The two `multi_dataset/jax_likelihood/rectangular*.py` scripts are **not** parked. + +Also note `autolens_workspace_test` parks `multi_dataset/jax_likelihood/delaunay.py` +as `NEEDS_FIX 2026-08-01 - hangs to the 1800s release cap in 3 of 5 release-integrate +runs` — a sibling in the same family, filed after this prompt, suggesting the JAX +likelihood timeout story has moved on independently of this prompt. + +**This changes what the prompt's premise means.** A parked script cannot fail in +release validation, so the 2026-08-07 release drive's Stage 3 result (51/51 jobs green, +`657p/0f/101s/0t` — note the **101 skips**) is *not* evidence these were fixed. Nothing +here is graded shipped; the failures are simply no longer being provoked. Note too that +the parkings are for **timeouts**, a different failure from the defect this prompt +describes — so unparking is a precondition for reproducing it at all. + +Not re-graded here: whether the underlying defect still reproduces. That needs real +runs, which a cloud session cannot do. + +--- ## Context Six JAX likelihood scripts failed with the rehearsed release stack. CI included diff --git a/draft/bug/health_fixes/jit_visualization_outputs.md b/draft/bug/health_fixes/jit_visualization_outputs.md index 3517d870..43a21f32 100644 --- a/draft/bug/health_fixes/jit_visualization_outputs.md +++ b/draft/bug/health_fixes/jit_visualization_outputs.md @@ -7,6 +7,60 @@ Autonomy: supervised Priority: high Status: formalised +## 2026-08-09 — EVERY SCRIPT PATH BELOW IS STALE (they moved, they were not deleted) + +Checked by the draft/ sweep. All 4 scripts named in § Scripts return **404** at the +paths written below, and **all 4 still exist** under a renamed layout. Do not read the +404s as "the scripts were deleted" or "this was fixed by removal". + +Two systematic renames landed in both test workspaces since this prompt was filed: + +``` +scripts/jax_likelihood_functions//X.py -> scripts//jax_likelihood/X.py +scripts//modeling_visualization_jit.py -> scripts//visualization/modeling_visualization_jit.py +scripts/multi/... -> scripts/multi_dataset/... +``` + +Verified in `autolens_workspace_test` (`4cea3f8c`, cloned) and +`autogalaxy_workspace_test` (raw, every path resolves 200). This is the **same rename +family** that cost a previous session time on +`draft/bug/autolens/jax_point_source_point_smoke_sentinel.md` — treat a 404 in this +cluster as path drift until proven otherwise. + +Corrected paths: + +| named in § Scripts | actual path on main | repo | +|---|---|---| +| `…/scripts/ellipse/modeling_visualization_jit.py` | `scripts/ellipse/visualization/modeling_visualization_jit.py` | autogalaxy_workspace_test | +| `…/scripts/imaging/modeling_visualization_jit.py` | `scripts/imaging/visualization/modeling_visualization_jit.py` | autogalaxy_workspace_test | +| `…/scripts/interferometer/modeling_visualization_jit.py` | `scripts/interferometer/visualization/modeling_visualization_jit.py` | autogalaxy_workspace_test | +| `…/scripts/point_source/modeling_visualization_jit.py` | `scripts/point_source/visualization/modeling_visualization_jit.py` | autolens_workspace_test | + +### Parking state — read before assuming CI still exercises these + +**1 of the 4 is parked.** `autolens_workspace_test` `config/build/no_run.yaml:30`: +`point_source/visualization/modeling_visualization_jit # SLOW 2026-07-08 - JIT + Part-2 +live Nautilus fit exceeds 300s cap`. The three `autogalaxy_workspace_test` scripts are +**not** parked and should still be running. + +Adjacent and worth knowing: `autolens_workspace_test` also parks its *own* +`imaging/visualization/modeling_visualization_jit` (SLOW 2026-07-30, "re-measured: times +out at the 300s cap") and `interferometer/visualization/modeling_visualization_jit` +(SLOW 2026-07-30, "local re-measurement OOM-killed"). Those two are not named in this +prompt but are the same script family, so the timeout problem is broader than the four +listed here. + +**This changes what the prompt's premise means.** A parked script cannot fail in +release validation, so the 2026-08-07 release drive's Stage 3 result (51/51 jobs green, +`657p/0f/101s/0t` — note the **101 skips**) is *not* evidence these were fixed. Nothing +here is graded shipped; the failures are simply no longer being provoked. Note too that +the parkings are for **timeouts**, a different failure from the defect this prompt +describes — so unparking is a precondition for reproducing it at all. + +Not re-graded here: whether the underlying defect still reproduces. That needs real +runs, which a cloud session cannot do. + +--- ## Context Four test-workspace scripts expect real release-profile searches to invoke the JIT-cached diff --git a/draft/bug/health_fixes/release_timeout_policy.md b/draft/bug/health_fixes/release_timeout_policy.md index 58e54635..08a191b9 100644 --- a/draft/bug/health_fixes/release_timeout_policy.md +++ b/draft/bug/health_fixes/release_timeout_policy.md @@ -7,6 +7,36 @@ Autonomy: supervised Priority: normal Status: formalised +## 2026-08-09 — 1 of the 5 is resolved via § Required work option 3b + +Checked by the draft/ sweep against the two workspaces' `config/build/no_run.yaml` on +main. The prompt's item 3 offers an explicit fork per script — optimize below the cap, +**or** add a documented `SLOW` entry. One script has taken the second branch: + +- **`autolens_workspace/scripts/cluster/start_here.py` — PARKED.** + `- cluster/start_here # SLOW 2026-07-22 - hits the full 1800s mode=release cap in + workspace-validation (PyAutoHeart run 29912642195). Script-specific…` That is exactly + the documented-parking outcome this prompt asks for, decided after it was filed. + +The other four are **not** in either `no_run.yaml`: + +- `autogalaxy_workspace/scripts/ellipse/multipoles.py` +- `autolens_workspace/scripts/cluster/modeling.py` +- `autolens_workspace/scripts/imaging/features/advanced/double_einstein_ring/chaining.py` +- `autolens_workspace/scripts/multi/features/slam/simultaneous.py` + +Which means either they were optimized under the cap, or the decision was never taken. +**This sweep cannot tell which** — item 1 requires benchmarking from a clean output tree, +which needs a real run. Do not read "absent from no_run.yaml" as "optimized"; that is the +same unproven inference the prompt's item 4 warns against. + +One caution for whoever picks this up: the caps themselves moved. This prompt is written +against a **300s** cap, while the 2026-07-22 parking above cites an **1800s +`mode=release`** cap. Re-establish which cap applies to each script before benchmarking, +or the numbers will not mean what the prompt assumes. + +--- + ## Context Five scripts exceeded PyAutoBuild's 300-second per-script cap in release run From 1b450cc149b4b838c40226ce82590836ad139f32 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 14:06:51 +0000 Subject: [PATCH 6/7] =?UTF-8?q?mind:=20fifth=20draft/=20sweep=20target=20?= =?UTF-8?q?=E2=80=94=20jax=5Fsubstructure,=20four=20shipped=20prompts=20at?= =?UTF-8?q?=20once?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The largest single find of the sweep, and the Mind had already written it down. complete/2026/06/jax-substructure-simulator.md was itself authored as a "retroactive close-out" on 2026-06-09 and says plainly that "the 4 jax_substructure/ prompts shipped to main". All four prompt files then sat in draft/feature/jax_substructure/ for a further two months, indistinguishable from unstarted work. Prompts 1 (vmap deflections), 2 (lax.scan multi-plane), 3 (e2e jit simulate) and 4 (vmap batched) are now folded into that record and removed from draft/. RE-VERIFIED UPSTREAM RATHER THAN TRUSTING THE RECORD Given the whole point of this sweep is that prompts lie about their own state, a record claiming shipped is not better evidence than a prompt claiming open. Checked all three repos: PyAutoLens autolens/lens/substructure_util.py defines all six named deliverables -- precompute_scaling_matrix, galaxies_to_halo_arrays, traced_grids_via_scan, simulate_substructure, los_realizations_to_arrays, batched_simulate_substructure PyAutoGalaxy has vmapped_deflections_from on the abstract mass profile autolens_workspace_test has all three scripts The workspace scripts 404'd on first lookup and are NOT under misc/ -- they live at scripts/imaging/substructure/. That is the third time in two sweeps a 404 has been path drift rather than absence, so it is now the default assumption. THE TWO FOLLOW-UPS ARE GENUINELY OPEN -- CONFIRMED, NOT ASSUMED The record queued prompts 5 and 6 as deferred sub-items. Both verified still open rather than inherited on trust: 5_prng_key_vmap_noise -- preprocess.poisson_noise_via_data_eps_from still has the signature (data_eps, exposure_time_map, seed=-1, xp=np) on PyAutoArray main. No prng_key parameter, so the OO SimulatorImaging path still cannot be vmapped over a batch of noise keys. 6_deflection_equivalence_test -- no standalone equivalence script exists; scripts/imaging/substructure/ holds only the three e2e/scan/batched scripts plus subhalo.py, and no workspace script references vmapped_deflections_from. Both stay in draft/. The prompt-4 stretch memory-estimator remains unbuilt and was deliberately never requested, so it is not filed. THE ARCHIVED EPIC TRACKER WAS ACTIVELY MISLEADING complete/archive/epics/jax_substructure_simulator.md still listed items 1-4 under "__Outstanding__ (sequenced)" with live relative links into draft/. Anyone navigating from the epic would have read four shipped tasks as the remaining work, and the links are now dead. Corrected in place with a pointer to the completion record and a note that only prompts 5 and 6 are live. METHOD NOTE This is the strongest case yet for signal 3 in draft/feature/pyautomind/draft_staleness_detection_signals.md -- a completion record naming the prompt's deliverable. Here the record names the prompts themselves, by directory, in its first clause. Any grep of record bodies for "jax_substructure" surfaces it instantly; slug similarity does not, because 1_vmap_subhalo_deflections shares no tokens with jax-substructure-simulator. lifecycle check / orphans / index --check all OK; pytest tests/ 118 passed. draft/ 144 to 141. Dashboard regenerated. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E3MuurHXi3xo9TLRpMLJA6 --- .../2026/06/jax-substructure-simulator.md | 680 ++++++++++++++++++ .../epics/jax_substructure_simulator.md | 10 + dashboard.md | 10 +- .../1_vmap_subhalo_deflections.md | 124 ---- .../jax_substructure/2_tracer_lax_scan.md | 172 ----- .../jax_substructure/3_simulator_jax_e2e.md | 186 ----- .../4_vmap_batched_simulation.md | 148 ---- 7 files changed, 693 insertions(+), 637 deletions(-) delete mode 100644 draft/feature/jax_substructure/1_vmap_subhalo_deflections.md delete mode 100644 draft/feature/jax_substructure/2_tracer_lax_scan.md delete mode 100644 draft/feature/jax_substructure/3_simulator_jax_e2e.md delete mode 100644 draft/feature/jax_substructure/4_vmap_batched_simulation.md diff --git a/complete/2026/06/jax-substructure-simulator.md b/complete/2026/06/jax-substructure-simulator.md index bb50e769..18040489 100644 --- a/complete/2026/06/jax-substructure-simulator.md +++ b/complete/2026/06/jax-substructure-simulator.md @@ -11,3 +11,683 @@ - https://github.com/PyAutoLabs/autolens_workspace_test/pull/129 - repos: PyAutoLens, PyAutoGalaxy, autolens_workspace_test - notes: Retroactive close-out — the 4 `jax_substructure/` prompts shipped to `main` over PRs PyAutoLens #543 (scan) + #544 (e2e), additional PyAutoLens commits `b744801` (batched_simulate) / `4e93ecc` (parameterize lens/source/light), PyAutoGalaxy direct commits (`8a317dfc` jnp.where NaN-safe mask, `a4b8ce22`) for the generic `vmapped_deflections_from` classmethod on the abstract mass profile, and workspace_test #127/#128/#129, but the work was never recorded here and issue #542 was left OPEN. Delivered: `autolens/lens/substructure_util.py` (`precompute_scaling_matrix`, `galaxies_to_halo_arrays`, `traced_grids_via_scan`, `simulate_substructure`, `los_realizations_to_arrays`, `batched_simulate_substructure`), the generic `Profile.vmapped_deflections_from` (covers any mass profile exposing `radial_deflection_from`), and 3 workspace_test scripts (`test_scan_multiplane.py`, `test_simulate_e2e.py`, `test_batched_simulate.py`). **Two prompt sub-items were sidestepped/deferred and are now queued as follow-up prompts:** (1) `jax_substructure/5_prng_key_vmap_noise.md` — prompt 3's Gap 1: `preprocess.poisson_noise_via_data_eps_from` still derives its `PRNGKey` internally from the int `seed` (`seed=-1` → `int(time.time())`), so the OO `SimulatorImaging` path can't be vmapped over a batch of noise keys; the standalone `simulate_substructure` sidesteps this by calling `jax.random.poisson(prng_key, ...)` directly. The fix is an optional `prng_key` param (PyAutoArray, → /ship_library). (2) `jax_substructure/6_deflection_equivalence_test.md` — prompt 1's dedicated old-vs-vmapped deflection-equivalence test (all 4 dark-matter profile types + masked-slot-zero) was never authored as a standalone script; validation is only folded into the scan/e2e tests. Note `galaxies_to_halo_arrays` only branches `cNFWSph` vs. truncated, so that test may surface a small MCR-variant extension (workspace_test, → /ship_workspace). Prompt-4 stretch memory-estimator / sub-batching helper remains unbuilt (not requested). + +## Lifecycle note — the four prompts retired from `draft/` on 2026-08-09 + +Record backfilled behaviour, second pass. This record was itself written as a +"retroactive close-out" on 2026-06-09 and states plainly that "the 4 +`jax_substructure/` prompts shipped to `main`" — yet all four prompt files stayed +in `draft/feature/jax_substructure/` for a further two months, indistinguishable +from unstarted work. The draft/ sweep retired them here; their bodies are folded +in below so nothing is lost. + +**Re-verified against upstream `main` before retiring** (2026-08-09), rather than +trusting this record's own claim: + +- **PyAutoLens** — `autolens/lens/substructure_util.py` exists and defines all six + named deliverables: `precompute_scaling_matrix`, `galaxies_to_halo_arrays`, + `traced_grids_via_scan`, `simulate_substructure`, `los_realizations_to_arrays`, + `batched_simulate_substructure`. +- **PyAutoGalaxy** — `vmapped_deflections_from` is present on the abstract mass + profile (`autogalaxy/profiles/mass/abstract/abstract.py`). +- **autolens_workspace_test** — all three scripts exist, at + `scripts/imaging/substructure/{test_scan_multiplane,test_simulate_e2e,test_batched_simulate}.py`. + Note the path: they are **not** under `misc/`, and a lookup there 404s. Same + path-drift trap as the health_fixes cluster. + +**The two follow-ups this record queued are still genuinely open** and stay in +`draft/feature/jax_substructure/` — confirmed, not assumed: + +- `5_prng_key_vmap_noise.md` — `preprocess.poisson_noise_via_data_eps_from` still + has the signature `(data_eps, exposure_time_map, seed=-1, xp=np)` on PyAutoArray + `main`. No `prng_key` parameter. Unchanged. +- `6_deflection_equivalence_test.md` — no standalone deflection-equivalence script + exists in `autolens_workspace_test`; `scripts/imaging/substructure/` holds only + the three e2e/scan/batched scripts plus `subhalo.py`, and no workspace script + references `vmapped_deflections_from`. Unchanged. + +The prompt-4 stretch memory-estimator / sub-batching helper remains unbuilt, as +recorded above — deliberately not requested, so not filed as a follow-up. + + +## Original prompt — `1_vmap_subhalo_deflections.md` + +# Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure + +Type: feature +Target: jax_substructure +Difficulty: too-large +Autonomy: supervised +Priority: normal +Status: formalised + +Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure +forward simulator. This is prompt 1 of 4 — building the vectorized per-plane +deflection computation that everything else stacks on top of. + +## Background + +Today, when a Tracer has N subhalos on a single plane, their deflections are +summed via a Python generator loop: + +```python +# tracer_util.py line 262 +deflections_yx_2d = sum( + (g.deflections_yx_2d_from(grid=scaled_grid, xp=xp) for g in galaxies) +) +``` + +Under `jax.jit`, JAX unrolls this into N separate traced operations. For 5 +galaxies that's fine. For 1000 halos it produces a massive XLA graph (slow +compilation) and recompiles whenever N changes between realizations. + +The fix is a **vmapped deflection function** that takes stacked parameter arrays +and computes all N deflections in a single GPU launch, then sums them. + +All four dark matter profiles already accept `xp=jnp` and produce correct +JAX-traced outputs — the individual deflection math is ready. What's missing +is the batching orchestration. + +## What to build + +A pure-function module (suggest `autolens/lens/substructure_util.py` or similar) +containing functions like: + +```python +def deflections_nfw_truncated_sph_from( + grid, # (M, 2) image-plane grid + params, # (N, 4) — mass_at_200, concentration, centre_y, centre_x + mask, # (N,) boolean — which slots are active halos + cosmology, # for MCR variants that need kappa_s / scale_radius + redshift, # halo redshift (scalar, shared across the batch) + xp=jnp, +): + """Compute summed deflections from N NFWTruncatedSph halos via vmap.""" + ... +``` + +The inner single-halo function should call the existing deflection math from +the profile classes. Look at how `NFWTruncatedSph.deflections_yx_2d_from` +works in `autogalaxy/profiles/mass/dark/nfw_truncated.py` — it calls through +the `@aa.decorators.transform` and `@aa.decorators.to_vector_yx` decorator +chain. For the vmapped path you'll want to call the underlying math directly +(pre-transform the grid by subtracting `centre`, call the radial deflection +functions, post-transform back) to avoid the decorator overhead that wraps +results in autoarray objects. + +The key profiles to cover: + +- `NFWTruncatedSph` — `autogalaxy/profiles/mass/dark/nfw_truncated.py` +- `cNFWSph` — `autogalaxy/profiles/mass/dark/cnfw.py` +- Their MCR Ludlow variants (`nfw_truncated_mcr.py`, `cnfw_mcr.py`) which + derive `kappa_s` and `scale_radius` from `mass_at_200` via + `autogalaxy/profiles/mass/dark/mcr_util.py` + +For the MCR variants, the Ludlow concentration-mass relation +(`mcr_util.kappa_s_and_scale_radius_for_ludlow` and +`mcr_util.kappa_s_scale_radius_and_core_radius_for_ludlow`) is already +JAX-native — it auto-detects JAX arrays and uses `jnp` internally. So you +can vmap through the full MCR → deflection chain. + +The `mask` parameter handles the padding: pad `params` to `max_N` rows, set +`mask=False` for unused slots, and zero out their deflection contribution +before summing. This way the array shape is fixed regardless of the actual +number of halos, so `jax.jit` compiles once. + +## Integration test + +This is the key validation: build a Tracer the normal way with ~10 subhalos +(using the existing Galaxy/profile API), compute deflections via the +Python-loop path, then compute the same deflections via the new vmapped +path, and assert they match to numerical tolerance. + +Put this in `autolens_workspace_test/scripts/jax_substructure/` (new directory). +Something like: + +```python +# 1. Build 10 NFWTruncatedSph halos as Galaxy objects +halos = [ag.Galaxy(redshift=0.5, mass=ag.mp.NFWTruncatedSph(...)) for _ in range(10)] +tracer = al.Tracer(galaxies=[macro_galaxy, *halos, source_galaxy]) + +# 2. Get deflections via existing path +deflections_old = tracer_util.traced_grid_2d_list_from(..., xp=jnp) + +# 3. Stack same parameters into arrays +params = jnp.array([[mass_i, conc_i, cy_i, cx_i] for ...]) +mask = jnp.ones(10, dtype=bool) + +# 4. Get deflections via new vmapped path +deflections_new = deflections_nfw_truncated_sph_from(grid, params, mask, ...) + +# 5. Assert match +assert jnp.allclose(deflections_old, deflections_new, atol=1e-8) +``` + +Do this for all four profile types. Also test that masked-out slots contribute +zero deflection. + +## Scope boundaries + +- This prompt covers **single-plane** vectorized deflections only. Multi-plane + scan is prompt 2. +- Don't modify the existing Tracer or Galaxy classes. This is a parallel path. +- The macro lens (PowerLaw + ExternalShear) doesn't need vmapping here — there's + only one macro lens per realization. It will be called directly in prompt 2. +- Light profiles (source image) are also not in scope here — just mass deflections. + + + +## Original prompt — `2_tracer_lax_scan.md` + +# Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1 + +Type: feature +Target: jax_substructure +Difficulty: too-large +Autonomy: supervised +Priority: normal +Status: formalised + +Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1 built vmapped per-plane +deflection functions. This prompt wires them into a `jax.lax.scan` over redshift +planes to replace the Python loops in multi-plane ray-tracing. + +## Background + +The current multi-plane ray-tracing lives in +`autolens/lens/tracer_util.py : traced_grid_2d_list_from` (lines 174-268). +It has three nested Python loops: + +1. **Outer loop** (line 232): `for plane_index, galaxies in enumerate(planes):` +2. **Scaling loop** (line 238): `for previous_plane_index in range(plane_index):` + — applies cosmological scaling factors from all previous planes +3. **Galaxy sum** (line 262): `sum(g.deflections_yx_2d_from(...) for g in galaxies)` + — sums deflections from all galaxies on the current plane + +For the substructure use case (~8 planes, ~1000 total halos), these Python loops +unroll into a huge XLA graph and recompile whenever the galaxy count changes. + +## What to build + +A standalone pure-function that does the same multi-plane ray-tracing but using +`jax.lax.scan` over planes and the vmapped deflection functions from prompt 1. +Suggest placing this in the same module as prompt 1 +(`autolens/lens/substructure_util.py`). + +### Input representation + +The key design decision is how to represent the per-plane halo populations as +fixed-shape arrays. The natural structure is: + +```python +# Per-plane halo parameters, padded to max_halos_per_plane +halo_params: jnp.array # shape (n_planes, max_halos_per_plane, n_halo_params) +halo_mask: jnp.array # shape (n_planes, max_halos_per_plane) — bool +plane_redshifts: jnp.array # shape (n_planes,) +``` + +The macro lens (PowerLaw + ExternalShear) should be handled separately from the +halo stacks — it's a single galaxy evaluated directly, not vmapped. The source +light profile is also separate (evaluated on the final traced grid). + +### Precomputed scaling-factor matrix + +The cosmological scaling factors between all plane pairs can be precomputed +**outside jit** as a `(n_planes, n_planes)` matrix: + +```python +# scaling_matrix[i, j] = scaling_factor from plane j to plane i (0 if j >= i) +scaling_matrix = precompute_scaling_matrix(plane_redshifts, cosmology) +``` + +The cosmology module at `autogalaxy/cosmology/model.py` already has +`scaling_factor_between_redshifts_from(redshift_0, redshift_1, redshift_final, xp)` +which is xp-threaded. Call it for each `(j, i)` pair where `j < i`. + +This matrix is a static input to the jitted function — it only depends on +redshifts, which are fixed for a given realization. + +### The scan function + +```python +def traced_grids_via_scan( + grid, # (M, 2) image-plane grid + macro_params, # dict or array of PowerLaw + ExternalShear params + halo_params, # (n_planes, max_N, n_halo_params) + halo_mask, # (n_planes, max_N) + scaling_matrix, # (n_planes, n_planes) + source_params, # Sersic params for the source + ... +): + def scan_step(carry, plane_inputs): + # carry: (current_grid, all_prev_deflections as (n_planes, M, 2) buffer) + # plane_inputs: (this_plane_halo_params, this_plane_mask, scaling_row) + + grid, deflection_buffer, plane_idx = carry + plane_halo_params, plane_mask, scaling_row = plane_inputs + + # 1. Apply scaled deflections from all previous planes + # scaling_row is (n_planes,) — entries for j >= plane_idx are 0 + scaled_deflections = jnp.einsum('p,pmd->md', scaling_row, deflection_buffer) + current_grid = grid - scaled_deflections + + # 2. Compute macro deflections (if this is the lens plane) + # ... call PowerLaw + ExternalShear deflection directly ... + + # 3. Compute halo deflections via vmapped function from prompt 1 + halo_deflections = deflections_nfw_truncated_sph_from( + current_grid, plane_halo_params, plane_mask, ... + ) + + # 4. Store total plane deflections in buffer + total_deflections = macro_deflections + halo_deflections + deflection_buffer = deflection_buffer.at[plane_idx].set(total_deflections) + + return (grid, deflection_buffer, plane_idx + 1), current_grid + + init_carry = (grid, jnp.zeros((n_planes, M, 2)), 0) + _, traced_grids = jax.lax.scan(scan_step, init_carry, plane_stack) + return traced_grids +``` + +The exact API will need refinement — the sketch above shows the idea. The macro +lens only contributes on one plane (the main lens plane), so use `jax.lax.cond` +or `jnp.where` to conditionally add its deflections based on `plane_idx`. + +### Where the macro lens fits + +The macro galaxy (PowerLaw + ExternalShear) is evaluated directly — not vmapped, +since there's only one. Its deflection function is already JAX-traceable +(`autogalaxy/profiles/mass/total/power_law.py` uses a `jax.lax.scan` series +expansion). Call it on the lens-plane grid and add it to that plane's deflection +buffer alongside the halo contribution. + +### Where the source fits + +After the scan produces `traced_grids` for all planes, evaluate the source light +profile (e.g. `SersicCore`) on the final plane's traced grid to produce the +lensed image. `SersicCore.image_2d_via_radii_from` already accepts `xp` — call +it directly on the source-plane grid. + +## Integration test + +Extend the test from prompt 1. Build a Tracer with: +- 1 PowerLaw + ExternalShear macro at z=0.5 +- 10 NFWTruncatedSph subhalos at z=0.5 (lens plane) +- 5 NFWTruncatedSph LOS halos at z=0.25 (foreground plane) +- 5 NFWTruncatedSph LOS halos at z=0.75 (background plane) +- 1 Sersic source at z=1.0 + +Compute the final source-plane grid via both paths: +1. `tracer_util.traced_grid_2d_list_from(planes, grid, cosmology, xp=jnp)` +2. `traced_grids_via_scan(grid, macro_params, halo_params, ...)` + +Assert the source-plane grids match to numerical tolerance. This validates that +the scan + vmap path reproduces the existing Python-loop path. + +Also test that the scan path compiles once and reuses the compiled code when +only parameter values change (same shapes, different halo masses/positions). + +Put tests in `autolens_workspace_test/scripts/jax_substructure/`. + +## Scope boundaries + +- This covers multi-plane ray-tracing and source-plane grid computation. +- PSF convolution and noise are prompt 3. +- The LOSSampler output stays as-is — it runs outside jit and produces the + parameter arrays that feed into this function. The conversion from + `LOSSampler.galaxies_from()` output to `(halo_params, halo_mask)` arrays + is a small helper, not a refactor of LOSSampler itself. +- Don't modify the existing Tracer class or tracer_util. This is a parallel path. + +## Existing patterns to follow + +- `jax.lax.scan` is already used in `autogalaxy/profiles/mass/total/jax_utils.py` + (omega series expansion) and `autoarray/operators/transformer.py` (chunked + NUFFT). Look at those for the carry/accumulator pattern. +- `jax.lax.fori_loop` is used in `autoarray/inversion/mesh/interpolator/knn.py`. +- Pytree registration: `autoarray/abstract_ndarray.py` has `register_instance_pytree`. + The new function takes raw arrays, so pytree registration isn't needed for + the function itself — just ensure inputs are plain `jnp.arrays`. + + + +## Original prompt — `3_simulator_jax_e2e.md` + +# Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2 + +Type: feature +Target: jax_substructure +Difficulty: too-large +Autonomy: supervised +Priority: normal +Status: formalised + +Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2 built the vectorized +deflection and scan-based ray-tracing. This prompt wires them through PSF +convolution and Poisson noise to produce the end-to-end `jax.jit(simulate)` +function. + +## Background + +The existing simulator call chain is: + +``` +SimulatorImaging.via_tracer_from(tracer, grid) + -> tracer.padded_image_2d_from(grid, psf_shape_2d) + -> image_2d_from (sum light profiles on traced grids) + -> SimulatorImaging.via_image_from(image) + -> PSF convolution (FFT or real-space, both JAX-ready) + -> add background sky + -> Poisson noise via jax.random.poisson (when xp=jnp) + -> return Imaging dataset +``` + +The downstream half (PSF convolution onward) is already JAX-friendly. The +upstream half (image from traced grids) is now handled by the scan path from +prompt 2. This prompt connects them and fixes the remaining gaps. + +## Gap 1: PRNGKey support for Poisson noise + +`autoarray/dataset/preprocess.py : poisson_noise_via_data_eps_from` (line 455) +currently takes an integer `seed` parameter. On the JAX path (line 488) it +converts this to a PRNGKey: + +```python +effective_seed = seed if seed != -1 else int(time.time() * 1e6) & 0xFFFFFFFF +key = jax.random.PRNGKey(effective_seed) +``` + +This works for single calls but blocks `vmap` over noise seeds — you can't +vmap a function that calls `int(time.time())` inside. + +Add an optional `prng_key` parameter: + +```python +def poisson_noise_via_data_eps_from( + data_eps, exposure_time_map, seed=-1, prng_key=None, xp=np +): + ... + if prng_key is not None: + key = prng_key + elif xp is not np: + effective_seed = seed if seed != -1 else int(time.time() * 1e6) & 0xFFFFFFFF + key = jax.random.PRNGKey(effective_seed) + ... +``` + +Thread this parameter through `data_eps_with_poisson_noise_added` (line 500) +and up through `SimulatorImaging.via_image_from` in +`autoarray/dataset/imaging/simulator.py`. + +## Gap 2: Over-sampler xp threading + +`Grid2D.padded_grid_from` in `autoarray/structures/grids/uniform_2d.py` +(line 1140) uses `np.pad` which is not xp-aware. Similarly the OverSampler +binning path uses numpy operations. + +For the substructure fast path, the simplest approach is to **skip the +autoarray grid/over-sampler machinery entirely** and handle padding and +sub-gridding with plain jnp operations in the standalone simulate function. +The grid is uniform and the over-sample factor is fixed, so this is +straightforward: + +```python +# Pad grid for PSF +padded_shape = image_shape + psf_shape - 1 +padded_grid = make_uniform_grid(padded_shape, pixel_scale) # pure jnp + +# Evaluate source on sub-grid if over_sample > 1 +sub_grid = make_sub_grid(padded_grid, over_sample_size) # pure jnp +sub_images = source_image_fn(sub_grid, source_params) +image = sub_images.reshape(...).mean(axis=-1) # bin down +``` + +This avoids modifying the autoarray grid classes while giving us a fully +jnp-native path. + +## The end-to-end simulate function + +Combine everything into a single jittable function: + +```python +@jax.jit +def simulate_substructure( + macro_params, # PowerLaw + ExternalShear parameters + halo_params, # (n_planes, max_N, n_halo_params) + halo_mask, # (n_planes, max_N) + source_params, # Sersic parameters + # --- static / precomputed (passed via jax.jit static_argnums or closure) --- + grid, # (M, 2) image-plane grid (padded for PSF) + psf_kernel, # (K, K) PSF array + scaling_matrix, # (n_planes, n_planes) + exposure_time, # scalar + background_sky, # scalar + prng_key, # jax.random.PRNGKey for Poisson noise +): + # 1. Multi-plane ray-trace (from prompt 2) + traced_grids = traced_grids_via_scan( + grid, macro_params, halo_params, halo_mask, scaling_matrix + ) + + # 2. Evaluate source light on final traced grid + source_grid = traced_grids[-1] + image = sersic_image_from(source_grid, source_params) + + # 3. PSF convolution (FFT) + image = jax.scipy.signal.fftconvolve(image, psf_kernel, mode='same') + + # 4. Add background sky + image = image + background_sky + + # 5. Poisson noise + image_counts = image * exposure_time + noisy_counts = jax.random.poisson(prng_key, image_counts) + noisy_image = noisy_counts / exposure_time + + # 6. Subtract sky + noisy_image = noisy_image - background_sky + + return noisy_image +``` + +The PSF convolution can use `jax.scipy.signal.fftconvolve` directly — the +existing Convolver FFT path in `autoarray/operators/convolver.py` already +does essentially this with `jnp.fft.rfft2 / irfft2`, so either approach works. +For the standalone function, the scipy one-liner is simpler. + +## Integration test / smoke test + +Build a representative substructure configuration and verify the end-to-end +simulate function against the existing OO path: + +```python +# Build via existing API +tracer = al.Tracer(galaxies=[macro, *subhalos_10, source]) +simulator = al.SimulatorImaging( + exposure_time=300.0, background_sky_level=1.0, + psf=al.Kernel2D.from_gaussian(shape_native=(11, 11), sigma=0.1, ...), + noise_seed=42, +) +imaging_old = simulator.via_tracer_from(tracer=tracer, grid=grid) + +# Build via new pure-function path (same parameters, same seed) +key = jax.random.PRNGKey(42) +image_new = simulate_substructure( + macro_params, halo_params, halo_mask, source_params, + grid, psf_kernel, scaling_matrix, 300.0, 1.0, key, +) + +# Compare (tolerance for Poisson noise RNG differences — compare +# the deterministic part first, then the noisy part with the same seed) +assert jnp.allclose(image_new, imaging_old.data, atol=1e-6) +``` + +Also verify that `jax.jit(simulate_substructure)` compiles successfully +and that calling it a second time with different parameter values (same +shapes) reuses the compiled code (no recompilation). + +Put tests in `autolens_workspace_test/scripts/jax_substructure/`. + +## Scope boundaries + +- This prompt produces a working `jit(simulate)` for a single realization. +- `vmap` over a batch of parameter vectors is prompt 4. +- The LOSSampler conversion helper (Galaxy list -> padded arrays) should be + a small utility, not a refactor. If it's simple enough, include it here; + otherwise defer to prompt 4. +- Don't modify the existing SimulatorImaging class beyond adding the + `prng_key` parameter to the noise functions in preprocess.py. + + + +## Original prompt — `4_vmap_batched_simulation.md` + +# Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal). + +Type: feature +Target: jax_substructure +Difficulty: too-large +Autonomy: supervised +Priority: normal +Status: formalised + +Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal). Prompts 1-3 built +`jax.jit(simulate_substructure)` for a single realization. This prompt extends +it to `vmap(jit(simulate))(thetas, keys)` for batched evaluation — ~1024 lensed +images per GPU launch. + +## Background + +The issue author's use case evaluates `theta -> noisy image` of order 10^6 +times. After prompt 3, each call is a single jitted GPU kernel. The next +speedup is batching: evaluate many theta vectors in one launch, saturating +GPU parallelism. + +## What to build + +### Batched simulate function + +```python +batched_simulate = jax.vmap(simulate_substructure, in_axes=( + 0, # macro_params: (batch, n_macro_params) — varies per realization + 0, # halo_params: (batch, n_planes, max_N, n_halo_params) — varies + 0, # halo_mask: (batch, n_planes, max_N) — varies (different N per draw) + 0, # source_params: (batch, n_source_params) — varies + None, # grid: shared across batch + None, # psf_kernel: shared + None, # scaling_matrix: shared (same redshift structure) + None, # exposure_time: shared + None, # background_sky: shared + 0, # prng_key: (batch,) — different key per realization +)) +``` + +Call with: + +```python +keys = jax.random.split(master_key, batch_size) +images = jax.jit(batched_simulate)( + macro_params_batch, # (1024, n_macro) + halo_params_batch, # (1024, n_planes, max_N, n_halo) + halo_mask_batch, # (1024, n_planes, max_N) + source_params_batch, # (1024, n_source) + grid, psf_kernel, scaling_matrix, exposure_time, background_sky, + keys, # (1024,) +) +# images shape: (1024, H, W) +``` + +### LOSSampler → padded array conversion + +The LOSSampler at `autolens/lens/los.py` produces a `List[ag.Galaxy]` per +realization. For the batched path, we need a helper that converts many +realizations into padded arrays: + +```python +def los_realizations_to_arrays( + realizations: List[List[ag.Galaxy]], + max_halos_per_plane: int, + n_planes: int, + plane_redshifts: np.ndarray, +): + """Convert a batch of LOSSampler outputs to padded arrays. + + Returns: + halo_params: (batch, n_planes, max_halos_per_plane, n_params) + halo_mask: (batch, n_planes, max_halos_per_plane) + """ + ... +``` + +This runs in numpy (outside jit) and produces the fixed-shape arrays that +feed into the vmapped function. The LOSSampler itself doesn't need to change. + +### Memory considerations + +1024 images of size 100x100 at float32 = 1024 * 100 * 100 * 4 bytes = ~40 MB. +Fine for any GPU. But the intermediate arrays (per-halo deflections across all +batch elements) can be larger: 1024 * max_N * M * 2 * 4 bytes. For max_N=200 +and M=10000 grid points, that's ~16 GB — may exceed GPU memory. + +Mitigation strategies: +- Process in sub-batches (e.g. 128 at a time) and concatenate results +- Reduce max_N by using separate halo types per plane (most planes have + few halos; only the lens plane has many subhalos) +- Use `jax.checkpoint` to trade compute for memory on the scan steps + +Include a utility that estimates peak memory for a given configuration and +suggests a batch size. + +### What varies vs what's shared across the batch + +For the issue author's use case (fixed lens macro, varying substructure): + +| Input | Varies? | Notes | +|-------|---------|-------| +| macro_params | Maybe | Could be fixed or sampled | +| halo_params | Yes | Different SHMF draw per realization | +| halo_mask | Yes | Different N per draw | +| source_params | Maybe | Could be fixed or sampled | +| grid | No | Same image grid | +| psf_kernel | No | Same instrument | +| scaling_matrix | No | Same redshift planes (if plane structure is fixed) | +| prng_key | Yes | Different noise per realization | + +If the plane redshift structure also varies between realizations (different +LOS plane redshifts per draw), then `scaling_matrix` would need to be batched +too. But the issue author mentions 8 fixed planes, so it's likely shared. + +## Integration test + +Verify batch consistency: + +```python +# Single-image results +images_single = [simulate_substructure(p, h, m, s, ..., k) + for p, h, m, s, k in zip(params...)] + +# Batched results +images_batch = batched_simulate(params_stacked..., keys) + +# Must match +for i in range(batch_size): + assert jnp.allclose(images_single[i], images_batch[i], atol=1e-6) +``` + +Also benchmark: measure wall-clock time for 1024 sequential calls vs one +batched call. The batched version should be significantly faster (the whole +point). + +Put tests in `autolens_workspace_test/scripts/jax_substructure/`. + +## Scope boundaries + +- This is the final prompt in the series. After this, the user has a complete + `vmap(jit(simulate))(thetas, keys)` path. +- If memory is a hard constraint, the sub-batching utility is sufficient — + don't try to implement gradient checkpointing in this prompt. +- The LOSSampler conversion helper is simple numpy reshaping, not a refactor + of the sampler itself. + + diff --git a/complete/archive/epics/jax_substructure_simulator.md b/complete/archive/epics/jax_substructure_simulator.md index 6cbf2a1f..f0677ace 100644 --- a/complete/archive/epics/jax_substructure_simulator.md +++ b/complete/archive/epics/jax_substructure_simulator.md @@ -30,6 +30,16 @@ __What's already done__ (no work needed): __Outstanding__ (sequenced): +> **CORRECTION 2026-08-09 (draft/ sweep): items 1-4 below are NOT outstanding — +> they SHIPPED on 2026-06-09.** See `complete/2026/06/jax-substructure-simulator.md` +> (PyAutoLens#543 + #544, PyAutoGalaxy direct commits, autolens_workspace_test +> #127/#128/#129), re-verified against upstream `main`. Their four prompt files +> stayed in `draft/` for two months after shipping and were retired into that +> record; the relative links below are therefore dead. The only live work from this +> epic is `draft/feature/jax_substructure/5_prng_key_vmap_noise.md` and +> `6_deflection_equivalence_test.md`, both confirmed still open. This tracker is +> archived material — read the completion record, not this list. + 1. [feature/jax_substructure/1_vmap_subhalo_deflections.md](../feature/jax_substructure/1_vmap_subhalo_deflections.md) — vectorized deflection path: represent N halos as `(max_N, n_params)` arrays, `jax.vmap` the profile deflection function, sum with mask. Integration test diff --git a/dashboard.md b/dashboard.md index 28309f58..1c5d5c02 100644 --- a/dashboard.md +++ b/dashboard.md @@ -2,12 +2,12 @@ -**143** filed prompts in the backlog · **6** already dispatched to issues (`active/`). Backlog view only — organism health lives with the Heart (`/health`), not here. +**139** filed prompts in the backlog · **6** already dispatched to issues (`active/`). Backlog view only — organism health lives with the Heart (`/health`), not here. | Work-type | Prompts | |-----------|--------:| | bug | 40 | -| feature | 34 | +| feature | 30 | | maintenance | 21 | | research | 18 | | docs | 17 | @@ -62,7 +62,7 @@ | [Tenant firewall: release_run.py carries an unlisted 'PyAutoLabs' instance fact](draft/bug/pyautoheart/tenant_firewall_release_run_instance_fact.md) | pyautoheart | small | safe | normal | | [`aplt.Output` stale-API drift in the remaining workspace repos](draft/bug/workspaces/aplt_output_drift_remaining_repos.md) | workspaces | small | supervised | normal | -## feature (34) +## feature (30) | Prompt | Target | Difficulty | Autonomy | Priority | |--------|--------|------------|----------|----------| @@ -85,10 +85,6 @@ | [Tune cluster-scale JOSS benchmarks toward their 5-minute targets](draft/feature/autolens_workspace/joss_cluster_benchmark_tuning.md) | autolens_workspace | medium | supervised | normal | | [Adopt oversampled PSFs in the start-here dataset chain (option a)](draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md) | autolens_workspace | large | supervised | normal | | [Scheduled runs — overnight queue passes with a morning report](draft/feature/autonomy/10_scheduled_runs.md) | autonomy | medium | supervised | low | -| [Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure](draft/feature/jax_substructure/1_vmap_subhalo_deflections.md) | jax_substructure | too-large | supervised | normal | -| [Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1](draft/feature/jax_substructure/2_tracer_lax_scan.md) | jax_substructure | too-large | supervised | normal | -| [Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2](draft/feature/jax_substructure/3_simulator_jax_e2e.md) | jax_substructure | too-large | supervised | normal | -| [Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal)](draft/feature/jax_substructure/4_vmap_batched_simulation.md) | jax_substructure | too-large | supervised | normal | | [Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the](draft/feature/jax_substructure/5_prng_key_vmap_noise.md) | jax_substructure | too-large | supervised | normal | | [Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the](draft/feature/jax_substructure/6_deflection_equivalence_test.md) | jax_substructure | too-large | supervised | normal | | [Extend the Profiling Agent scope to track JAX compile/eval times](draft/feature/profiling/profiling_agent_jax_compile_time_scope.md) | profiling | large | supervised | high | diff --git a/draft/feature/jax_substructure/1_vmap_subhalo_deflections.md b/draft/feature/jax_substructure/1_vmap_subhalo_deflections.md deleted file mode 100644 index 4b3ce401..00000000 --- a/draft/feature/jax_substructure/1_vmap_subhalo_deflections.md +++ /dev/null @@ -1,124 +0,0 @@ -# Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure - -Type: feature -Target: jax_substructure -Difficulty: too-large -Autonomy: supervised -Priority: normal -Status: formalised - -Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure -forward simulator. This is prompt 1 of 4 — building the vectorized per-plane -deflection computation that everything else stacks on top of. - -## Background - -Today, when a Tracer has N subhalos on a single plane, their deflections are -summed via a Python generator loop: - -```python -# tracer_util.py line 262 -deflections_yx_2d = sum( - (g.deflections_yx_2d_from(grid=scaled_grid, xp=xp) for g in galaxies) -) -``` - -Under `jax.jit`, JAX unrolls this into N separate traced operations. For 5 -galaxies that's fine. For 1000 halos it produces a massive XLA graph (slow -compilation) and recompiles whenever N changes between realizations. - -The fix is a **vmapped deflection function** that takes stacked parameter arrays -and computes all N deflections in a single GPU launch, then sums them. - -All four dark matter profiles already accept `xp=jnp` and produce correct -JAX-traced outputs — the individual deflection math is ready. What's missing -is the batching orchestration. - -## What to build - -A pure-function module (suggest `autolens/lens/substructure_util.py` or similar) -containing functions like: - -```python -def deflections_nfw_truncated_sph_from( - grid, # (M, 2) image-plane grid - params, # (N, 4) — mass_at_200, concentration, centre_y, centre_x - mask, # (N,) boolean — which slots are active halos - cosmology, # for MCR variants that need kappa_s / scale_radius - redshift, # halo redshift (scalar, shared across the batch) - xp=jnp, -): - """Compute summed deflections from N NFWTruncatedSph halos via vmap.""" - ... -``` - -The inner single-halo function should call the existing deflection math from -the profile classes. Look at how `NFWTruncatedSph.deflections_yx_2d_from` -works in `autogalaxy/profiles/mass/dark/nfw_truncated.py` — it calls through -the `@aa.decorators.transform` and `@aa.decorators.to_vector_yx` decorator -chain. For the vmapped path you'll want to call the underlying math directly -(pre-transform the grid by subtracting `centre`, call the radial deflection -functions, post-transform back) to avoid the decorator overhead that wraps -results in autoarray objects. - -The key profiles to cover: - -- `NFWTruncatedSph` — `autogalaxy/profiles/mass/dark/nfw_truncated.py` -- `cNFWSph` — `autogalaxy/profiles/mass/dark/cnfw.py` -- Their MCR Ludlow variants (`nfw_truncated_mcr.py`, `cnfw_mcr.py`) which - derive `kappa_s` and `scale_radius` from `mass_at_200` via - `autogalaxy/profiles/mass/dark/mcr_util.py` - -For the MCR variants, the Ludlow concentration-mass relation -(`mcr_util.kappa_s_and_scale_radius_for_ludlow` and -`mcr_util.kappa_s_scale_radius_and_core_radius_for_ludlow`) is already -JAX-native — it auto-detects JAX arrays and uses `jnp` internally. So you -can vmap through the full MCR → deflection chain. - -The `mask` parameter handles the padding: pad `params` to `max_N` rows, set -`mask=False` for unused slots, and zero out their deflection contribution -before summing. This way the array shape is fixed regardless of the actual -number of halos, so `jax.jit` compiles once. - -## Integration test - -This is the key validation: build a Tracer the normal way with ~10 subhalos -(using the existing Galaxy/profile API), compute deflections via the -Python-loop path, then compute the same deflections via the new vmapped -path, and assert they match to numerical tolerance. - -Put this in `autolens_workspace_test/scripts/jax_substructure/` (new directory). -Something like: - -```python -# 1. Build 10 NFWTruncatedSph halos as Galaxy objects -halos = [ag.Galaxy(redshift=0.5, mass=ag.mp.NFWTruncatedSph(...)) for _ in range(10)] -tracer = al.Tracer(galaxies=[macro_galaxy, *halos, source_galaxy]) - -# 2. Get deflections via existing path -deflections_old = tracer_util.traced_grid_2d_list_from(..., xp=jnp) - -# 3. Stack same parameters into arrays -params = jnp.array([[mass_i, conc_i, cy_i, cx_i] for ...]) -mask = jnp.ones(10, dtype=bool) - -# 4. Get deflections via new vmapped path -deflections_new = deflections_nfw_truncated_sph_from(grid, params, mask, ...) - -# 5. Assert match -assert jnp.allclose(deflections_old, deflections_new, atol=1e-8) -``` - -Do this for all four profile types. Also test that masked-out slots contribute -zero deflection. - -## Scope boundaries - -- This prompt covers **single-plane** vectorized deflections only. Multi-plane - scan is prompt 2. -- Don't modify the existing Tracer or Galaxy classes. This is a parallel path. -- The macro lens (PowerLaw + ExternalShear) doesn't need vmapping here — there's - only one macro lens per realization. It will be called directly in prompt 2. -- Light profiles (source image) are also not in scope here — just mass deflections. - - diff --git a/draft/feature/jax_substructure/2_tracer_lax_scan.md b/draft/feature/jax_substructure/2_tracer_lax_scan.md deleted file mode 100644 index f3fbba6d..00000000 --- a/draft/feature/jax_substructure/2_tracer_lax_scan.md +++ /dev/null @@ -1,172 +0,0 @@ -# Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1 - -Type: feature -Target: jax_substructure -Difficulty: too-large -Autonomy: supervised -Priority: normal -Status: formalised - -Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1 built vmapped per-plane -deflection functions. This prompt wires them into a `jax.lax.scan` over redshift -planes to replace the Python loops in multi-plane ray-tracing. - -## Background - -The current multi-plane ray-tracing lives in -`autolens/lens/tracer_util.py : traced_grid_2d_list_from` (lines 174-268). -It has three nested Python loops: - -1. **Outer loop** (line 232): `for plane_index, galaxies in enumerate(planes):` -2. **Scaling loop** (line 238): `for previous_plane_index in range(plane_index):` - — applies cosmological scaling factors from all previous planes -3. **Galaxy sum** (line 262): `sum(g.deflections_yx_2d_from(...) for g in galaxies)` - — sums deflections from all galaxies on the current plane - -For the substructure use case (~8 planes, ~1000 total halos), these Python loops -unroll into a huge XLA graph and recompile whenever the galaxy count changes. - -## What to build - -A standalone pure-function that does the same multi-plane ray-tracing but using -`jax.lax.scan` over planes and the vmapped deflection functions from prompt 1. -Suggest placing this in the same module as prompt 1 -(`autolens/lens/substructure_util.py`). - -### Input representation - -The key design decision is how to represent the per-plane halo populations as -fixed-shape arrays. The natural structure is: - -```python -# Per-plane halo parameters, padded to max_halos_per_plane -halo_params: jnp.array # shape (n_planes, max_halos_per_plane, n_halo_params) -halo_mask: jnp.array # shape (n_planes, max_halos_per_plane) — bool -plane_redshifts: jnp.array # shape (n_planes,) -``` - -The macro lens (PowerLaw + ExternalShear) should be handled separately from the -halo stacks — it's a single galaxy evaluated directly, not vmapped. The source -light profile is also separate (evaluated on the final traced grid). - -### Precomputed scaling-factor matrix - -The cosmological scaling factors between all plane pairs can be precomputed -**outside jit** as a `(n_planes, n_planes)` matrix: - -```python -# scaling_matrix[i, j] = scaling_factor from plane j to plane i (0 if j >= i) -scaling_matrix = precompute_scaling_matrix(plane_redshifts, cosmology) -``` - -The cosmology module at `autogalaxy/cosmology/model.py` already has -`scaling_factor_between_redshifts_from(redshift_0, redshift_1, redshift_final, xp)` -which is xp-threaded. Call it for each `(j, i)` pair where `j < i`. - -This matrix is a static input to the jitted function — it only depends on -redshifts, which are fixed for a given realization. - -### The scan function - -```python -def traced_grids_via_scan( - grid, # (M, 2) image-plane grid - macro_params, # dict or array of PowerLaw + ExternalShear params - halo_params, # (n_planes, max_N, n_halo_params) - halo_mask, # (n_planes, max_N) - scaling_matrix, # (n_planes, n_planes) - source_params, # Sersic params for the source - ... -): - def scan_step(carry, plane_inputs): - # carry: (current_grid, all_prev_deflections as (n_planes, M, 2) buffer) - # plane_inputs: (this_plane_halo_params, this_plane_mask, scaling_row) - - grid, deflection_buffer, plane_idx = carry - plane_halo_params, plane_mask, scaling_row = plane_inputs - - # 1. Apply scaled deflections from all previous planes - # scaling_row is (n_planes,) — entries for j >= plane_idx are 0 - scaled_deflections = jnp.einsum('p,pmd->md', scaling_row, deflection_buffer) - current_grid = grid - scaled_deflections - - # 2. Compute macro deflections (if this is the lens plane) - # ... call PowerLaw + ExternalShear deflection directly ... - - # 3. Compute halo deflections via vmapped function from prompt 1 - halo_deflections = deflections_nfw_truncated_sph_from( - current_grid, plane_halo_params, plane_mask, ... - ) - - # 4. Store total plane deflections in buffer - total_deflections = macro_deflections + halo_deflections - deflection_buffer = deflection_buffer.at[plane_idx].set(total_deflections) - - return (grid, deflection_buffer, plane_idx + 1), current_grid - - init_carry = (grid, jnp.zeros((n_planes, M, 2)), 0) - _, traced_grids = jax.lax.scan(scan_step, init_carry, plane_stack) - return traced_grids -``` - -The exact API will need refinement — the sketch above shows the idea. The macro -lens only contributes on one plane (the main lens plane), so use `jax.lax.cond` -or `jnp.where` to conditionally add its deflections based on `plane_idx`. - -### Where the macro lens fits - -The macro galaxy (PowerLaw + ExternalShear) is evaluated directly — not vmapped, -since there's only one. Its deflection function is already JAX-traceable -(`autogalaxy/profiles/mass/total/power_law.py` uses a `jax.lax.scan` series -expansion). Call it on the lens-plane grid and add it to that plane's deflection -buffer alongside the halo contribution. - -### Where the source fits - -After the scan produces `traced_grids` for all planes, evaluate the source light -profile (e.g. `SersicCore`) on the final plane's traced grid to produce the -lensed image. `SersicCore.image_2d_via_radii_from` already accepts `xp` — call -it directly on the source-plane grid. - -## Integration test - -Extend the test from prompt 1. Build a Tracer with: -- 1 PowerLaw + ExternalShear macro at z=0.5 -- 10 NFWTruncatedSph subhalos at z=0.5 (lens plane) -- 5 NFWTruncatedSph LOS halos at z=0.25 (foreground plane) -- 5 NFWTruncatedSph LOS halos at z=0.75 (background plane) -- 1 Sersic source at z=1.0 - -Compute the final source-plane grid via both paths: -1. `tracer_util.traced_grid_2d_list_from(planes, grid, cosmology, xp=jnp)` -2. `traced_grids_via_scan(grid, macro_params, halo_params, ...)` - -Assert the source-plane grids match to numerical tolerance. This validates that -the scan + vmap path reproduces the existing Python-loop path. - -Also test that the scan path compiles once and reuses the compiled code when -only parameter values change (same shapes, different halo masses/positions). - -Put tests in `autolens_workspace_test/scripts/jax_substructure/`. - -## Scope boundaries - -- This covers multi-plane ray-tracing and source-plane grid computation. -- PSF convolution and noise are prompt 3. -- The LOSSampler output stays as-is — it runs outside jit and produces the - parameter arrays that feed into this function. The conversion from - `LOSSampler.galaxies_from()` output to `(halo_params, halo_mask)` arrays - is a small helper, not a refactor of LOSSampler itself. -- Don't modify the existing Tracer class or tracer_util. This is a parallel path. - -## Existing patterns to follow - -- `jax.lax.scan` is already used in `autogalaxy/profiles/mass/total/jax_utils.py` - (omega series expansion) and `autoarray/operators/transformer.py` (chunked - NUFFT). Look at those for the carry/accumulator pattern. -- `jax.lax.fori_loop` is used in `autoarray/inversion/mesh/interpolator/knn.py`. -- Pytree registration: `autoarray/abstract_ndarray.py` has `register_instance_pytree`. - The new function takes raw arrays, so pytree registration isn't needed for - the function itself — just ensure inputs are plain `jnp.arrays`. - - diff --git a/draft/feature/jax_substructure/3_simulator_jax_e2e.md b/draft/feature/jax_substructure/3_simulator_jax_e2e.md deleted file mode 100644 index 46a40596..00000000 --- a/draft/feature/jax_substructure/3_simulator_jax_e2e.md +++ /dev/null @@ -1,186 +0,0 @@ -# Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2 - -Type: feature -Target: jax_substructure -Difficulty: too-large -Autonomy: supervised -Priority: normal -Status: formalised - -Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2 built the vectorized -deflection and scan-based ray-tracing. This prompt wires them through PSF -convolution and Poisson noise to produce the end-to-end `jax.jit(simulate)` -function. - -## Background - -The existing simulator call chain is: - -``` -SimulatorImaging.via_tracer_from(tracer, grid) - -> tracer.padded_image_2d_from(grid, psf_shape_2d) - -> image_2d_from (sum light profiles on traced grids) - -> SimulatorImaging.via_image_from(image) - -> PSF convolution (FFT or real-space, both JAX-ready) - -> add background sky - -> Poisson noise via jax.random.poisson (when xp=jnp) - -> return Imaging dataset -``` - -The downstream half (PSF convolution onward) is already JAX-friendly. The -upstream half (image from traced grids) is now handled by the scan path from -prompt 2. This prompt connects them and fixes the remaining gaps. - -## Gap 1: PRNGKey support for Poisson noise - -`autoarray/dataset/preprocess.py : poisson_noise_via_data_eps_from` (line 455) -currently takes an integer `seed` parameter. On the JAX path (line 488) it -converts this to a PRNGKey: - -```python -effective_seed = seed if seed != -1 else int(time.time() * 1e6) & 0xFFFFFFFF -key = jax.random.PRNGKey(effective_seed) -``` - -This works for single calls but blocks `vmap` over noise seeds — you can't -vmap a function that calls `int(time.time())` inside. - -Add an optional `prng_key` parameter: - -```python -def poisson_noise_via_data_eps_from( - data_eps, exposure_time_map, seed=-1, prng_key=None, xp=np -): - ... - if prng_key is not None: - key = prng_key - elif xp is not np: - effective_seed = seed if seed != -1 else int(time.time() * 1e6) & 0xFFFFFFFF - key = jax.random.PRNGKey(effective_seed) - ... -``` - -Thread this parameter through `data_eps_with_poisson_noise_added` (line 500) -and up through `SimulatorImaging.via_image_from` in -`autoarray/dataset/imaging/simulator.py`. - -## Gap 2: Over-sampler xp threading - -`Grid2D.padded_grid_from` in `autoarray/structures/grids/uniform_2d.py` -(line 1140) uses `np.pad` which is not xp-aware. Similarly the OverSampler -binning path uses numpy operations. - -For the substructure fast path, the simplest approach is to **skip the -autoarray grid/over-sampler machinery entirely** and handle padding and -sub-gridding with plain jnp operations in the standalone simulate function. -The grid is uniform and the over-sample factor is fixed, so this is -straightforward: - -```python -# Pad grid for PSF -padded_shape = image_shape + psf_shape - 1 -padded_grid = make_uniform_grid(padded_shape, pixel_scale) # pure jnp - -# Evaluate source on sub-grid if over_sample > 1 -sub_grid = make_sub_grid(padded_grid, over_sample_size) # pure jnp -sub_images = source_image_fn(sub_grid, source_params) -image = sub_images.reshape(...).mean(axis=-1) # bin down -``` - -This avoids modifying the autoarray grid classes while giving us a fully -jnp-native path. - -## The end-to-end simulate function - -Combine everything into a single jittable function: - -```python -@jax.jit -def simulate_substructure( - macro_params, # PowerLaw + ExternalShear parameters - halo_params, # (n_planes, max_N, n_halo_params) - halo_mask, # (n_planes, max_N) - source_params, # Sersic parameters - # --- static / precomputed (passed via jax.jit static_argnums or closure) --- - grid, # (M, 2) image-plane grid (padded for PSF) - psf_kernel, # (K, K) PSF array - scaling_matrix, # (n_planes, n_planes) - exposure_time, # scalar - background_sky, # scalar - prng_key, # jax.random.PRNGKey for Poisson noise -): - # 1. Multi-plane ray-trace (from prompt 2) - traced_grids = traced_grids_via_scan( - grid, macro_params, halo_params, halo_mask, scaling_matrix - ) - - # 2. Evaluate source light on final traced grid - source_grid = traced_grids[-1] - image = sersic_image_from(source_grid, source_params) - - # 3. PSF convolution (FFT) - image = jax.scipy.signal.fftconvolve(image, psf_kernel, mode='same') - - # 4. Add background sky - image = image + background_sky - - # 5. Poisson noise - image_counts = image * exposure_time - noisy_counts = jax.random.poisson(prng_key, image_counts) - noisy_image = noisy_counts / exposure_time - - # 6. Subtract sky - noisy_image = noisy_image - background_sky - - return noisy_image -``` - -The PSF convolution can use `jax.scipy.signal.fftconvolve` directly — the -existing Convolver FFT path in `autoarray/operators/convolver.py` already -does essentially this with `jnp.fft.rfft2 / irfft2`, so either approach works. -For the standalone function, the scipy one-liner is simpler. - -## Integration test / smoke test - -Build a representative substructure configuration and verify the end-to-end -simulate function against the existing OO path: - -```python -# Build via existing API -tracer = al.Tracer(galaxies=[macro, *subhalos_10, source]) -simulator = al.SimulatorImaging( - exposure_time=300.0, background_sky_level=1.0, - psf=al.Kernel2D.from_gaussian(shape_native=(11, 11), sigma=0.1, ...), - noise_seed=42, -) -imaging_old = simulator.via_tracer_from(tracer=tracer, grid=grid) - -# Build via new pure-function path (same parameters, same seed) -key = jax.random.PRNGKey(42) -image_new = simulate_substructure( - macro_params, halo_params, halo_mask, source_params, - grid, psf_kernel, scaling_matrix, 300.0, 1.0, key, -) - -# Compare (tolerance for Poisson noise RNG differences — compare -# the deterministic part first, then the noisy part with the same seed) -assert jnp.allclose(image_new, imaging_old.data, atol=1e-6) -``` - -Also verify that `jax.jit(simulate_substructure)` compiles successfully -and that calling it a second time with different parameter values (same -shapes) reuses the compiled code (no recompilation). - -Put tests in `autolens_workspace_test/scripts/jax_substructure/`. - -## Scope boundaries - -- This prompt produces a working `jit(simulate)` for a single realization. -- `vmap` over a batch of parameter vectors is prompt 4. -- The LOSSampler conversion helper (Galaxy list -> padded arrays) should be - a small utility, not a refactor. If it's simple enough, include it here; - otherwise defer to prompt 4. -- Don't modify the existing SimulatorImaging class beyond adding the - `prng_key` parameter to the noise functions in preprocess.py. - - diff --git a/draft/feature/jax_substructure/4_vmap_batched_simulation.md b/draft/feature/jax_substructure/4_vmap_batched_simulation.md deleted file mode 100644 index cd31b7cd..00000000 --- a/draft/feature/jax_substructure/4_vmap_batched_simulation.md +++ /dev/null @@ -1,148 +0,0 @@ -# Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal). - -Type: feature -Target: jax_substructure -Difficulty: too-large -Autonomy: supervised -Priority: normal -Status: formalised - -Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal). Prompts 1-3 built -`jax.jit(simulate_substructure)` for a single realization. This prompt extends -it to `vmap(jit(simulate))(thetas, keys)` for batched evaluation — ~1024 lensed -images per GPU launch. - -## Background - -The issue author's use case evaluates `theta -> noisy image` of order 10^6 -times. After prompt 3, each call is a single jitted GPU kernel. The next -speedup is batching: evaluate many theta vectors in one launch, saturating -GPU parallelism. - -## What to build - -### Batched simulate function - -```python -batched_simulate = jax.vmap(simulate_substructure, in_axes=( - 0, # macro_params: (batch, n_macro_params) — varies per realization - 0, # halo_params: (batch, n_planes, max_N, n_halo_params) — varies - 0, # halo_mask: (batch, n_planes, max_N) — varies (different N per draw) - 0, # source_params: (batch, n_source_params) — varies - None, # grid: shared across batch - None, # psf_kernel: shared - None, # scaling_matrix: shared (same redshift structure) - None, # exposure_time: shared - None, # background_sky: shared - 0, # prng_key: (batch,) — different key per realization -)) -``` - -Call with: - -```python -keys = jax.random.split(master_key, batch_size) -images = jax.jit(batched_simulate)( - macro_params_batch, # (1024, n_macro) - halo_params_batch, # (1024, n_planes, max_N, n_halo) - halo_mask_batch, # (1024, n_planes, max_N) - source_params_batch, # (1024, n_source) - grid, psf_kernel, scaling_matrix, exposure_time, background_sky, - keys, # (1024,) -) -# images shape: (1024, H, W) -``` - -### LOSSampler → padded array conversion - -The LOSSampler at `autolens/lens/los.py` produces a `List[ag.Galaxy]` per -realization. For the batched path, we need a helper that converts many -realizations into padded arrays: - -```python -def los_realizations_to_arrays( - realizations: List[List[ag.Galaxy]], - max_halos_per_plane: int, - n_planes: int, - plane_redshifts: np.ndarray, -): - """Convert a batch of LOSSampler outputs to padded arrays. - - Returns: - halo_params: (batch, n_planes, max_halos_per_plane, n_params) - halo_mask: (batch, n_planes, max_halos_per_plane) - """ - ... -``` - -This runs in numpy (outside jit) and produces the fixed-shape arrays that -feed into the vmapped function. The LOSSampler itself doesn't need to change. - -### Memory considerations - -1024 images of size 100x100 at float32 = 1024 * 100 * 100 * 4 bytes = ~40 MB. -Fine for any GPU. But the intermediate arrays (per-halo deflections across all -batch elements) can be larger: 1024 * max_N * M * 2 * 4 bytes. For max_N=200 -and M=10000 grid points, that's ~16 GB — may exceed GPU memory. - -Mitigation strategies: -- Process in sub-batches (e.g. 128 at a time) and concatenate results -- Reduce max_N by using separate halo types per plane (most planes have - few halos; only the lens plane has many subhalos) -- Use `jax.checkpoint` to trade compute for memory on the scan steps - -Include a utility that estimates peak memory for a given configuration and -suggests a batch size. - -### What varies vs what's shared across the batch - -For the issue author's use case (fixed lens macro, varying substructure): - -| Input | Varies? | Notes | -|-------|---------|-------| -| macro_params | Maybe | Could be fixed or sampled | -| halo_params | Yes | Different SHMF draw per realization | -| halo_mask | Yes | Different N per draw | -| source_params | Maybe | Could be fixed or sampled | -| grid | No | Same image grid | -| psf_kernel | No | Same instrument | -| scaling_matrix | No | Same redshift planes (if plane structure is fixed) | -| prng_key | Yes | Different noise per realization | - -If the plane redshift structure also varies between realizations (different -LOS plane redshifts per draw), then `scaling_matrix` would need to be batched -too. But the issue author mentions 8 fixed planes, so it's likely shared. - -## Integration test - -Verify batch consistency: - -```python -# Single-image results -images_single = [simulate_substructure(p, h, m, s, ..., k) - for p, h, m, s, k in zip(params...)] - -# Batched results -images_batch = batched_simulate(params_stacked..., keys) - -# Must match -for i in range(batch_size): - assert jnp.allclose(images_single[i], images_batch[i], atol=1e-6) -``` - -Also benchmark: measure wall-clock time for 1024 sequential calls vs one -batched call. The batched version should be significantly faster (the whole -point). - -Put tests in `autolens_workspace_test/scripts/jax_substructure/`. - -## Scope boundaries - -- This is the final prompt in the series. After this, the user has a complete - `vmap(jit(simulate))(thetas, keys)` path. -- If memory is a hard constraint, the sub-batching utility is sufficient — - don't try to implement gradient checkpointing in this prompt. -- The LOSSampler conversion helper is simple numpy reshaping, not a refactor - of the sampler itself. - - From 4db0fcdb7ee9df20e681f693b6a0b0f1996edf26 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 14:43:33 +0000 Subject: [PATCH 7/7] =?UTF-8?q?mind:=20final=20sweep=20=E2=80=94=20the=20v?= =?UTF-8?q?alidated=20cheap=20signals=20run=20across=20the=20whole=20backl?= =?UTF-8?q?og?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sweeps 1-5 deep-graded 60 prompts against upstream. Rather than deep-grade a sixth cluster, this pass runs the three signals those sweeps validated across all 139 remaining drafts, so the rest of the backlog gets the cheap coverage even where nobody reads it line by line. SIGNAL 3 (completion-record bodies naming a draft's stem): 14 of 139 flagged, 12 already known from earlier sweeps. Two genuinely new, both benign on inspection -- group_subhalo_sensitivity is recorded as "deferred at the user's discretion... remains for future revival", and complete_archive_wiki is cited as a follow-up rather than as delivered. Two false positives worth noting for anyone automating this: z_features and jax_zero_contour collide lexically with the retired z_features/ folder name and with generic prose, so stem matching needs a length or specificity floor. SIGNAL 1 (gates): a URL-only extractor found just 2 refs, both self-references in the detection prompt filed earlier -- because prompts cite issues as Repo#NNN, not as URLs. Broadening to that form found 8 gate-shaped refs, of which 3 were new. All 3 gates are closed: PyAutoArray#431 merged 2026-08-03 PyAutoFit#1373 closed completed 2026-07-15 PyAutoLens#565 closed completed 2026-07-10 #1373 turned out to be a FALSE POSITIVE on reading the sentence -- it is cited descriptively ("is tiled by jax.lax.map(..., batch_size=)"), not as a dependency. Recorded because it is the failure mode any automated gate detector will hit: proximity to gate words is not a gate. FOUR PROMPTS UNBLOCKED OR RE-SCOPED unpark_imaging_scaling_relation_slam -- its gate has TWO clauses, "until PyAutoArray PR#431 merges AND reaches the installed stack", and both are met: #431 merged 2026-08-03 and shipped in the 2026-08-07 PyPI release at 2026.8.7.1 (PyAutoArray 828d5c13, downstream of the merge). The park is still in place at no_run.yaml:46, so the work is genuinely outstanding. #431's own test plan says the unpark "can be removed in a separate workspace PR" -- this prompt is that PR. Also recorded the trap: do NOT unpark the multi_galaxy sibling at line 48, which hits a separate latent bug this sweep confirmed unfixed. preloads_advanced_workspace_guide -- verified the API rather than the issue, which mattered: the gate is "the Preloads API landing in @PyAutoArray" while the tracking issue is in PyAutoLens. PyAutoArray __init__.py exports AbstractPreloads, PreloadsImaging and PreloadsInterferometer. Landed. complete_archive_wiki -- SUBSTANTIALLY SHIPPED. complete/index.md already links 952 records, is generated by lifecycle.py index with index --check gating staleness, carries the CURATED band, and its header states this prompt's lookup protocol almost verbatim. What is left is the editorial curation: the Highlights band is still the empty placeholder. Re-scoped in place -- drop the build legs, keep the curate legs. oversampled_psf_dataset_adoption -- noted that PyAutoArray#362 closed as "series complete", so its "after the k x s series completes" condition is met, not just the merges it already knew about. A NOTE ON PATH GUESSING Guessing module paths was wrong a fourth time here (autoarray/preloads.py and two variants all 404 while the API is exported and real). Combined with the three script-path drifts in earlier sweeps, the rule is now explicit in the annotations: resolve via the import surface or a clone, never by constructing a plausible path. lifecycle check / orphans / index --check all OK; pytest tests/ 118 passed. draft/ unchanged at 141 -- nothing retired, four annotated. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E3MuurHXi3xo9TLRpMLJA6 --- .../preloads_advanced_workspace_guide.md | 20 ++++++++++++ .../oversampled_psf_dataset_adoption.md | 8 +++++ .../pyautomind/complete_archive_wiki.md | 31 +++++++++++++++++++ .../unpark_imaging_scaling_relation_slam.md | 29 +++++++++++++++++ 4 files changed, 88 insertions(+) diff --git a/draft/docs/workspaces/preloads_advanced_workspace_guide.md b/draft/docs/workspaces/preloads_advanced_workspace_guide.md index b060cd8d..30e0b946 100644 --- a/draft/docs/workspaces/preloads_advanced_workspace_guide.md +++ b/draft/docs/workspaces/preloads_advanced_workspace_guide.md @@ -15,6 +15,26 @@ Primary repos: **@autolens_workspace**, **@autogalaxy_workspace** (workspace doc `Preloads` API landing in **@PyAutoArray** (the `datacube-shared-state` task, PyAutoLens#565 / sub-task B of the `analysis_shared_state` epic). +## 2026-08-09 — UNBLOCKED, the dependency has landed + +Checked by the draft/ sweep. Verified the **API**, not merely the issue state: + +- `PyAutoArray/autoarray/__init__.py` on `main` exports `AbstractPreloads`, + `PreloadsImaging` and `PreloadsInterferometer`. The `Preloads` API this guide + documents exists and is public. +- PyAutoLens#565 (datacube shared-state, sub-task B) closed `completed` + 2026-07-10. + +Worth checking the issue state alone would have been weaker evidence: the gate is +phrased as "the `Preloads` API landing in @PyAutoArray", which is a different repo +from the issue that tracked the work. + +The guide can now be written against the shipped API. One caution: locate the +preloads module before citing paths — guesses at `autoarray/preloads.py`, +`autoarray/dataset/preloads.py` and `autoarray/inversion/inversion/preloads.py` +all 404, so the import in `__init__.py` is the reliable entry point. Path-guessing +has been wrong four times in this sweep. + ## Why this guide is needed `Preloads` let a caller compute an invariant fit/inversion quantity once (e.g. the curvature matrix diff --git a/draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md b/draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md index 1ca6cec4..c35f8364 100644 --- a/draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md +++ b/draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md @@ -60,3 +60,11 @@ the re-baselining survey. After the k×s series completes (phase 3 workspace tests + refactor exercise). The k×s machinery it depends on is merged (PyAutoArray#363, PyAutoGalaxy#486, autolens_workspace#236). + +**2026-08-09 (draft/ sweep): the series is CLOSED, not merely merged.** +PyAutoArray#362 closed as "series complete" — all five phases plus the extra +cache/refactor legs shipped, recorded across `complete/2026/07/kxs-*.md` and +[[oversampling-kxs-coupling]]. So the "after the series completes" condition is +met and this is ready to start. It remains the correct live residue of that +series' § 5: [[kxs-core]] records the phase-4 fork resolved as option (c) — +executed simulators stay `s=1` — with option (a) split out as this prompt. diff --git a/draft/feature/pyautomind/complete_archive_wiki.md b/draft/feature/pyautomind/complete_archive_wiki.md index 21330a58..611a874b 100644 --- a/draft/feature/pyautomind/complete_archive_wiki.md +++ b/draft/feature/pyautomind/complete_archive_wiki.md @@ -16,6 +16,37 @@ Status: formalised `complete/YYYY/MM/.md` per-task rich records to exist first. **Do not issue this until Phase 1 nears shipping** (`feedback_no_bulk_issue_queues`). +## 2026-08-09 — SUBSTANTIALLY SHIPPED; only the curation is left + +Checked by the draft/ sweep. Phase 1 shipped +([[lifecycle-state-split]], monolithic `complete.md` retired 2026-07-16, issue #81), +so the dependency above is long satisfied — **and most of what this prompt asks for +shipped with it.** + +`complete/index.md` exists on `main` and already is the token-light index this +prompt specifies: + +- **952 records** linked, grouped by dated bucket. +- Its own header states the lookup protocol this prompt describes almost verbatim + — *"read this, follow one or two links, and only then grep a dated bucket."* +- **Generated**, by `scripts/lifecycle.py index`, with `index --check` gating + staleness in CI — so it cannot rot. +- It has the curated band: `` … ``, + documented as surviving regeneration. +- `complete/AGENTS.md` § "How to look something up (token-light — RAG is dead)" + carries the same doctrine this prompt opens with. + +**What is actually left is the curation, not the machinery.** The Highlights band +is empty — it reads `_(curate hard-won records here — survives regeneration.)_`. +So the remaining work is the editorial pass: pick the hard-won records worth +surfacing and write the one-line hooks, in the `autolens_assistant/wiki` style +this prompt says to study. + +Re-scope before issuing: drop the "build the index" legs, keep the "curate it" +legs, and re-read § "Model to emulate" against what `lifecycle.py index` already +generates rather than against a blank slate. `Difficulty:` medium is now +generous. + ## Problem Once `complete/` holds hundreds of per-task records, an agent still can't look diff --git a/draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md b/draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md index 8feae813..a11a0384 100644 --- a/draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md +++ b/draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md @@ -12,6 +12,35 @@ Status: formalised BLOCKED until PyAutoArray PR#431 merges and reaches the installed stack. Do not start before then — the script only passes with that loader fix in place. +## 2026-08-09 — UNBLOCKED, both clauses satisfied + +Checked by the draft/ sweep. This gate has two conditions and **both** are met: + +1. **Merged** — PyAutoArray#431 merged `2026-08-03T18:03:14Z` (`5006f347`, "fix: + relabel at-or-below-cap data at the capped pixel scale", fixes #430). +2. **Reached the installed stack** — #431 carried the `pending-release` label, and + the 2026-08-07 release drive published all five libraries to PyPI at + **2026.8.7.1** with PyAutoArray at `828d5c13`, downstream of the merge. So the + loader fix is in a released wheel, not just on `main`. (See `active.md` + § release-drive-2026-08-07.) + +**The park is still in place**, so the work is genuinely outstanding: +`autolens_workspace/config/build/no_run.yaml:46` still carries +`imaging/features/scaling_relation/slam # NEEDS_FIX 2026-07-30 - measures its +luminosities from a preceding light stage…`. + +#431's own test plan states the outcome directly: *"`imaging/features/scaling_relation/slam` +→ now exit 0 (6 real searches). Its `NEEDS_FIX` park in +`autolens_workspace/config/build/no_run.yaml` can be removed in a separate +workspace PR."* This prompt is that PR. + +**Do not also unpark the `multi_galaxy/` sibling** at line 48. The same test plan +records that it gets past the 0.0-luminosity cause but then hits a separate latent +script bug — `slam.py:863` computing `image_half_width` from a hardcoded +`pixel_scale` while the mask uses `dataset_full.pixel_scales`. That is +`draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md`, +confirmed still unfixed on main by this sweep. It stays parked. + ## What Remove this NEEDS_FIX line from `autolens_workspace/config/build/no_run.yaml`: