diff --git a/complete/2026/06/jax-substructure-simulator.md b/complete/2026/06/jax-substructure-simulator.md index bb50e769..18040489 100644 --- a/complete/2026/06/jax-substructure-simulator.md +++ b/complete/2026/06/jax-substructure-simulator.md @@ -11,3 +11,683 @@ - https://github.com/PyAutoLabs/autolens_workspace_test/pull/129 - repos: PyAutoLens, PyAutoGalaxy, autolens_workspace_test - notes: Retroactive close-out — the 4 `jax_substructure/` prompts shipped to `main` over PRs PyAutoLens #543 (scan) + #544 (e2e), additional PyAutoLens commits `b744801` (batched_simulate) / `4e93ecc` (parameterize lens/source/light), PyAutoGalaxy direct commits (`8a317dfc` jnp.where NaN-safe mask, `a4b8ce22`) for the generic `vmapped_deflections_from` classmethod on the abstract mass profile, and workspace_test #127/#128/#129, but the work was never recorded here and issue #542 was left OPEN. Delivered: `autolens/lens/substructure_util.py` (`precompute_scaling_matrix`, `galaxies_to_halo_arrays`, `traced_grids_via_scan`, `simulate_substructure`, `los_realizations_to_arrays`, `batched_simulate_substructure`), the generic `Profile.vmapped_deflections_from` (covers any mass profile exposing `radial_deflection_from`), and 3 workspace_test scripts (`test_scan_multiplane.py`, `test_simulate_e2e.py`, `test_batched_simulate.py`). **Two prompt sub-items were sidestepped/deferred and are now queued as follow-up prompts:** (1) `jax_substructure/5_prng_key_vmap_noise.md` — prompt 3's Gap 1: `preprocess.poisson_noise_via_data_eps_from` still derives its `PRNGKey` internally from the int `seed` (`seed=-1` → `int(time.time())`), so the OO `SimulatorImaging` path can't be vmapped over a batch of noise keys; the standalone `simulate_substructure` sidesteps this by calling `jax.random.poisson(prng_key, ...)` directly. The fix is an optional `prng_key` param (PyAutoArray, → /ship_library). (2) `jax_substructure/6_deflection_equivalence_test.md` — prompt 1's dedicated old-vs-vmapped deflection-equivalence test (all 4 dark-matter profile types + masked-slot-zero) was never authored as a standalone script; validation is only folded into the scan/e2e tests. Note `galaxies_to_halo_arrays` only branches `cNFWSph` vs. truncated, so that test may surface a small MCR-variant extension (workspace_test, → /ship_workspace). Prompt-4 stretch memory-estimator / sub-batching helper remains unbuilt (not requested). + +## Lifecycle note — the four prompts retired from `draft/` on 2026-08-09 + +Record backfilled behaviour, second pass. This record was itself written as a +"retroactive close-out" on 2026-06-09 and states plainly that "the 4 +`jax_substructure/` prompts shipped to `main`" — yet all four prompt files stayed +in `draft/feature/jax_substructure/` for a further two months, indistinguishable +from unstarted work. The draft/ sweep retired them here; their bodies are folded +in below so nothing is lost. + +**Re-verified against upstream `main` before retiring** (2026-08-09), rather than +trusting this record's own claim: + +- **PyAutoLens** — `autolens/lens/substructure_util.py` exists and defines all six + named deliverables: `precompute_scaling_matrix`, `galaxies_to_halo_arrays`, + `traced_grids_via_scan`, `simulate_substructure`, `los_realizations_to_arrays`, + `batched_simulate_substructure`. +- **PyAutoGalaxy** — `vmapped_deflections_from` is present on the abstract mass + profile (`autogalaxy/profiles/mass/abstract/abstract.py`). +- **autolens_workspace_test** — all three scripts exist, at + `scripts/imaging/substructure/{test_scan_multiplane,test_simulate_e2e,test_batched_simulate}.py`. + Note the path: they are **not** under `misc/`, and a lookup there 404s. Same + path-drift trap as the health_fixes cluster. + +**The two follow-ups this record queued are still genuinely open** and stay in +`draft/feature/jax_substructure/` — confirmed, not assumed: + +- `5_prng_key_vmap_noise.md` — `preprocess.poisson_noise_via_data_eps_from` still + has the signature `(data_eps, exposure_time_map, seed=-1, xp=np)` on PyAutoArray + `main`. No `prng_key` parameter. Unchanged. +- `6_deflection_equivalence_test.md` — no standalone deflection-equivalence script + exists in `autolens_workspace_test`; `scripts/imaging/substructure/` holds only + the three e2e/scan/batched scripts plus `subhalo.py`, and no workspace script + references `vmapped_deflections_from`. Unchanged. + +The prompt-4 stretch memory-estimator / sub-batching helper remains unbuilt, as +recorded above — deliberately not requested, so not filed as a follow-up. + + +## Original prompt — `1_vmap_subhalo_deflections.md` + +# Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure + +Type: feature +Target: jax_substructure +Difficulty: too-large +Autonomy: supervised +Priority: normal +Status: formalised + +Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure +forward simulator. This is prompt 1 of 4 — building the vectorized per-plane +deflection computation that everything else stacks on top of. + +## Background + +Today, when a Tracer has N subhalos on a single plane, their deflections are +summed via a Python generator loop: + +```python +# tracer_util.py line 262 +deflections_yx_2d = sum( + (g.deflections_yx_2d_from(grid=scaled_grid, xp=xp) for g in galaxies) +) +``` + +Under `jax.jit`, JAX unrolls this into N separate traced operations. For 5 +galaxies that's fine. For 1000 halos it produces a massive XLA graph (slow +compilation) and recompiles whenever N changes between realizations. + +The fix is a **vmapped deflection function** that takes stacked parameter arrays +and computes all N deflections in a single GPU launch, then sums them. + +All four dark matter profiles already accept `xp=jnp` and produce correct +JAX-traced outputs — the individual deflection math is ready. What's missing +is the batching orchestration. + +## What to build + +A pure-function module (suggest `autolens/lens/substructure_util.py` or similar) +containing functions like: + +```python +def deflections_nfw_truncated_sph_from( + grid, # (M, 2) image-plane grid + params, # (N, 4) — mass_at_200, concentration, centre_y, centre_x + mask, # (N,) boolean — which slots are active halos + cosmology, # for MCR variants that need kappa_s / scale_radius + redshift, # halo redshift (scalar, shared across the batch) + xp=jnp, +): + """Compute summed deflections from N NFWTruncatedSph halos via vmap.""" + ... +``` + +The inner single-halo function should call the existing deflection math from +the profile classes. Look at how `NFWTruncatedSph.deflections_yx_2d_from` +works in `autogalaxy/profiles/mass/dark/nfw_truncated.py` — it calls through +the `@aa.decorators.transform` and `@aa.decorators.to_vector_yx` decorator +chain. For the vmapped path you'll want to call the underlying math directly +(pre-transform the grid by subtracting `centre`, call the radial deflection +functions, post-transform back) to avoid the decorator overhead that wraps +results in autoarray objects. + +The key profiles to cover: + +- `NFWTruncatedSph` — `autogalaxy/profiles/mass/dark/nfw_truncated.py` +- `cNFWSph` — `autogalaxy/profiles/mass/dark/cnfw.py` +- Their MCR Ludlow variants (`nfw_truncated_mcr.py`, `cnfw_mcr.py`) which + derive `kappa_s` and `scale_radius` from `mass_at_200` via + `autogalaxy/profiles/mass/dark/mcr_util.py` + +For the MCR variants, the Ludlow concentration-mass relation +(`mcr_util.kappa_s_and_scale_radius_for_ludlow` and +`mcr_util.kappa_s_scale_radius_and_core_radius_for_ludlow`) is already +JAX-native — it auto-detects JAX arrays and uses `jnp` internally. So you +can vmap through the full MCR → deflection chain. + +The `mask` parameter handles the padding: pad `params` to `max_N` rows, set +`mask=False` for unused slots, and zero out their deflection contribution +before summing. This way the array shape is fixed regardless of the actual +number of halos, so `jax.jit` compiles once. + +## Integration test + +This is the key validation: build a Tracer the normal way with ~10 subhalos +(using the existing Galaxy/profile API), compute deflections via the +Python-loop path, then compute the same deflections via the new vmapped +path, and assert they match to numerical tolerance. + +Put this in `autolens_workspace_test/scripts/jax_substructure/` (new directory). +Something like: + +```python +# 1. Build 10 NFWTruncatedSph halos as Galaxy objects +halos = [ag.Galaxy(redshift=0.5, mass=ag.mp.NFWTruncatedSph(...)) for _ in range(10)] +tracer = al.Tracer(galaxies=[macro_galaxy, *halos, source_galaxy]) + +# 2. Get deflections via existing path +deflections_old = tracer_util.traced_grid_2d_list_from(..., xp=jnp) + +# 3. Stack same parameters into arrays +params = jnp.array([[mass_i, conc_i, cy_i, cx_i] for ...]) +mask = jnp.ones(10, dtype=bool) + +# 4. Get deflections via new vmapped path +deflections_new = deflections_nfw_truncated_sph_from(grid, params, mask, ...) + +# 5. Assert match +assert jnp.allclose(deflections_old, deflections_new, atol=1e-8) +``` + +Do this for all four profile types. Also test that masked-out slots contribute +zero deflection. + +## Scope boundaries + +- This prompt covers **single-plane** vectorized deflections only. Multi-plane + scan is prompt 2. +- Don't modify the existing Tracer or Galaxy classes. This is a parallel path. +- The macro lens (PowerLaw + ExternalShear) doesn't need vmapping here — there's + only one macro lens per realization. It will be called directly in prompt 2. +- Light profiles (source image) are also not in scope here — just mass deflections. + + + +## Original prompt — `2_tracer_lax_scan.md` + +# Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1 + +Type: feature +Target: jax_substructure +Difficulty: too-large +Autonomy: supervised +Priority: normal +Status: formalised + +Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1 built vmapped per-plane +deflection functions. This prompt wires them into a `jax.lax.scan` over redshift +planes to replace the Python loops in multi-plane ray-tracing. + +## Background + +The current multi-plane ray-tracing lives in +`autolens/lens/tracer_util.py : traced_grid_2d_list_from` (lines 174-268). +It has three nested Python loops: + +1. **Outer loop** (line 232): `for plane_index, galaxies in enumerate(planes):` +2. **Scaling loop** (line 238): `for previous_plane_index in range(plane_index):` + — applies cosmological scaling factors from all previous planes +3. **Galaxy sum** (line 262): `sum(g.deflections_yx_2d_from(...) for g in galaxies)` + — sums deflections from all galaxies on the current plane + +For the substructure use case (~8 planes, ~1000 total halos), these Python loops +unroll into a huge XLA graph and recompile whenever the galaxy count changes. + +## What to build + +A standalone pure-function that does the same multi-plane ray-tracing but using +`jax.lax.scan` over planes and the vmapped deflection functions from prompt 1. +Suggest placing this in the same module as prompt 1 +(`autolens/lens/substructure_util.py`). + +### Input representation + +The key design decision is how to represent the per-plane halo populations as +fixed-shape arrays. The natural structure is: + +```python +# Per-plane halo parameters, padded to max_halos_per_plane +halo_params: jnp.array # shape (n_planes, max_halos_per_plane, n_halo_params) +halo_mask: jnp.array # shape (n_planes, max_halos_per_plane) — bool +plane_redshifts: jnp.array # shape (n_planes,) +``` + +The macro lens (PowerLaw + ExternalShear) should be handled separately from the +halo stacks — it's a single galaxy evaluated directly, not vmapped. The source +light profile is also separate (evaluated on the final traced grid). + +### Precomputed scaling-factor matrix + +The cosmological scaling factors between all plane pairs can be precomputed +**outside jit** as a `(n_planes, n_planes)` matrix: + +```python +# scaling_matrix[i, j] = scaling_factor from plane j to plane i (0 if j >= i) +scaling_matrix = precompute_scaling_matrix(plane_redshifts, cosmology) +``` + +The cosmology module at `autogalaxy/cosmology/model.py` already has +`scaling_factor_between_redshifts_from(redshift_0, redshift_1, redshift_final, xp)` +which is xp-threaded. Call it for each `(j, i)` pair where `j < i`. + +This matrix is a static input to the jitted function — it only depends on +redshifts, which are fixed for a given realization. + +### The scan function + +```python +def traced_grids_via_scan( + grid, # (M, 2) image-plane grid + macro_params, # dict or array of PowerLaw + ExternalShear params + halo_params, # (n_planes, max_N, n_halo_params) + halo_mask, # (n_planes, max_N) + scaling_matrix, # (n_planes, n_planes) + source_params, # Sersic params for the source + ... +): + def scan_step(carry, plane_inputs): + # carry: (current_grid, all_prev_deflections as (n_planes, M, 2) buffer) + # plane_inputs: (this_plane_halo_params, this_plane_mask, scaling_row) + + grid, deflection_buffer, plane_idx = carry + plane_halo_params, plane_mask, scaling_row = plane_inputs + + # 1. Apply scaled deflections from all previous planes + # scaling_row is (n_planes,) — entries for j >= plane_idx are 0 + scaled_deflections = jnp.einsum('p,pmd->md', scaling_row, deflection_buffer) + current_grid = grid - scaled_deflections + + # 2. Compute macro deflections (if this is the lens plane) + # ... call PowerLaw + ExternalShear deflection directly ... + + # 3. Compute halo deflections via vmapped function from prompt 1 + halo_deflections = deflections_nfw_truncated_sph_from( + current_grid, plane_halo_params, plane_mask, ... + ) + + # 4. Store total plane deflections in buffer + total_deflections = macro_deflections + halo_deflections + deflection_buffer = deflection_buffer.at[plane_idx].set(total_deflections) + + return (grid, deflection_buffer, plane_idx + 1), current_grid + + init_carry = (grid, jnp.zeros((n_planes, M, 2)), 0) + _, traced_grids = jax.lax.scan(scan_step, init_carry, plane_stack) + return traced_grids +``` + +The exact API will need refinement — the sketch above shows the idea. The macro +lens only contributes on one plane (the main lens plane), so use `jax.lax.cond` +or `jnp.where` to conditionally add its deflections based on `plane_idx`. + +### Where the macro lens fits + +The macro galaxy (PowerLaw + ExternalShear) is evaluated directly — not vmapped, +since there's only one. Its deflection function is already JAX-traceable +(`autogalaxy/profiles/mass/total/power_law.py` uses a `jax.lax.scan` series +expansion). Call it on the lens-plane grid and add it to that plane's deflection +buffer alongside the halo contribution. + +### Where the source fits + +After the scan produces `traced_grids` for all planes, evaluate the source light +profile (e.g. `SersicCore`) on the final plane's traced grid to produce the +lensed image. `SersicCore.image_2d_via_radii_from` already accepts `xp` — call +it directly on the source-plane grid. + +## Integration test + +Extend the test from prompt 1. Build a Tracer with: +- 1 PowerLaw + ExternalShear macro at z=0.5 +- 10 NFWTruncatedSph subhalos at z=0.5 (lens plane) +- 5 NFWTruncatedSph LOS halos at z=0.25 (foreground plane) +- 5 NFWTruncatedSph LOS halos at z=0.75 (background plane) +- 1 Sersic source at z=1.0 + +Compute the final source-plane grid via both paths: +1. `tracer_util.traced_grid_2d_list_from(planes, grid, cosmology, xp=jnp)` +2. `traced_grids_via_scan(grid, macro_params, halo_params, ...)` + +Assert the source-plane grids match to numerical tolerance. This validates that +the scan + vmap path reproduces the existing Python-loop path. + +Also test that the scan path compiles once and reuses the compiled code when +only parameter values change (same shapes, different halo masses/positions). + +Put tests in `autolens_workspace_test/scripts/jax_substructure/`. + +## Scope boundaries + +- This covers multi-plane ray-tracing and source-plane grid computation. +- PSF convolution and noise are prompt 3. +- The LOSSampler output stays as-is — it runs outside jit and produces the + parameter arrays that feed into this function. The conversion from + `LOSSampler.galaxies_from()` output to `(halo_params, halo_mask)` arrays + is a small helper, not a refactor of LOSSampler itself. +- Don't modify the existing Tracer class or tracer_util. This is a parallel path. + +## Existing patterns to follow + +- `jax.lax.scan` is already used in `autogalaxy/profiles/mass/total/jax_utils.py` + (omega series expansion) and `autoarray/operators/transformer.py` (chunked + NUFFT). Look at those for the carry/accumulator pattern. +- `jax.lax.fori_loop` is used in `autoarray/inversion/mesh/interpolator/knn.py`. +- Pytree registration: `autoarray/abstract_ndarray.py` has `register_instance_pytree`. + The new function takes raw arrays, so pytree registration isn't needed for + the function itself — just ensure inputs are plain `jnp.arrays`. + + + +## Original prompt — `3_simulator_jax_e2e.md` + +# Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2 + +Type: feature +Target: jax_substructure +Difficulty: too-large +Autonomy: supervised +Priority: normal +Status: formalised + +Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2 built the vectorized +deflection and scan-based ray-tracing. This prompt wires them through PSF +convolution and Poisson noise to produce the end-to-end `jax.jit(simulate)` +function. + +## Background + +The existing simulator call chain is: + +``` +SimulatorImaging.via_tracer_from(tracer, grid) + -> tracer.padded_image_2d_from(grid, psf_shape_2d) + -> image_2d_from (sum light profiles on traced grids) + -> SimulatorImaging.via_image_from(image) + -> PSF convolution (FFT or real-space, both JAX-ready) + -> add background sky + -> Poisson noise via jax.random.poisson (when xp=jnp) + -> return Imaging dataset +``` + +The downstream half (PSF convolution onward) is already JAX-friendly. The +upstream half (image from traced grids) is now handled by the scan path from +prompt 2. This prompt connects them and fixes the remaining gaps. + +## Gap 1: PRNGKey support for Poisson noise + +`autoarray/dataset/preprocess.py : poisson_noise_via_data_eps_from` (line 455) +currently takes an integer `seed` parameter. On the JAX path (line 488) it +converts this to a PRNGKey: + +```python +effective_seed = seed if seed != -1 else int(time.time() * 1e6) & 0xFFFFFFFF +key = jax.random.PRNGKey(effective_seed) +``` + +This works for single calls but blocks `vmap` over noise seeds — you can't +vmap a function that calls `int(time.time())` inside. + +Add an optional `prng_key` parameter: + +```python +def poisson_noise_via_data_eps_from( + data_eps, exposure_time_map, seed=-1, prng_key=None, xp=np +): + ... + if prng_key is not None: + key = prng_key + elif xp is not np: + effective_seed = seed if seed != -1 else int(time.time() * 1e6) & 0xFFFFFFFF + key = jax.random.PRNGKey(effective_seed) + ... +``` + +Thread this parameter through `data_eps_with_poisson_noise_added` (line 500) +and up through `SimulatorImaging.via_image_from` in +`autoarray/dataset/imaging/simulator.py`. + +## Gap 2: Over-sampler xp threading + +`Grid2D.padded_grid_from` in `autoarray/structures/grids/uniform_2d.py` +(line 1140) uses `np.pad` which is not xp-aware. Similarly the OverSampler +binning path uses numpy operations. + +For the substructure fast path, the simplest approach is to **skip the +autoarray grid/over-sampler machinery entirely** and handle padding and +sub-gridding with plain jnp operations in the standalone simulate function. +The grid is uniform and the over-sample factor is fixed, so this is +straightforward: + +```python +# Pad grid for PSF +padded_shape = image_shape + psf_shape - 1 +padded_grid = make_uniform_grid(padded_shape, pixel_scale) # pure jnp + +# Evaluate source on sub-grid if over_sample > 1 +sub_grid = make_sub_grid(padded_grid, over_sample_size) # pure jnp +sub_images = source_image_fn(sub_grid, source_params) +image = sub_images.reshape(...).mean(axis=-1) # bin down +``` + +This avoids modifying the autoarray grid classes while giving us a fully +jnp-native path. + +## The end-to-end simulate function + +Combine everything into a single jittable function: + +```python +@jax.jit +def simulate_substructure( + macro_params, # PowerLaw + ExternalShear parameters + halo_params, # (n_planes, max_N, n_halo_params) + halo_mask, # (n_planes, max_N) + source_params, # Sersic parameters + # --- static / precomputed (passed via jax.jit static_argnums or closure) --- + grid, # (M, 2) image-plane grid (padded for PSF) + psf_kernel, # (K, K) PSF array + scaling_matrix, # (n_planes, n_planes) + exposure_time, # scalar + background_sky, # scalar + prng_key, # jax.random.PRNGKey for Poisson noise +): + # 1. Multi-plane ray-trace (from prompt 2) + traced_grids = traced_grids_via_scan( + grid, macro_params, halo_params, halo_mask, scaling_matrix + ) + + # 2. Evaluate source light on final traced grid + source_grid = traced_grids[-1] + image = sersic_image_from(source_grid, source_params) + + # 3. PSF convolution (FFT) + image = jax.scipy.signal.fftconvolve(image, psf_kernel, mode='same') + + # 4. Add background sky + image = image + background_sky + + # 5. Poisson noise + image_counts = image * exposure_time + noisy_counts = jax.random.poisson(prng_key, image_counts) + noisy_image = noisy_counts / exposure_time + + # 6. Subtract sky + noisy_image = noisy_image - background_sky + + return noisy_image +``` + +The PSF convolution can use `jax.scipy.signal.fftconvolve` directly — the +existing Convolver FFT path in `autoarray/operators/convolver.py` already +does essentially this with `jnp.fft.rfft2 / irfft2`, so either approach works. +For the standalone function, the scipy one-liner is simpler. + +## Integration test / smoke test + +Build a representative substructure configuration and verify the end-to-end +simulate function against the existing OO path: + +```python +# Build via existing API +tracer = al.Tracer(galaxies=[macro, *subhalos_10, source]) +simulator = al.SimulatorImaging( + exposure_time=300.0, background_sky_level=1.0, + psf=al.Kernel2D.from_gaussian(shape_native=(11, 11), sigma=0.1, ...), + noise_seed=42, +) +imaging_old = simulator.via_tracer_from(tracer=tracer, grid=grid) + +# Build via new pure-function path (same parameters, same seed) +key = jax.random.PRNGKey(42) +image_new = simulate_substructure( + macro_params, halo_params, halo_mask, source_params, + grid, psf_kernel, scaling_matrix, 300.0, 1.0, key, +) + +# Compare (tolerance for Poisson noise RNG differences — compare +# the deterministic part first, then the noisy part with the same seed) +assert jnp.allclose(image_new, imaging_old.data, atol=1e-6) +``` + +Also verify that `jax.jit(simulate_substructure)` compiles successfully +and that calling it a second time with different parameter values (same +shapes) reuses the compiled code (no recompilation). + +Put tests in `autolens_workspace_test/scripts/jax_substructure/`. + +## Scope boundaries + +- This prompt produces a working `jit(simulate)` for a single realization. +- `vmap` over a batch of parameter vectors is prompt 4. +- The LOSSampler conversion helper (Galaxy list -> padded arrays) should be + a small utility, not a refactor. If it's simple enough, include it here; + otherwise defer to prompt 4. +- Don't modify the existing SimulatorImaging class beyond adding the + `prng_key` parameter to the noise functions in preprocess.py. + + + +## Original prompt — `4_vmap_batched_simulation.md` + +# Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal). + +Type: feature +Target: jax_substructure +Difficulty: too-large +Autonomy: supervised +Priority: normal +Status: formalised + +Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal). Prompts 1-3 built +`jax.jit(simulate_substructure)` for a single realization. This prompt extends +it to `vmap(jit(simulate))(thetas, keys)` for batched evaluation — ~1024 lensed +images per GPU launch. + +## Background + +The issue author's use case evaluates `theta -> noisy image` of order 10^6 +times. After prompt 3, each call is a single jitted GPU kernel. The next +speedup is batching: evaluate many theta vectors in one launch, saturating +GPU parallelism. + +## What to build + +### Batched simulate function + +```python +batched_simulate = jax.vmap(simulate_substructure, in_axes=( + 0, # macro_params: (batch, n_macro_params) — varies per realization + 0, # halo_params: (batch, n_planes, max_N, n_halo_params) — varies + 0, # halo_mask: (batch, n_planes, max_N) — varies (different N per draw) + 0, # source_params: (batch, n_source_params) — varies + None, # grid: shared across batch + None, # psf_kernel: shared + None, # scaling_matrix: shared (same redshift structure) + None, # exposure_time: shared + None, # background_sky: shared + 0, # prng_key: (batch,) — different key per realization +)) +``` + +Call with: + +```python +keys = jax.random.split(master_key, batch_size) +images = jax.jit(batched_simulate)( + macro_params_batch, # (1024, n_macro) + halo_params_batch, # (1024, n_planes, max_N, n_halo) + halo_mask_batch, # (1024, n_planes, max_N) + source_params_batch, # (1024, n_source) + grid, psf_kernel, scaling_matrix, exposure_time, background_sky, + keys, # (1024,) +) +# images shape: (1024, H, W) +``` + +### LOSSampler → padded array conversion + +The LOSSampler at `autolens/lens/los.py` produces a `List[ag.Galaxy]` per +realization. For the batched path, we need a helper that converts many +realizations into padded arrays: + +```python +def los_realizations_to_arrays( + realizations: List[List[ag.Galaxy]], + max_halos_per_plane: int, + n_planes: int, + plane_redshifts: np.ndarray, +): + """Convert a batch of LOSSampler outputs to padded arrays. + + Returns: + halo_params: (batch, n_planes, max_halos_per_plane, n_params) + halo_mask: (batch, n_planes, max_halos_per_plane) + """ + ... +``` + +This runs in numpy (outside jit) and produces the fixed-shape arrays that +feed into the vmapped function. The LOSSampler itself doesn't need to change. + +### Memory considerations + +1024 images of size 100x100 at float32 = 1024 * 100 * 100 * 4 bytes = ~40 MB. +Fine for any GPU. But the intermediate arrays (per-halo deflections across all +batch elements) can be larger: 1024 * max_N * M * 2 * 4 bytes. For max_N=200 +and M=10000 grid points, that's ~16 GB — may exceed GPU memory. + +Mitigation strategies: +- Process in sub-batches (e.g. 128 at a time) and concatenate results +- Reduce max_N by using separate halo types per plane (most planes have + few halos; only the lens plane has many subhalos) +- Use `jax.checkpoint` to trade compute for memory on the scan steps + +Include a utility that estimates peak memory for a given configuration and +suggests a batch size. + +### What varies vs what's shared across the batch + +For the issue author's use case (fixed lens macro, varying substructure): + +| Input | Varies? | Notes | +|-------|---------|-------| +| macro_params | Maybe | Could be fixed or sampled | +| halo_params | Yes | Different SHMF draw per realization | +| halo_mask | Yes | Different N per draw | +| source_params | Maybe | Could be fixed or sampled | +| grid | No | Same image grid | +| psf_kernel | No | Same instrument | +| scaling_matrix | No | Same redshift planes (if plane structure is fixed) | +| prng_key | Yes | Different noise per realization | + +If the plane redshift structure also varies between realizations (different +LOS plane redshifts per draw), then `scaling_matrix` would need to be batched +too. But the issue author mentions 8 fixed planes, so it's likely shared. + +## Integration test + +Verify batch consistency: + +```python +# Single-image results +images_single = [simulate_substructure(p, h, m, s, ..., k) + for p, h, m, s, k in zip(params...)] + +# Batched results +images_batch = batched_simulate(params_stacked..., keys) + +# Must match +for i in range(batch_size): + assert jnp.allclose(images_single[i], images_batch[i], atol=1e-6) +``` + +Also benchmark: measure wall-clock time for 1024 sequential calls vs one +batched call. The batched version should be significantly faster (the whole +point). + +Put tests in `autolens_workspace_test/scripts/jax_substructure/`. + +## Scope boundaries + +- This is the final prompt in the series. After this, the user has a complete + `vmap(jit(simulate))(thetas, keys)` path. +- If memory is a hard constraint, the sub-batching utility is sufficient — + don't try to implement gradient checkpointing in this prompt. +- The LOSSampler conversion helper is simple numpy reshaping, not a refactor + of the sampler itself. + + diff --git a/complete/2026/07/latent-samples-none-on-resumed-fit.md b/complete/2026/07/latent-samples-none-on-resumed-fit.md new file mode 100644 index 00000000..28d21fc9 --- /dev/null +++ b/complete/2026/07/latent-samples-none-on-resumed-fit.md @@ -0,0 +1,87 @@ +# latent-samples-none-on-resumed-fit + +- shipped: 2026-07-25 (the same day the prompt recorded the finding) +- library-pr: https://github.com/PyAutoLabs/PyAutoFit/pull/1418 (merged) — "fix: raise SamplesException from latent computation when samples is None (resumed fits)" +- repos: + - PyAutoFit + +## Summary + +The prompt was filed from the 2026-07-25 full health sweep and fixed upstream the +same day. Nothing in PyAutoMind recorded it, so the prompt sat in `draft/` for two +weeks looking like open work. + +Recorded 2026-08-09 by the draft/ sweep. No work is owed. + +## Verified against PyAutoFit main (`3b960609`), 2026-08-09 + +The prompt's § Task offers two options — (a) reload persisted samples on the +resume path, or (b) raise a clear guarded error. **Option (b) is what landed**, +and its § Acceptance is met exactly: "either computes latent samples or fails with +an intentional, documented message — never an `AttributeError` from inside +`latent_samples_from`." + +`autofit/non_linear/analysis/latent.py` now opens `latent_samples_from` with an +explicit `if samples is None:` guard raising `exc.SamplesException`, and the +message names the cause and both remedies — samples output disabled via +`output.yaml`'s `samples: false` or `general.yaml`'s `samples_to_csv: false`, so +`result.samples` comes back `None` on the reload; enable samples output and re-run +from cleared output, or pass a `Samples` object explicitly. It also states why +`samples_summary.json` is not a substitute (latents are per-posterior-sample). + +The function's `Raises` docstring carries the same diagnosis the prompt reached +independently — that a completed fit short-circuits to +`NonLinearSearch.result_via_completed_fit`, which reloads from `samples.csv`. + +The prompt's open sub-question ("May be test-mode-specific — check whether a real +non-bypass resumed fit also returns samples=None") was answered by the fix taking +the general path: the guard is unconditional, not gated on `PYAUTO_TEST_MODE`. + +## Why it was missed + +This is the one class of drift the Mind cannot see by itself. Unlike the +PyAutoArray k×s finding, there is **no completion record for #1418 anywhere in +`complete/`** — the fix went in upstream without a Mind entry, so no amount of +cross-referencing draft prompts against the ledger would surface it. Only reading +the prompt's acceptance criteria against the upstream tree finds this shape. + +## Original prompt +# compute_latent_samples crashes on a resumed completed fit (samples is None) + +Type: bug +Target: autofit +Repos: +- @PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: low +Status: draft + +## Finding (2026-07-25 full health sweep) + +Running `autolens_workspace_test` `misc/latent/latent_variables_smoke.py` (and +`latent_nan_robustness.py`) twice in the same output tree fails on the second +run: the search resumes ("Fit Already Completed: skipping non-linear search"), +`result.samples` comes back `None` on the resume path (PYAUTO_TEST_MODE=2 +bypass), and + + autofit/non_linear/analysis/latent.py:113 latent_samples_from + -> samples.model -> AttributeError: 'NoneType' object has no attribute 'model' + +A fresh run (output cleared) passes. So the latent pipeline works, but the +resume/load path hands `compute_latent_samples` a `None` samples object +instead of the persisted samples (or a clear error). + +## Task + +Determine whether the resume path should (a) reload persisted samples so +latent computation works on resumed results, or (b) raise a clear, guarded +error from `compute_latent_samples` when samples are unavailable. May be +test-mode-specific — check whether a real (non-bypass) resumed fit also +returns samples=None. + +## Acceptance + +Second invocation of the latent smoke scripts in an existing output tree +either computes latent samples or fails with an intentional, documented +message — never an AttributeError from inside latent_samples_from. diff --git a/draft/feature/autoarray/oversampling_kxs_coupling.md b/complete/2026/07/oversampling-kxs-coupling.md similarity index 61% rename from draft/feature/autoarray/oversampling_kxs_coupling.md rename to complete/2026/07/oversampling-kxs-coupling.md index 09862598..19c048b1 100644 --- a/draft/feature/autoarray/oversampling_kxs_coupling.md +++ b/complete/2026/07/oversampling-kxs-coupling.md @@ -1,3 +1,74 @@ +# oversampling-kxs-coupling + +- shipped: 2026-07-09 (series closed; the prompt never left `draft/`) +- issue: https://github.com/PyAutoLabs/PyAutoArray/issues/362 (CLOSED — "series complete") +- prs: PyAutoArray#363, #364, #365 · PyAutoGalaxy#486, #489 · autolens_workspace#236 · autolens_workspace_test#154 (all merged) +- repos: + - PyAutoArray, PyAutoGalaxy + - autolens_workspace, autolens_workspace_test +- phase-records: [[kxs-design]] [[kxs-core]] [[kxs-cache]] [[kxs-workspace-tests]] [[kxs-refactor]] [[kxs-surface-refactor]] + +## Summary + +This is the **source prompt** for the k×s coupling series — the five-phase plan +it lays out was executed in full over 2026-07-08/09 and the series tracker +PyAutoArray#362 was closed as "series complete". Six dated completion records +already existed for the individual phases; only the prompt they all came from +was never advanced out of `draft/feature/autoarray/`. + +Recorded 2026-08-09 by the draft/ sweep. No work is owed. + +## Verified against PyAutoArray main (`efaf3041`), 2026-08-09 + +Graded against the prompt's own § Scope, not inferred from the phase records: + +1. **§1 relax the coupling rule — SHIPPED.** + `_validate_convolve_over_sample_size` (`autoarray/dataset/imaging/dataset.py:23`) + accepts any `over_sample_size` (int or adaptive `Array2D`) whose every entry + is divisible by `convolve_over_sample_size`, and raises `DatasetException` + naming the rule otherwise. Its docstring states the mechanism in the prompt's + own words — "the k x s coupling, whereby values evaluated at per-pixel sizes + k_i * s are partially binned to the uniform s the convolution requires". + The pre-change equality rule is gone. + +2. **§2 partial pre-bin — SHIPPED**, as + `over_sample_util.binned_to_convolve_size_from` + (`autoarray/operators/over_sampling/over_sample_util.py:205`). The placement + fork the prompt required a design paragraph for was settled and is recorded in + [[kxs-design]]. Covered by `test_over_sample_util.py` on both legs the prompt + asked for — `binned_to_convolve_size_from__uniform_k__equals_manual_reshape_mean` + and `__adaptive_k__and_identity_fast_path`. + +3. **§3 PyAutoGalaxy callers — SHIPPED** via PyAutoGalaxy#486 ([[kxs-core]]). + +4. **§4 tests — SHIPPED.** Library numpy-only tests as above; the + `convolution_over_sampled.py` adaptive + s=2 `FitImaging` leg the prompt + specifies landed as autolens_workspace_test#154 ([[kxs-workspace-tests]]: + lp round trip chi2=0, adaptive pixelized 6.6e-5, divisibility guard, + adaptive simulate→fit 1.1e-29). + +5. **§5 simulator adoption — DELIBERATELY RE-SCOPED, not skipped.** + [[kxs-core]] records the fork resolved as option (c): executed simulators stay + at `s=1`, and option (a) was split out as + `draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md`. That + prompt is correctly still a live draft — it is the residue of this §5, and is + NOT covered by this record. + +6. **Phase-5 refactor exercise — SHIPPED** as PyAutoGalaxy#489 + ([[kxs-surface-refactor]]), plus the extra [[kxs-cache]] and [[kxs-refactor]] + legs the series added on top of the original plan. + +## Why it was missed + +Nothing upstream drifted — the trackers were right the whole time, exactly as in +the 2026-08-08 `planned.md` prune. The prompt is stale purely because `draft/` is +graded by no check, and slug-similarity is too weak to catch it: this file's stem +against `kxs-core` scores a Jaccard of 0.25, well under any workable threshold. +What found it was reading the prompt's acceptance criteria against the tree — and +it was provable from PyAutoMind alone, since the six phase records were already +sitting in `complete/2026/07/`. + +## Original prompt # Oversampled PSF: k×s evaluation/convolution coupling + simulator adoption Type: feature diff --git a/draft/feature/autoarray/rectangular_adapt_constant_split_guard.md b/complete/2026/07/rectangular-adapt-constant-split-guard.md similarity index 51% rename from draft/feature/autoarray/rectangular_adapt_constant_split_guard.md rename to complete/2026/07/rectangular-adapt-constant-split-guard.md index ecfb8712..3a0a70ae 100644 --- a/draft/feature/autoarray/rectangular_adapt_constant_split_guard.md +++ b/complete/2026/07/rectangular-adapt-constant-split-guard.md @@ -1,3 +1,80 @@ +# rectangular-adapt-constant-split-guard + +- shipped: 2026-07-28 (phase 1 of the @rhayes777 audit epic; the prompt never left `draft/`) +- epic: https://github.com/PyAutoLabs/PyAutoArray/issues/415 (open — phases 2-4 remain) +- issue: https://github.com/PyAutoLabs/PyAutoArray/issues/332 (the reporter's finding), tracker #416 closed +- prs: PyAutoArray#417 (`9411904d`) + PyAutoLens#662 (`2a3f1a63`), both merged +- repos: + - PyAutoArray +- see-also: `draft/bug/autoarray/rhayes_audit_validation_and_crashes.md` § "Phase 1 completion record — 2026-07-28" + +## Summary + +A duplicate prompt for work that shipped as phase 1 of the @rhayes777 API-audit +epic. The Mind already held the completion record — inside the *sibling* prompt +`rhayes_audit_validation_and_crashes.md`, which planned.md tracks — but this +second, independently-filed prompt for the same surface never learned about it. + +Recorded 2026-08-09 by the draft/ sweep. No work is owed on the library leg; +one leg of the prompt's § Verification is noted below as unconfirmed. + +## Verified against PyAutoArray main (`efaf3041`), 2026-08-09 + +The prompt asks for "an explicit validation guard which rejects this unsupported +configuration early, with an error message that tells users what to do instead". +That guard is on main: + +- **Guard**: `Pixelization.__init__` (`autoarray/inversion/pixelization.py:154`) + raises `exc.PixelizationException` when a split regularization meets a mesh + that does not support it. The message names both classes and tells the user + the two ways out — an adaptive mesh (`Delaunay`/`KNNBarycentric`) with the same + regularization, or a non-split scheme (`Constant` for `ConstantSplit`, `Adapt` + for `AdaptSplit`) with the same mesh. Substantively the prompt's suggested text. +- **Mechanism**: two capability flags rather than a type blacklist — + `AbstractMesh.supports_split_regularization` (default `True`, set `False` on + the rectangular family) × `AbstractRegularization.is_split_regularization` + (default `False`, set `True` on `ConstantSplit`/`AdaptSplit`/`AdaptSplitZeroth`). +- **The false pass-through is gone**: `InterpolatorRectangular`'s claim that split + "reuses the same mappings" — the source of the `IndexError: index 4 is out of + bounds for axis 0 with size 4` this prompt reproduces — no longer stands; + `interpolator/rectangular.py:466` now records that the combination is rejected + at construction instead. + +Against the prompt's § Verification: + +1. **Criterion 1 (concrete construction raises) — MET.** + `test_autoarray/inversion/pixelization/test_split_regularization_support.py` + parametrizes all **9** rectangular × split combinations (the prompt reported + 1) and asserts both class names appear in the message. +2. **Criterion 2 (`af.Model` composition form fails before Nautilus starts) — + NOT CONFIRMED.** The guard sits in `Pixelization.__init__`, so it fires + whenever the model is instantiated rather than at composition time. A search + for `supports_split_regularization` in PyAutoLens returns nothing, so no + separate pre-fit model-inspection guard was added. In practice the concrete + guard is reached on the first instantiation, which is early — but that this + precedes sampling was not verified here and would need a run to settle. The + prompt itself allowed this ordering ("add the concrete PyAutoArray guard + first and add a companion AutoLens / analysis guard where the prior model can + be inspected"), so this is a possible residue, not a regression. +3. **Criterion 3 (allowed combinations still work) — MET.** Rectangular + + `Constant`, adaptive + split, and rectangular + no regularization all have + passing parametrized tests. +4. **Criterion 4 (low-level regression test) — MET in the form the fix took.** + Because the capability is deliberately absent rather than repaired, the tests + assert the *clear failure*; the test module says so explicitly. The prompt's + "do not paper over this by clipping indices" instruction was honoured. + +## Why it was missed + +Two prompts described one surface. `rhayes_audit_validation_and_crashes.md` came +in through the audit and is tracked by `planned.md`, so it was updated when phase +1 shipped; this one came in separately from a user repro +(`z_help/jacob/HerBS-28…`) and, sitting in `draft/`, was graded by nothing. The +related `draft/feature/autoarray/regularization_jax_gradient_gaps.md` § 3 flagged +the same surface a third time and asked "merge at intake if so" — that merge is +now moot, and its leg 3 has been marked done in place. + +## Original prompt # Users keep combining `RectangularAdaptDensity` meshes with `ConstantSplit` Type: feature diff --git a/draft/feature/autofit/test_mode_representative_outputs_size_realistic.md b/complete/2026/07/test-mode-representative-outputs-size-realistic.md similarity index 68% rename from draft/feature/autofit/test_mode_representative_outputs_size_realistic.md rename to complete/2026/07/test-mode-representative-outputs-size-realistic.md index 7270a849..75fbbdbc 100644 --- a/draft/feature/autofit/test_mode_representative_outputs_size_realistic.md +++ b/complete/2026/07/test-mode-representative-outputs-size-realistic.md @@ -1,3 +1,45 @@ +# test-mode-representative-outputs-size-realistic + +- shipped: 2026-07-17 (the epic's own closing condition was met and never actioned) +- issues: PyAutoFit#1378 (phase 1), PyAutoFit#1381 + PyAutoConf#126 (phase 2), autolens_profiling#70 (the closing gate — CLOSED as `completed` 2026-07-17) +- phase-records: [[test-mode-representative-samples-phase-1-design]] [[test-mode-representative-samples-phase-2-core-api]] +- repos: + - PyAutoFit, PyAutoConf (now PyAutoNerves), autolens_profiling + +## Summary + +An umbrella prompt whose four phases were all resolved, tracked in the prompt's +own status line, and whose stated exit condition then quietly came true. + +The prompt is explicit that it is the umbrella — *"issue the phase files, not this +one"* — and its 2026-07-17 status block already recorded: + +- phase 1 (design) **COMPLETE**, PyAutoFit#1378, record in `complete/2026/07/` +- phase 2 (core API) **COMPLETE + MERGED**, PyAutoConf#126 + PyAutoFit#1381 +- phase 3 **ABSORBED** into slam-resume-profiling (autolens_profiling#70), prompt retired +- phase 4 **ABSORBED** (survey found RTD has no test-mode surface), prompt retired +- *"EPIC CLOSES when #70 ships its recipe leg."* + +## Verified 2026-08-09 + +**autolens_profiling#70 is closed, `state_reason: completed`, closed 2026-07-17 by +Jammy2211** — the same day the prompt's status block was last written. So the only +outstanding condition the umbrella named was satisfied within hours of the note +being taken, and nothing advanced the file. + +Recorded by the draft/ sweep. No work is owed. + +## Why it was missed + +This one needed neither a clone nor a code read — the prompt states its own exit +condition, and one issue-state lookup settles it. That makes it the cheapest class +of draft/ drift to detect and a good argument for extending the advisory +`issues --drafts` pass: #70 is named in the prompt body but is not the kind of +citation that pass currently nets, because the prompt cites it as a *gate* rather +than as its own tracking issue. Worth considering whether an "epic closes when X" +idiom should be machine-readable. + +## Original prompt # Test-mode representative outputs: size-realistic samples for instant pipeline runs Type: feature diff --git a/draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md b/complete/2026/08/normalise-auto-simulate-guard-idiom.md similarity index 50% rename from draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md rename to complete/2026/08/normalise-auto-simulate-guard-idiom.md index d4429732..0a48ebf1 100644 --- a/draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md +++ b/complete/2026/08/normalise-auto-simulate-guard-idiom.md @@ -1,3 +1,71 @@ +# normalise-auto-simulate-guard-idiom + +- shipped: verified on autolens_workspace main `9974f891` (the prompt never left `draft/`) +- follows: [[auto-simulate-guard-targets]] (autolens_workspace#359 → #364, autogalaxy_workspace#175) and the `should_simulate` migration (autolens_workspace#354) +- repos: + - autolens_workspace + +## Summary + +The prompt asks for four hand-rolled auto-simulate guards to be converted to +`al.util.dataset.should_simulate`, with one specific caution about not dropping a +stricter check while doing it. All four are converted on main, and the caution was +honoured — more carefully than the prompt asked. + +Recorded 2026-08-09 by the draft/ sweep. No work is owed. + +## Verified against autolens_workspace main (`9974f891`), 2026-08-09 + +Against the prompt's § Proposed work: + +1. **All four idiom-B sites converted — DONE.** `cluster/likelihood_function.py`, + `interferometer/features/pixelization/many_visibilities_preparation.py`, + `imaging/features/advanced/subhalo/sensitivity/slam_source_parametric.py` and + `…/slam_source_pixelized.py` all call `should_simulate` now. A repo-wide sweep + finds **zero** remaining hand-rolled `data.fits … .exists()` simulator guards + (492 files use the standard idiom). + +2. **The `mass.csv` check was preserved — DONE, and better than specified.** The + prompt warned "do not silently drop it" and offered two ways out. The shipped + form takes the first *and* fixes an ordering subtlety the prompt did not raise: + + ```python + if ( + al.util.dataset.should_simulate(str(dataset_path)) + or not (dataset_path / "mass.csv").exists() + ): + ``` + + with a comment recording that `should_simulate` is evaluated **first** so its + `PYAUTO_SMALL_DATASETS` rebuild always runs. Written the other way round, a + present `mass.csv` would short-circuit past the capped-rebuild side effect — + which is precisely the failure the conversion exists to prevent. + +3. **Re-running under the capped profile** — not independently re-run here; that + needs a real execution environment. + +4. **`required_files=[...]` on `should_simulate` — NOT taken.** PyAutoArray's + `should_simulate(dataset_path)` still has the single argument. This leg was + explicitly optional in the prompt ("consider whether…", and it would have made + the task library+workspace rather than workspace-only). The `mass.csv` clause + living in the script is the alternative the prompt allowed. + +**Not the same class, correctly left alone:** two `if not data_fits_path.exists():` +sites survive, in `multi_dataset/features/imaging_and_point_source/modeling.py:75` +and `cluster/start_here.py:153`. Both guard a one-off `urllib` download of real HST +data (RXJ1131, Abell 2744), not a simulator invocation, so `should_simulate`'s +capped-rebuild semantics would be actively wrong there — it would delete a +downloaded file to re-download it. Converting these would be a regression. + +## Note on detection + +This is the one finding across three sweeps that `pyauto-brain intake reconcile` +ranked `high` for the right reason: [[auto-simulate-guard-targets]] names the +prompt's path directly in its body. That is signal 3 in +`draft/feature/pyautomind/draft_staleness_detection_signals.md` working as +intended — though it still arrived among 51 other `high`s. + +## Original prompt # Normalise the two auto-simulate guard idioms Type: maintenance diff --git a/complete/archive/epics/jax_substructure_simulator.md b/complete/archive/epics/jax_substructure_simulator.md index 6cbf2a1f..f0677ace 100644 --- a/complete/archive/epics/jax_substructure_simulator.md +++ b/complete/archive/epics/jax_substructure_simulator.md @@ -30,6 +30,16 @@ __What's already done__ (no work needed): __Outstanding__ (sequenced): +> **CORRECTION 2026-08-09 (draft/ sweep): items 1-4 below are NOT outstanding — +> they SHIPPED on 2026-06-09.** See `complete/2026/06/jax-substructure-simulator.md` +> (PyAutoLens#543 + #544, PyAutoGalaxy direct commits, autolens_workspace_test +> #127/#128/#129), re-verified against upstream `main`. Their four prompt files +> stayed in `draft/` for two months after shipping and were retired into that +> record; the relative links below are therefore dead. The only live work from this +> epic is `draft/feature/jax_substructure/5_prng_key_vmap_noise.md` and +> `6_deflection_equivalence_test.md`, both confirmed still open. This tracker is +> archived material — read the completion record, not this list. + 1. [feature/jax_substructure/1_vmap_subhalo_deflections.md](../feature/jax_substructure/1_vmap_subhalo_deflections.md) — vectorized deflection path: represent N halos as `(max_N, n_params)` arrays, `jax.vmap` the profile deflection function, sum with mask. Integration test diff --git a/draft/bug/autoarray/reg_matrix_logdet_nonfinite_fix.md b/complete/archive/shelved/reg_matrix_logdet_nonfinite_fix.md similarity index 100% rename from draft/bug/autoarray/reg_matrix_logdet_nonfinite_fix.md rename to complete/archive/shelved/reg_matrix_logdet_nonfinite_fix.md diff --git a/complete/index.md b/complete/index.md index 80964b70..aaf02fbb 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -948 records across 7 buckets. +953 records across 7 buckets. ## Highlights @@ -50,6 +50,7 @@ _(curate hard-won records here — survives regeneration.)_ - [missing-auto-simulate-guards](2026/08/missing-auto-simulate-guards.md) - [multi-start-auto-convergence-real-search](2026/08/multi-start-auto-convergence-real-search.md) — `scripts/jax_assertions/multi_start_gradient_auto_convergence.py` failed - [nautilus-1core-serial-pool](2026/08/nautilus-1core-serial-pool.md) — corrective for the Heart RED "release validation FAILED (stage +- [normalise-auto-simulate-guard-idiom](2026/08/normalise-auto-simulate-guard-idiom.md) - [notebook-setup-notebook-regen-drift](2026/08/notebook-setup-notebook-regen-drift.md) - [nufft-parity-full-datasets](2026/08/nufft-parity-full-datasets.md) - [plot-array-stale-kwargs](2026/08/plot-array-stale-kwargs.md) @@ -311,6 +312,7 @@ _(curate hard-won records here — survives regeneration.)_ - [kxs-refactor](2026/07/kxs-refactor.md) - [kxs-surface-refactor](2026/07/kxs-surface-refactor.md) - [kxs-workspace-tests](2026/07/kxs-workspace-tests.md) +- [latent-samples-none-on-resumed-fit](2026/07/latent-samples-none-on-resumed-fit.md) - [lenstool-example](2026/07/lenstool-example.md) - [lenstool-scaling-reference-magnitude](2026/07/lenstool-scaling-reference-magnitude.md) — PR1 cluster + PR2 group/imaging — complete; PR3 SLaM deferred - [lifecycle-drift-self-heal](2026/07/lifecycle-drift-self-heal.md) @@ -384,6 +386,7 @@ _(curate hard-won records here — survives regeneration.)_ - [optional-none-default-typos](2026/07/optional-none-default-typos.md) — Fixed three PyAutoLens sites using the typing construct - [opus-wrapup](2026/07/opus-wrapup.md) — executed on Fable - [over-sample-trailing-one-to-two](2026/07/over-sample-trailing-one-to-two.md) — Replaced every trailing sub_size 1 in adaptive over-sampling schemes across five repos (169 files, 186 sites: … +- [oversampling-kxs-coupling](2026/07/oversampling-kxs-coupling.md) - [parked-sweep](2026/07/parked-sweep.md) - [per-frame-psf](2026/07/per-frame-psf.md) — per-frame native ePSFs live (psf/frame_epsf.py: sky-subtracted, DQ local-median patch in ESTIMATOR input only … - [per-project-literature](2026/07/per-project-literature.md) — task 3 of the autolens_assistant batch — hybrid literature rule live: Create scaffolds wiki/project/bibliograp… @@ -466,6 +469,7 @@ _(curate hard-won records here — survives regeneration.)_ - [python_312_floor_phase_4d_euclid_assistant](2026/07/python_312_floor_phase_4d_euclid_assistant.md) - [raw-guard-migration](2026/07/raw-guard-migration.md) — Leg 3 of the dataset-bulk series. Migrated 116 (autolens, 113 scripts) + 61 (autogalaxy, 59 scripts) raw `if n… - [rect-adapt](2026/07/rect-adapt.md) — rectangular adaptive-mesh edges — MERGED +- [rectangular-adapt-constant-split-guard](2026/07/rectangular-adapt-constant-split-guard.md) - [rectangular-kernel-cdf-mesh](2026/07/rectangular-kernel-cdf-mesh.md) — kernel-density CDF meshes RectangularKernelAdapt{Density,Image} (Enzi RTU) shipped opt-in — strict FD certifie… - [rectangular-mesh-consolidation](2026/07/rectangular-mesh-consolidation.md) — Consolidated PyAutoArray's rectangular mesh family from 8 mesh classes / 5 interpolators / 2 geometries down t… - [refactor-conductor](2026/07/refactor-conductor.md) @@ -543,6 +547,7 @@ _(curate hard-won records here — survives regeneration.)_ - [tenant-firewall-drift](2026/07/tenant-firewall-drift.md) - [test-mirror-restructure](2026/07/test-mirror-restructure.md) - [test-mode-bypass-completed](2026/07/test-mode-bypass-completed.md) — One-line fix — _fit_bypass_test_mode now calls paths.completed() before returning (mirrors start_resume_fit), … +- [test-mode-representative-outputs-size-realistic](2026/07/test-mode-representative-outputs-size-realistic.md) - [test-mode-representative-samples-phase-1-design](2026/07/test-mode-representative-samples-phase-1-design.md) - [test-mode-representative-samples-phase-2-core-api](2026/07/test-mode-representative-samples-phase-2-core-api.md) - [test-results-relayout](2026/07/test-results-relayout.md) — Hands#193 + Heart#106, lockstep pair diff --git a/dashboard.md b/dashboard.md index 86d4ee39..1c5d5c02 100644 --- a/dashboard.md +++ b/dashboard.md @@ -1,14 +1,14 @@ # PyAutoMind backlog dashboard - + -**148** filed prompts in the backlog · **6** already dispatched to issues (`active/`). Backlog view only — organism health lives with the Heart (`/health`), not here. +**139** filed prompts in the backlog · **6** already dispatched to issues (`active/`). Backlog view only — organism health lives with the Heart (`/health`), not here. | Work-type | Prompts | |-----------|--------:| -| bug | 42 | -| feature | 36 | -| maintenance | 22 | +| bug | 40 | +| feature | 30 | +| maintenance | 21 | | research | 18 | | docs | 17 | | refactor | 5 | @@ -17,17 +17,15 @@ | experiment | 1 | | release | 1 | -## bug (42) +## bug (40) | Prompt | Target | Difficulty | Autonomy | Priority | |--------|--------|------------|----------|----------| | [PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional?](draft/bug/autoarray/PROBE_adapt_double_square_coefficient.md) | autoarray | medium | supervised | normal | | [ConstantZeroth regularization is broken twice over — dead code presenting](draft/bug/autoarray/constant_zeroth_broken_dead_code.md) | autoarray | small | supervised | normal | | [PyNUFFT dev extra is incompatible with current SciPy on Python](draft/bug/autoarray/pynufft_scipy_pinv2_dev_extra.md) | autoarray | small | supervised | normal | -| [[WITHDRAWN 2026-07-17] superseded by draft/feature/autoarray/gradient_safe_logdet_settings_option.md — the verdict ruled out](draft/bug/autoarray/reg_matrix_logdet_nonfinite_fix.md) | autoarray | medium | supervised | high | | [@rhayes777's 2026-05-23 API audit — all 16 findings re-verified, still](draft/bug/autoarray/rhayes_audit_validation_and_crashes.md) | autoarray | medium | supervised | high | | [EP: cure the hierarchical parent-scale collapse basin (and make F10](draft/bug/autofit/ep_hierarchical_scale_collapse_moment_match.md) | autofit | medium | supervised | high | -| [compute_latent_samples crashes on a resumed completed fit (samples is None)](draft/bug/autofit/latent_samples_none_on_resumed_fit.md) | autofit | medium | supervised | low | | [Make autofit.messages safe under JAX jit trace (xp.array → xp.stack)](draft/bug/autofit/messages_xp_stack_jax_trace.md) | autofit | large | supervised | normal | | [`autofit.plot` functions accept `**kwargs` and silently discard them](draft/bug/autofit/plot_functions_discard_kwargs.md) | autofit | small | supervised | normal | | [TEST_MODE bypass crashes on ordered-parameter assertion ties](draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md) | autofit | small | supervised | normal | @@ -64,22 +62,19 @@ | [Tenant firewall: release_run.py carries an unlisted 'PyAutoLabs' instance fact](draft/bug/pyautoheart/tenant_firewall_release_run_instance_fact.md) | pyautoheart | small | safe | normal | | [`aplt.Output` stale-API drift in the remaining workspace repos](draft/bug/workspaces/aplt_output_drift_remaining_repos.md) | workspaces | small | supervised | normal | -## feature (36) +## feature (30) | Prompt | Target | Difficulty | Autonomy | Priority | |--------|--------|------------|----------|----------| | [Claude Development Prompt: Arcsecond Tick Label Decimal Placement](draft/feature/autoarray/arcsecond_to_decimal.md) | autoarray | large | supervised | normal | | [Can create a list of InversionMatrix objects for each dataset](draft/feature/autoarray/multiwavelength_inversion.md) | autoarray | medium | supervised | normal | | [Column-chunk the interferometer inversion mapping-matrix NUFFT so alma_high fits on](draft/feature/autoarray/nufft_mapping_matrix_column_chunking.md) | autoarray | large | supervised | normal | -| [The `al.SimulatorInterferometer` path that uses `al.TransformerNUFFT` (nufftax-backed) can't scale to](draft/feature/autoarray/nufft_simulator_chunking.md) | autoarray | too-large | supervised | high | -| [Oversampled PSF: k×s evaluation/convolution coupling + simulator adoption](draft/feature/autoarray/oversampling_kxs_coupling.md) | autoarray | large | supervised | normal | -| [Users keep combining `RectangularAdaptDensity` meshes with `ConstantSplit`](draft/feature/autoarray/rectangular_adapt_constant_split_guard.md) | autoarray | too-large | supervised | high | +| [The `al.SimulatorInterferometer` path that uses `al.TransformerNUFFT` (nufftax-backed) can't scale to](draft/feature/autoarray/nufft_simulator_chunking.md) | autoarray | small | supervised | high | | [Follow-up to `rectangular_adapt_cdf.md` (issue #322) and Path A](draft/feature/autoarray/rectangular_multi_submesh.md) | autoarray | too-large | supervised | normal | | [Regularization JAX gradient gaps — xp-ports + kernel-scheme linear algebra](draft/feature/autoarray/regularization_jax_gradient_gaps.md) | autoarray | medium | supervised | normal | | [EP analytic updates — implement the four planned work packages](draft/feature/autofit/ep_analytic_updates.md) | autofit | large | supervised | normal | | [The project @z_projects/ic50_workspace is our IC50 use case which we](draft/feature/autofit/ep_lbfgs_jax.md) | autofit | medium | safe | normal | | [Give PyAutoFit searches a `seed` — today no search can](draft/feature/autofit/search_seed_reproducibility.md) | autofit | medium | supervised | medium | -| [Test-mode representative outputs: size-realistic samples for instant pipeline runs](draft/feature/autofit/test_mode_representative_outputs_size_realistic.md) | autofit | medium | supervised | normal | | [Remote-MCP deployment tiers (2 + 3) for the results-inspector server](draft/feature/autofit_assistant/remote_mcp_deployment_tiers.md) | autofit_assistant | large | human-required | normal | | [dPIE: optional central-dispersion (sigma_0) parameterization](draft/feature/autogalaxy/dpie_sigma0_parameterization.md) | autogalaxy | small | supervised | low | | [`PIEMass.potential_2d_from`: implement the missing lensing potential](draft/feature/autogalaxy/piemass_potential.md) | autogalaxy | too-large | supervised | normal | @@ -90,22 +85,19 @@ | [Tune cluster-scale JOSS benchmarks toward their 5-minute targets](draft/feature/autolens_workspace/joss_cluster_benchmark_tuning.md) | autolens_workspace | medium | supervised | normal | | [Adopt oversampled PSFs in the start-here dataset chain (option a)](draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md) | autolens_workspace | large | supervised | normal | | [Scheduled runs — overnight queue passes with a morning report](draft/feature/autonomy/10_scheduled_runs.md) | autonomy | medium | supervised | low | -| [Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure](draft/feature/jax_substructure/1_vmap_subhalo_deflections.md) | jax_substructure | too-large | supervised | normal | -| [Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1](draft/feature/jax_substructure/2_tracer_lax_scan.md) | jax_substructure | too-large | supervised | normal | -| [Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2](draft/feature/jax_substructure/3_simulator_jax_e2e.md) | jax_substructure | too-large | supervised | normal | -| [Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal)](draft/feature/jax_substructure/4_vmap_batched_simulation.md) | jax_substructure | too-large | supervised | normal | | [Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the](draft/feature/jax_substructure/5_prng_key_vmap_noise.md) | jax_substructure | too-large | supervised | normal | | [Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the](draft/feature/jax_substructure/6_deflection_equivalence_test.md) | jax_substructure | too-large | supervised | normal | | [Extend the Profiling Agent scope to track JAX compile/eval times](draft/feature/profiling/profiling_agent_jax_compile_time_scope.md) | profiling | large | supervised | high | | [SamplerSurface: scan the autolens-side findings-lane tiers](draft/feature/pyautobrain/samplers_surface_autolens_tiers.md) | pyautobrain | small | safe | normal | | [Token-light wiki index over the complete/ archive](draft/feature/pyautomind/complete_archive_wiki.md) | pyautomind | medium | supervised | normal | +| [Make draft/ staleness detectable — `intake reconcile` measured, and the](draft/feature/pyautomind/draft_staleness_detection_signals.md) | pyautomind | medium | supervised | high | | [LACosmic per-frame CR masking option + decouple PSF-star pass from](draft/feature/pyautoreduce/lacosmic_cr_option_and_star_pass_decoupling.md) | pyautoreduce | medium | supervised | high | | [Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier](draft/feature/workspaces/gallery_runner_missing_tiers.md) | workspaces | small | supervised | low | | [The imaging `features/advanced/los_halos` example needs improving and padding out before](draft/feature/workspaces/group_los_halos.md) | workspaces | medium | safe | normal | | [The imaging `features/advanced/subhalo/sensitivity` example needs improving and padding out before](draft/feature/workspaces/group_subhalo_sensitivity.md) | workspaces | medium | safe | normal | | [Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver](draft/feature/workspaces/restore_multiple_sources_lensing_of_lens.md) | workspaces | too-large | supervised | normal | -## maintenance (22) +## maintenance (21) | Prompt | Target | Difficulty | Autonomy | Priority | |--------|--------|------------|----------|----------| @@ -113,7 +105,6 @@ | [autolens_profiling is now a mature project, with a good separation](draft/maintenance/autolens_profiling/polish.md) | autolens_profiling | large | supervised | normal | | [cosmos_web_ring stores boolean masks as float64, wasting ~3.4 MB of](draft/maintenance/autolens_workspace/cosmos_web_ring_mask_dtype.md) | autolens_workspace | small | supervised | low | | [LaTeX in non-raw docstrings emits SyntaxWarning: invalid escape sequence](draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md) | autolens_workspace | small | supervised | low | -| [Normalise the two auto-simulate guard idioms](draft/maintenance/autolens_workspace/normalise_auto_simulate_guard_idiom.md) | autolens_workspace | small | supervised | low | | [autolens_workspace_developer rectangular experiments — Gut stash + rename](draft/maintenance/autolens_workspace_developer/rectangular_experiments_gut_stash.md) | autolens_workspace_developer | small | supervised | normal | | [autolens_workspace_developer: broad stale-API rot (56 symbols, no CI)](draft/maintenance/autolens_workspace_developer/stale_api_rot_audit.md) | autolens_workspace_developer | medium | supervised | normal | | [Auto-request GitHub Copilot code review on every PR, org-wide](draft/maintenance/ci/copilot_auto_review.md) | ci | large | supervised | normal | diff --git a/draft/bug/autofit/latent_samples_none_on_resumed_fit.md b/draft/bug/autofit/latent_samples_none_on_resumed_fit.md deleted file mode 100644 index 28d3bab6..00000000 --- a/draft/bug/autofit/latent_samples_none_on_resumed_fit.md +++ /dev/null @@ -1,39 +0,0 @@ -# compute_latent_samples crashes on a resumed completed fit (samples is None) - -Type: bug -Target: autofit -Repos: -- @PyAutoFit -Difficulty: medium -Autonomy: supervised -Priority: low -Status: draft - -## Finding (2026-07-25 full health sweep) - -Running `autolens_workspace_test` `misc/latent/latent_variables_smoke.py` (and -`latent_nan_robustness.py`) twice in the same output tree fails on the second -run: the search resumes ("Fit Already Completed: skipping non-linear search"), -`result.samples` comes back `None` on the resume path (PYAUTO_TEST_MODE=2 -bypass), and - - autofit/non_linear/analysis/latent.py:113 latent_samples_from - -> samples.model -> AttributeError: 'NoneType' object has no attribute 'model' - -A fresh run (output cleared) passes. So the latent pipeline works, but the -resume/load path hands `compute_latent_samples` a `None` samples object -instead of the persisted samples (or a clear error). - -## Task - -Determine whether the resume path should (a) reload persisted samples so -latent computation works on resumed results, or (b) raise a clear, guarded -error from `compute_latent_samples` when samples are unavailable. May be -test-mode-specific — check whether a real (non-bypass) resumed fit also -returns samples=None. - -## Acceptance - -Second invocation of the latent smoke scripts in an existing output tree -either computes latent samples or fails with an intentional, documented -message — never an AttributeError from inside latent_samples_from. diff --git a/draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md b/draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md index 9f0e6811..3149fe80 100644 --- a/draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md +++ b/draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md @@ -7,7 +7,52 @@ Repos: Difficulty: small Autonomy: supervised Priority: normal -Status: formalised +Status: formalised — STILL REPRODUCES; see the 2026-08-09 note before grading this against main + +## 2026-08-09 — do NOT mistake the adjacent FitException catch for this fix + +Checked by the draft/ sweep against PyAutoFit main (`3b960609`). The bypass path +in `abstract_search.py` **now catches `exc.FitException`** and continues with the +`-1e99` sentinel, logging "TEST MODE 2: likelihood verification raised +FitException … treating as a resample-rejected instance". That reads exactly like +this prompt's suggested fix. **It is not.** The bug below still reproduces. + +The catch wraps only the likelihood call. The model instantiation is on the line +*before* the `try`: + +```python +if call_likelihood: + instance = model.instance_from_vector(vector=parameter_vector) # <-- outside + try: + log_likelihood = float(analysis.log_likelihood_function(instance)) + except exc.FitException as e: + ... +``` + +and `instance_from_vector` → `instance_for_arguments` → `check_assertions` +(`autofit/mapper/prior_model/abstract.py:193`) is precisely what raises +`exc.FitException("N assertions failed!")` when an ordering assertion ties at the +prior medians. `ignore_assertions` defaults to `False` and the bypass does not +pass it. So the assertion exception escapes the guard entirely and still +hard-fails the run. + +The upside: the fix is now a one-liner rather than the "catch and retry with a +perturbation" design sketched below. Two options, both cheap and both +deterministic: + +- move the `instance_from_vector` call inside the existing `try` — the sentinel + path already does the right thing for a rejected instance; or +- pass `ignore_assertions=True` at the bypass instantiation, on the grounds that + a verification eval at the medians is not a sampled point and assertions exist + to steer sampling. + +The second is probably the better semantics (a tied median is not a pathological +model), but it changes what the verification eval attests to — pick deliberately. +Prefer either over adding perturbation logic. + +`Difficulty:` stays small. The § Blocks note below still holds. + +--- Found during the CTI resurrection epic (Phase 4, 2026-07-17). `PYAUTO_TEST_MODE=2/3` bypass evaluates the model at the **prior medians**. A model whose components have diff --git a/draft/bug/health_fixes/README.md b/draft/bug/health_fixes/README.md index ed3f5592..26636959 100644 --- a/draft/bug/health_fixes/README.md +++ b/draft/bug/health_fixes/README.md @@ -15,7 +15,7 @@ Each failing script is assigned to exactly one prompt: |---|---:|---| | [samples_parameter_paths.md](samples_parameter_paths.md) — ⚠️ parked, does not reproduce on current `main` ([PyAutoFit#1327](https://github.com/PyAutoLabs/PyAutoFit/issues/1327), blocked on clean-CI re-validation) | 9 | PyAutoFit result/sample path resolution | | [autofit_sampler_database.md](autofit_sampler_database.md) | 9 | Emcee NaNs and database output discovery | -| [aggregator_output_contracts.md](aggregator_output_contracts.md) | 7 | Result/aggregator prerequisites and generated paths | +| ~~aggregator_output_contracts.md~~ — ✅ **SHIPPED 2026-07-07**, record `complete/2026/07/aggregator-output-contracts.md` (PyAutoFit#1324; autogalaxy_workspace#122, autolens_workspace#229, autolens_workspace_test#146 all merged) | 7 | Result/aggregator prerequisites and generated paths | | [jax_runtime_and_parity.md](jax_runtime_and_parity.md) | 6 | JAX/TFP compatibility and likelihood parity | | [jit_visualization_outputs.md](jit_visualization_outputs.md) | 4 | Quick-update visualizations not producing images | | [numerical_inversion_failures.md](numerical_inversion_failures.md) | 2 | Non-positive-definite inversion matrices | @@ -24,3 +24,27 @@ Each failing script is assigned to exactly one prompt: Total: **42 scripts**. Scripts that pass on current `main` remain listed because they still require a clean-worktree, directory-order reproduction before being declared fixed. Do not rebaseline assertions or edit tutorials to conceal a library regression. + +## 2026-08-09 sweep — three things to know before working this folder + +1. **One of the seven has shipped** (aggregator_output_contracts, struck through above); + its file is gone from this folder and lives in `complete/2026/07/`. The table said + nothing about that for a month. + +2. **Script paths in `jit_visualization_outputs.md` and `jax_runtime_and_parity.md` are + all stale** — every one 404s, and every one still exists under a renamed layout + (`scripts/jax_likelihood_functions//` → `scripts//jax_likelihood/`; + `/modeling_visualization_jit.py` → `/visualization/…`; `multi/` → + `multi_dataset/`). Corrected tables are in each prompt. A 404 in this cluster means + drift, not deletion. + +3. **Many of the named scripts are now parked** in their workspace's + `config/build/no_run.yaml`, mostly `SLOW` for cap timeouts, with dates *after* these + prompts were filed. A parked script cannot fail release validation, so a green + release run is not evidence of a fix — the 2026-08-07 Stage 3 integrate reports + `657p/0f/101s/0t`, and those **101 skips** are where this cluster's scripts went. The + parkings also cite a **different failure** (timeout) from the defects these prompts + describe, so unparking is a precondition for reproducing any of them. + +None of the five remaining prompts was graded shipped. What this sweep could establish +offline is recorded in each file; the reproduction legs all need real runs. diff --git a/draft/bug/health_fixes/jax_runtime_and_parity.md b/draft/bug/health_fixes/jax_runtime_and_parity.md index c493f12c..e164a6f8 100644 --- a/draft/bug/health_fixes/jax_runtime_and_parity.md +++ b/draft/bug/health_fixes/jax_runtime_and_parity.md @@ -7,6 +7,64 @@ Autonomy: supervised Priority: high Status: formalised +## 2026-08-09 — EVERY SCRIPT PATH BELOW IS STALE (they moved, they were not deleted) + +Checked by the draft/ sweep. All 6 scripts named in § Scripts return **404** at the +paths written below, and **all 6 still exist** under a renamed layout. Do not read the +404s as "the scripts were deleted" or "this was fixed by removal". + +Two systematic renames landed in both test workspaces since this prompt was filed: + +``` +scripts/jax_likelihood_functions//X.py -> scripts//jax_likelihood/X.py +scripts//modeling_visualization_jit.py -> scripts//visualization/modeling_visualization_jit.py +scripts/multi/... -> scripts/multi_dataset/... +``` + +Verified in `autolens_workspace_test` (`4cea3f8c`, cloned) and +`autogalaxy_workspace_test` (raw, every path resolves 200). This is the **same rename +family** that cost a previous session time on +`draft/bug/autolens/jax_point_source_point_smoke_sentinel.md` — treat a 404 in this +cluster as path drift until proven otherwise. + +Corrected paths: + +| named in § Scripts | actual path on main | repo | +|---|---|---| +| `…/jax_likelihood_functions/imaging/delaunay_mge.py` | `scripts/imaging/jax_likelihood/delaunay_mge.py` | autogalaxy_workspace_test | +| `…/jax_likelihood_functions/interferometer/delaunay_mge.py` | `scripts/interferometer/jax_likelihood/delaunay_mge.py` | autogalaxy_workspace_test | +| `…/jax_likelihood_functions/multi/delaunay_mge.py` | `scripts/multi_dataset/jax_likelihood/delaunay_mge.py` | autogalaxy_workspace_test | +| `…/jax_likelihood_functions/interferometer/delaunay_mge.py` | `scripts/interferometer/jax_likelihood/delaunay_mge.py` | autolens_workspace_test | +| `…/jax_likelihood_functions/multi/rectangular.py` | `scripts/multi_dataset/jax_likelihood/rectangular.py` | autolens_workspace_test | +| `…/jax_likelihood_functions/multi/rectangular_mge.py` | `scripts/multi_dataset/jax_likelihood/rectangular_mge.py` | autolens_workspace_test | + +### Parking state — read before assuming CI still exercises these + +**4 of the 6 are parked SLOW**, all dated 2026-07-14 and all citing PyAutoHeart#74 +(flaking at the 1800s `mode=release` cap): + +- `autogalaxy_workspace_test` `no_run.yaml`: `imaging/jax_likelihood/delaunay_mge.py`, + `interferometer/jax_likelihood/delaunay_mge.py`, `multi_dataset/jax_likelihood/delaunay_mge` +- `autolens_workspace_test` `no_run.yaml`: `interferometer/jax_likelihood/delaunay_mge.py` + +The two `multi_dataset/jax_likelihood/rectangular*.py` scripts are **not** parked. + +Also note `autolens_workspace_test` parks `multi_dataset/jax_likelihood/delaunay.py` +as `NEEDS_FIX 2026-08-01 - hangs to the 1800s release cap in 3 of 5 release-integrate +runs` — a sibling in the same family, filed after this prompt, suggesting the JAX +likelihood timeout story has moved on independently of this prompt. + +**This changes what the prompt's premise means.** A parked script cannot fail in +release validation, so the 2026-08-07 release drive's Stage 3 result (51/51 jobs green, +`657p/0f/101s/0t` — note the **101 skips**) is *not* evidence these were fixed. Nothing +here is graded shipped; the failures are simply no longer being provoked. Note too that +the parkings are for **timeouts**, a different failure from the defect this prompt +describes — so unparking is a precondition for reproducing it at all. + +Not re-graded here: whether the underlying defect still reproduces. That needs real +runs, which a cloud session cannot do. + +--- ## Context Six JAX likelihood scripts failed with the rehearsed release stack. CI included diff --git a/draft/bug/health_fixes/jit_visualization_outputs.md b/draft/bug/health_fixes/jit_visualization_outputs.md index 3517d870..43a21f32 100644 --- a/draft/bug/health_fixes/jit_visualization_outputs.md +++ b/draft/bug/health_fixes/jit_visualization_outputs.md @@ -7,6 +7,60 @@ Autonomy: supervised Priority: high Status: formalised +## 2026-08-09 — EVERY SCRIPT PATH BELOW IS STALE (they moved, they were not deleted) + +Checked by the draft/ sweep. All 4 scripts named in § Scripts return **404** at the +paths written below, and **all 4 still exist** under a renamed layout. Do not read the +404s as "the scripts were deleted" or "this was fixed by removal". + +Two systematic renames landed in both test workspaces since this prompt was filed: + +``` +scripts/jax_likelihood_functions//X.py -> scripts//jax_likelihood/X.py +scripts//modeling_visualization_jit.py -> scripts//visualization/modeling_visualization_jit.py +scripts/multi/... -> scripts/multi_dataset/... +``` + +Verified in `autolens_workspace_test` (`4cea3f8c`, cloned) and +`autogalaxy_workspace_test` (raw, every path resolves 200). This is the **same rename +family** that cost a previous session time on +`draft/bug/autolens/jax_point_source_point_smoke_sentinel.md` — treat a 404 in this +cluster as path drift until proven otherwise. + +Corrected paths: + +| named in § Scripts | actual path on main | repo | +|---|---|---| +| `…/scripts/ellipse/modeling_visualization_jit.py` | `scripts/ellipse/visualization/modeling_visualization_jit.py` | autogalaxy_workspace_test | +| `…/scripts/imaging/modeling_visualization_jit.py` | `scripts/imaging/visualization/modeling_visualization_jit.py` | autogalaxy_workspace_test | +| `…/scripts/interferometer/modeling_visualization_jit.py` | `scripts/interferometer/visualization/modeling_visualization_jit.py` | autogalaxy_workspace_test | +| `…/scripts/point_source/modeling_visualization_jit.py` | `scripts/point_source/visualization/modeling_visualization_jit.py` | autolens_workspace_test | + +### Parking state — read before assuming CI still exercises these + +**1 of the 4 is parked.** `autolens_workspace_test` `config/build/no_run.yaml:30`: +`point_source/visualization/modeling_visualization_jit # SLOW 2026-07-08 - JIT + Part-2 +live Nautilus fit exceeds 300s cap`. The three `autogalaxy_workspace_test` scripts are +**not** parked and should still be running. + +Adjacent and worth knowing: `autolens_workspace_test` also parks its *own* +`imaging/visualization/modeling_visualization_jit` (SLOW 2026-07-30, "re-measured: times +out at the 300s cap") and `interferometer/visualization/modeling_visualization_jit` +(SLOW 2026-07-30, "local re-measurement OOM-killed"). Those two are not named in this +prompt but are the same script family, so the timeout problem is broader than the four +listed here. + +**This changes what the prompt's premise means.** A parked script cannot fail in +release validation, so the 2026-08-07 release drive's Stage 3 result (51/51 jobs green, +`657p/0f/101s/0t` — note the **101 skips**) is *not* evidence these were fixed. Nothing +here is graded shipped; the failures are simply no longer being provoked. Note too that +the parkings are for **timeouts**, a different failure from the defect this prompt +describes — so unparking is a precondition for reproducing it at all. + +Not re-graded here: whether the underlying defect still reproduces. That needs real +runs, which a cloud session cannot do. + +--- ## Context Four test-workspace scripts expect real release-profile searches to invoke the JIT-cached diff --git a/draft/bug/health_fixes/release_timeout_policy.md b/draft/bug/health_fixes/release_timeout_policy.md index 58e54635..08a191b9 100644 --- a/draft/bug/health_fixes/release_timeout_policy.md +++ b/draft/bug/health_fixes/release_timeout_policy.md @@ -7,6 +7,36 @@ Autonomy: supervised Priority: normal Status: formalised +## 2026-08-09 — 1 of the 5 is resolved via § Required work option 3b + +Checked by the draft/ sweep against the two workspaces' `config/build/no_run.yaml` on +main. The prompt's item 3 offers an explicit fork per script — optimize below the cap, +**or** add a documented `SLOW` entry. One script has taken the second branch: + +- **`autolens_workspace/scripts/cluster/start_here.py` — PARKED.** + `- cluster/start_here # SLOW 2026-07-22 - hits the full 1800s mode=release cap in + workspace-validation (PyAutoHeart run 29912642195). Script-specific…` That is exactly + the documented-parking outcome this prompt asks for, decided after it was filed. + +The other four are **not** in either `no_run.yaml`: + +- `autogalaxy_workspace/scripts/ellipse/multipoles.py` +- `autolens_workspace/scripts/cluster/modeling.py` +- `autolens_workspace/scripts/imaging/features/advanced/double_einstein_ring/chaining.py` +- `autolens_workspace/scripts/multi/features/slam/simultaneous.py` + +Which means either they were optimized under the cap, or the decision was never taken. +**This sweep cannot tell which** — item 1 requires benchmarking from a clean output tree, +which needs a real run. Do not read "absent from no_run.yaml" as "optimized"; that is the +same unproven inference the prompt's item 4 warns against. + +One caution for whoever picks this up: the caps themselves moved. This prompt is written +against a **300s** cap, while the 2026-07-22 parking above cites an **1800s +`mode=release`** cap. Re-establish which cap applies to each script before benchmarking, +or the numbers will not mean what the prompt assumes. + +--- + ## Context Five scripts exceeded PyAutoBuild's 300-second per-script cap in release run diff --git a/draft/bug/priors/11_transformed_message_semantics_doc.md b/draft/bug/priors/11_transformed_message_semantics_doc.md index 7feb6353..211693a7 100644 --- a/draft/bug/priors/11_transformed_message_semantics_doc.md +++ b/draft/bug/priors/11_transformed_message_semantics_doc.md @@ -5,7 +5,36 @@ Target: priors Difficulty: large Autonomy: supervised Priority: normal -Status: formalised +Status: HALF SHIPPED — edge 1 is documented, edge 2 is not (2026-08-09) + +## 2026-08-09 — the reversal-convention half landed; the reciprocal half did not + +Checked by the draft/ sweep against PyAutoFit main (`3b960609`). The EP framework +review's Phase 2 (PyAutoFit#1334, shipped 2026-07-08) explicitly carried +"transform composition convention **incl. bug/priors/11 doc half**" — see +[[ep-framework-review]]. Verified which half: + +**Edge 1 (asymmetric reversal convention) — DONE.** `_transform` and +`_inverse_transform` in `messages/composed_transform.py` now both carry +docstrings naming the direction each maps ("physical → base" / "base → physical") +and stating that "the asymmetry with `_transform` is deliberate and load-bearing +(module docstring)". `autofit/graphical/README.md` § 2 also points readers at the +module docstring for the composition-order convention. That is the foot-gun this +prompt's § 1 exists to defuse. + +**Edge 2 (`LinearShiftTransform` stores the inverse of the intuitive scale) — NOT +DONE.** `messages/transform.py:171` is unchanged: the class still has **no +docstring at all**, and `super().__init__(DiagonalMatrix(np.reciprocal(self.scale)))` +sits bare — not even the `# Jacobian = 1/scale` inline comment this prompt writes +out. The physical-vs-base kwarg confusion that compounded the LogUniform sign bug +is still undocumented. + +**Remaining work is § 2 only.** Add the class docstring stating that `shift`/`scale` +describe physical space while the stored parent Jacobian is `1/scale` because the +transform runs physical → base, and keep the `log_det` sign explanation with it. +`Difficulty:` drops `large` → `small` accordingly; skip § 1 entirely. + +--- Found during the priors/messages audit (see `PyAutoPrompt/autofit/priors_and_messages_math_audit.md`, finding C6). diff --git a/draft/bug/priors/12_single_source_density_refactor.md b/draft/bug/priors/12_single_source_density_refactor.md index ed5ff590..9d58599b 100644 --- a/draft/bug/priors/12_single_source_density_refactor.md +++ b/draft/bug/priors/12_single_source_density_refactor.md @@ -160,6 +160,20 @@ kwarg as today; everything else inherits. into one issue. +## 2026-08-09 — PREREQUISITE HAS LANDED; this is unblocked + +Checked by the draft/ sweep. The Fable verdict below sequences this behind +Phases 1-2 of `research/graphical_ep/ep_framework_review.md`. That review +**completed 2026-07-08**, all 8 phases — record [[ep-framework-review]]. Phase 1 +(EP statistics correctness review) is complete with findings F1-F9 on +PyAutoFit#1332; Phase 2 (formal documentation of the graphical package) **shipped** +as PyAutoFit#1334, adding `autofit/graphical/README.md` (16 numbered +code-anchored equations) plus the statistical docstrings. + +So the design input this work package was waiting on — the EP review's inventory +of which message operations the factor graph actually needs — exists. The gate is +open; this is ready to plan, not blocked. Nothing here has shipped. + ## Fable verdict (2026-07-08, PyAutoFit main @ 0f26ff2d8; PyAutoFit#1330) **Verdict: still valid — design discussion; sequence behind the EP review.** diff --git a/draft/bug/priors/13_collapse_prior_and_message.md b/draft/bug/priors/13_collapse_prior_and_message.md index 2e59136b..1dcbbb52 100644 --- a/draft/bug/priors/13_collapse_prior_and_message.md +++ b/draft/bug/priors/13_collapse_prior_and_message.md @@ -176,6 +176,20 @@ the codebase ~half as legible as it could be. 7. **Stop. No code changes until a design is approved.** +## 2026-08-09 — PREREQUISITE HAS LANDED; this is unblocked + +Checked by the draft/ sweep. The Fable verdict below sequences this behind +Phases 1-2 of `research/graphical_ep/ep_framework_review.md`. That review +**completed 2026-07-08**, all 8 phases — record [[ep-framework-review]]. Phase 1 +(EP statistics correctness review) is complete with findings F1-F9 on +PyAutoFit#1332; Phase 2 (formal documentation of the graphical package) **shipped** +as PyAutoFit#1334, adding `autofit/graphical/README.md` (16 numbered +code-anchored equations) plus the statistical docstrings. + +So the design input this work package was waiting on — the EP review's inventory +of which message operations the factor graph actually needs — exists. The gate is +open; this is ready to plan, not blocked. Nothing here has shipped. + ## Fable verdict (2026-07-08, PyAutoFit main @ 0f26ff2d8; PyAutoFit#1330) **Verdict: still valid — bundle with 12 behind the EP review.** diff --git a/draft/bug/priors/14_replace_transform_stack_with_bijectors.md b/draft/bug/priors/14_replace_transform_stack_with_bijectors.md index 99280f5d..07cc25f3 100644 --- a/draft/bug/priors/14_replace_transform_stack_with_bijectors.md +++ b/draft/bug/priors/14_replace_transform_stack_with_bijectors.md @@ -198,6 +198,20 @@ one release cycle. the transform stack underneath `TransformedMessage`. +## 2026-08-09 — PREREQUISITE HAS LANDED; this is unblocked + +Checked by the draft/ sweep. The Fable verdict below sequences this behind +Phases 1-2 of `research/graphical_ep/ep_framework_review.md`. That review +**completed 2026-07-08**, all 8 phases — record [[ep-framework-review]]. Phase 1 +(EP statistics correctness review) is complete with findings F1-F9 on +PyAutoFit#1332; Phase 2 (formal documentation of the graphical package) **shipped** +as PyAutoFit#1334, adding `autofit/graphical/README.md` (16 numbered +code-anchored equations) plus the statistical docstrings. + +So the design input this work package was waiting on — the EP review's inventory +of which message operations the factor graph actually needs — exists. The gate is +open; this is ready to plan, not blocked. Nothing here has shipped. + ## Fable verdict (2026-07-08, PyAutoFit main @ 0f26ff2d8; PyAutoFit#1330) **Verdict: still valid — go/no-go after 12/13.** diff --git a/draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md b/draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md index 4d666569..77f7cf44 100644 --- a/draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md +++ b/draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md @@ -3,6 +3,31 @@ Follow-up to **PyAutoFit#1434 / PR#1436**, which moved the on-the-fly update cadence message into the library. Do not start until #1436 has merged. +## 2026-08-09 — UNBLOCKED, and the counts refreshed + +Checked by the draft/ sweep. **PyAutoFit#1436 merged 2026-07-30T21:49:25Z** +(`daa0dbcb`, closes #1434), so the gate above is satisfied — this is ready to +start, not waiting. Nothing here has shipped. + +Counts re-measured against autolens_workspace main (`9974f891`), which sharpens +the table below (it estimated "14+"): + +| surface | count | +|---|--:| +| `.py` scripts still printing `On-the-fly updates every iterations_per_quick_update` | **19** | +| of those, still carrying the "notebook cell **with** progress" typo | **16** | +| `.ipynb` notebooks carrying the line | **54** | + +The notebook count is much larger than the script count because the line also +appears in generated notebooks across sibling directories — confirm whether those +are all regenerated from the 19 scripts, or whether some notebooks are authored +directly, before assuming a regeneration pass covers them. + +For what the library now emits in its place, see PyAutoFit#1436's own summary: the +message has two branches, because the packaged default cadence is the inf-like +`1e99` never-sentinel, so the replacement text is either a real integer cadence or +a statement that updates are disabled naming the config key. + ## Problem 23 workspace scripts print this block before `search.fit(...)`: diff --git a/draft/docs/workspaces/preloads_advanced_workspace_guide.md b/draft/docs/workspaces/preloads_advanced_workspace_guide.md index b060cd8d..30e0b946 100644 --- a/draft/docs/workspaces/preloads_advanced_workspace_guide.md +++ b/draft/docs/workspaces/preloads_advanced_workspace_guide.md @@ -15,6 +15,26 @@ Primary repos: **@autolens_workspace**, **@autogalaxy_workspace** (workspace doc `Preloads` API landing in **@PyAutoArray** (the `datacube-shared-state` task, PyAutoLens#565 / sub-task B of the `analysis_shared_state` epic). +## 2026-08-09 — UNBLOCKED, the dependency has landed + +Checked by the draft/ sweep. Verified the **API**, not merely the issue state: + +- `PyAutoArray/autoarray/__init__.py` on `main` exports `AbstractPreloads`, + `PreloadsImaging` and `PreloadsInterferometer`. The `Preloads` API this guide + documents exists and is public. +- PyAutoLens#565 (datacube shared-state, sub-task B) closed `completed` + 2026-07-10. + +Worth checking the issue state alone would have been weaker evidence: the gate is +phrased as "the `Preloads` API landing in @PyAutoArray", which is a different repo +from the issue that tracked the work. + +The guide can now be written against the shipped API. One caution: locate the +preloads module before citing paths — guesses at `autoarray/preloads.py`, +`autoarray/dataset/preloads.py` and `autoarray/inversion/inversion/preloads.py` +all 404, so the import in `__init__.py` is the reliable entry point. Path-guessing +has been wrong four times in this sweep. + ## Why this guide is needed `Preloads` let a caller compute an invariant fit/inversion quantity once (e.g. the curvature matrix diff --git a/draft/feature/autoarray/nufft_simulator_chunking.md b/draft/feature/autoarray/nufft_simulator_chunking.md index e7abe793..bff52abf 100644 --- a/draft/feature/autoarray/nufft_simulator_chunking.md +++ b/draft/feature/autoarray/nufft_simulator_chunking.md @@ -2,10 +2,52 @@ Type: feature Target: PyAutoArray -Difficulty: too-large +Difficulty: small Autonomy: supervised Priority: high -Status: formalised +Status: OVERTAKEN — re-scoped 2026-08-09, see the block below before reading further + +## 2026-08-09 — the library work below is SHIPPED; only the wiring is left + +Found by the `draft/` sweep. Verified against PyAutoArray main (`efaf3041`). + +**Everything in § "The fix" option 1 is already on main**, delivered by +**PyAutoArray#330 ("TransformerNUFFT: add chunk_size knob to cap nufftax gather +buffer"), merged 2026-05-22** — roughly seven weeks BEFORE the Intake Agent +retroactively formalised this prompt on 2026-07-08. It landed to the letter of +the plumbing section below: + +- `TransformerNUFFT.__init__` takes `chunk_size: Optional[int] = None` — this + prompt's suggested name and its "no chunking by default" default, so the + small-N `sma` callers pay nothing. A non-positive value raises `ValueError`. +- `_forward_native` (`autoarray/operators/transformer.py:681`) splits the + visibility axis and iterates with **`jax.lax.scan`** on the JAX path (a Python + loop only on the numpy path) — exactly the "do NOT use a Python `for`, it + unrolls and blows up JIT compile time" requirement below. +- `image_from` — the adjoint via `nufft2d1`, which this prompt flagged as "out + of scope today, but flag it" — **is chunked too**, in the same shape. + +The sibling blocker this prompt names has also shipped: the `apply_sparse_operator` +alma-scale precompute OOM closed 2026-05-22 as PyAutoArray#329 +([[alma-apply-sparse-operator-oom]]). + +**What is actually left is one wiring leg.** `complete/2026/07/interferometer-jax-jit.md` +records it: "`chunk_size` is a `TransformerNUFFT.__init__` argument that +`SimulatorInterferometer` **NEVER sets**, so the `lax.scan` branch is unreachable +via the simulator." So the capability exists and is untested-in-anger from the +simulator side. The remaining task is to plumb `chunk_size` from +`SimulatorInterferometer` (a default chosen by the memory budget already derived +below, ~1M for nspread=14/complex64 on a 40 GB working budget), then run the +§ Verification below to confirm alma_high actually lands. + +`Difficulty:` accordingly drops `too-large` → `small`; `Priority:` stays high +because the profiling sweep it unblocks is still blocked. Do **not** re-implement +the chunking. + +Note also that `option 2` below (upstream `nufftax` `chunk_size`) was never the +chosen scope and remains untouched — still a legitimate follow-up, still optional. + +--- The `al.SimulatorInterferometer` path that uses `al.TransformerNUFFT` (nufftax-backed) can't scale to ALMA-realistic visibility counts. At ~5M visibilities on an 800×800 real-space grid it OOMs on an A100 (80 GB) with a single ~15.7 GB allocation; at 10M it's ~31 GB. The likelihood path scales fine to the same regime because `apply_sparse_operator` precomputes a small W-Tilde matrix bounded by `N_source_pixels` (~thousands), not by `N_visibilities`. The simulator has no equivalent escape valve — every forward call does one dense nufftax spread. diff --git a/draft/feature/autoarray/rectangular_multi_submesh.md b/draft/feature/autoarray/rectangular_multi_submesh.md index 18b9f271..de1f8be2 100644 --- a/draft/feature/autoarray/rectangular_multi_submesh.md +++ b/draft/feature/autoarray/rectangular_multi_submesh.md @@ -5,7 +5,39 @@ Target: PyAutoArray Difficulty: too-large Autonomy: supervised Priority: normal -Status: formalised +Status: STALE PREMISE — needs re-basing before it can be issued (2026-08-09) + +## 2026-08-09 — the implementation base named below no longer exists + +Found by the `draft/` sweep, verified against PyAutoArray main (`efaf3041`). + +Path B is written as "subclass or compose alongside" `RectangularRotatedAdaptImage` +(Path A's mesh class), building one `RectangularSplineAdaptImage` per detected +mode. **Neither class is on main.** The mesh package is now exactly: + +``` +autoarray/inversion/mesh/mesh/{delaunay,knn,rectangular_adapt_density, + rectangular_adapt_image,rectangular_uniform}.py +``` + +They were removed by the rectangular-mesh consolidation (#402/#403), which also +moved the kernel-CDF machinery into `mesh_geometry/rectangular.py` and +`interpolator/rectangular.py`. So the class names, the subclassing plan, and the +"each sub-mesh runs the existing single-mode CDF code unchanged" claim all need +re-deriving against the consolidated classes before this can be issued. + +**The research context survives intact** — all four artefacts this prompt tells +you to read first are still present in the repo: `files/cdf_audit.md`, +`files/ghost_peak_findings.md`, `files/ghost_peak.png`, `files/pca_rotation.png` +(plus `ghost_peak_experiment.py` / `pca_rotation_experiment.py`). The separability +problem and the A/B/C fork are unaffected; only the code the plan attaches to moved. + +Before issuing: re-read the consolidated mesh classes, decide which one now plays +Path A's role (the PCA-rotation behaviour may live as a parameter rather than a +class), and rewrite § "The approach in more detail" step 2 against it. Everything +below this line predates the consolidation. + +--- Follow-up to `rectangular_adapt_cdf.md` (issue #322) and Path A (`RectangularRotatedAdaptImage`). The PCA-rotation hack we shipped fully diff --git a/draft/feature/autoarray/regularization_jax_gradient_gaps.md b/draft/feature/autoarray/regularization_jax_gradient_gaps.md index 6a609dad..754dcfa7 100644 --- a/draft/feature/autoarray/regularization_jax_gradient_gaps.md +++ b/draft/feature/autoarray/regularization_jax_gradient_gaps.md @@ -56,7 +56,21 @@ Reformulation candidates (in the spirit of the opt-in slogdet, PyAutoArray#391): Gate any change on the `regularization.py` jax_grad script re-passing and on FoM parity on the numpy path. -## 3. Split-family shape guard on rectangular meshes (papercut) +## 3. Split-family shape guard on rectangular meshes (papercut) — DONE + +*Shipped 2026-07-28 as phase 1 of the @rhayes777 audit epic — +PyAutoArray#417 (`9411904d`), merged. The first of the two options below is what +landed: `Pixelization.__init__` (`autoarray/inversion/pixelization.py:154`) +raises `PixelizationException` on any split regularization × rectangular mesh, +driven by `AbstractMesh.supports_split_regularization` × +`AbstractRegularization.is_split_regularization`, with all 9 combinations +covered by `test_autoarray/inversion/pixelization/test_split_regularization_support.py`. +True pixel-centre crosses for the rectangular geometry were deliberately NOT +implemented — the capability is absent by design. The +`rectangular_adapt_constant_split_guard.md` merge question this section raises is +therefore moot; that prompt was recorded as complete on 2026-08-09 +([[rectangular-adapt-constant-split-guard]]). Verified by the draft/ sweep against +main `efaf3041`. **Leg 2 is now the only open work in this prompt.*** `ConstantSplit`/`AdaptSplit`/`AdaptSplitZeroth` on a rectangular mesh fail with a raw broadcasting `TypeError` ((784,784) vs (3808,3808)): the split diff --git a/draft/feature/autofit/ep_analytic_updates.md b/draft/feature/autofit/ep_analytic_updates.md index 6f8f11f9..f610866d 100644 --- a/draft/feature/autofit/ep_analytic_updates.md +++ b/draft/feature/autofit/ep_analytic_updates.md @@ -18,6 +18,9 @@ do not bulk-issue): 1. **WP1** — exact PriorFactor updates in the declarative path + document the analytic-projection contract (~2–3 days). Start here. Note: rebase over / land after PyAutoFit#1334 (owns README.md). + **UNBLOCKED 2026-08-09 (draft/ sweep): PyAutoFit#1334 is MERGED** — it shipped + `autofit/graphical/README.md` as Phase 2 of the EP framework review on + 2026-07-08 ([[ep-framework-review]]). Rebase, don't wait. 2. **WP2** — first-class linear-Gaussian factor; validate against an IC50-shaped integration script (~1 week). 3. **WP3** — Gamma–Poisson / Beta–Bernoulli conjugate factors. diff --git a/draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md b/draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md index 1ca6cec4..c35f8364 100644 --- a/draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md +++ b/draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md @@ -60,3 +60,11 @@ the re-baselining survey. After the k×s series completes (phase 3 workspace tests + refactor exercise). The k×s machinery it depends on is merged (PyAutoArray#363, PyAutoGalaxy#486, autolens_workspace#236). + +**2026-08-09 (draft/ sweep): the series is CLOSED, not merely merged.** +PyAutoArray#362 closed as "series complete" — all five phases plus the extra +cache/refactor legs shipped, recorded across `complete/2026/07/kxs-*.md` and +[[oversampling-kxs-coupling]]. So the "after the series completes" condition is +met and this is ready to start. It remains the correct live residue of that +series' § 5: [[kxs-core]] records the phase-4 fork resolved as option (c) — +executed simulators stay `s=1` — with option (a) split out as this prompt. diff --git a/draft/feature/jax_substructure/1_vmap_subhalo_deflections.md b/draft/feature/jax_substructure/1_vmap_subhalo_deflections.md deleted file mode 100644 index 4b3ce401..00000000 --- a/draft/feature/jax_substructure/1_vmap_subhalo_deflections.md +++ /dev/null @@ -1,124 +0,0 @@ -# Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure - -Type: feature -Target: jax_substructure -Difficulty: too-large -Autonomy: supervised -Priority: normal -Status: formalised - -Context: PyAutoLens issue #542 asks for a JIT/vmap-able multi-plane substructure -forward simulator. This is prompt 1 of 4 — building the vectorized per-plane -deflection computation that everything else stacks on top of. - -## Background - -Today, when a Tracer has N subhalos on a single plane, their deflections are -summed via a Python generator loop: - -```python -# tracer_util.py line 262 -deflections_yx_2d = sum( - (g.deflections_yx_2d_from(grid=scaled_grid, xp=xp) for g in galaxies) -) -``` - -Under `jax.jit`, JAX unrolls this into N separate traced operations. For 5 -galaxies that's fine. For 1000 halos it produces a massive XLA graph (slow -compilation) and recompiles whenever N changes between realizations. - -The fix is a **vmapped deflection function** that takes stacked parameter arrays -and computes all N deflections in a single GPU launch, then sums them. - -All four dark matter profiles already accept `xp=jnp` and produce correct -JAX-traced outputs — the individual deflection math is ready. What's missing -is the batching orchestration. - -## What to build - -A pure-function module (suggest `autolens/lens/substructure_util.py` or similar) -containing functions like: - -```python -def deflections_nfw_truncated_sph_from( - grid, # (M, 2) image-plane grid - params, # (N, 4) — mass_at_200, concentration, centre_y, centre_x - mask, # (N,) boolean — which slots are active halos - cosmology, # for MCR variants that need kappa_s / scale_radius - redshift, # halo redshift (scalar, shared across the batch) - xp=jnp, -): - """Compute summed deflections from N NFWTruncatedSph halos via vmap.""" - ... -``` - -The inner single-halo function should call the existing deflection math from -the profile classes. Look at how `NFWTruncatedSph.deflections_yx_2d_from` -works in `autogalaxy/profiles/mass/dark/nfw_truncated.py` — it calls through -the `@aa.decorators.transform` and `@aa.decorators.to_vector_yx` decorator -chain. For the vmapped path you'll want to call the underlying math directly -(pre-transform the grid by subtracting `centre`, call the radial deflection -functions, post-transform back) to avoid the decorator overhead that wraps -results in autoarray objects. - -The key profiles to cover: - -- `NFWTruncatedSph` — `autogalaxy/profiles/mass/dark/nfw_truncated.py` -- `cNFWSph` — `autogalaxy/profiles/mass/dark/cnfw.py` -- Their MCR Ludlow variants (`nfw_truncated_mcr.py`, `cnfw_mcr.py`) which - derive `kappa_s` and `scale_radius` from `mass_at_200` via - `autogalaxy/profiles/mass/dark/mcr_util.py` - -For the MCR variants, the Ludlow concentration-mass relation -(`mcr_util.kappa_s_and_scale_radius_for_ludlow` and -`mcr_util.kappa_s_scale_radius_and_core_radius_for_ludlow`) is already -JAX-native — it auto-detects JAX arrays and uses `jnp` internally. So you -can vmap through the full MCR → deflection chain. - -The `mask` parameter handles the padding: pad `params` to `max_N` rows, set -`mask=False` for unused slots, and zero out their deflection contribution -before summing. This way the array shape is fixed regardless of the actual -number of halos, so `jax.jit` compiles once. - -## Integration test - -This is the key validation: build a Tracer the normal way with ~10 subhalos -(using the existing Galaxy/profile API), compute deflections via the -Python-loop path, then compute the same deflections via the new vmapped -path, and assert they match to numerical tolerance. - -Put this in `autolens_workspace_test/scripts/jax_substructure/` (new directory). -Something like: - -```python -# 1. Build 10 NFWTruncatedSph halos as Galaxy objects -halos = [ag.Galaxy(redshift=0.5, mass=ag.mp.NFWTruncatedSph(...)) for _ in range(10)] -tracer = al.Tracer(galaxies=[macro_galaxy, *halos, source_galaxy]) - -# 2. Get deflections via existing path -deflections_old = tracer_util.traced_grid_2d_list_from(..., xp=jnp) - -# 3. Stack same parameters into arrays -params = jnp.array([[mass_i, conc_i, cy_i, cx_i] for ...]) -mask = jnp.ones(10, dtype=bool) - -# 4. Get deflections via new vmapped path -deflections_new = deflections_nfw_truncated_sph_from(grid, params, mask, ...) - -# 5. Assert match -assert jnp.allclose(deflections_old, deflections_new, atol=1e-8) -``` - -Do this for all four profile types. Also test that masked-out slots contribute -zero deflection. - -## Scope boundaries - -- This prompt covers **single-plane** vectorized deflections only. Multi-plane - scan is prompt 2. -- Don't modify the existing Tracer or Galaxy classes. This is a parallel path. -- The macro lens (PowerLaw + ExternalShear) doesn't need vmapping here — there's - only one macro lens per realization. It will be called directly in prompt 2. -- Light profiles (source image) are also not in scope here — just mass deflections. - - diff --git a/draft/feature/jax_substructure/2_tracer_lax_scan.md b/draft/feature/jax_substructure/2_tracer_lax_scan.md deleted file mode 100644 index f3fbba6d..00000000 --- a/draft/feature/jax_substructure/2_tracer_lax_scan.md +++ /dev/null @@ -1,172 +0,0 @@ -# Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1 - -Type: feature -Target: jax_substructure -Difficulty: too-large -Autonomy: supervised -Priority: normal -Status: formalised - -Context: PyAutoLens issue #542, prompt 2 of 4. Prompt 1 built vmapped per-plane -deflection functions. This prompt wires them into a `jax.lax.scan` over redshift -planes to replace the Python loops in multi-plane ray-tracing. - -## Background - -The current multi-plane ray-tracing lives in -`autolens/lens/tracer_util.py : traced_grid_2d_list_from` (lines 174-268). -It has three nested Python loops: - -1. **Outer loop** (line 232): `for plane_index, galaxies in enumerate(planes):` -2. **Scaling loop** (line 238): `for previous_plane_index in range(plane_index):` - — applies cosmological scaling factors from all previous planes -3. **Galaxy sum** (line 262): `sum(g.deflections_yx_2d_from(...) for g in galaxies)` - — sums deflections from all galaxies on the current plane - -For the substructure use case (~8 planes, ~1000 total halos), these Python loops -unroll into a huge XLA graph and recompile whenever the galaxy count changes. - -## What to build - -A standalone pure-function that does the same multi-plane ray-tracing but using -`jax.lax.scan` over planes and the vmapped deflection functions from prompt 1. -Suggest placing this in the same module as prompt 1 -(`autolens/lens/substructure_util.py`). - -### Input representation - -The key design decision is how to represent the per-plane halo populations as -fixed-shape arrays. The natural structure is: - -```python -# Per-plane halo parameters, padded to max_halos_per_plane -halo_params: jnp.array # shape (n_planes, max_halos_per_plane, n_halo_params) -halo_mask: jnp.array # shape (n_planes, max_halos_per_plane) — bool -plane_redshifts: jnp.array # shape (n_planes,) -``` - -The macro lens (PowerLaw + ExternalShear) should be handled separately from the -halo stacks — it's a single galaxy evaluated directly, not vmapped. The source -light profile is also separate (evaluated on the final traced grid). - -### Precomputed scaling-factor matrix - -The cosmological scaling factors between all plane pairs can be precomputed -**outside jit** as a `(n_planes, n_planes)` matrix: - -```python -# scaling_matrix[i, j] = scaling_factor from plane j to plane i (0 if j >= i) -scaling_matrix = precompute_scaling_matrix(plane_redshifts, cosmology) -``` - -The cosmology module at `autogalaxy/cosmology/model.py` already has -`scaling_factor_between_redshifts_from(redshift_0, redshift_1, redshift_final, xp)` -which is xp-threaded. Call it for each `(j, i)` pair where `j < i`. - -This matrix is a static input to the jitted function — it only depends on -redshifts, which are fixed for a given realization. - -### The scan function - -```python -def traced_grids_via_scan( - grid, # (M, 2) image-plane grid - macro_params, # dict or array of PowerLaw + ExternalShear params - halo_params, # (n_planes, max_N, n_halo_params) - halo_mask, # (n_planes, max_N) - scaling_matrix, # (n_planes, n_planes) - source_params, # Sersic params for the source - ... -): - def scan_step(carry, plane_inputs): - # carry: (current_grid, all_prev_deflections as (n_planes, M, 2) buffer) - # plane_inputs: (this_plane_halo_params, this_plane_mask, scaling_row) - - grid, deflection_buffer, plane_idx = carry - plane_halo_params, plane_mask, scaling_row = plane_inputs - - # 1. Apply scaled deflections from all previous planes - # scaling_row is (n_planes,) — entries for j >= plane_idx are 0 - scaled_deflections = jnp.einsum('p,pmd->md', scaling_row, deflection_buffer) - current_grid = grid - scaled_deflections - - # 2. Compute macro deflections (if this is the lens plane) - # ... call PowerLaw + ExternalShear deflection directly ... - - # 3. Compute halo deflections via vmapped function from prompt 1 - halo_deflections = deflections_nfw_truncated_sph_from( - current_grid, plane_halo_params, plane_mask, ... - ) - - # 4. Store total plane deflections in buffer - total_deflections = macro_deflections + halo_deflections - deflection_buffer = deflection_buffer.at[plane_idx].set(total_deflections) - - return (grid, deflection_buffer, plane_idx + 1), current_grid - - init_carry = (grid, jnp.zeros((n_planes, M, 2)), 0) - _, traced_grids = jax.lax.scan(scan_step, init_carry, plane_stack) - return traced_grids -``` - -The exact API will need refinement — the sketch above shows the idea. The macro -lens only contributes on one plane (the main lens plane), so use `jax.lax.cond` -or `jnp.where` to conditionally add its deflections based on `plane_idx`. - -### Where the macro lens fits - -The macro galaxy (PowerLaw + ExternalShear) is evaluated directly — not vmapped, -since there's only one. Its deflection function is already JAX-traceable -(`autogalaxy/profiles/mass/total/power_law.py` uses a `jax.lax.scan` series -expansion). Call it on the lens-plane grid and add it to that plane's deflection -buffer alongside the halo contribution. - -### Where the source fits - -After the scan produces `traced_grids` for all planes, evaluate the source light -profile (e.g. `SersicCore`) on the final plane's traced grid to produce the -lensed image. `SersicCore.image_2d_via_radii_from` already accepts `xp` — call -it directly on the source-plane grid. - -## Integration test - -Extend the test from prompt 1. Build a Tracer with: -- 1 PowerLaw + ExternalShear macro at z=0.5 -- 10 NFWTruncatedSph subhalos at z=0.5 (lens plane) -- 5 NFWTruncatedSph LOS halos at z=0.25 (foreground plane) -- 5 NFWTruncatedSph LOS halos at z=0.75 (background plane) -- 1 Sersic source at z=1.0 - -Compute the final source-plane grid via both paths: -1. `tracer_util.traced_grid_2d_list_from(planes, grid, cosmology, xp=jnp)` -2. `traced_grids_via_scan(grid, macro_params, halo_params, ...)` - -Assert the source-plane grids match to numerical tolerance. This validates that -the scan + vmap path reproduces the existing Python-loop path. - -Also test that the scan path compiles once and reuses the compiled code when -only parameter values change (same shapes, different halo masses/positions). - -Put tests in `autolens_workspace_test/scripts/jax_substructure/`. - -## Scope boundaries - -- This covers multi-plane ray-tracing and source-plane grid computation. -- PSF convolution and noise are prompt 3. -- The LOSSampler output stays as-is — it runs outside jit and produces the - parameter arrays that feed into this function. The conversion from - `LOSSampler.galaxies_from()` output to `(halo_params, halo_mask)` arrays - is a small helper, not a refactor of LOSSampler itself. -- Don't modify the existing Tracer class or tracer_util. This is a parallel path. - -## Existing patterns to follow - -- `jax.lax.scan` is already used in `autogalaxy/profiles/mass/total/jax_utils.py` - (omega series expansion) and `autoarray/operators/transformer.py` (chunked - NUFFT). Look at those for the carry/accumulator pattern. -- `jax.lax.fori_loop` is used in `autoarray/inversion/mesh/interpolator/knn.py`. -- Pytree registration: `autoarray/abstract_ndarray.py` has `register_instance_pytree`. - The new function takes raw arrays, so pytree registration isn't needed for - the function itself — just ensure inputs are plain `jnp.arrays`. - - diff --git a/draft/feature/jax_substructure/3_simulator_jax_e2e.md b/draft/feature/jax_substructure/3_simulator_jax_e2e.md deleted file mode 100644 index 46a40596..00000000 --- a/draft/feature/jax_substructure/3_simulator_jax_e2e.md +++ /dev/null @@ -1,186 +0,0 @@ -# Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2 - -Type: feature -Target: jax_substructure -Difficulty: too-large -Autonomy: supervised -Priority: normal -Status: formalised - -Context: PyAutoLens issue #542, prompt 3 of 4. Prompts 1-2 built the vectorized -deflection and scan-based ray-tracing. This prompt wires them through PSF -convolution and Poisson noise to produce the end-to-end `jax.jit(simulate)` -function. - -## Background - -The existing simulator call chain is: - -``` -SimulatorImaging.via_tracer_from(tracer, grid) - -> tracer.padded_image_2d_from(grid, psf_shape_2d) - -> image_2d_from (sum light profiles on traced grids) - -> SimulatorImaging.via_image_from(image) - -> PSF convolution (FFT or real-space, both JAX-ready) - -> add background sky - -> Poisson noise via jax.random.poisson (when xp=jnp) - -> return Imaging dataset -``` - -The downstream half (PSF convolution onward) is already JAX-friendly. The -upstream half (image from traced grids) is now handled by the scan path from -prompt 2. This prompt connects them and fixes the remaining gaps. - -## Gap 1: PRNGKey support for Poisson noise - -`autoarray/dataset/preprocess.py : poisson_noise_via_data_eps_from` (line 455) -currently takes an integer `seed` parameter. On the JAX path (line 488) it -converts this to a PRNGKey: - -```python -effective_seed = seed if seed != -1 else int(time.time() * 1e6) & 0xFFFFFFFF -key = jax.random.PRNGKey(effective_seed) -``` - -This works for single calls but blocks `vmap` over noise seeds — you can't -vmap a function that calls `int(time.time())` inside. - -Add an optional `prng_key` parameter: - -```python -def poisson_noise_via_data_eps_from( - data_eps, exposure_time_map, seed=-1, prng_key=None, xp=np -): - ... - if prng_key is not None: - key = prng_key - elif xp is not np: - effective_seed = seed if seed != -1 else int(time.time() * 1e6) & 0xFFFFFFFF - key = jax.random.PRNGKey(effective_seed) - ... -``` - -Thread this parameter through `data_eps_with_poisson_noise_added` (line 500) -and up through `SimulatorImaging.via_image_from` in -`autoarray/dataset/imaging/simulator.py`. - -## Gap 2: Over-sampler xp threading - -`Grid2D.padded_grid_from` in `autoarray/structures/grids/uniform_2d.py` -(line 1140) uses `np.pad` which is not xp-aware. Similarly the OverSampler -binning path uses numpy operations. - -For the substructure fast path, the simplest approach is to **skip the -autoarray grid/over-sampler machinery entirely** and handle padding and -sub-gridding with plain jnp operations in the standalone simulate function. -The grid is uniform and the over-sample factor is fixed, so this is -straightforward: - -```python -# Pad grid for PSF -padded_shape = image_shape + psf_shape - 1 -padded_grid = make_uniform_grid(padded_shape, pixel_scale) # pure jnp - -# Evaluate source on sub-grid if over_sample > 1 -sub_grid = make_sub_grid(padded_grid, over_sample_size) # pure jnp -sub_images = source_image_fn(sub_grid, source_params) -image = sub_images.reshape(...).mean(axis=-1) # bin down -``` - -This avoids modifying the autoarray grid classes while giving us a fully -jnp-native path. - -## The end-to-end simulate function - -Combine everything into a single jittable function: - -```python -@jax.jit -def simulate_substructure( - macro_params, # PowerLaw + ExternalShear parameters - halo_params, # (n_planes, max_N, n_halo_params) - halo_mask, # (n_planes, max_N) - source_params, # Sersic parameters - # --- static / precomputed (passed via jax.jit static_argnums or closure) --- - grid, # (M, 2) image-plane grid (padded for PSF) - psf_kernel, # (K, K) PSF array - scaling_matrix, # (n_planes, n_planes) - exposure_time, # scalar - background_sky, # scalar - prng_key, # jax.random.PRNGKey for Poisson noise -): - # 1. Multi-plane ray-trace (from prompt 2) - traced_grids = traced_grids_via_scan( - grid, macro_params, halo_params, halo_mask, scaling_matrix - ) - - # 2. Evaluate source light on final traced grid - source_grid = traced_grids[-1] - image = sersic_image_from(source_grid, source_params) - - # 3. PSF convolution (FFT) - image = jax.scipy.signal.fftconvolve(image, psf_kernel, mode='same') - - # 4. Add background sky - image = image + background_sky - - # 5. Poisson noise - image_counts = image * exposure_time - noisy_counts = jax.random.poisson(prng_key, image_counts) - noisy_image = noisy_counts / exposure_time - - # 6. Subtract sky - noisy_image = noisy_image - background_sky - - return noisy_image -``` - -The PSF convolution can use `jax.scipy.signal.fftconvolve` directly — the -existing Convolver FFT path in `autoarray/operators/convolver.py` already -does essentially this with `jnp.fft.rfft2 / irfft2`, so either approach works. -For the standalone function, the scipy one-liner is simpler. - -## Integration test / smoke test - -Build a representative substructure configuration and verify the end-to-end -simulate function against the existing OO path: - -```python -# Build via existing API -tracer = al.Tracer(galaxies=[macro, *subhalos_10, source]) -simulator = al.SimulatorImaging( - exposure_time=300.0, background_sky_level=1.0, - psf=al.Kernel2D.from_gaussian(shape_native=(11, 11), sigma=0.1, ...), - noise_seed=42, -) -imaging_old = simulator.via_tracer_from(tracer=tracer, grid=grid) - -# Build via new pure-function path (same parameters, same seed) -key = jax.random.PRNGKey(42) -image_new = simulate_substructure( - macro_params, halo_params, halo_mask, source_params, - grid, psf_kernel, scaling_matrix, 300.0, 1.0, key, -) - -# Compare (tolerance for Poisson noise RNG differences — compare -# the deterministic part first, then the noisy part with the same seed) -assert jnp.allclose(image_new, imaging_old.data, atol=1e-6) -``` - -Also verify that `jax.jit(simulate_substructure)` compiles successfully -and that calling it a second time with different parameter values (same -shapes) reuses the compiled code (no recompilation). - -Put tests in `autolens_workspace_test/scripts/jax_substructure/`. - -## Scope boundaries - -- This prompt produces a working `jit(simulate)` for a single realization. -- `vmap` over a batch of parameter vectors is prompt 4. -- The LOSSampler conversion helper (Galaxy list -> padded arrays) should be - a small utility, not a refactor. If it's simple enough, include it here; - otherwise defer to prompt 4. -- Don't modify the existing SimulatorImaging class beyond adding the - `prng_key` parameter to the noise functions in preprocess.py. - - diff --git a/draft/feature/jax_substructure/4_vmap_batched_simulation.md b/draft/feature/jax_substructure/4_vmap_batched_simulation.md deleted file mode 100644 index cd31b7cd..00000000 --- a/draft/feature/jax_substructure/4_vmap_batched_simulation.md +++ /dev/null @@ -1,148 +0,0 @@ -# Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal). - -Type: feature -Target: jax_substructure -Difficulty: too-large -Autonomy: supervised -Priority: normal -Status: formalised - -Context: PyAutoLens issue #542, prompt 4 of 4 (stretch goal). Prompts 1-3 built -`jax.jit(simulate_substructure)` for a single realization. This prompt extends -it to `vmap(jit(simulate))(thetas, keys)` for batched evaluation — ~1024 lensed -images per GPU launch. - -## Background - -The issue author's use case evaluates `theta -> noisy image` of order 10^6 -times. After prompt 3, each call is a single jitted GPU kernel. The next -speedup is batching: evaluate many theta vectors in one launch, saturating -GPU parallelism. - -## What to build - -### Batched simulate function - -```python -batched_simulate = jax.vmap(simulate_substructure, in_axes=( - 0, # macro_params: (batch, n_macro_params) — varies per realization - 0, # halo_params: (batch, n_planes, max_N, n_halo_params) — varies - 0, # halo_mask: (batch, n_planes, max_N) — varies (different N per draw) - 0, # source_params: (batch, n_source_params) — varies - None, # grid: shared across batch - None, # psf_kernel: shared - None, # scaling_matrix: shared (same redshift structure) - None, # exposure_time: shared - None, # background_sky: shared - 0, # prng_key: (batch,) — different key per realization -)) -``` - -Call with: - -```python -keys = jax.random.split(master_key, batch_size) -images = jax.jit(batched_simulate)( - macro_params_batch, # (1024, n_macro) - halo_params_batch, # (1024, n_planes, max_N, n_halo) - halo_mask_batch, # (1024, n_planes, max_N) - source_params_batch, # (1024, n_source) - grid, psf_kernel, scaling_matrix, exposure_time, background_sky, - keys, # (1024,) -) -# images shape: (1024, H, W) -``` - -### LOSSampler → padded array conversion - -The LOSSampler at `autolens/lens/los.py` produces a `List[ag.Galaxy]` per -realization. For the batched path, we need a helper that converts many -realizations into padded arrays: - -```python -def los_realizations_to_arrays( - realizations: List[List[ag.Galaxy]], - max_halos_per_plane: int, - n_planes: int, - plane_redshifts: np.ndarray, -): - """Convert a batch of LOSSampler outputs to padded arrays. - - Returns: - halo_params: (batch, n_planes, max_halos_per_plane, n_params) - halo_mask: (batch, n_planes, max_halos_per_plane) - """ - ... -``` - -This runs in numpy (outside jit) and produces the fixed-shape arrays that -feed into the vmapped function. The LOSSampler itself doesn't need to change. - -### Memory considerations - -1024 images of size 100x100 at float32 = 1024 * 100 * 100 * 4 bytes = ~40 MB. -Fine for any GPU. But the intermediate arrays (per-halo deflections across all -batch elements) can be larger: 1024 * max_N * M * 2 * 4 bytes. For max_N=200 -and M=10000 grid points, that's ~16 GB — may exceed GPU memory. - -Mitigation strategies: -- Process in sub-batches (e.g. 128 at a time) and concatenate results -- Reduce max_N by using separate halo types per plane (most planes have - few halos; only the lens plane has many subhalos) -- Use `jax.checkpoint` to trade compute for memory on the scan steps - -Include a utility that estimates peak memory for a given configuration and -suggests a batch size. - -### What varies vs what's shared across the batch - -For the issue author's use case (fixed lens macro, varying substructure): - -| Input | Varies? | Notes | -|-------|---------|-------| -| macro_params | Maybe | Could be fixed or sampled | -| halo_params | Yes | Different SHMF draw per realization | -| halo_mask | Yes | Different N per draw | -| source_params | Maybe | Could be fixed or sampled | -| grid | No | Same image grid | -| psf_kernel | No | Same instrument | -| scaling_matrix | No | Same redshift planes (if plane structure is fixed) | -| prng_key | Yes | Different noise per realization | - -If the plane redshift structure also varies between realizations (different -LOS plane redshifts per draw), then `scaling_matrix` would need to be batched -too. But the issue author mentions 8 fixed planes, so it's likely shared. - -## Integration test - -Verify batch consistency: - -```python -# Single-image results -images_single = [simulate_substructure(p, h, m, s, ..., k) - for p, h, m, s, k in zip(params...)] - -# Batched results -images_batch = batched_simulate(params_stacked..., keys) - -# Must match -for i in range(batch_size): - assert jnp.allclose(images_single[i], images_batch[i], atol=1e-6) -``` - -Also benchmark: measure wall-clock time for 1024 sequential calls vs one -batched call. The batched version should be significantly faster (the whole -point). - -Put tests in `autolens_workspace_test/scripts/jax_substructure/`. - -## Scope boundaries - -- This is the final prompt in the series. After this, the user has a complete - `vmap(jit(simulate))(thetas, keys)` path. -- If memory is a hard constraint, the sub-batching utility is sufficient — - don't try to implement gradient checkpointing in this prompt. -- The LOSSampler conversion helper is simple numpy reshaping, not a refactor - of the sampler itself. - - diff --git a/draft/feature/pyautomind/complete_archive_wiki.md b/draft/feature/pyautomind/complete_archive_wiki.md index 21330a58..611a874b 100644 --- a/draft/feature/pyautomind/complete_archive_wiki.md +++ b/draft/feature/pyautomind/complete_archive_wiki.md @@ -16,6 +16,37 @@ Status: formalised `complete/YYYY/MM/.md` per-task rich records to exist first. **Do not issue this until Phase 1 nears shipping** (`feedback_no_bulk_issue_queues`). +## 2026-08-09 — SUBSTANTIALLY SHIPPED; only the curation is left + +Checked by the draft/ sweep. Phase 1 shipped +([[lifecycle-state-split]], monolithic `complete.md` retired 2026-07-16, issue #81), +so the dependency above is long satisfied — **and most of what this prompt asks for +shipped with it.** + +`complete/index.md` exists on `main` and already is the token-light index this +prompt specifies: + +- **952 records** linked, grouped by dated bucket. +- Its own header states the lookup protocol this prompt describes almost verbatim + — *"read this, follow one or two links, and only then grep a dated bucket."* +- **Generated**, by `scripts/lifecycle.py index`, with `index --check` gating + staleness in CI — so it cannot rot. +- It has the curated band: `` … ``, + documented as surviving regeneration. +- `complete/AGENTS.md` § "How to look something up (token-light — RAG is dead)" + carries the same doctrine this prompt opens with. + +**What is actually left is the curation, not the machinery.** The Highlights band +is empty — it reads `_(curate hard-won records here — survives regeneration.)_`. +So the remaining work is the editorial pass: pick the hard-won records worth +surfacing and write the one-line hooks, in the `autolens_assistant/wiki` style +this prompt says to study. + +Re-scope before issuing: drop the "build the index" legs, keep the "curate it" +legs, and re-read § "Model to emulate" against what `lifecycle.py index` already +generates rather than against a blank slate. `Difficulty:` medium is now +generous. + ## Problem Once `complete/` holds hundreds of per-task records, an agent still can't look diff --git a/draft/feature/pyautomind/draft_staleness_detection_signals.md b/draft/feature/pyautomind/draft_staleness_detection_signals.md new file mode 100644 index 00000000..7475f12c --- /dev/null +++ b/draft/feature/pyautomind/draft_staleness_detection_signals.md @@ -0,0 +1,159 @@ +# Make draft/ staleness detectable — `intake reconcile` measured, and the three signals that actually worked + +Type: feature +Target: PyAutoMind +Repos: +- PyAutoMind +- PyAutoBrain +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +## Why + +`draft/` is ~145 prompts graded by **no check at all**. The 2026-08-09 sweep read +acceptance criteria against upstream `main` for two target clusters — 18 +PyAutoArray prompts, then 22 PyAutoFit-domain prompts — and found roughly a third +carrying stale state: + +| outcome | count (of 40) | +|---|---| +| shipped, recorded to `complete/` | 4 | +| half-shipped (scope narrowed in place) | 2 | +| unblocked by a since-closed prerequisite | 4 | +| unstartable (premise removed upstream) | 1 | +| withdrawn, archived | 1 | +| **mis-gradeable** (an adjacent upstream fix reads like the prompt's) | 1 | + +At that rate the remaining ~105 prompts hold real drift. Doing it by hand is +expensive; the question is what can be mechanised. + +## The measurement — `intake reconcile` as it stands + +A reconcile pass already exists (`pyauto-brain intake reconcile` — "rank backlog +prompts that look already-shipped … always read-only"). It was run against the +pre-sweep tree (PyAutoMind `f25e154e`) so its output could be scored against +findings that were later confirmed by reading upstream source. Result: + +- **96 suspects of 148 scanned** — a 65% flag rate (52 `high`, 20 `medium`, 24 `low`). +- Of the 5 confirmed findings, it flagged **2**: the test-mode umbrella (`high`) + and the latent-samples bug (`low`, i.e. buried). +- It **missed the three largest** — `oversampling_kxs_coupling` (a whole shipped + 5-phase series), `rectangular_adapt_constant_split_guard` (shipped as + PyAutoArray#417), `nufft_simulator_chunking` (shipped as PyAutoArray#330). + +So ~40% recall, and the one true positive at `high` is indistinguishable from 51 +other `high`s. **This is a precision problem, not a missing-tool problem.** The +existing matchers — cross-file references and shared topic words — fire on +prompts that merely *mention* each other, which is most of them. Do not rewrite +reconcile from scratch; make its ranking discriminative. + +Not a criticism of the tool's existence: it is read-only by design and retiring a +prompt is meant to stay human. The goal here is a signal a human can act on. + +## The three signals that actually found things + +Each is what surfaced a specific confirmed finding, so each is grounded rather +than speculative. + +### 1. Machine-readable "epic closes when X" gates (cheapest; highest precision) + +`test_mode_representative_outputs_size_realistic.md` stated its own exit +condition in prose — *"EPIC CLOSES when #70 ships its recipe leg"* — and +autolens_profiling#70 closed as `completed` on 2026-07-17, **the same day that +status line was written**. One issue-state lookup settles the prompt; no clone, +no code read. + +`lifecycle.py issues --drafts` does not net this, because it treats a cited issue +as *context* and this one is a *gate*. Proposal: a header key, e.g. + +``` +Closes-when: https://github.com/PyAutoLabs/autolens_profiling/issues/70 +Blocked-by: https://github.com/PyAutoLabs/PyAutoFit/issues/1331 +``` + +`Closes-when` closed → the prompt is **done**; `Blocked-by` closed → the prompt is +**newly unblocked**. Both are actionable and the two readings are opposite, which +is exactly the ambiguity that makes today's `--drafts` advisory-only. Backfill the +key on the prompts that already say it in prose (this sweep annotated four +unblocked-by-a-closed-gate cases by hand: `bug/priors/12`, `13`, `14`, and +`ep_analytic_updates` WP1). + +### 2. Prompt names an identifier that now exists upstream + +`oversampling_kxs_coupling.md` § Scope named `_validate_convolve_over_sample_size` +and a partial pre-bin util. Both are on PyAutoArray `main`, and the validator's +**docstring uses the prompt's own phrase** ("the k x s coupling"). Likewise +`nufft_simulator_chunking.md` asked for a `chunk_size` kwarg using `jax.lax.scan`; +`TransformerNUFFT.__init__` has exactly that, under the prompt's suggested name. + +Mechanisable: extract backticked `snake_case` / `CamelCase` identifiers from a +prompt's scope/acceptance sections, grep the target repo's `main` (an anonymous +treeless clone is seconds — `GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1 +--filter=blob:none`), and rank on **hits for identifiers the prompt says do not +exist yet**. Far more discriminative than shared topic words, and it is what a +human grader is doing anyway. + +### 3. A completion record already names the prompt's deliverable + +`complete/2026/07/interferometer-jax-jit.md` says outright: "`chunk_size` is a +`TransformerNUFFT.__init__` argument that `SimulatorInterferometer` NEVER sets". +That sentence resolves `nufft_simulator_chunking.md` — library shipped, wiring +not. Similarly the split-guard prompt's twin, `rhayes_audit_validation_and_crashes.md`, +carried a full "Phase 1 completion record" for the same surface. + +So: search record **bodies** for the prompt's identifiers, not record **slugs** for +the prompt's slug. Slug similarity was measured on this sweep and is useless here +— `oversampling_kxs_coupling` against `kxs-core` scores a Jaccard of **0.25**, +under any workable threshold, and the whole scan missed that finding. + +## Hard limit — worth stating so nobody over-promises + +**One of the five findings had no signal in PyAutoMind at all.** The +`latent_samples_none_on_resumed_fit` bug was fixed by PyAutoFit#1418 the same day +it was filed, and **no completion record for #1418 exists anywhere in `complete/`**. +No amount of ledger cross-referencing surfaces that. Any design must accept that +the upstream read is load-bearing and the Mind-only passes are a cheap pre-filter, +not a substitute. + +Related, and arguably the deeper fix: work that ships without a Mind record is the +root cause here. Worth asking separately whether the ship skills can fail louder. + +## The trap any such tool must not create + +`test_mode_bypass_ordered_assertion_ties.md` reads as shipped and is not. Main now +catches `exc.FitException` in the TEST_MODE bypass — which looks exactly like the +prompt's requested fix — but the catch wraps only the likelihood call, while +`model.instance_from_vector` (where `check_assertions` actually raises on an +ordering tie) sits on the line *before* the `try`. An identifier-presence matcher +would score this "shipped" with high confidence and be wrong. + +**So the tool must rank for human review and never retire a prompt itself** — which +is already reconcile's stated contract. Keep it. + +## Scope + +1. Add `Closes-when:` / `Blocked-by:` header keys (README "Prompt file format"), + backfill from prose where prompts already state a gate, and grade them in + `lifecycle.py issues --drafts` with the two readings reported separately. + This leg alone is small and worth landing first. +2. Re-rank `intake reconcile`: demote bare cross-references and shared topic + words; promote identifier-presence and record-body-names-deliverable. Target a + flag rate that a human can actually work through — under ~15% of the backlog, + against the 65% measured today. +3. Optional: a `--repo ` mode that does the treeless clone and runs leg 2 + for one target cluster, which is the shape the manual sweep took. + +## Acceptance + +- Re-running the ranker against PyAutoMind `f25e154e` (the pre-sweep tree, which + is the labelled set this prompt establishes) puts **all five** confirmed + findings in the top band, and flags materially fewer than 96 prompts. +- `test_mode_bypass_ordered_assertion_ties.md` is NOT reported as shipped. +- No prompt is moved or retired by the tool. + + diff --git a/draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md b/draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md index 19ba9e30..b1c011fc 100644 --- a/draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md +++ b/draft/maintenance/autolens_workspace/latex_docstrings_invalid_escape_warnings.md @@ -7,7 +7,64 @@ Repos: Difficulty: small Autonomy: supervised Priority: low -Status: draft +Status: draft — the § "Scope to establish first" measurement is DONE for autolens_workspace (2026-08-09) + +## 2026-08-09 — the sweep this prompt asks for, run + +Still open; nothing has been fixed. But § "Scope to establish first" says to report +the count and affected repos **before** proposing a change, so here is that number +for autolens_workspace main (`9974f891`): + +**80 warnings across 17 files** in `scripts/` — six times the "roughly a dozen +per script across the four" this prompt was filed on, and well beyond +potential_correction. So the prompt's own instinct ("do NOT assume this is +confined to potential_correction") was right. + +By escape sequence: + +| seq | n | | seq | n | | seq | n | +|---|--:|---|---|--:|---|---|--:| +| `\d` | 15 | | `\e` | 7 | | `\l` | 4 | +| `\c` | 15 | | `\k` | 5 | | `\*` | 3 | +| `\s` | 10 | | `\,` | 5 | | `\p` | 2 | +| `\m` | 9 | | `\o` | 4 | | `\h` | 1 | + +Worst files (all `likelihood_function.py`, i.e. the maths-heavy prose): + +``` +12 scripts/imaging/features/advanced/potential_correction/likelihood_function.py +10 scripts/interferometer/features/advanced/potential_correction/likelihood_function.py + 8 scripts/interferometer/features/pixelization/likelihood_function.py + 8 scripts/imaging/features/pixelization/likelihood_function.py + 7 scripts/interferometer/likelihood_function.py + 7 scripts/imaging/likelihood_function.py +``` + +The concentration in `likelihood_function.py` files is a useful shape: these are +the scripts that carry LaTeX-heavy derivations, so option 1 (raw docstrings) would +touch mostly files whose rendered output is already equation-dense — check the +`r"""` prefix's appearance there specifically before choosing. + +**The sibling workspaces and HowTo* repos are still uncounted** — this measurement +covers autolens_workspace only. + +### TRAP — the command in § "Scope to establish first" silently reports zero + +Both snippets below use `SyntaxWarning`. These escapes are only a `SyntaxWarning` +on **Python 3.12+**; on 3.11 and earlier they are a `DeprecationWarning`, so + +```bash +python3 -W always::SyntaxWarning -m compileall -q scripts/ # → 0 hits on 3.11 +python3 -W always::DeprecationWarning -m compileall -q scripts/ # → 80 hits on 3.11 +``` + +The first form reports a clean sweep on an older interpreter and looks exactly +like "already fixed". This cost a mis-grade during the sweep before the source was +read directly and found unchanged. Use `-f` too, or `__pycache__` suppresses +recompilation and the counts silently drop. Anyone picking this up on a 3.11 +environment should run both. + +--- ## Origin diff --git a/draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md b/draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md index 8feae813..a11a0384 100644 --- a/draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md +++ b/draft/maintenance/workspaces/unpark_imaging_scaling_relation_slam.md @@ -12,6 +12,35 @@ Status: formalised BLOCKED until PyAutoArray PR#431 merges and reaches the installed stack. Do not start before then — the script only passes with that loader fix in place. +## 2026-08-09 — UNBLOCKED, both clauses satisfied + +Checked by the draft/ sweep. This gate has two conditions and **both** are met: + +1. **Merged** — PyAutoArray#431 merged `2026-08-03T18:03:14Z` (`5006f347`, "fix: + relabel at-or-below-cap data at the capped pixel scale", fixes #430). +2. **Reached the installed stack** — #431 carried the `pending-release` label, and + the 2026-08-07 release drive published all five libraries to PyPI at + **2026.8.7.1** with PyAutoArray at `828d5c13`, downstream of the merge. So the + loader fix is in a released wheel, not just on `main`. (See `active.md` + § release-drive-2026-08-07.) + +**The park is still in place**, so the work is genuinely outstanding: +`autolens_workspace/config/build/no_run.yaml:46` still carries +`imaging/features/scaling_relation/slam # NEEDS_FIX 2026-07-30 - measures its +luminosities from a preceding light stage…`. + +#431's own test plan states the outcome directly: *"`imaging/features/scaling_relation/slam` +→ now exit 0 (6 real searches). Its `NEEDS_FIX` park in +`autolens_workspace/config/build/no_run.yaml` can be removed in a separate +workspace PR."* This prompt is that PR. + +**Do not also unpark the `multi_galaxy/` sibling** at line 48. The same test plan +records that it gets past the 0.0-luminosity cause but then hits a separate latent +script bug — `slam.py:863` computing `image_half_width` from a hardcoded +`pixel_scale` while the mask uses `dataset_full.pixel_scales`. That is +`draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md`, +confirmed still unfixed on main by this sweep. It stays parked. + ## What Remove this NEEDS_FIX line from `autolens_workspace/config/build/no_run.yaml`: diff --git a/draft/refactor/pyautofit/remove_eden_packaging_tooling.md b/draft/refactor/pyautofit/remove_eden_packaging_tooling.md index b1376d35..2003f8b9 100644 --- a/draft/refactor/pyautofit/remove_eden_packaging_tooling.md +++ b/draft/refactor/pyautofit/remove_eden_packaging_tooling.md @@ -8,7 +8,28 @@ Repos: Difficulty: medium Autonomy: supervised Priority: normal -Status: formalised +Status: PARTIALLY SHIPPED — the module is gone, `eden.yaml` is not (2026-08-09) + +## 2026-08-09 — half the primary scope has already landed + +Checked by the draft/ sweep against PyAutoFit main (`3b960609`): + +- **`autofit/tools/edenise` — GONE.** `autofit/tools/` now holds only + `__init__.py`, `add_notebook_quotes.py`, `namer.py`, `util.py`. A repo-wide + grep for `edenise` returns zero Python hits, so the § Guardrails + "confirm nothing imports it" check is settled by the removal itself. +- **Root `eden.yaml` — STILL PRESENT.** The second half of the § Scope + PyAutoFit bullet is outstanding. + +Also note the § Scope follow-up names **PyAutoConf**, which has since been renamed +**PyAutoNerves** (the `autonerves` package). Re-target `scripts/edenise.py` there +before acting, and confirm it survived the rename rather than assuming the path. + +What is left is deleting one dead config file plus the PyAutoNerves driver, so +`Difficulty:` drops `medium` → `small`. The guardrail about re-scoping if a live +consumer surfaces no longer applies to the PyAutoFit leg. + +--- ## Why