diff --git a/active.md b/active.md index 5f046002..421eb944 100644 --- a/active.md +++ b/active.md @@ -1,5 +1,23 @@ # Active Tasks +## mge-lane-death +- issue: https://github.com/PyAutoLabs/autolens_profiling/issues/128 +- prompt: active/mge_lane_death.md +- status: CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed confirmation, and the two follow-ups below. +- worktree: ~/Code/PyAutoLabs-wt/mge-lane-death (not yet created) +- repos: + - autolens_profiling: research/mge-lane-death (not yet created) +- CAUSE: the deaths are in the PRIOR term, not the likelihood. The objective is `fom = -2 * (log_likelihood + sum(log_prior_list))` (`Fitness(fom_is_log_likelihood=False)`); a `UniformPrior` is `-inf` outside its box; `MultiStartGradient` steps in PHYSICAL space with no projection back onto that box. A lane crossing a hard prior edge reads as non-finite, and `resurrect=False` never redraws it, so it stays dead for every remaining step — that accumulation IS the 62%. **The likelihood never went non-finite in ~7200 lane-steps across three arms.** +- evidence: per-lane autopsy at the death vectors — 11/14 have finite likelihood at every pipeline stage and `sum(log_prior) = -inf` with 1-2 params outside a `UniformPrior`; 2/14 have NaN params (the gradient path); 1/14 unexplained. Decisive arm: neutering `log_prior_list_from_vector` -> zeros drops value-NaN 1446 -> 215 (60.25% -> 8.96%) and survivors 2 -> 13, with all 3 residual deaths being NaN-params. A narrower hypothesis (widen the shear box, which was 10 of the 11 exits) was REFUTED — deaths moved later and got marginally worse, because widening one box only moves the wall. +- reproduction: 16x150 cloud CPU gave 1446/18/0/0 and `alive 2/16` against the filed 1498/9/0/0 and the same 2/16. The survival identity is exact: `sum(150 - k_i) = 14*150 - 654 = 1446` = `n_value_nan_lane_steps`. +- COUNTER-FINDING, corrects the framing: the `ell_comps` plateau was MASKED, not cleared. The baseline's `n_constrained_lane_steps = 0` was a correctly-measured zero (the positive control was sound) but it meant "nothing got that far" — lanes died of prior-exit first. With the prior deaths removed the constrained count is 667 (27.79%). #1475's trapped-lane counter is measuring a live failure mode on this cell, hidden behind a larger one. Lanes stop being dead and start being STUCK. +- follow-ups owed (both out of this task's boundary): (1) PyAutoFit — bounded stepping (projection/clipping onto prior support) or soft-walled priors; `resurrect=True` is NOT the fix, it redraws a lane that then walks out again. (2) the ell_comps trapping at 27.79%, now that it is visible. +- caveat: one baseline death (lane 9, step 39) re-evaluates finite in every term with all params inside their boxes — the jitted/vmapped float32 path differs from the eager recompute there, unexplained. Single seed per arm, CPU, x64 off. +- reading the number: 62% is a survival integral, not a hazard rate — a frozen lane keeps counting every subsequent step, so the same death curve reports ~75% at 300 steps. Inverting it gives a mean death step of ~43 of 150 (mid-descent, not bad initial draws). Grade any re-run on the alive-versus-step CURVE, not on recovering the scalar. +- ordering (deliberate, do not revert): cause-finding FIRST on the existing ~6-min CPU run, then the `resurrect=True` budget-recovery measurement, then the production/GPU/seed confirmation. Do not queue for a GPU before the cause step has been attempted. +- boundary: investigation only. Changing the `resurrect` default is a separate PyAutoFit task — it would shift every existing multi-start benchmark. +- upstream: PyAutoFit#1475 (`004f798`) + PyAutoGalaxy#572 (`695b27c`) shipped the trapped-lane counter; record in `complete/2026/08/frozen-lane-counter.md`. + ## pix-prodigy-gpu-compat - issue: https://github.com/PyAutoLabs/autolens_workspace_developer/issues/125 - prompt: active/pixelized_prodigy_laptop_gpu_phase_1_compatibility.md diff --git a/active/mge_lane_death.md b/active/mge_lane_death.md new file mode 100644 index 00000000..4255163a --- /dev/null +++ b/active/mge_lane_death.md @@ -0,0 +1,463 @@ +# Find what kills MGE multi-start lanes — it is not the ell_comps plateau + +Type: research +Target: autolens_profiling +Repos: +- autolens_profiling +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +# Find what kills MGE multi-start lanes — it is not the ell_comps plateau + +@autolens_profiling + +The frozen-lane counter shipped (PyAutoFit #1475, PyAutoGalaxy #572) and its +first run cleared the ell_comps plateau as a suspect for the MGE cell — and +surfaced a much larger effect nobody has characterised. + +Real `imaging/mge` cell (production dataset, model and analysis via +`build_for_cell`; 16 starts x 150 steps, cloud CPU, 352s): + +| counter | lane-steps | share | +|---|---:|---:| +| `n_value_nan_lane_steps` | 1498 | **62.42%** | +| `n_grad_nan_lane_steps` | 9 | 0.38% | +| `n_constrained_lane_steps` | 0 | 0.00% | + +The population fell from `alive 16/16` to `alive 2/16`. **Roughly seven of every +eight starts die, and the best-fit is being found by two survivors.** + +## Why this is worth a task + +It contradicts a documented assumption. `AbstractMultiStartGradient`'s +`resurrect` docstring states the default `resurrect=False` is safe because "the +parametric (MGE-class) cell has only the measure-zero singularity, so the +`apply_if_finite` guard suffices". The measurement says otherwise: value-NaN, +not gradient-NaN, dominates at 62% of lane-steps, and `apply_if_finite` does not +rescue a value-NaN lane — it only zeroes the step, so the lane stays dead and +`resurrect=False` never redraws it. + +If that holds at production budget, the MGE cell is running at a small fraction +of its nominal start count, and `n_starts` is not buying what it appears to. + +`n_resurrections: 0` in the JSON closes the mechanism: the counter fired 1498 +times and nothing was ever redrawn, which is exactly `resurrect=False` behaving +as documented. Nothing is broken — the default is simply wrong for this cell. + +## Read 62% as a survival curve, not a hazard rate + +**The percentage is budget-dependent by construction, so it is not a portable +number and must not be compared across budgets as if it were.** A value-NaN lane +is frozen by `apply_if_finite` and never redrawn, so it keeps counting a +value-NaN lane-step on *every* subsequent step. The 1498 is therefore the +integral under the death curve, not a per-step hazard. + +That makes the number invertible, which is worth doing before spending GPU time. +With 14 of 16 lanes dead at the end and no resurrections, a lane dying at step +`k` contributes `150 - k`: + +``` +sum(150 - k_i) = 1498 over 14 lanes + => sum(k_i) = 14*150 - 1498 = 602 + => mean death step ≈ 43 of 150 +``` + +So the deaths are **neither at step 0 nor spread evenly** — they concentrate in +roughly the first third. That already discriminates between the two hypotheses in +"When they happen" below: it is not bad initial draws (`_broad_starts` filtered +those, and they would give mean ≈ 0), it is trajectories walking into a wall +early in the descent. + +The corollary matters for step planning: hold that same death curve fixed and run +300 steps instead of 150, and the *same physics* reports **75%**, not 62%. A +production-budget run that comes back higher than 62% is therefore not +automatically a worse result — it may be the identical landscape measured over a +longer window. **Compare the alive-versus-step curve across budgets; use the +scalar percentage only within a fixed budget.** + +## What to establish + +Ordered by information per unit time, which is **not** the order they were +originally listed in. The cause is the unanswered question and it is the cheap +one; the GPU reproduction confirms an effect that has already been measured once. +Do not queue for a GPU before step 1 has been attempted. + +1. **Where the NaNs come from** — the actual question. Which parameters/regions + produce a non-finite likelihood in the MGE cell: the mask edge, a Gaussian + `sigma` collapsing, the linear inversion / NNLS solve, the `sqrt` at r=0, + something else. **This does not need production budget or a GPU.** The + existing ~6-minute 16x150 CPU run already produces 1498 deaths; instrument it + to dump each lane's parameter vector at the step its value first goes + non-finite, and the distribution of those vectors is the answer. That is a + minutes-long iteration loop, not a queued one. Record the finding in this + repo's hazard index. +2. **Whether `resurrect=True` recovers the budget** on this cell, and what it + costs. The docstring says resurrection is for pixelized sources; this evidence + suggests the parametric cell may need it too. Measurement of the candidate + remedy — not adoption of it (see Boundary). +3. **Whether it holds at production budget, on GPU, across seeds.** The + measurement above is one seed at reduced budget on CPU, and the float64 GPU + path is not the same numerics. **Grade this on the alive-versus-step curve, + not on recovering "62%"** — per the survival-curve section above, the scalar + is budget-dependent by construction and the same landscape reports ~75% at 300 + steps. Log `alive N/16` per step so the curves can be overlaid across budgets; + a run that only reports the final percentage cannot be compared to this one. + +**When they happen** is already partly answered and should not be re-derived from +scratch: inverting the 1498 gives a mean death step of ~43 of 150, so deaths +concentrate in the first third — trajectories walking into a wall mid-descent, +not bad initial draws that `_broad_starts` failed to filter. Step 1's per-lane +death steps should confirm or overturn that inversion; if they disagree with a +mean of ~43, the disagreement is itself a finding (it would mean lanes are +recovering and re-dying, which `resurrect=False` says they cannot). + +## Boundary + +Investigation and measurement, in `@autolens_profiling`. Do not change the +`resurrect` default or any search behaviour as part of this — if the evidence +supports a change, that is a separate PyAutoFit task with its own benchmark +impact, since it would shift every existing multi-start result. + +## Provenance + +Measured 2026-08-15 with the frozen-lane counter; see +`complete/2026/08/frozen-lane-counter.md` for the full run, the positive control +that validates the zero, and the environment notes (`jaxnnls` is a required +extra; the runner writes into `dataset/`). + + + +## CAUSE FOUND (2026-08-15, cloud CPU) — it is the prior, not the likelihood + +Written up in full on autolens_profiling#128. Summary, so the file stands alone: + +`_fit` builds its objective as `Fitness(..., fom_is_log_likelihood=False, +convert_to_chi_squared=True)`: + +``` +fom = -2 * (log_likelihood + sum(log_prior_list)) +``` + +A `UniformPrior` is `-inf` outside its box, and `MultiStartGradient` steps in +**physical** parameter space with **no projection back onto that box**. A lane +crossing a hard prior edge reads as non-finite; `resurrect=False` never redraws +it; it stays dead for every remaining step. That accumulation is the 62%. + +**The likelihood never went non-finite** — not once in ~7200 lane-steps across +three arms. Every pipeline stage (positions penalty, deflections, convergence, +model data, residuals, chi-squared, NNLS reconstruction, log-determinants, +`figure_of_merit`) was finite at every death vector. + +| arm (16x150, cloud CPU) | value-NaN | grad-NaN | constrained | dead | alive end | +|---|---:|---:|---:|---:|---:| +| reproduction | 1446 (60.25%) | 18 | 0 | 14/16 | 2 | +| shear box widened to ±1 | 1422 (59.25%) | 36 | 13 | 15/16 | 1 | +| **prior term neutered** | **215 (8.96%)** | 27 | **667 (27.79%)** | **3/16** | **13** | + +Per-lane autopsy at the death vectors: 11/14 finite likelihood with +`sum(log_prior) = -inf`; 2/14 NaN params; 1/14 unexplained. All three residual +deaths in the neutered arm are NaN-params, not likelihood deaths. + +The survival identity is exact: `sum(150 - k_i) = 14*150 - 654 = 1446`, which is +`n_value_nan_lane_steps` to the unit — the counter *is* the area under the death +curve. + +**A narrower hypothesis was refuted.** 10 of the 11 box exits were the shear +`UniformPrior(-0.3, 0.3)`, but widening only those two priors did not collapse +the deaths — they moved later and got marginally worse. Widening one box only +moves the wall. Test the mechanism, not the parameter. + +### The docstring claim is correct, and irrelevant + +The parametric MGE likelihood does appear to have only measure-zero +singularities. What is wrong is using that to justify `resurrect=False`, since +the deaths come from the prior, which the claim never covered. And this is not +MGE-specific: any model with a hard-box prior and unbounded stepping has it. + +### Counter-finding: the ell_comps plateau was MASKED, not cleared + +The baseline's `n_constrained_lane_steps = 0` was correctly measured and the +positive control was sound — but it meant *"nothing got that far"*, not +*"nothing gets trapped there"*. Lanes died of prior-exit before reaching the +saturation plateau. Remove the prior deaths and the constrained count is **667 +(27.79%)**. #1475's counter is measuring a live failure mode on this cell that +was hidden behind a larger one. Lanes stop being dead and start being **stuck**. + +### Out of this task's boundary, now owed elsewhere + +1. **PyAutoFit** — bounded stepping (projection/clipping onto prior support) or + soft-walled priors. `resurrect=True` is *not* the fix: it redraws a lane that + then walks out again. +2. **The ell_comps trapping at 27.79%**, now that it is visible. + +### Still owed here + +GPU / float64 / multi-seed confirmation, graded on the alive-versus-step curve. +One caveat: baseline lane 9 (step 39) re-evaluates finite in every term with all +params inside their boxes — the jitted/vmapped float32 path differs from the +eager recompute there, unexplained. + +## Reproducer + +Both scripts below are self-contained and were used to produce the numbers in +this prompt. They live here rather than in `@autolens_profiling` because they +are throwaway measurement drivers, not part of that repo's script tiers — copy +them out to run, do not commit them there. + +### The measured result (verbatim) + +```json +{ + "cell": "imaging/mge", + "instrument": "hst", + "hardware": "cloud_cpu", + "n_starts": 16, + "n_steps": 150, + "batch_size": null, + "lane_steps": 2400, + "wall_s": 352.21100759506226, + "free_parameters": 15, + "counters": { + "n_value_nan_lane_steps": 1498, + "n_grad_nan_lane_steps": 9, + "n_constrained_lane_steps": 0, + "n_resurrections": 0 + } +}``` + +### `rerun_cell.py` — runs one production cell at reduced budget + +Invoke as `SEARCHES_DISABLE_VIZ=1 N_STARTS=16 N_STEPS=150 python rerun_cell.py mge`. +Environment: Python 3.12+, `pip install jaxnnls`, and install `autolens` with +`--no-deps` if using editable local `autofit`/`autogalaxy`. + +The `MESH_SHAPE` / `HILBERT_PIXELS` block is **inert for this cell** — it only +touches the pixelization knobs, and `mge` has no mesh. The 62% was measured on +the unmodified production MGE cell; nothing was shrunk. The block is there for +the pixelized cells, and for those the image-plane grid at `mask_radius 3.5` +dominates the cost anyway, so shrinking the mesh is the wrong lever there too. +Leave both unset. + +```python +"""Re-run one autolens_profiling search cell on CPU at reduced budget, to read +the new constrained-lane counter. + +Uses the repo's own `build_for_cell` so the dataset, model and analysis are +exactly the production ones. Only the search budget is reduced (n_starts, +n_steps, batch_size) — that changes how thoroughly the space is explored, not +the shape of the likelihood surface, which is what the counter reports on. +""" + +import os, sys, time, json +from pathlib import Path + +ROOT = Path("/workspace/pyautolabs/autolens_profiling") +sys.path.insert(0, str(ROOT / "scripts" / "misc")) +sys.path.insert(0, str(ROOT)) + +# Dataset paths in `_setup.py` are relative to the repo root. +os.chdir(ROOT) + +import numpy as np +import autofit as af + +from searches import _setup +from searches._setup import build_for_cell + +# Optional mesh shrink. The production fiducial is a (39, 39) = 1521-pixel mesh +# with 1500 Hilbert pixels, whose JIT compile alone exceeds an hour on CPU. A +# shrunken mesh is NOT the production cell — the landscape differs — but it +# keeps the same clamp, the same unbounded stepping, and the same mesh-shaped +# likelihood, so it can still say whether lanes reach the plateau at all. +MESH = os.environ.get("MESH_SHAPE") +HILBERT = os.environ.get("HILBERT_PIXELS") +if MESH: + n = int(MESH) + _setup._PIXELIZATION_MESH_SHAPE = (n, n) + print(f" [shrunk] pixelization mesh {(n, n)} (production is (39, 39))") +if HILBERT: + _setup._HILBERT_PIXELS = int(HILBERT) + print(f" [shrunk] hilbert pixels {HILBERT} (production is 1500)") + +MODEL_TYPE = sys.argv[1] if len(sys.argv) > 1 else "mge" +N_STARTS = int(os.environ.get("N_STARTS", "16")) +N_STEPS = int(os.environ.get("N_STEPS", "150")) +BATCH = os.environ.get("BATCH_SIZE") +INSTRUMENT = os.environ.get("INSTRUMENT", "hst") + +print(f"=== cell: imaging/{MODEL_TYPE} [{INSTRUMENT}] " + f"starts={N_STARTS} steps={N_STEPS} batch={BATCH or 'all'} ===") + +t0 = time.time() +dataset, model, analysis = build_for_cell( + dataset_class="imaging", + model_type=MODEL_TYPE, + instrument=INSTRUMENT, + use_jax=True, + use_mixed_precision=False, +) +print(f" build: {time.time() - t0:.1f}s free parameters: {model.total_free_parameters}") + +constrained = model.constrained_model_tuples() +print(f" components declaring a model constraint: {len(constrained)}") +for path, sub in constrained[:8]: + print(f" {'.'.join(p for p in path if p) or ''} -> {sub.cls.__name__}") + +kwargs = dict( + name=f"rerun_{MODEL_TYPE}", + n_starts=N_STARTS, + n_steps=N_STEPS, + iterations_per_log=25, + convergence=af.MultiStartGradientConvergence(check_for_convergence=False), +) +if BATCH: + kwargs["batch_size"] = int(BATCH) + +search = af.MultiStartProdigy(**kwargs) + +t0 = time.time() +result = search.fit(model=model, analysis=analysis) +wall = time.time() - t0 + +primary = result[0] if isinstance(result, list) else result +si = primary.search_internal +if not isinstance(si, dict): + si = getattr(si, "__dict__", {}) or {} + +counters = { + k: si.get(k) + for k in ( + "n_value_nan_lane_steps", + "n_grad_nan_lane_steps", + "n_constrained_lane_steps", + "n_resurrections", + ) +} +lane_steps = N_STARTS * N_STEPS + +print(f"\n=== imaging/{MODEL_TYPE} — {wall:.1f}s wall, {lane_steps} lane-steps ===") +for k, v in counters.items(): + pct = f"({100.0 * v / lane_steps:.2f}% of lane-steps)" if isinstance(v, int) and lane_steps else "" + print(f" {k:<28} {v} {pct}") + +out = { + "cell": f"imaging/{MODEL_TYPE}", + "instrument": INSTRUMENT, + "hardware": "cloud_cpu", + "n_starts": N_STARTS, + "n_steps": N_STEPS, + "batch_size": int(BATCH) if BATCH else None, + "lane_steps": lane_steps, + "wall_s": wall, + "free_parameters": model.total_free_parameters, + "counters": counters, +} +dest = Path(f"/tmp/claude-0/-home-user/ef0adef1-5fcd-5111-9cdf-bcb1014fc23d/scratchpad/rerun_{MODEL_TYPE}.json") +dest.write_text(json.dumps(out, indent=2)) +print(f"\nwrote {dest}") +``` + +### `validate_zero.py` — the positive control that makes the zero meaningful + +Proves the counter was watching that exact model and would have fired. Note it +probes with `jnp` arrays: concrete Python floats trip `validate_ell_comps` and +raise before the constraint is ever reached. + +What it covers, precisely: constraint **discovery** on the production model, +**evaluation** (zero inside / positive outside), and the **predicate**. All four +checks call those pieces directly with hand-built arrays, so none of them +exercises the search loop that actually accumulates the counter. The thing that +closes that last gap is in the result JSON rather than in this script: +`n_value_nan_lane_steps` and `n_constrained_lane_steps` come back as integer `0` +and `1498`, not `null` — and `rerun_cell.py` reads them with `si.get(k)`, which +would have given `null` had `_fit` never written the key. So the zero is a +measured zero, not an absent one. **Keep that distinction when re-running: a +`null` in the counters block means the plumbing broke, and it is not the same +finding as a `0`.** + +```python +"""Positive control: prove the zero from imaging/mge is a real zero. + +A count of 0 is only evidence if the counter would have fired had a lane been +trapped. This builds the SAME production model the mge cell used, then checks: + + 1. the constraint is discovered on it at all; + 2. a vector inside the valid region reports zero violation; + 3. a vector placed beyond the clamp reports a positive violation; + 4. the lane-count predicate turns that into a counted lane. + +If (1) failed, the zero would mean "nothing was watching", not "nothing happened". +""" + +import os, sys +from pathlib import Path + +ROOT = Path("/workspace/pyautolabs/autolens_profiling") +sys.path.insert(0, str(ROOT / "scripts" / "misc")) +sys.path.insert(0, str(ROOT)) +os.chdir(ROOT) + +import numpy as np +import jax.numpy as jnp +import autofit as af +from autofit.non_linear.search.mle.multi_start_gradient.search import ( + AbstractMultiStartGradient, +) +from searches._setup import build_for_cell + +dataset, model, analysis = build_for_cell( + dataset_class="imaging", + model_type="mge", + instrument="hst", + use_jax=True, + use_mixed_precision=False, +) + +# --- 1. discovery on the real production model +constrained = model.constrained_model_tuples() +print(f"1. constrained components discovered: {len(constrained)}") +for path, sub in constrained: + print(f" {'.'.join(p for p in path if p) or ''} -> {sub.cls.__name__}") +assert constrained, "NOTHING WAS WATCHING — the zero would be meaningless" + +# JAX arrays, not Python floats: concrete scalars trip the component's own +# `validate_ell_comps` guard before the constraint is ever reached (exactly as +# the method's docstring warns). The real search path is traced, where that +# guard returns early — jnp arrays reproduce that. +vector = [jnp.asarray(float(v)) for v in model.physical_values_from_prior_medians] + +# --- 2. valid region -> zero +inside = float(model.model_constraint_from_vector(vector, xp=jnp)) +print(f"\n2. violation at prior medians: {inside}") + +# --- 3. beyond the clamp -> positive. +# Brute-force which parameters drive the constraint rather than guessing names: +# push each one past the clamp in turn and see which move the violation. +responders = [] +for i in range(len(vector)): + probe = list(vector) + probe[i] = jnp.asarray(3.0) + v = float(model.model_constraint_from_vector(probe, xp=jnp)) + if v > 0.0: + responders.append((i, v)) + +print(f"3. parameters that drive the constraint when pushed to 3.0: " + f"{len(responders)} of {len(vector)}") +for i, v in responders[:6]: + print(f" index {i:>3} -> violation {v:.4f}") +outside = responders[0][1] if responders else 0.0 + +# --- 4. the predicate counts it +counted = AbstractMultiStartGradient._constrained_lane_count( + alive=np.array([True, True]), + grad_finite=np.array([True, True]), + constraint_violation=np.array([inside, outside]), +) +print(f"4. lanes counted from [valid, trapped]: {counted}") + +ok = bool(constrained) and inside == 0.0 and outside > 0.0 and counted == 1 +print(f"\nVERDICT: the mge zero is {'a REAL zero' if ok else 'NOT TRUSTWORTHY'}") +sys.exit(0 if ok else 1) +``` diff --git a/complete/2026/08/frozen-lane-counter.md b/complete/2026/08/frozen-lane-counter.md new file mode 100644 index 00000000..1570ad34 --- /dev/null +++ b/complete/2026/08/frozen-lane-counter.md @@ -0,0 +1,485 @@ +- library-prs: https://github.com/PyAutoLabs/PyAutoFit/pull/1475, https://github.com/PyAutoLabs/PyAutoGalaxy/pull/572 +- merge-commits: PyAutoFit `004f798a89e621ab7320b46fe0494201720260fd`; PyAutoGalaxy `695b27c545fa1328a34366d56e579b1bbc55f95e` +- issue: none — filed straight from investigation via `/intake`, never issued +- summary: Added a third disjoint lane counter to the multi-start gradient search + for lanes that are finite *and* differentiable but sit on a saturating plateau + with no gradient along the saturated direction, so they can never leave. Both + existing counters are blind to them and the flat figure of merit reads as + convergence. Detection reads a new class-declared `__model_constraint__` + protocol; `EllProfile` declares the `ell_comps` saturation region, reaching + every elliptical light and mass profile through the one site where `ell_comps` + is assigned. +- validation: PyAutoFit 1745 passed +15 new, CI green on 3.12/3.13/docs; + PyAutoGalaxy 1101 passed 1 skipped +12 new, CI green on 3.12/3.13/docs. Five + PyAutoFit failures are pre-existing, verified identical on a stashed clean tree + (missing `astropy`; a nautilus pool test). +- release: not performed; both merged PRs remain in the pending-release queue. + +## The finding that shaped the design + +A component-wise zero-gradient test — the obvious detector, and the one this +task was originally filed to build — **does not work**, and a JAX reproduction +proved it before any code was written. The clamp kills only the *radial* +derivative; the angle still comes from an unclamped `arctan2`, so in general +position both `ell_comps` components carry a large non-zero gradient while their +radial projection is zero only to floating-point residue. Scored against 17 +genuinely trapped lanes: + +| candidate detector | caught | +|---|---:| +| gradient component exactly zero | 0/17 | +| all components exactly zero | 0/17 | +| radial derivative exactly zero | 6/17 | +| per-parameter prior-limit escape | 17/17 | +| declared model constraint | 17/17 + corner region | + +Prior-limit escape scores well only because Prodigy overshoots grossly (trapped +magnitudes ran 1.62 to 6.78). It provably cannot see the corner region — both +components inside `(-1, 1)` with magnitude above 1, the whole `4 - pi` area +between the unit disc and the unit square. + +## Why the exception could not be reused + +`validate_ell_comps` already owns the geometry but signals by raising, which +needs a concrete boolean. A `raise` on a traced condition gives +`TracerBoolConversionError`, which is why `validate.py:153-154` returns early for +non-concrete scalars — that escape hatch is load-bearing, not an oversight. +`jax.experimental.checkify` does survive `vmap` + `value_and_grad` with finite +values and gradients, but collapses a batch to one abort-shaped error, so it +cannot serve a per-lane counter that must let surviving lanes finish. The +predicate returned as a traced value gives 32 independent verdicts with no +exception machinery. + +## Two thresholds, and drift that already existed + +The clamp saturates at `0.999` (`convert.py`); the guard rejects at `1.0` +(`validate.py`). They answer different questions — where the *gradient* dies +versus where the *geometry* stops meaning anything — and the annulus between +them is reachable: at magnitude 0.9995 the radial derivative is already exactly +zero while `validate_ell_comps` still calls the point valid. The constraint is +therefore keyed to the clamp, not the guard. + +The clamp was a bare literal at **three** sites — `convert.py`'s JAX and NumPy +branches and the Sersic Cartesian eccentric-radius path from PyAutoGalaxy#571 — +with the guard's `1.0` in a fourth file and nothing relating them. +`ELL_COMPS_MAGNITUDE_CLAMP` now states it once. Value unchanged at every site. + +## Architecture + +The protocol is **assertions with two changes**: declared on the class rather +than attached per model instance, and evaluated as a traced non-negative measure +rather than raised. It attaches through `Model.__init__`, which is already a +class-introspection site (`gather_namespaces`, `get_type_hints`, per-argument +prior resolution) — not through `__default_fields__`, which is a narrow +`ConfigException` escape hatch with two usages and is *not* a constraint +registry. No user-facing composition call changes. + +Nothing flows *up* through the likelihood: the constraint is a pure function of +the parameter vector, evaluated beside `fitness.call`, which is untouched. It +rides the fused `value_and_grad` as a fourth output on a device→host sync that +already happens. Cost is a flat +20 HLO lines at every scale tested (grid +31→256, starts 32→128) — fixed, not proportional to likelihood cost. + +## Evidence the counter fires + +End to end on a real `ag.mp.Isothermal` under `af.MultiStartProdigy`, the +constraint arriving purely through inheritance: + +``` +prodigy step 300/300 | best log_post -431.7808 | alive 22/32 | constrained 6/32 +n_value_nan_lane_steps 2344 +n_grad_nan_lane_steps 0 +n_constrained_lane_steps 1057 +best lane einstein_radius = 1.5968 (truth 1.6) +``` + +Severity is honest: the trap wastes start budget rather than corrupting results +— surviving lanes still recovered the truth. It becomes a correctness risk only +where `n_starts` is small or the good basin is rare, which is the pixelized-mesh +regime. + +## Traps + +- **The original question is still unanswered.** Whether the *real* Prodigy mesh + or MGE runs entered the region cannot be determined retrospectively: + `autolens_profiling` stores summary records with no per-start traces, and the + Prodigy mesh cells (`scripts/imaging/searches/multi_start_prodigy/`) were never + persisted to `results/searches/multi_start_prodigy/` at all, which holds only + `point_source/` and `cluster/`. Re-running those cells with this counter is the + follow-up that answers it. +- **Spherical profiles inherit the constraint.** `IsothermalSph` subclasses + `Isothermal`, so it carries the declaration with `ell_comps` pinned at `(0, 0)` + — always satisfied, a few wasted ops. A test asserting the opposite failed and + was corrected; the real behaviour is now pinned. +- **A grid with a pixel at the exact centre masks this hazard entirely.** The + first reproduction returned NaN gradients everywhere from `sqrt(0)` — the + *separate* r=0 non-finite-gradient hazard — hiding the plateau under + investigation. Use an off-centre grid, as the hazard scans do. +- **`black` is not enforced on the PyAutoFit files touched.** It wants to + reformat all three at `main` too, so running it would bury the change in + unrelated churn. Left alone deliberately. + +## What the penalty term inherits + +The counter reads only `violation > 0.0`, a sign test, so it is scale-free and +carries no lambda. That leaves two properties of the measure untested by anything +shipped, both landing on the penalty task: + +- **Units.** The measure is in the constraint's own units (ellipticity + magnitude); the figure of merit is in log-likelihood. That mismatch is what a + lambda has to absorb, and why a constant cannot work across cells whose scales + differ by orders of magnitude. +- **The reduction.** `model_constraint_from_vector` combines components with + `maximum` — correct for counting, wrong for a penalty, where two constraints in + different units would let whichever is numerically larger silently dominate. + Per-constraint lambdas or normalised measures, not one max. + +Also unresolved for the penalty: it must be moved *inside* the differentiated +call (today the violation is computed after `value_and_grad`, so it has no +gradient effect), and kept out of the reported likelihood or made invertible in +`log_likelihood_from`. + +## Original prompt + +# Count frozen lanes in the multi-start gradient search + +Type: feature +Target: PyAutoFit +Repos: +- PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: formalised + +@PyAutoFit + +Add a third lane counter to `AbstractMultiStartGradient._nan_lane_counts` +(`autofit/non_linear/search/mle/multi_start_gradient/search.py:219-255`). + +It counts two disjoint failure modes today — **value-NaN** (likelihood +undefined; the death `resurrect` triggers on) and **gradient-NaN** (defined but +not differentiable; `apply_if_finite` zeroes the update). A third mode escapes +both: a lane whose value and gradient are both finite, but which sits on a +saturating plateau with **no gradient along the saturated direction**, so it can +never leave. + +## Why it matters + +The reference case is the ellipticity magnitude clamp `jax.lax.min(fac, 0.999)` +in PyAutoGalaxy's `convert.py:71-77`. Past `|ell_comps| >= 0.999` the axis ratio +pins at `q = 5.0025e-4`, so the likelihood is exactly constant along the radial +direction — measured identical to 10 significant figures at `|ell_comps|` of +1.0, 1.2 and 5.0. For a multi-start gradient search that is an absorbing trap: + +- Starts are safe: `_broad_starts` draws in the unit cube at `(0.15, 0.85)`, + capping start magnitude near 0.44 under the default `TruncatedGaussian(0, 0.3)`. + Measured start range in the toy run was 0.087 to 0.362. +- But `optax.apply_updates(params, updates)` (`search.py:743`) steps the physical + vector with no re-projection into prior limits, so trajectories walk in — 17 of + 32 lanes did, reaching `|ell_comps|` as high as 6.78. +- Once inside there is no radial restoring force, so the lane cannot come back. +- `apply_if_finite` and `resurrect` are both no-ops here — value and gradient are + finite. +- `multi_start_prodigy_autoconv` runs `check_for_convergence=True`, so trapped + lanes flatten the figure of merit and can false-trigger early stopping. + +A trapped lane is therefore indistinguishable from a converged one in the +figure-of-merit trace, which is exactly the hazard `_nan_lane_counts` was +written to expose for the gradient-NaN case. + +## Scope + +Instrumentation only. Add the count alongside the existing two, accumulate it +across steps, record it into `search_internal`, restore it on resume, and report +it on the progress line — mirroring `n_value_nan_lane_steps` and +`n_grad_nan_lane_steps` exactly. Keep it pure-NumPy and free of search state so +it stays directly testable like its two siblings. + +The search must produce identical results with the counter present. Do not add a +penalty term, and do not touch `resurrect`, `apply_if_finite`, the convergence +check, stepping behaviour, or the clamp itself. + +## Detector: use the saturation predicate, not a zero-gradient test + +A JAX toy reproduction (32 starts, 400 Prodigy steps, PyAutoGalaxy's clamp +verbatim, `apply_if_finite` + unbounded `apply_updates`, truth at +`|ell_comps| = 0.90`) settled the detector question empirically. **Do not +implement a component-wise zero-gradient test — it does not work.** + +Measured on the 17 of 32 lanes that ended on the plateau: + +| candidate detector | caught | false pos | +|---|---:|---:| +| gradient component exactly zero | **0/17** | 0 | +| all gradient components exactly zero | **0/17** | 0 | +| radial derivative exactly zero | 6/17 | 0 | +| **per-parameter prior-limit escape** | **17/17** | **0** | + +The reason is that the clamp kills only the *radial* derivative. The angle still +comes from an unclamped `arctan2`, so in general position both `ell_comps` +components carry a large non-zero gradient while their radial projection is zero +— at `ell_comps = (0.867, 0.593)` the components are `+2.7e4` and `-3.9e4` while +the radial projection is `-3.4e-12`. Only on the measure-zero axis where one +component is exactly zero does a component itself read zero. Floating-point +residue from the rotation is also why the radial test scores 6/17 rather than +17/17: the projection is ~1e-12, not an exact zero. + +### Reuse the guard's predicate, not its exception + +`validate_ell_comps` already owns the authoritative constraint, but it cannot +raise under a trace and deliberately does not try — `validate.py:153-154` returns +early for non-concrete scalars. That escape hatch is load-bearing: a plain +`raise` on a traced condition gives `TracerBoolConversionError`, so without it +every jitted likelihood would crash rather than sample. + +Measured on the three candidate mechanisms: + +| mechanism | works under `vmap` + `grad`? | shape | +|---|---|---| +| plain `raise` | no — `TracerBoolConversionError` | — | +| `jax.experimental.checkify` | yes, values/grads stay finite | **one error per batch**, abort-shaped | +| **guard predicate as a traced boolean** | yes | **per-lane verdict** | + +`checkify` is the genuine JAX exception mechanism and it does survive +`vmap`+`value_and_grad`, but it collapses a batch to a single error and is built +to abort. That is the wrong shape here: the run must continue so the surviving +lanes finish, and the counter needs 32 independent verdicts, not one. + +The predicate as a traced boolean gives exactly that, returned alongside the FoM +— no exception machinery at all. Scored against the 17 real trapped lanes plus +three synthetic corner-region lanes (both components inside `(-1, 1)`, magnitude +above 1): + +| detector | caught | missed | +|---|---:|---:| +| per-parameter prior-limit escape | 17/20 | 3 | +| **saturation predicate `|ell_comps| >= 0.999`** | **20/20** | **0** | + +**Use the clamp's threshold (0.999), not the guard's (1.0).** They differ, and +the gap is reachable: at `|ell_comps| = 0.9995` and `0.99999` the radial +derivative is already exactly zero while `validate_ell_comps` still calls the +point valid. A detector keyed to the guard's `magnitude_squared >= 1.0` misses +that annulus; one keyed to the clamp does not. + +This supersedes the prior-limit-escape recommendation below, which is provably +incomplete — the corner region it misses is the whole `4 - pi` area between the +unit disc and the unit square. + +### How it attaches (no new composition call) + +The model-composition API does not gain a user-facing call — `af.Model(al.mp.Isothermal)` +is unchanged. + +What makes that possible is that `Model.__init__` is **already** a class +introspection site: it runs `gather_namespaces(cls)` and +`typing.get_type_hints(cls.__init__, ...)`, walks the constructor signature and +resolves a prior per argument. Per-class knowledge is already gathered there, so +this is one more lookup in an existing mechanism, not a new one. + +Do **not** build this on `__default_fields__`. That is a narrow escape hatch — +`prior_model.py:200` consults it only when `make_prior` returns a +`ConfigException`, and it has exactly two usages (`messages/normal.py:412`, +`truncated_normal.py:485`, both `("log_norm", "id_")`). It marks "this argument +is not a model parameter"; it is not a constraint registry. + +The real relative is **`add_assertion`** (`abstract.py:441`), PyAutoFit's +existing constraint concept. It is wrong here on exactly two counts: it attaches +per *model instance* rather than per class (so every elliptical profile would +need the user to remember it), and it raises `FitException`, which is numpy-only +for the reasons above. + +So frame the work as **assertions with two changes** — class-declared rather than +per-model, and a traced predicate rather than a raise — not as a new validity +subsystem. The two should likely share a home rather than sit as unrelated +features. + +Placement is clean because `validate_ell_comps` has exactly **one** call site, +`geometry_profiles.py:237` in `EllProfile.__init__` — the single base every +elliptical light and mass profile inherits. The declaration goes on that class, +beside the existing call. + +Do **not** let this become a second statement of the constraint. Extract the +predicate out of `validate_ell_comps` as a pure `xp`-generic function; leave +`validate_ell_comps` raising on concrete scalars as it does today, now calling +that predicate; point the class-level declaration at the same function. + +Note that the drift this guards against **already exists**: the clamp is `0.999` +in `convert.py:71-77`, the guard is `1.0` in `validate.py:158`, in different +files with nothing relating them. That gap is precisely the reachable annulus +where the radial gradient is already dead while the guard still calls the point +valid. The two thresholds answer different questions and both should survive — +but their relationship should be stated in one place, which this work is the +opportunity to do. + +### What it costs, and what it cannot see + +Nothing flows *up* through the likelihood. The saturation predicate is a pure +function of the parameter vector, so it is evaluated at the top beside the +likelihood, which is left untouched — measured with the toy's +`log_likelihood` imported unmodified, gradients bit-identical (`atol=0`). + +Cost on the toy: +23 HLO lines (700 -> 723, +3.3%). Wall-clock overhead measured +at +40 us on a ~1090 us call, which is **below the noise band** on a shared CPU +(interleaved burst spreads overlapped) — treat it as an upper bound, and expect +it to be proportionally smaller against a real pixelized likelihood. + +The ceiling is worth stating before anyone assumes one mechanism covers the +whole hazard index: this works *only* for parameter-only properties. The tier-2 +likelihood hazards — which basis components the NNLS active set pinned at zero, +how a conditioning floor bit against real flux — are genuine likelihood-internal +state, are not recoverable from the parameter vector, and would need real upward +plumbing. + +### Scope consequence + +Asking the model "is this instance saturated?" is a **validity channel** between +PyAutoFit and the profile libraries, not something PyAutoFit can answer alone. +It is also the same hook the later penalty term needs, so building it once serves +both. If this task stays PyAutoFit-only it must fall back to prior-limit escape +and accept the corner-region miss; see the open question at the end. + +### Superseded: prior-limit escape + +Detect the **cause** — a lane that has left its priors' support, which is +possible at all only because `apply_updates` steps the physical vector with no +re-projection. That is fully generic (every `Prior` already carries +`lower_limit`/`upper_limit`), needs no model semantics, no gradient inspection, +and no tolerance. + +Record its limit honestly in the docstring: it is a proxy for the cause, not the +effect. It caught every trapped lane here because Prodigy overshoots grossly +(trapped `|ell_comps|` ran 1.62 to 6.78, max component 6.45), not because it is +complete. A lane in the corner region — both components inside `(-1, 1)` but +magnitude above 1, which is the whole of the `4 - pi` area between the unit disc +and the unit square — is beyond the clamp yet inside every per-parameter limit, +and this detector will miss it. + +## Severity, for prioritisation + +The same run shows the trap **wastes budget rather than corrupting results**: +17/32 lanes died on the plateau, yet the surviving lanes still recovered the +truth (best lane `|ell_comps| = 0.9001`, logL −427.5 against a truth logL of +−430.4). Multi-start redundancy absorbs it. It becomes a correctness risk only +when `n_starts` is small or the good basin is rare — which is exactly the +pixelized-mesh regime the counter is meant to observe. + +## Other decisions + +- Keep the buckets disjoint: a lane already counted as value-NaN or gradient-NaN + must not also count as escaped. +- Do not force a per-step device sync if it costs measurable run time. The + existing NaN accounting measured 0.0004% of step time; stay in that class. + + + + +## Open question for start_dev + +Whether to widen this task to the validity channel (PyAutoFit + PyAutoGalaxy, +complete detector, shared with the later penalty term) or keep it PyAutoFit-only +(prior-limit escape, misses the corner region). Widening changes the header: +`Repos:` gains PyAutoGalaxy and difficulty rises from `medium`. Decide before +issuing, not during. + +## What the penalty inherits (shipped counter is unaffected) + +The counter uses only `violation > 0.0` — a sign test, so it is scale-free and +no lambda exists anywhere in it. Two properties of the shipped measure are +therefore **untested by anything shipped**, because only its sign is ever read, +and both land on the penalty task: + +- **Units.** `max(|ell_comps| - 0.999, 0)` is in units of ellipticity magnitude; + the figure of merit is in log-likelihood. That mismatch is exactly what a + lambda has to absorb, and it is why a single constant cannot work across cells + whose likelihood scales differ by orders of magnitude (point-source ~10s, + pixelized ~30,000s). +- **The reduction.** `model_constraint_from_vector` combines components with + `xp.maximum`. Correct for counting — any violation makes the lane constrained + — but wrong for a penalty: two constraints in different units (ellipticity + magnitude vs a radius, say) would be reduced by `max`, so whichever is + numerically larger silently dominates. A penalty likely wants per-constraint + lambdas, or measures normalised to a common scale, rather than one max. + +Neither is a defect in the counter. Both are decisions to make before the +measure is multiplied into the figure of merit. + +## Status + +The counter shipped: PyAutoFit PR #1475 (branch +`claude/jax-sampling-flat-gradients-ptmqnl`), CI green on 3.12, 3.13 and docs. +1745 tests pass, +15 new. PyAutoGalaxy declares no constraint yet, so the +counter reads zero on real lens models until `EllProfile` opts in — a ~4-line +method at the site that already calls `validate_ell_comps`. + +## Follow-up: first rerun with the counter (2026-08-15, cloud CPU) + +Half the original question is now answered, and the other half is bounded. + +**`imaging/mge` — zero constrained lane-steps.** Run through the repo's own +`build_for_cell` (real HST dataset, real 15-parameter model, real analysis; +only the search budget reduced), 16 starts x 150 steps, 352s on 4 CPU cores: + +| counter | lane-steps | share | +|---|---:|---:| +| `n_value_nan_lane_steps` | 1498 | 62.42% | +| `n_grad_nan_lane_steps` | 9 | 0.38% | +| `n_constrained_lane_steps` | **0** | **0.00%** | + +So for the MGE cell the flat-gradient region is **not** what hurts it. Read the +first row alongside the third, though: the population fell to `alive 2/16` and +NaN death dominates, so lanes are dying by a different mechanism before they +could reach the plateau. The zero is real but partly "they never got the +chance". Reduced budget, one seed. + +**The pixelized mesh cells cannot run on CPU — compile, not memory.** Two +attempts, both killed by timeout having emitted **zero steps**: + +- production mesh (39x39, 1500 Hilbert), `batch_size=1` — 58 min, no step 1; +- shrunk mesh (12x12), `batch_size=1` — 40 min, no step 1. + +Memory is solvable (`batch_size=1` clears the 18.5 GB OOM); the JIT compile is +not. These need the GPU, which matches the search's own docstring on compile +cost. The mesh reruns therefore remain the open tail. + +**Shrinking the source mesh is the wrong lever.** Going 39x39 -> 12x12 barely +moved the allocation (an unbatched 8-start draw still asked for 67 GB). The cost +is dominated by the image-plane grid and mapping matrices at `mask_radius` 3.5, +not by the source mesh — worth knowing before anyone tries to make these cells +cheap by shrinking the mesh. + +**Trap: the runner writes to `dataset/`.** Building the imaging cell rewrote +`dataset/imaging/hst/*.fits` + `tracer.json`, added `positions.json`, and +emitted `results/simulators/imaging_hst_summary_*`. Harmless in a throwaway +clone, but do not assume the dataset directory is read-only. + +**Environment note.** The stack ran on a cloud box with `autolens` installed +`--no-deps` over editable local `autofit`/`autogalaxy`; `jaxnnls` is a required +extra for the JAX NNLS solver path and is not pulled in by default. + +## The zero was validated, not assumed + +A count of zero is only evidence if the counter would have fired. Positive +control against the **same production mge model**: + +1. **44 constrained components discovered** on it (MGE `Basis` Gaussians, the + `Isothermal` mass, `ExternalShear`) — something was watching. +2. Violation at prior medians: **0.0**. +3. **6 of 15** free parameters drive the constraint when pushed past the clamp + (the three `ell_comps` pairs), each reporting violation 2.001. +4. The lane predicate turns that into **1 counted lane**. + +So `n_constrained_lane_steps = 0` on the mge cell means *nothing entered the +plateau*, not *nothing was watching*. + +Two inheritance cases confirmed benign, both discovered but unable to violate +because their `ell_comps` are pinned at `(0, 0)`: spherical profiles (they +subclass their elliptical counterpart) and `ExternalShear` (a subclass of +`EllProfile` whose free parameters are `gamma_1`/`gamma_2`). Shear magnitude is +**not** constrained by this rule, which is correct — the unit-disc bound is +about the axis ratio, not shear. + +**Trap worth repeating:** probing the constraint with concrete Python floats +raises `ModelParameterException` from the component's own `validate_ell_comps` +before the constraint is ever evaluated. That is the documented NumPy-path +behaviour, not a fault. Reproduce the traced path with `jnp` arrays. diff --git a/complete/index.md b/complete/index.md index 10bf1887..aeaa4626 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -991 records across 7 buckets. +992 records across 7 buckets. ## Highlights @@ -50,6 +50,7 @@ _(curate hard-won records here — survives regeneration.)_ - [ep-optimise-updater](2026/08/ep-optimise-updater.md) - [feature-ranker-ignores-header-keys](2026/08/feature-ranker-ignores-header-keys.md) — the Feature Agent's ranker now reads the prompt metadata header it was - [file-path-guard-decision](2026/08/file-path-guard-decision.md) — The file-path leg split from raw-guard-migration (leg 3 of the dataset-bulk series, autolens_workspace#354). D… +- [frozen-lane-counter](2026/08/frozen-lane-counter.md) — Added a third disjoint lane counter to the multi-start gradient search - [group-data-preparation-readme](2026/08/group-data-preparation-readme.md) - [hazard-profiling-likelihood-tier](2026/08/hazard-profiling-likelihood-tier.md) — Tier-2 likelihood profiling landed with five persistent findings and corrected NNLS continuity semantics. - [health-conductor-stale-verdict](2026/08/health-conductor-stale-verdict.md) diff --git a/dashboard.md b/dashboard.md index 45231d88..1ba7b353 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,19 +8,21 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 7 | +| [In flight](#in-flight) (`active/`) | 8 | | [Parked](#parked) (`parked.md`) | 6 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 136 | +| [Backlog](#backlog) (`draft/`) | 138 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) ## Start here -**Highest priority** (filed as `high`) — showing 12 of 30 +**Highest priority** (filed as `high`) — showing 12 of 32 - [pre_build stages untracked files, publishing uncommitted human work](draft/bug/pyautohands/pre_build_stages_untracked_wip.md) — pyautohands · small · supervised · high - [TRIAGE: needs manual review before routing](draft/triage/jax_zero_contour.md) — medium · safe · high +- [Clipper: demonstrate and validate on the profiling search cells](draft/feature/autofit/clipper_validation_campaign.md) — autofit · medium · supervised · high +- [Search-agnostic prior-support enforcement: a Clipper class](draft/feature/autofit/prior_support_clipper.md) — autofit · medium · supervised · high - [PyAutoLens RTD docs: three-regime restructure (multi_galaxy / group / cluster)](draft/docs/autolens/docs_three_regime_restructure.md) — autolens · medium · supervised · high - [Optimize pixelized Prodigy settings on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu_phase_2_settings.md) — autolens_workspace_developer · medium · human-required · high - [Release does not sync __version__ stamps and workspace pins back](draft/bug/pyautobuild/release_version_sync_back_to_main.md) — pyautobuild · medium · supervised · high @@ -29,8 +31,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A - [Cluster package: point-source-default narrative + extended-source follow-up feature](draft/docs/workspaces/cluster_regime_narrative.md) — workspaces · medium · supervised · high - [multi_galaxy package: new regime package in autolens_workspace](draft/docs/autolens/multi_galaxy_package.md) — autolens · large · supervised · high - [Tune the JAX multi-start optimizers into a standard option (MGE](draft/experiment/autolens_profiling/jax_optimizer_settings_tuning.md) — autolens_profiling · large · supervised · high -- [Profile and speed up JAX likelihood-function compile times (all use](draft/feature/autolens_profiling/jax_compile_time_profiling.md) — autolens_profiling · large · supervised · high -- [Optimize MultiStartProdigy for pixelized meshes on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu.md) — autolens_workspace_developer · large · human-required · high **Quick wins** (small enough, and safe enough to run unattended) @@ -55,6 +55,7 @@ Issued — each has an open GitHub issue and usually a branch. The full record f - [Address ECEB editorial comments on ECLIPSE-C](active/euclid_eceb_editorial_revision.md) - [JAX-native posterior sampler wave — ranked shortlist from the 2026-07-16](active/jax_native_posterior_sampler_wave.md) — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius… - [Remove standalone matplotlib-inline comments](active/matplotlib_inline_standalones.md) +- [Find what kills MGE multi-start lanes — it is not](active/mge_lane_death.md) — [issue #128](https://github.com/PyAutoLabs/autolens_profiling/issues/128) — CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed… - [Pixelized Prodigy laptop-GPU compatibility across four meshes](active/pixelized_prodigy_laptop_gpu_phase_1_compatibility.md) — [issue #125](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/125) — workspace-dev — phase 1 and phase 2 COMPLETE 2026-08-13, all 13 cells landed, PR #126 ready for review - [PyAutoReduce validation: slacs1430+4105 ACS reduction vs trusted legacy dataset](active/pyautoreduce_slacs1430_acs_comparison.md) - [Research profiling experiment in the autolens_profiling repo](active/research_profiling_experiment_in_the_autolens_pr.md) — [issue #82](https://github.com/PyAutoLabs/autolens_profiling/issues/82) @@ -94,7 +95,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**136** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**138** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
bug — 40 @@ -143,8 +144,10 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-feature — 25 +feature — 27 +- [Clipper: demonstrate and validate on the profiling search cells](draft/feature/autofit/clipper_validation_campaign.md) — autofit · medium · supervised · high +- [Search-agnostic prior-support enforcement: a Clipper class](draft/feature/autofit/prior_support_clipper.md) — autofit · medium · supervised · high - [Make draft/ staleness detectable — `intake reconcile` measured, and the](draft/feature/pyautomind/draft_staleness_detection_signals.md) — pyautomind · medium · supervised · high - [LACosmic per-frame CR masking option + decouple PSF-star pass from](draft/feature/pyautoreduce/lacosmic_cr_option_and_star_pass_decoupling.md) — pyautoreduce · medium · supervised · high - [Profile and speed up JAX likelihood-function compile times (all use](draft/feature/autolens_profiling/jax_compile_time_profiling.md) — autolens_profiling · large · supervised · high diff --git a/draft/feature/autofit/clipper_validation_campaign.md b/draft/feature/autofit/clipper_validation_campaign.md new file mode 100644 index 00000000..efdacdcd --- /dev/null +++ b/draft/feature/autofit/clipper_validation_campaign.md @@ -0,0 +1,172 @@ +# Clipper: demonstrate and validate on the profiling search cells + +Type: feature +Target: autolens_profiling +Repos: +- autolens_profiling +- PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +## What this is + +The **validation phase** of the prior-support fix. It runs after +`draft/feature/autofit/prior_support_clipper.md` lands the `Clipper` in +`@PyAutoFit`, and it produces the evidence that justifies flipping the default. + +Three phases, three PRs, in this order — do not merge them out of order: + +| phase | repo | what | +|---|---|---| +| 1 | `@PyAutoFit` | `Clipper` class, opt-in, bit-identical by default (`prior_support_clipper.md`) | +| **2** | **`@autolens_profiling`** | **this task — demonstrate and validate across the search cells** | +| 3 | `@PyAutoFit` | flip the default to `ClipperPriorBox`, carrying the phase-2 re-baseline | + +Phase 2 is where the claim "clipping recovers the lost lanes without changing the +answer" either survives or does not. Phase 3 must not be written until phase 2 +has run. + +Background and the full root-cause investigation: **autolens_profiling#128**. +One-line version: the objective is `-2 * (log_likelihood + sum(log_prior))`, a +`UniformPrior` is `-inf` outside its box, the search steps in physical space with +nothing holding it there, and a lane that oversteps by ~3% of a box width is +marked dead and never redrawn — while continuing to step and consume full +likelihood-and-gradient cost with its output discarded. + +## The truth bar already exists — use it + +`results/searches/nautilus/imaging/mge/hst/hpc_a100_fp64.json` is a Nautilus run +on **the same cell, the same 15-parameter model**: + +``` +max_log_likelihood = 31786.782462488976 +log_evidence = 31690.47079355404 +posterior_samples = 63800 (A100, fp64, n_live 200) +``` + +Nautilus samples in **unit-cube coordinates**, so it is structurally immune to +this bug. That makes it the reference answer, not merely another data point. + +**This is the load-bearing validation.** "Fewer lanes die" is a weak claim on its +own — a change that keeps lanes alive by making them useless would satisfy it. +The claim worth testing is: **does clipped MultiStartProdigy get closer to the +Nautilus maximum log-likelihood than unclipped does?** If lane deaths fall and +best-fit logL does *not* improve toward 31786.8, clipping is cosmetic and phase 3 +should not happen. + +## Arms + +Per cell, at minimum: + +1. `clipper=None` / `ClipperNone` — control, must reproduce today's numbers. +2. `ClipperPriorBox` — the candidate. +3. `ClipperPriorBox` **+ momentum reset** on clipped coordinates, if phase 1 + shipped it. The prototype left 5/16 lanes *pinned* to a bound with Prodigy's + state still pushing outward; this arm is what says whether that matters. + +Record for every arm: `n_value_nan_lane_steps`, `n_grad_nan_lane_steps`, +`n_constrained_lane_steps`, `n_resurrections`, **`alive N/n_starts` per step**, +best-fit log-likelihood, wall time, and the count of lanes ending pinned to a +bound. + +**At least two seeds per arm.** Single-seed CPU numbers are what this whole +investigation had to go back and re-derive. + +## Cells + +Start with the characterised one, then widen: + +- **`imaging/mge` (hst)** — the reference. ~250s at 16x150 on cloud CPU, so it + iterates fast. Known baseline: 1446/2400 value-NaN (60.25%), `alive 16/16 -> + 2/16`, 14 lanes dead at steps `[34,36,36,37,38,39,39,40,40,41,41,52,56,125]`. + Prototype clipping gave 425 (17.71%), 5 dead, `alive -> 11`. +- **The pixelized mesh cells** (`delaunay`, `pixelization`, DelaunayNN) — **GPU + required**. These are the cells `resurrect=True` was introduced for, so they are + the strongest test of whether clipping changes the resurrection story. Two prior + attempts timed out on CPU with zero steps emitted; that is **JIT compile, not + memory** (`batch_size=1` clears the OOM). Do not shrink the source mesh to make + them cheaper — the image-plane grid at `mask_radius 3.5` dominates, not the mesh. +- **`point_source`** cells — different model family and different prior structure; + confirms the fix is not MGE-shaped. +- **A negative control**: a model whose priors are all unbounded (`GaussianPrior`). + `ClipperPriorBox` must be a *no-op* there. If it changes anything, the bounds + extraction is wrong. + +## What would falsify the fix + +Write these down before running, and report them honestly if they happen: + +- Lane deaths fall but best-fit logL does **not** move toward the Nautilus + reference → clipping keeps lanes alive without making them useful. +- Most surviving lanes end pinned to a bound → the wall is absorbing the + population; momentum reset is mandatory, or projection is the wrong strategy. +- Wall time per step rises materially → a clip on `(n_starts, ndim)` should be + unmeasurable; if it is not, something is wrong with where it was inserted. +- The pixelized cells get *worse* → clipping and `resurrect=True` interact badly, + and phase 3 must be scoped per-search rather than globally. + +## Pinning is a result, not a failure + +Where the likelihood genuinely prefers a value outside the prior, a clipped lane +sitting on the bound is the **correct MAP answer under the declared prior**. In +the reference cell the shear escapes were mixed-sign (`+0.353`, `-0.341`, +`+0.301`, `-0.312`), which reads more like a poorly-constrained parameter +diffusing out than a true value sitting outside — but that is a hypothesis, not a +finding. If clipped runs pin `gamma` at `±0.3` reproducibly, that is evidence the +shear prior is fighting the data and belongs in the write-up as a science finding, +not swept up as a clipping artefact. + +## Deliverables + +- Results JSONs under `results/searches/` alongside the existing NaN-accounting + artefacts, following the conventions already there. +- A note under `results/notes/` — the comparison table, the Nautilus-reference + verdict, and an explicit recommendation for or against phase 3. +- The hazard-index entry for the prior-exit failure mode (owed from #128). + +## Environment + +- **Python 3.12+** (autonerves). `pip install jaxnnls` is **required** for the JAX + NNLS solver path and is not pulled in by default. `optax` likewise. +- Install `autolens` with **`--no-deps`** when running editable local + `autofit`/`autogalaxy`, or the released wheels clobber them. Phase 1 is + unreleased, so this task **must** run against `@PyAutoFit` `main` (or the phase-1 + branch), not a PyPI wheel — verify `autofit.__file__` resolves to the checkout + before trusting any number. +- `build_for_cell` **writes into `dataset/`** (rewrites the HST FITS, adds + `positions.json`, emits `results/simulators/*`). Not read-only. +- Cell scripts honour `SEARCHES_N_STARTS` / `SEARCHES_N_STEPS` / + `SEARCHES_BATCH_SIZE` / `SEARCHES_DISABLE_VIZ`. +- On A100 set `jax_enable_x64` **explicitly** — it is not inherited under `sbatch`, + and float32 would understate the quantity under test. All #128 numbers are + float32 CPU; expect them to move on fp64 and do not treat a difference as a + regression without checking precision first. + +## Traps, all paid for already + +- **Grade on the alive-versus-step curve, not the percentage.** `n_value_nan_lane_steps` + is a *survival integral*: a dead lane keeps counting every subsequent step, so + the same death curve reports 60% at 150 steps and ~75% at 300. Verified exactly: + `sum(150 - k_i) = 14*150 - 654 = 1446` = the counter, to the unit. Two arms at + different budgets cannot be compared on the scalar. +- **A crashed run poisons the next run of the same `name`.** A half-written output + JSON makes the next search try to *resume* and fail — a 4-second no-op that reads + as a clean result (zero deaths, because zero steps). Delete `output//` + between arms, or use unique names, and **assert the recorded step count equals + `n_steps`** before believing any counter. +- **`float32` breaks `save_json`.** `autofit/non_linear/paths/directory.py:80` + raises `TypeError: Object of type float32 is not JSON serializable` at the end of + a *successful* run. It does not fire when most lanes are dead, so it will start + firing exactly when the fix works. If phase 1 has not fixed it, capture counters + independently of the result object. +- **`0` and `null` are different findings.** Read counters with `.get()` and a + `null` means the search never wrote the key — broken plumbing, not a clean cell. + +## Deliberately out of scope + +- Flipping any default (phase 3). +- NUTS. It targets the log posterior from a physical start and *diverges* rather + than dying — a different mechanism needing its own investigation. +- Unit-cube stepping. Rejected for now in `prior_support_clipper.md`, with reasons. diff --git a/draft/feature/autofit/prior_support_clipper.md b/draft/feature/autofit/prior_support_clipper.md new file mode 100644 index 00000000..04954d4d --- /dev/null +++ b/draft/feature/autofit/prior_support_clipper.md @@ -0,0 +1,214 @@ +# Search-agnostic prior-support enforcement: a Clipper class + +Type: feature +Target: PyAutoFit +Repos: +- PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +## Why + +`@PyAutoFit/autofit/non_linear/search/mle/multi_start_gradient/search.py` builds +its objective as + +``` +fom = -2 * (log_likelihood + sum(log_prior_list)) +``` + +A `UniformPrior` returns `log_prior = -inf` outside its box, and the search steps +in **physical** parameter space with nothing constraining it to that box. A lane +that oversteps a hard prior edge reads as non-finite, is marked dead, and with +`resurrect=False` is never redrawn. + +Measured on the real `imaging/mge` profiling cell (16 starts x 150 steps, cloud +CPU) — full investigation and evidence in autolens_profiling#128: + +| arm | value-NaN lane-steps | lanes dead | alive at end | +|---|---:|---:|---:| +| baseline | 1446 (60.25%) | 14/16 | 2 | +| shear box widened to ±1 | 1422 (59.25%) | 15/16 | 1 | +| prior term neutered (diagnostic) | 215 (8.96%) | 3/16 | 13 | +| **clip to prior box (prototype)** | **425 (17.71%)** | **5/16** | **11** | + +**The likelihood never went non-finite** in ~7200 lane-steps. This is entirely a +prior-support problem. + +The behaviour is worse than "frozen": the overshoot is tiny (median 3% of box +width, min 0.21%), and because `log_prior = -inf` is *constant* outside the box +its derivative is zero, so the total gradient is the finite **likelihood** +gradient. `optax.apply_if_finite` therefore never fires and the dead lane **keeps +stepping forever** — full likelihood-and-gradient cost every step, output +discarded, wandering far (one parameter went `0.30 -> -1.76`). 0/16 lanes ever +revive. + +## The exposure is not MultiStart-only + +This is why the fix should not live inside one search: + +- **`MultiStartGradient`** (`MultiStartAdam` / `MultiStartADABelief` / + `MultiStartLion` / `MultiStartProdigy` all share one `_fit`) — measured above. +- **`@PyAutoFit/autofit/non_linear/search/mle/bfgs/search.py`** — same + `Fitness(fom_is_log_likelihood=False, resample_figure_of_merit=-np.inf, + convert_to_chi_squared=True)`, steps in physical space, and calls + `optimize.minimize(fun=..., x0=..., method=self.method, options=..., tol=...)` + with **no `bounds=` argument**. `L-BFGS-B` supports box bounds natively; they + are simply not passed. Being single-start, this presents as a failed fit rather + than a population collapse, so it is easier to misattribute. +- **NUTS** (`@PyAutoFit/autofit/non_linear/search/mcmc/blackjax/nuts/search.py`) + also targets the log posterior from a physical `initial_position`. HMC entering + a `-inf` region *diverges* rather than freezing. **Out of scope here** — different + mechanism, needs its own investigation. See "Deliberately out of scope". + +Not exposed, and correctly so: the nested samplers already work in unit-cube +coordinates, and the MCMC samplers reject `-inf` proposals so the walker stays +put. **Rejection is the restoring mechanism that gradient methods lack.** + +## The design + +A `Clipper`, modelled on `@PyAutoFit/autofit/non_linear/initializer.py` — a +pluggable, per-search strategy object with a config-resolved default. + +**One place the `Initializer` analogy does not carry.** `Initializer` has a single +consumption pattern (`samples_from_model`). `Clipper` has **two structurally +different consumers** and must serve both from one source of truth: + +- `MultiStartGradient` enforces the constraint itself, every step → wants an + imperative `project(...)`. +- `LBFGS` hands bounds to scipy and lets *scipy* enforce → wants a declarative + `bounds`. + +Proposed contract: + +```python +class AbstractClipper(ABC): + @abstractmethod + def bounds_from_model(self, model) -> tuple[np.ndarray, np.ndarray]: + """(lower, upper) in PHYSICAL parameter order. Unbounded -> -inf/+inf.""" + + @abstractmethod + def project(self, vector, model, xp=np): + """Return (projected_vector, clipped_mask). Identity where unbounded.""" + + +class ClipperNone(AbstractClipper): + """No-op. Bounds are ±inf, project is the identity. THE DEFAULT (see below).""" + + +class ClipperPriorBox(AbstractClipper): + """Hard projection onto the prior support, inset by a margin.""" +``` + +`project` **must return which coordinates it clipped**, not just the new vector. +That mask is what lets a caller zero the optimiser momentum along clipped +directions. It is needed: the prototype left 5 of 16 lanes pinned to a bound at +the end of the run because the parameters were projected while Prodigy's +accumulated state kept pushing outward. The `Clipper` cannot fix that itself — it +does not own `opt_state` — so it must expose enough for the search to. + +Later strategies (`ClipperReflect`, a soft-wall variant) drop in without touching +callers. A soft wall must be a *Clipper* (search-local), **never** a change to the +`Prior` classes — that would silently alter the objective for the nested samplers, +where the hard box currently works correctly. + +## Scope — PR 1 (this task) + +1. `AbstractClipper` + `ClipperNone` + `ClipperPriorBox` in a new + `@PyAutoFit/autofit/non_linear/clipper.py`. +2. Bounds extraction covering **every** prior type. Confirmed present in the + reference model: `UniformPrior` (finite both sides), `TruncatedGaussianPrior` + (finite both sides, e.g. `(-1, 1)` for `ell_comps`), `GaussianPrior` + (`±inf` — must pass through untouched). Audit the rest (`LogUniformPrior`, + `LogGaussianPrior`, any `Constant`/deterministic entries). +3. Wire into `AbstractMultiStartGradient._fit`, applied after + `optax.apply_updates`, **opt-in**. +4. Wire into `LBFGS`, passing `bounds=` through to `optimize.minimize`, + **opt-in**. Only valid for bound-supporting methods (`L-BFGS-B`, `TNC`, + `SLSQP`) — guard or warn for plain `BFGS`. +5. `clipper: AbstractClipper = None` constructor arg on the searches, resolved + like `initializer`. + +**Default is `ClipperNone`, and PR 1 must be bit-identical with it.** Follow the +precedent set by PyAutoFit#1475, whose models declaring no constraint +short-circuit to bit-identical behaviour. Flipping the default is a real +behaviour change that shifts every stored multi-start benchmark, which is exactly +the comparability argument PyAutoFit#1472 made when it deferred its own policy +change. + +## Scope — PR 2 (separate prompt, file after PR 1 lands) + +Flip `MultiStartGradient`'s default to `ClipperPriorBox`, **with** the benchmark +re-baseline, plus the momentum-reset-on-clip decision informed by how bad the +pinning actually is at production budget. + +## Traps, measured + +- **Parameter ordering is load-bearing and silent if wrong.** + `model.priors_ordered_by_id` was used for the prototype and lined up correctly + with `model.instance_from_vector`, but a mismatch would clip the *wrong + parameter* with no error. Assert the correspondence in a test rather than + trusting it. +- **Boundary semantics.** Decide and document whether `log_prior` at *exactly* the + limit is finite. The prototype inset by `1e-6` of the box width to stay strictly + inside; that margin is a guess and should be a justified constant. +- **Pinning is correct behaviour, not a bug.** Where the likelihood genuinely + prefers a value outside the prior, a clipped lane sitting on the bound is the + correct MAP answer under the declared prior. It is worth surfacing (it says the + prior is fighting the data) rather than hiding. In the reference cell the shear + escapes were mixed-sign (`+0.353`, `-0.341`, `+0.301`, `-0.312`), which reads + more like a poorly-constrained parameter diffusing out than a true value sitting + outside. +- **Clipping does not fix every death.** 5/16 lanes still died in the prototype; + those are the NaN-gradient population (likelihood NaN in the *jitted* path, + which the `Fitness` guard maps to `-inf` and whose `where` makes the gradient + NaN). Separate mechanism, do not expect this task to remove it. + +## Two incidental bugs found while investigating — do not lose these + +Both surfaced only because clipping let lanes *survive*, i.e. on a code path this +cell had apparently never taken: + +1. **`float32` is not JSON serializable in result output.** + `@PyAutoFit/autofit/non_linear/paths/directory.py:80` `save_json` raises + `TypeError: Object of type float32 is not JSON serializable` at the end of a + successful clipped run. Did not fire on the baseline runs, where 14/16 lanes + were dead. File separately if confirmed. +2. **A crashed run poisons the next run of the same name.** The half-written + output left by (1) caused the next search with the same `name` to fail with + `JSONDecodeError` while trying to resume — a 4-second no-op run that *looked + like* a clean result (zero deaths, because zero steps). This is a new form of + the cached-result hazard already recorded in + `complete/2026/08/multistart-nan-step-diagnostics.md`. + +## Deliberately out of scope + +- **NUTS.** Divergence, not lane death; may need a transform or a soft wall rather + than projection. Its own task. +- **Unit-cube stepping.** The more principled long-term fix — PyAutoFit's prior + machinery is already unit-cube and the nested samplers work that way, and it + would also normalise parameter scales (`einstein_radius ∈ [0,8]` alongside + `ell_comps ∈ [-1,1]`). Rejected *for now* on three grounds: a logit + reparameterisation sends the optimum to infinity when it genuinely sits on a + boundary, which this cell demonstrably has; the inverse-CDF transform for + non-uniform priors has `∂θ/∂u -> ∞` at the cube faces, trading one numerical + hazard for another; and it invalidates every stored benchmark. If pursued, note + that reparameterising the *search path* does not move the optimum **provided the + objective is still the physical-space posterior evaluated at `θ(u)`** — optimise + the density *of u* instead and the Jacobian makes the MAP non-invariant, which + fails silently. +- **Changing `resurrect` defaults.** Not the fix: a redrawn lane walks out again. + +## Testing + +- Bounds extraction per prior type, including `±inf` passthrough for `GaussianPrior`. +- Ordering assertion (see traps). +- `ClipperNone` is bit-identical: same seed, same final parameters, on both + `MultiStartGradient` and `LBFGS`. +- A lane deliberately stepped across a boundary is projected back inside, and the + returned mask names exactly the crossed coordinates. +- `LBFGS` passes bounds through and rejects/warns for non-bound-supporting methods. +- Regression: with `ClipperPriorBox` on a model with a tight `UniformPrior`, the + value-NaN rate falls substantially. The reference numbers above are CPU/float32, + single seed — assert a direction and a large margin, not an exact figure.