From 8863797bc69b3685ec33844ea1bbba51695e55db Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 13:50:27 +0000 Subject: [PATCH 01/20] intake: count frozen lanes in the multi-start gradient search MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Files the PyAutoFit instrumentation task uncovered while assessing the ell_comps flat-gradient hazard under JAX sampling. _nan_lane_counts counts value-NaN and gradient-NaN lanes, but a lane on a finite zero-gradient plateau (the |ell_comps| >= 0.999 clamp) escapes both, is absorbing, and can false-trigger the autoconv convergence check. Detection first: search results store no per-start traces today, so whether real runs enter the region is currently unanswerable. Scoped as instrumentation only — the penalty term is a separate follow-up. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- dashboard.md | 7 +- draft/feature/autofit/frozen_lane_counter.md | 67 ++++++++++++++++++++ 2 files changed, 71 insertions(+), 3 deletions(-) create mode 100644 draft/feature/autofit/frozen_lane_counter.md diff --git a/dashboard.md b/dashboard.md index 45231d88..cede1068 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | [In flight](#in-flight) (`active/`) | 7 | | [Parked](#parked) (`parked.md`) | 6 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 136 | +| [Backlog](#backlog) (`draft/`) | 137 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) @@ -94,7 +94,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**136** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**137** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
bug — 40 @@ -143,7 +143,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-feature — 25 +feature — 26 - [Make draft/ staleness detectable — `intake reconcile` measured, and the](draft/feature/pyautomind/draft_staleness_detection_signals.md) — pyautomind · medium · supervised · high - [LACosmic per-frame CR masking option + decouple PSF-star pass from](draft/feature/pyautoreduce/lacosmic_cr_option_and_star_pass_decoupling.md) — pyautoreduce · medium · supervised · high @@ -152,6 +152,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. - [Give PyAutoFit searches a `seed` — today no search can](draft/feature/autofit/search_seed_reproducibility.md) — autofit · medium · supervised · medium - [Can create a list of InversionMatrix objects for each dataset](draft/feature/autoarray/multiwavelength_inversion.md) — autoarray · medium · supervised · normal - [The project @z_projects/ic50_workspace is our IC50 use case which we](draft/feature/autofit/ep_lbfgs_jax.md) — autofit · medium · safe · normal +- [Count frozen lanes in the multi-start gradient search](draft/feature/autofit/frozen_lane_counter.md) — autofit · medium · supervised · normal - [Tune cluster-scale JOSS benchmarks toward their 5-minute targets](draft/feature/autolens_workspace/joss_cluster_benchmark_tuning.md) — autolens_workspace · medium · supervised · normal - [Token-light wiki index over the complete/ archive](draft/feature/pyautomind/complete_archive_wiki.md) — pyautomind · medium · supervised · normal - [The imaging `features/advanced/los_halos` example needs improving and padding out before](draft/feature/workspaces/group_los_halos.md) — workspaces · medium · safe · normal diff --git a/draft/feature/autofit/frozen_lane_counter.md b/draft/feature/autofit/frozen_lane_counter.md new file mode 100644 index 00000000..920afb65 --- /dev/null +++ b/draft/feature/autofit/frozen_lane_counter.md @@ -0,0 +1,67 @@ +# Count frozen lanes in the multi-start gradient search + +Type: feature +Target: PyAutoFit +Repos: +- PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: formalised + +@PyAutoFit + +Add a third lane counter to `AbstractMultiStartGradient._nan_lane_counts` +(`autofit/non_linear/search/mle/multi_start_gradient/search.py:219-255`). + +It counts two disjoint failure modes today — **value-NaN** (likelihood +undefined; the death `resurrect` triggers on) and **gradient-NaN** (defined but +not differentiable; `apply_if_finite` zeroes the update). A third mode escapes +both: a lane whose value is finite and whose gradient is finite but **exactly +zero**, sitting on a saturating plateau. + +## Why it matters + +The reference case is the ellipticity magnitude clamp `jax.lax.min(fac, 0.999)` +in PyAutoGalaxy's `convert.py:71-77`. Past `|ell_comps| >= 0.999` the axis ratio +pins at `q = 5.0025e-4` and the derivative w.r.t. both components is exactly +zero. For a multi-start gradient search that is an absorbing trap: + +- Starts are safe: `_broad_starts` draws in the unit cube at `(0.15, 0.85)`, + capping start magnitude near 0.44 under the default `TruncatedGaussian(0, 0.3)`. +- But `optax.apply_updates(params, updates)` (`search.py:743`) steps the physical + vector with no re-projection into prior limits, so trajectories can walk in. +- Once inside there is no restoring force, so the lane cannot leave. +- `apply_if_finite` and `resurrect` are both no-ops here — value and gradient are + finite. +- `multi_start_prodigy_autoconv` runs `check_for_convergence=True`, so frozen + lanes flatten the figure of merit and can false-trigger early stopping. + +A frozen lane is therefore indistinguishable from a converged one in the +figure-of-merit trace, which is exactly the hazard `_nan_lane_counts` was +written to expose for the gradient-NaN case. + +## Scope + +Instrumentation only. Add the count alongside the existing two, accumulate it +across steps, record it into `search_internal`, restore it on resume, and report +it on the progress line — mirroring `n_value_nan_lane_steps` and +`n_grad_nan_lane_steps` exactly. Keep it pure-NumPy and free of search state so +it stays directly testable like its two siblings. + +The search must produce identical results with the counter present. Do not add a +penalty term, and do not touch `resurrect`, `apply_if_finite`, the convergence +check, stepping behaviour, or the clamp itself. + +## Decisions to make + +- Keep the buckets disjoint: a lane already counted as value-NaN or gradient-NaN + must not also count as frozen. +- Define "exactly zero" — all coordinates versus any coordinate, and exact `== 0` + versus a threshold. An exact test is the honest default for a `lax.min` + plateau, but the realistic case is a partially frozen lane (ellipticity dead, + other coordinates live), which is the more useful signal. +- Do not force a per-step device sync if it costs measurable run time. The + existing NaN accounting measured 0.0004% of step time; stay in that class. + + From a0e6c8f59aecf4c2af76d0ba8f1bb78b05c0a538 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 14:12:13 +0000 Subject: [PATCH 02/20] prompt: correct the frozen-lane detector spec from a JAX reproduction Ran a toy reproduction (32 starts, 400 Prodigy steps, PyAutoGalaxy's clamp verbatim, apply_if_finite + unbounded apply_updates) and it falsified the detector this prompt originally specified. The clamp kills only the radial derivative; the angle still comes from an unclamped arctan2, so in general position both ell_comps components carry a large non-zero gradient while their radial projection is ~1e-12. A component-wise zero test caught 0 of the 17 trapped lanes. Per-parameter prior-limit escape caught 17/17 with no false positives, needs no model semantics, and detects the actual cause. Also records severity: the trap wasted 53% of the start budget but the surviving lanes still recovered the truth, so it is a robustness cost rather than a correctness fault except where n_starts is small or the good basin is rare. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- draft/feature/autofit/frozen_lane_counter.md | 76 ++++++++++++++++---- 1 file changed, 62 insertions(+), 14 deletions(-) diff --git a/draft/feature/autofit/frozen_lane_counter.md b/draft/feature/autofit/frozen_lane_counter.md index 920afb65..1b1031bb 100644 --- a/draft/feature/autofit/frozen_lane_counter.md +++ b/draft/feature/autofit/frozen_lane_counter.md @@ -17,27 +17,31 @@ Add a third lane counter to `AbstractMultiStartGradient._nan_lane_counts` It counts two disjoint failure modes today — **value-NaN** (likelihood undefined; the death `resurrect` triggers on) and **gradient-NaN** (defined but not differentiable; `apply_if_finite` zeroes the update). A third mode escapes -both: a lane whose value is finite and whose gradient is finite but **exactly -zero**, sitting on a saturating plateau. +both: a lane whose value and gradient are both finite, but which sits on a +saturating plateau with **no gradient along the saturated direction**, so it can +never leave. ## Why it matters The reference case is the ellipticity magnitude clamp `jax.lax.min(fac, 0.999)` in PyAutoGalaxy's `convert.py:71-77`. Past `|ell_comps| >= 0.999` the axis ratio -pins at `q = 5.0025e-4` and the derivative w.r.t. both components is exactly -zero. For a multi-start gradient search that is an absorbing trap: +pins at `q = 5.0025e-4`, so the likelihood is exactly constant along the radial +direction — measured identical to 10 significant figures at `|ell_comps|` of +1.0, 1.2 and 5.0. For a multi-start gradient search that is an absorbing trap: - Starts are safe: `_broad_starts` draws in the unit cube at `(0.15, 0.85)`, capping start magnitude near 0.44 under the default `TruncatedGaussian(0, 0.3)`. + Measured start range in the toy run was 0.087 to 0.362. - But `optax.apply_updates(params, updates)` (`search.py:743`) steps the physical - vector with no re-projection into prior limits, so trajectories can walk in. -- Once inside there is no restoring force, so the lane cannot leave. + vector with no re-projection into prior limits, so trajectories walk in — 17 of + 32 lanes did, reaching `|ell_comps|` as high as 6.78. +- Once inside there is no radial restoring force, so the lane cannot come back. - `apply_if_finite` and `resurrect` are both no-ops here — value and gradient are finite. -- `multi_start_prodigy_autoconv` runs `check_for_convergence=True`, so frozen +- `multi_start_prodigy_autoconv` runs `check_for_convergence=True`, so trapped lanes flatten the figure of merit and can false-trigger early stopping. -A frozen lane is therefore indistinguishable from a converged one in the +A trapped lane is therefore indistinguishable from a converged one in the figure-of-merit trace, which is exactly the hazard `_nan_lane_counts` was written to expose for the gradient-NaN case. @@ -53,14 +57,58 @@ The search must produce identical results with the counter present. Do not add a penalty term, and do not touch `resurrect`, `apply_if_finite`, the convergence check, stepping behaviour, or the clamp itself. -## Decisions to make +## Use prior-limit escape, not a zero-gradient test + +A JAX toy reproduction (32 starts, 400 Prodigy steps, PyAutoGalaxy's clamp +verbatim, `apply_if_finite` + unbounded `apply_updates`, truth at +`|ell_comps| = 0.90`) settled the detector question empirically. **Do not +implement a component-wise zero-gradient test — it does not work.** + +Measured on the 17 of 32 lanes that ended on the plateau: + +| candidate detector | caught | false pos | +|---|---:|---:| +| gradient component exactly zero | **0/17** | 0 | +| all gradient components exactly zero | **0/17** | 0 | +| radial derivative exactly zero | 6/17 | 0 | +| **per-parameter prior-limit escape** | **17/17** | **0** | + +The reason is that the clamp kills only the *radial* derivative. The angle still +comes from an unclamped `arctan2`, so in general position both `ell_comps` +components carry a large non-zero gradient while their radial projection is zero +— at `ell_comps = (0.867, 0.593)` the components are `+2.7e4` and `-3.9e4` while +the radial projection is `-3.4e-12`. Only on the measure-zero axis where one +component is exactly zero does a component itself read zero. Floating-point +residue from the rotation is also why the radial test scores 6/17 rather than +17/17: the projection is ~1e-12, not an exact zero. + +So detect the **cause** — a lane that has left its priors' support, which is +possible at all only because `apply_updates` steps the physical vector with no +re-projection. That is fully generic (every `Prior` already carries +`lower_limit`/`upper_limit`), needs no model semantics, no gradient inspection, +and no tolerance. + +Record its limit honestly in the docstring: it is a proxy for the cause, not the +effect. It caught every trapped lane here because Prodigy overshoots grossly +(trapped `|ell_comps|` ran 1.62 to 6.78, max component 6.45), not because it is +complete. A lane in the corner region — both components inside `(-1, 1)` but +magnitude above 1, which is the whole of the `4 - pi` area between the unit disc +and the unit square — is beyond the clamp yet inside every per-parameter limit, +and this detector will miss it. + +## Severity, for prioritisation + +The same run shows the trap **wastes budget rather than corrupting results**: +17/32 lanes died on the plateau, yet the surviving lanes still recovered the +truth (best lane `|ell_comps| = 0.9001`, logL −427.5 against a truth logL of +−430.4). Multi-start redundancy absorbs it. It becomes a correctness risk only +when `n_starts` is small or the good basin is rare — which is exactly the +pixelized-mesh regime the counter is meant to observe. + +## Other decisions - Keep the buckets disjoint: a lane already counted as value-NaN or gradient-NaN - must not also count as frozen. -- Define "exactly zero" — all coordinates versus any coordinate, and exact `== 0` - versus a threshold. An exact test is the honest default for a `lax.min` - plateau, but the realistic case is a partially frozen lane (ellipticity dead, - other coordinates live), which is the more useful signal. + must not also count as escaped. - Do not force a per-step device sync if it costs measurable run time. The existing NaN accounting measured 0.0004% of step time; stay in that class. From a38592c7b6210e411e825bf1fc8d08399eb24fde Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 14:30:07 +0000 Subject: [PATCH 03/20] prompt: reuse the guard predicate as a traced boolean, not an exception Probed whether autogalaxy's FitException guard can be reused under JAX. It cannot raise -- validate.py:153-154 returns early for non-concrete scalars, and a plain raise on a traced condition gives TracerBoolConversionError, so the escape hatch is load-bearing rather than an oversight. checkify does survive vmap + value_and_grad with finite values and grads, but collapses a batch to one abort-shaped error, so it cannot serve a per-lane counter. The guard's predicate returned as a traced boolean does, and scores 20/20 against 17 real trapped lanes plus 3 synthetic corner-region lanes where prior-limit escape scores 17/20. Records that the clamp threshold (0.999) and the guard threshold (1.0) differ, and the gap is reachable: the radial derivative is already exactly zero at |ell_comps| = 0.9995 while validate_ell_comps still calls the point valid. Leaves the widen-to-validity-channel decision open for start_dev rather than silently re-scoping the header. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- draft/feature/autofit/frozen_lane_counter.md | 64 +++++++++++++++++++- 1 file changed, 62 insertions(+), 2 deletions(-) diff --git a/draft/feature/autofit/frozen_lane_counter.md b/draft/feature/autofit/frozen_lane_counter.md index 1b1031bb..38f6ce89 100644 --- a/draft/feature/autofit/frozen_lane_counter.md +++ b/draft/feature/autofit/frozen_lane_counter.md @@ -57,7 +57,7 @@ The search must produce identical results with the counter present. Do not add a penalty term, and do not touch `resurrect`, `apply_if_finite`, the convergence check, stepping behaviour, or the clamp itself. -## Use prior-limit escape, not a zero-gradient test +## Detector: use the saturation predicate, not a zero-gradient test A JAX toy reproduction (32 starts, 400 Prodigy steps, PyAutoGalaxy's clamp verbatim, `apply_if_finite` + unbounded `apply_updates`, truth at @@ -82,7 +82,58 @@ component is exactly zero does a component itself read zero. Floating-point residue from the rotation is also why the radial test scores 6/17 rather than 17/17: the projection is ~1e-12, not an exact zero. -So detect the **cause** — a lane that has left its priors' support, which is +### Reuse the guard's predicate, not its exception + +`validate_ell_comps` already owns the authoritative constraint, but it cannot +raise under a trace and deliberately does not try — `validate.py:153-154` returns +early for non-concrete scalars. That escape hatch is load-bearing: a plain +`raise` on a traced condition gives `TracerBoolConversionError`, so without it +every jitted likelihood would crash rather than sample. + +Measured on the three candidate mechanisms: + +| mechanism | works under `vmap` + `grad`? | shape | +|---|---|---| +| plain `raise` | no — `TracerBoolConversionError` | — | +| `jax.experimental.checkify` | yes, values/grads stay finite | **one error per batch**, abort-shaped | +| **guard predicate as a traced boolean** | yes | **per-lane verdict** | + +`checkify` is the genuine JAX exception mechanism and it does survive +`vmap`+`value_and_grad`, but it collapses a batch to a single error and is built +to abort. That is the wrong shape here: the run must continue so the surviving +lanes finish, and the counter needs 32 independent verdicts, not one. + +The predicate as a traced boolean gives exactly that, returned alongside the FoM +— no exception machinery at all. Scored against the 17 real trapped lanes plus +three synthetic corner-region lanes (both components inside `(-1, 1)`, magnitude +above 1): + +| detector | caught | missed | +|---|---:|---:| +| per-parameter prior-limit escape | 17/20 | 3 | +| **saturation predicate `|ell_comps| >= 0.999`** | **20/20** | **0** | + +**Use the clamp's threshold (0.999), not the guard's (1.0).** They differ, and +the gap is reachable: at `|ell_comps| = 0.9995` and `0.99999` the radial +derivative is already exactly zero while `validate_ell_comps` still calls the +point valid. A detector keyed to the guard's `magnitude_squared >= 1.0` misses +that annulus; one keyed to the clamp does not. + +This supersedes the prior-limit-escape recommendation below, which is provably +incomplete — the corner region it misses is the whole `4 - pi` area between the +unit disc and the unit square. + +### Scope consequence + +Asking the model "is this instance saturated?" is a **validity channel** between +PyAutoFit and the profile libraries, not something PyAutoFit can answer alone. +It is also the same hook the later penalty term needs, so building it once serves +both. If this task stays PyAutoFit-only it must fall back to prior-limit escape +and accept the corner-region miss; see the open question at the end. + +### Superseded: prior-limit escape + +Detect the **cause** — a lane that has left its priors' support, which is possible at all only because `apply_updates` steps the physical vector with no re-projection. That is fully generic (every `Prior` already carries `lower_limit`/`upper_limit`), needs no model semantics, no gradient inspection, @@ -112,4 +163,13 @@ pixelized-mesh regime the counter is meant to observe. - Do not force a per-step device sync if it costs measurable run time. The existing NaN accounting measured 0.0004% of step time; stay in that class. + + +## Open question for start_dev + +Whether to widen this task to the validity channel (PyAutoFit + PyAutoGalaxy, +complete detector, shared with the later penalty term) or keep it PyAutoFit-only +(prior-limit escape, misses the corner region). Widening changes the header: +`Repos:` gains PyAutoGalaxy and difficulty rises from `medium`. Decide before +issuing, not during. From b6e5ff5ba9e06db9978c45178a5fa76c15bc3d0c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 14:48:08 +0000 Subject: [PATCH 04/20] prompt: record how the constraint attaches, and the threshold drift it fixes No user-facing model-composition call is needed: PyAutoFit already discovers class-level metadata off the wrapped class (prior_model.py:200 reads __default_fields__), and validate_ell_comps has exactly one call site, at geometry_profiles.py:237 in EllProfile.__init__ -- the single base every elliptical profile inherits. Records that the predicate must be extracted and shared rather than restated, and that the drift this prevents already exists today: the clamp's 0.999 in convert.py and the guard's 1.0 in validate.py sit in different files with nothing relating them, which is exactly the reachable annulus where the radial gradient is dead while the guard still reports the point valid. Also records the measured cost of the parameter-only validity channel (+23 HLO lines on the toy; timing overhead below noise) and its ceiling: it cannot see likelihood-internal hazards such as NNLS active-set pinning. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- draft/feature/autofit/frozen_lane_counter.md | 25 ++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/draft/feature/autofit/frozen_lane_counter.md b/draft/feature/autofit/frozen_lane_counter.md index 38f6ce89..500422a4 100644 --- a/draft/feature/autofit/frozen_lane_counter.md +++ b/draft/feature/autofit/frozen_lane_counter.md @@ -123,6 +123,31 @@ This supersedes the prior-limit-escape recommendation below, which is provably incomplete — the corner region it misses is the whole `4 - pi` area between the unit disc and the unit square. +### How it attaches (no new composition call) + +The model-composition API does not gain a user-facing call — `af.Model(al.mp.Isothermal)` +is unchanged. PyAutoFit already discovers class-level metadata off the wrapped +class during composition (`prior_model.py:200` reads `cls.__default_fields__`), +and that is the pattern to follow. + +Placement is clean because `validate_ell_comps` has exactly **one** call site, +`geometry_profiles.py:237` in `EllProfile.__init__` — the single base every +elliptical light and mass profile inherits. The declaration goes on that class, +beside the existing call. + +Do **not** let this become a second statement of the constraint. Extract the +predicate out of `validate_ell_comps` as a pure `xp`-generic function; leave +`validate_ell_comps` raising on concrete scalars as it does today, now calling +that predicate; point the class-level declaration at the same function. + +Note that the drift this guards against **already exists**: the clamp is `0.999` +in `convert.py:71-77`, the guard is `1.0` in `validate.py:158`, in different +files with nothing relating them. That gap is precisely the reachable annulus +where the radial gradient is already dead while the guard still calls the point +valid. The two thresholds answer different questions and both should survive — +but their relationship should be stated in one place, which this work is the +opportunity to do. + ### Scope consequence Asking the model "is this instance saturated?" is a **validity channel** between From 0aeb6ac73e84e5d06c9f4c7cc416a9925e3e8b65 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 14:48:41 +0000 Subject: [PATCH 05/20] prompt: add the validity-channel cost and ceiling b6e5ff5b claimed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit message said it recorded the measured cost and the likelihood-internal ceiling; it did not — that material had only been discussed, not written. Adding it here rather than rewriting a pushed message. Records that nothing flows up through the likelihood (predicate is a pure function of the parameter vector, gradients bit-identical with the likelihood untouched), the measured +23 HLO lines with wall-clock overhead below the noise band, and the ceiling: parameter-only predicates cannot see NNLS active-set pinning or conditioning-floor behaviour. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- draft/feature/autofit/frozen_lane_counter.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/draft/feature/autofit/frozen_lane_counter.md b/draft/feature/autofit/frozen_lane_counter.md index 500422a4..ef6706ad 100644 --- a/draft/feature/autofit/frozen_lane_counter.md +++ b/draft/feature/autofit/frozen_lane_counter.md @@ -148,6 +148,25 @@ valid. The two thresholds answer different questions and both should survive — but their relationship should be stated in one place, which this work is the opportunity to do. +### What it costs, and what it cannot see + +Nothing flows *up* through the likelihood. The saturation predicate is a pure +function of the parameter vector, so it is evaluated at the top beside the +likelihood, which is left untouched — measured with the toy's +`log_likelihood` imported unmodified, gradients bit-identical (`atol=0`). + +Cost on the toy: +23 HLO lines (700 -> 723, +3.3%). Wall-clock overhead measured +at +40 us on a ~1090 us call, which is **below the noise band** on a shared CPU +(interleaved burst spreads overlapped) — treat it as an upper bound, and expect +it to be proportionally smaller against a real pixelized likelihood. + +The ceiling is worth stating before anyone assumes one mechanism covers the +whole hazard index: this works *only* for parameter-only properties. The tier-2 +likelihood hazards — which basis components the NNLS active set pinned at zero, +how a conditioning floor bit against real flux — are genuine likelihood-internal +state, are not recoverable from the parameter vector, and would need real upward +plumbing. + ### Scope consequence Asking the model "is this instance saturated?" is a **validity channel** between From 6de039cdf0d260de9bfdce8b3d5deaf08828541b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 14:59:14 +0000 Subject: [PATCH 06/20] prompt: point the attachment at Model introspection, not __default_fields__ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An earlier revision cited __default_fields__ as the pattern to follow, which would send an implementer at the wrong hook. It is a narrow escape hatch consulted only when make_prior returns a ConfigException, with two usages, both internal message classes; it marks "not a model parameter" rather than registering a constraint. The real enabler is that Model.__init__ already introspects the class (gather_namespaces, get_type_hints, per-argument prior resolution), so this is one more lookup in an existing mechanism. Also names the closer relative, add_assertion: the right concept, attached per model instance instead of per class and raising FitException. Reframes the work as assertions with two changes — class-declared, traced predicate — rather than a new validity subsystem. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- draft/feature/autofit/frozen_lane_counter.md | 27 +++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/draft/feature/autofit/frozen_lane_counter.md b/draft/feature/autofit/frozen_lane_counter.md index ef6706ad..7dbac4ab 100644 --- a/draft/feature/autofit/frozen_lane_counter.md +++ b/draft/feature/autofit/frozen_lane_counter.md @@ -126,9 +126,30 @@ unit disc and the unit square. ### How it attaches (no new composition call) The model-composition API does not gain a user-facing call — `af.Model(al.mp.Isothermal)` -is unchanged. PyAutoFit already discovers class-level metadata off the wrapped -class during composition (`prior_model.py:200` reads `cls.__default_fields__`), -and that is the pattern to follow. +is unchanged. + +What makes that possible is that `Model.__init__` is **already** a class +introspection site: it runs `gather_namespaces(cls)` and +`typing.get_type_hints(cls.__init__, ...)`, walks the constructor signature and +resolves a prior per argument. Per-class knowledge is already gathered there, so +this is one more lookup in an existing mechanism, not a new one. + +Do **not** build this on `__default_fields__`. That is a narrow escape hatch — +`prior_model.py:200` consults it only when `make_prior` returns a +`ConfigException`, and it has exactly two usages (`messages/normal.py:412`, +`truncated_normal.py:485`, both `("log_norm", "id_")`). It marks "this argument +is not a model parameter"; it is not a constraint registry. + +The real relative is **`add_assertion`** (`abstract.py:441`), PyAutoFit's +existing constraint concept. It is wrong here on exactly two counts: it attaches +per *model instance* rather than per class (so every elliptical profile would +need the user to remember it), and it raises `FitException`, which is numpy-only +for the reasons above. + +So frame the work as **assertions with two changes** — class-declared rather than +per-model, and a traced predicate rather than a raise — not as a new validity +subsystem. The two should likely share a home rather than sit as unrelated +features. Placement is clean because `validate_ell_comps` has exactly **one** call site, `geometry_profiles.py:237` in `EllProfile.__init__` — the single base every From 8e039b6518cdb016c9cfc65542ad31b0779dd627 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 15:31:06 +0000 Subject: [PATCH 07/20] prompt: record what the penalty inherits, and the shipped status The counter reads only the sign of the violation measure, so it is scale-free and carries no lambda. That leaves two properties of the measure untested by anything shipped, both of which land on the penalty task: its units are the constraint's own rather than the figure of merit's, which is what a lambda has to absorb; and model_constraint_from_vector reduces components with maximum, which is correct for counting but would let one constraint silently dominate another in different units once the measure is multiplied into the FoM. Also records that the counter shipped as PyAutoFit #1475 with CI green, and that PyAutoGalaxy declares no constraint yet so it reads zero on real lens models until EllProfile opts in. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- draft/feature/autofit/frozen_lane_counter.md | 30 ++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/draft/feature/autofit/frozen_lane_counter.md b/draft/feature/autofit/frozen_lane_counter.md index 7dbac4ab..444d61f5 100644 --- a/draft/feature/autofit/frozen_lane_counter.md +++ b/draft/feature/autofit/frozen_lane_counter.md @@ -238,3 +238,33 @@ complete detector, shared with the later penalty term) or keep it PyAutoFit-only (prior-limit escape, misses the corner region). Widening changes the header: `Repos:` gains PyAutoGalaxy and difficulty rises from `medium`. Decide before issuing, not during. + +## What the penalty inherits (shipped counter is unaffected) + +The counter uses only `violation > 0.0` — a sign test, so it is scale-free and +no lambda exists anywhere in it. Two properties of the shipped measure are +therefore **untested by anything shipped**, because only its sign is ever read, +and both land on the penalty task: + +- **Units.** `max(|ell_comps| - 0.999, 0)` is in units of ellipticity magnitude; + the figure of merit is in log-likelihood. That mismatch is exactly what a + lambda has to absorb, and it is why a single constant cannot work across cells + whose likelihood scales differ by orders of magnitude (point-source ~10s, + pixelized ~30,000s). +- **The reduction.** `model_constraint_from_vector` combines components with + `xp.maximum`. Correct for counting — any violation makes the lane constrained + — but wrong for a penalty: two constraints in different units (ellipticity + magnitude vs a radius, say) would be reduced by `max`, so whichever is + numerically larger silently dominates. A penalty likely wants per-constraint + lambdas, or measures normalised to a common scale, rather than one max. + +Neither is a defect in the counter. Both are decisions to make before the +measure is multiplied into the figure of merit. + +## Status + +The counter shipped: PyAutoFit PR #1475 (branch +`claude/jax-sampling-flat-gradients-ptmqnl`), CI green on 3.12, 3.13 and docs. +1745 tests pass, +15 new. PyAutoGalaxy declares no constraint yet, so the +counter reads zero on real lens models until `EllProfile` opts in — a ~4-line +method at the site that already calls `validate_ell_comps`. From ffd8e80d37f9a0ca54daedc46616af7ccc27492e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 15:59:38 +0000 Subject: [PATCH 08/20] prompt: record frozen-lane-counter as shipped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both halves merged: PyAutoFit #1475 (004f798a) adds the counter and the class-declared __model_constraint__ protocol; PyAutoGalaxy #572 (695b27c5) declares the ell_comps saturation region on EllProfile and gives the clamp one definition instead of three bare literals. The record keeps the reasoning that would otherwise be re-derived: that a component-wise zero-gradient detector caught 0 of 17 trapped lanes and why, that the exception could not be reused and checkify is the wrong shape, that the clamp's 0.999 and the guard's 1.0 answer different questions with a reachable annulus between them, and the traps (off-centre grid, spherical inheritance, black not enforced). Also records what the penalty term inherits — units, the max reduction, moving the term inside the differentiated call — and the honest open tail: whether the real Prodigy mesh/MGE runs entered the region is still unanswered, because those cells were never persisted and the stored records carry no per-start traces. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- .../2026/08/frozen-lane-counter.md | 143 ++++++++++++++++++ complete/index.md | 3 +- dashboard.md | 7 +- 3 files changed, 148 insertions(+), 5 deletions(-) rename draft/feature/autofit/frozen_lane_counter.md => complete/2026/08/frozen-lane-counter.md (64%) diff --git a/draft/feature/autofit/frozen_lane_counter.md b/complete/2026/08/frozen-lane-counter.md similarity index 64% rename from draft/feature/autofit/frozen_lane_counter.md rename to complete/2026/08/frozen-lane-counter.md index 444d61f5..8c151f15 100644 --- a/draft/feature/autofit/frozen_lane_counter.md +++ b/complete/2026/08/frozen-lane-counter.md @@ -1,3 +1,146 @@ +- library-prs: https://github.com/PyAutoLabs/PyAutoFit/pull/1475, https://github.com/PyAutoLabs/PyAutoGalaxy/pull/572 +- merge-commits: PyAutoFit `004f798a89e621ab7320b46fe0494201720260fd`; PyAutoGalaxy `695b27c545fa1328a34366d56e579b1bbc55f95e` +- issue: none — filed straight from investigation via `/intake`, never issued +- summary: Added a third disjoint lane counter to the multi-start gradient search + for lanes that are finite *and* differentiable but sit on a saturating plateau + with no gradient along the saturated direction, so they can never leave. Both + existing counters are blind to them and the flat figure of merit reads as + convergence. Detection reads a new class-declared `__model_constraint__` + protocol; `EllProfile` declares the `ell_comps` saturation region, reaching + every elliptical light and mass profile through the one site where `ell_comps` + is assigned. +- validation: PyAutoFit 1745 passed +15 new, CI green on 3.12/3.13/docs; + PyAutoGalaxy 1101 passed 1 skipped +12 new, CI green on 3.12/3.13/docs. Five + PyAutoFit failures are pre-existing, verified identical on a stashed clean tree + (missing `astropy`; a nautilus pool test). +- release: not performed; both merged PRs remain in the pending-release queue. + +## The finding that shaped the design + +A component-wise zero-gradient test — the obvious detector, and the one this +task was originally filed to build — **does not work**, and a JAX reproduction +proved it before any code was written. The clamp kills only the *radial* +derivative; the angle still comes from an unclamped `arctan2`, so in general +position both `ell_comps` components carry a large non-zero gradient while their +radial projection is zero only to floating-point residue. Scored against 17 +genuinely trapped lanes: + +| candidate detector | caught | +|---|---:| +| gradient component exactly zero | 0/17 | +| all components exactly zero | 0/17 | +| radial derivative exactly zero | 6/17 | +| per-parameter prior-limit escape | 17/17 | +| declared model constraint | 17/17 + corner region | + +Prior-limit escape scores well only because Prodigy overshoots grossly (trapped +magnitudes ran 1.62 to 6.78). It provably cannot see the corner region — both +components inside `(-1, 1)` with magnitude above 1, the whole `4 - pi` area +between the unit disc and the unit square. + +## Why the exception could not be reused + +`validate_ell_comps` already owns the geometry but signals by raising, which +needs a concrete boolean. A `raise` on a traced condition gives +`TracerBoolConversionError`, which is why `validate.py:153-154` returns early for +non-concrete scalars — that escape hatch is load-bearing, not an oversight. +`jax.experimental.checkify` does survive `vmap` + `value_and_grad` with finite +values and gradients, but collapses a batch to one abort-shaped error, so it +cannot serve a per-lane counter that must let surviving lanes finish. The +predicate returned as a traced value gives 32 independent verdicts with no +exception machinery. + +## Two thresholds, and drift that already existed + +The clamp saturates at `0.999` (`convert.py`); the guard rejects at `1.0` +(`validate.py`). They answer different questions — where the *gradient* dies +versus where the *geometry* stops meaning anything — and the annulus between +them is reachable: at magnitude 0.9995 the radial derivative is already exactly +zero while `validate_ell_comps` still calls the point valid. The constraint is +therefore keyed to the clamp, not the guard. + +The clamp was a bare literal at **three** sites — `convert.py`'s JAX and NumPy +branches and the Sersic Cartesian eccentric-radius path from PyAutoGalaxy#571 — +with the guard's `1.0` in a fourth file and nothing relating them. +`ELL_COMPS_MAGNITUDE_CLAMP` now states it once. Value unchanged at every site. + +## Architecture + +The protocol is **assertions with two changes**: declared on the class rather +than attached per model instance, and evaluated as a traced non-negative measure +rather than raised. It attaches through `Model.__init__`, which is already a +class-introspection site (`gather_namespaces`, `get_type_hints`, per-argument +prior resolution) — not through `__default_fields__`, which is a narrow +`ConfigException` escape hatch with two usages and is *not* a constraint +registry. No user-facing composition call changes. + +Nothing flows *up* through the likelihood: the constraint is a pure function of +the parameter vector, evaluated beside `fitness.call`, which is untouched. It +rides the fused `value_and_grad` as a fourth output on a device→host sync that +already happens. Cost is a flat +20 HLO lines at every scale tested (grid +31→256, starts 32→128) — fixed, not proportional to likelihood cost. + +## Evidence the counter fires + +End to end on a real `ag.mp.Isothermal` under `af.MultiStartProdigy`, the +constraint arriving purely through inheritance: + +``` +prodigy step 300/300 | best log_post -431.7808 | alive 22/32 | constrained 6/32 +n_value_nan_lane_steps 2344 +n_grad_nan_lane_steps 0 +n_constrained_lane_steps 1057 +best lane einstein_radius = 1.5968 (truth 1.6) +``` + +Severity is honest: the trap wastes start budget rather than corrupting results +— surviving lanes still recovered the truth. It becomes a correctness risk only +where `n_starts` is small or the good basin is rare, which is the pixelized-mesh +regime. + +## Traps + +- **The original question is still unanswered.** Whether the *real* Prodigy mesh + or MGE runs entered the region cannot be determined retrospectively: + `autolens_profiling` stores summary records with no per-start traces, and the + Prodigy mesh cells (`scripts/imaging/searches/multi_start_prodigy/`) were never + persisted to `results/searches/multi_start_prodigy/` at all, which holds only + `point_source/` and `cluster/`. Re-running those cells with this counter is the + follow-up that answers it. +- **Spherical profiles inherit the constraint.** `IsothermalSph` subclasses + `Isothermal`, so it carries the declaration with `ell_comps` pinned at `(0, 0)` + — always satisfied, a few wasted ops. A test asserting the opposite failed and + was corrected; the real behaviour is now pinned. +- **A grid with a pixel at the exact centre masks this hazard entirely.** The + first reproduction returned NaN gradients everywhere from `sqrt(0)` — the + *separate* r=0 non-finite-gradient hazard — hiding the plateau under + investigation. Use an off-centre grid, as the hazard scans do. +- **`black` is not enforced on the PyAutoFit files touched.** It wants to + reformat all three at `main` too, so running it would bury the change in + unrelated churn. Left alone deliberately. + +## What the penalty term inherits + +The counter reads only `violation > 0.0`, a sign test, so it is scale-free and +carries no lambda. That leaves two properties of the measure untested by anything +shipped, both landing on the penalty task: + +- **Units.** The measure is in the constraint's own units (ellipticity + magnitude); the figure of merit is in log-likelihood. That mismatch is what a + lambda has to absorb, and why a constant cannot work across cells whose scales + differ by orders of magnitude. +- **The reduction.** `model_constraint_from_vector` combines components with + `maximum` — correct for counting, wrong for a penalty, where two constraints in + different units would let whichever is numerically larger silently dominate. + Per-constraint lambdas or normalised measures, not one max. + +Also unresolved for the penalty: it must be moved *inside* the differentiated +call (today the violation is computed after `value_and_grad`, so it has no +gradient effect), and kept out of the reported likelihood or made invertible in +`log_likelihood_from`. + +## Original prompt + # Count frozen lanes in the multi-start gradient search Type: feature diff --git a/complete/index.md b/complete/index.md index 10bf1887..aeaa4626 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -991 records across 7 buckets. +992 records across 7 buckets. ## Highlights @@ -50,6 +50,7 @@ _(curate hard-won records here — survives regeneration.)_ - [ep-optimise-updater](2026/08/ep-optimise-updater.md) - [feature-ranker-ignores-header-keys](2026/08/feature-ranker-ignores-header-keys.md) — the Feature Agent's ranker now reads the prompt metadata header it was - [file-path-guard-decision](2026/08/file-path-guard-decision.md) — The file-path leg split from raw-guard-migration (leg 3 of the dataset-bulk series, autolens_workspace#354). D… +- [frozen-lane-counter](2026/08/frozen-lane-counter.md) — Added a third disjoint lane counter to the multi-start gradient search - [group-data-preparation-readme](2026/08/group-data-preparation-readme.md) - [hazard-profiling-likelihood-tier](2026/08/hazard-profiling-likelihood-tier.md) — Tier-2 likelihood profiling landed with five persistent findings and corrected NNLS continuity semantics. - [health-conductor-stale-verdict](2026/08/health-conductor-stale-verdict.md) diff --git a/dashboard.md b/dashboard.md index cede1068..45231d88 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | [In flight](#in-flight) (`active/`) | 7 | | [Parked](#parked) (`parked.md`) | 6 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 137 | +| [Backlog](#backlog) (`draft/`) | 136 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) @@ -94,7 +94,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**137** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**136** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
bug — 40 @@ -143,7 +143,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-feature — 26 +feature — 25 - [Make draft/ staleness detectable — `intake reconcile` measured, and the](draft/feature/pyautomind/draft_staleness_detection_signals.md) — pyautomind · medium · supervised · high - [LACosmic per-frame CR masking option + decouple PSF-star pass from](draft/feature/pyautoreduce/lacosmic_cr_option_and_star_pass_decoupling.md) — pyautoreduce · medium · supervised · high @@ -152,7 +152,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. - [Give PyAutoFit searches a `seed` — today no search can](draft/feature/autofit/search_seed_reproducibility.md) — autofit · medium · supervised · medium - [Can create a list of InversionMatrix objects for each dataset](draft/feature/autoarray/multiwavelength_inversion.md) — autoarray · medium · supervised · normal - [The project @z_projects/ic50_workspace is our IC50 use case which we](draft/feature/autofit/ep_lbfgs_jax.md) — autofit · medium · safe · normal -- [Count frozen lanes in the multi-start gradient search](draft/feature/autofit/frozen_lane_counter.md) — autofit · medium · supervised · normal - [Tune cluster-scale JOSS benchmarks toward their 5-minute targets](draft/feature/autolens_workspace/joss_cluster_benchmark_tuning.md) — autolens_workspace · medium · supervised · normal - [Token-light wiki index over the complete/ archive](draft/feature/pyautomind/complete_archive_wiki.md) — pyautomind · medium · supervised · normal - [The imaging `features/advanced/los_halos` example needs improving and padding out before](draft/feature/workspaces/group_los_halos.md) — workspaces · medium · safe · normal From 9d826e9778a075bfc33fffe8096baf50bd4fa3e9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 18:33:53 +0000 Subject: [PATCH 09/20] =?UTF-8?q?prompt:=20record=20the=20first=20counter?= =?UTF-8?q?=20rerun=20=E2=80=94=20MGE=20clean,=20mesh=20needs=20GPU?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the real imaging/mge cell through the repo's own build_for_cell on cloud CPU with the merged counter: zero constrained lane-steps across 2400 lane-steps. For MGE the flat-gradient region is not the problem — though NaN death dominates at 62% and the population fell to alive 2/16, so lanes are dying by another mechanism before they could reach the plateau. The pixelized mesh cells could not be run: two attempts, production and a shrunk 12x12 mesh, both timed out having emitted zero steps. Memory is solvable via batch_size=1; the JIT compile is not. They need the GPU, so the mesh half of the original question stays open. Also records that shrinking the source mesh is the wrong lever for cost (the image-plane grid dominates, not the mesh), that the runner writes into dataset/, and that jaxnnls is a required extra. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- complete/2026/08/frozen-lane-counter.md | 45 +++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/complete/2026/08/frozen-lane-counter.md b/complete/2026/08/frozen-lane-counter.md index 8c151f15..beb24bed 100644 --- a/complete/2026/08/frozen-lane-counter.md +++ b/complete/2026/08/frozen-lane-counter.md @@ -411,3 +411,48 @@ The counter shipped: PyAutoFit PR #1475 (branch 1745 tests pass, +15 new. PyAutoGalaxy declares no constraint yet, so the counter reads zero on real lens models until `EllProfile` opts in — a ~4-line method at the site that already calls `validate_ell_comps`. + +## Follow-up: first rerun with the counter (2026-08-15, cloud CPU) + +Half the original question is now answered, and the other half is bounded. + +**`imaging/mge` — zero constrained lane-steps.** Run through the repo's own +`build_for_cell` (real HST dataset, real 15-parameter model, real analysis; +only the search budget reduced), 16 starts x 150 steps, 352s on 4 CPU cores: + +| counter | lane-steps | share | +|---|---:|---:| +| `n_value_nan_lane_steps` | 1498 | 62.42% | +| `n_grad_nan_lane_steps` | 9 | 0.38% | +| `n_constrained_lane_steps` | **0** | **0.00%** | + +So for the MGE cell the flat-gradient region is **not** what hurts it. Read the +first row alongside the third, though: the population fell to `alive 2/16` and +NaN death dominates, so lanes are dying by a different mechanism before they +could reach the plateau. The zero is real but partly "they never got the +chance". Reduced budget, one seed. + +**The pixelized mesh cells cannot run on CPU — compile, not memory.** Two +attempts, both killed by timeout having emitted **zero steps**: + +- production mesh (39x39, 1500 Hilbert), `batch_size=1` — 58 min, no step 1; +- shrunk mesh (12x12), `batch_size=1` — 40 min, no step 1. + +Memory is solvable (`batch_size=1` clears the 18.5 GB OOM); the JIT compile is +not. These need the GPU, which matches the search's own docstring on compile +cost. The mesh reruns therefore remain the open tail. + +**Shrinking the source mesh is the wrong lever.** Going 39x39 -> 12x12 barely +moved the allocation (an unbatched 8-start draw still asked for 67 GB). The cost +is dominated by the image-plane grid and mapping matrices at `mask_radius` 3.5, +not by the source mesh — worth knowing before anyone tries to make these cells +cheap by shrinking the mesh. + +**Trap: the runner writes to `dataset/`.** Building the imaging cell rewrote +`dataset/imaging/hst/*.fits` + `tracer.json`, added `positions.json`, and +emitted `results/simulators/imaging_hst_summary_*`. Harmless in a throwaway +clone, but do not assume the dataset directory is read-only. + +**Environment note.** The stack ran on a cloud box with `autolens` installed +`--no-deps` over editable local `autofit`/`autogalaxy`; `jaxnnls` is a required +extra for the JAX NNLS solver path and is not pulled in by default. From c38a1ae3a735fc5d6d70daae403507ac65ee2385 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 18:42:21 +0000 Subject: [PATCH 10/20] prompt: validate the mge zero, and file what is actually killing the lanes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Positive control against the same production mge model: 44 constrained components discovered, zero violation at prior medians, 6 of 15 free parameters drive the constraint when pushed past the clamp, and the predicate counts the trapped lane. So the zero means nothing entered the plateau, not that nothing was watching. Confirms two inheritance cases are benign — spherical profiles and ExternalShear both carry the declaration with ell_comps pinned at (0, 0), so shear magnitude is correctly not constrained by the unit-disc rule — and notes that probing with concrete floats raises from validate_ell_comps before the constraint is reached, which is the documented NumPy-path behaviour. Files draft/research/autolens_profiling/mge_lane_death.md for the effect the counter surfaced instead: 62% of lane-steps value-NaN and the population falling to alive 2/16, which contradicts the resurrect docstring's claim that the parametric MGE-class cell has only the measure-zero singularity. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- complete/2026/08/frozen-lane-counter.md | 27 +++++++ dashboard.md | 11 +-- .../autolens_profiling/mge_lane_death.md | 74 +++++++++++++++++++ 3 files changed, 107 insertions(+), 5 deletions(-) create mode 100644 draft/research/autolens_profiling/mge_lane_death.md diff --git a/complete/2026/08/frozen-lane-counter.md b/complete/2026/08/frozen-lane-counter.md index beb24bed..1570ad34 100644 --- a/complete/2026/08/frozen-lane-counter.md +++ b/complete/2026/08/frozen-lane-counter.md @@ -456,3 +456,30 @@ clone, but do not assume the dataset directory is read-only. **Environment note.** The stack ran on a cloud box with `autolens` installed `--no-deps` over editable local `autofit`/`autogalaxy`; `jaxnnls` is a required extra for the JAX NNLS solver path and is not pulled in by default. + +## The zero was validated, not assumed + +A count of zero is only evidence if the counter would have fired. Positive +control against the **same production mge model**: + +1. **44 constrained components discovered** on it (MGE `Basis` Gaussians, the + `Isothermal` mass, `ExternalShear`) — something was watching. +2. Violation at prior medians: **0.0**. +3. **6 of 15** free parameters drive the constraint when pushed past the clamp + (the three `ell_comps` pairs), each reporting violation 2.001. +4. The lane predicate turns that into **1 counted lane**. + +So `n_constrained_lane_steps = 0` on the mge cell means *nothing entered the +plateau*, not *nothing was watching*. + +Two inheritance cases confirmed benign, both discovered but unable to violate +because their `ell_comps` are pinned at `(0, 0)`: spherical profiles (they +subclass their elliptical counterpart) and `ExternalShear` (a subclass of +`EllProfile` whose free parameters are `gamma_1`/`gamma_2`). Shear magnitude is +**not** constrained by this rule, which is correct — the unit-disc bound is +about the axis ratio, not shear. + +**Trap worth repeating:** probing the constraint with concrete Python floats +raises `ModelParameterException` from the component's own `validate_ell_comps` +before the constraint is ever evaluated. That is the documented NumPy-path +behaviour, not a fault. Reproduce the traced path with `jnp` arrays. diff --git a/dashboard.md b/dashboard.md index 45231d88..14970a4a 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,17 +11,18 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | [In flight](#in-flight) (`active/`) | 7 | | [Parked](#parked) (`parked.md`) | 6 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 136 | +| [Backlog](#backlog) (`draft/`) | 137 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) ## Start here -**Highest priority** (filed as `high`) — showing 12 of 30 +**Highest priority** (filed as `high`) — showing 12 of 31 - [pre_build stages untracked files, publishing uncommitted human work](draft/bug/pyautohands/pre_build_stages_untracked_wip.md) — pyautohands · small · supervised · high - [TRIAGE: needs manual review before routing](draft/triage/jax_zero_contour.md) — medium · safe · high - [PyAutoLens RTD docs: three-regime restructure (multi_galaxy / group / cluster)](draft/docs/autolens/docs_three_regime_restructure.md) — autolens · medium · supervised · high +- [Find what kills MGE multi-start lanes — it is not](draft/research/autolens_profiling/mge_lane_death.md) — autolens_profiling · medium · supervised · high - [Optimize pixelized Prodigy settings on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu_phase_2_settings.md) — autolens_workspace_developer · medium · human-required · high - [Release does not sync __version__ stamps and workspace pins back](draft/bug/pyautobuild/release_version_sync_back_to_main.md) — pyautobuild · medium · supervised · high - [Make draft/ staleness detectable — `intake reconcile` measured, and the](draft/feature/pyautomind/draft_staleness_detection_signals.md) — pyautomind · medium · supervised · high @@ -30,7 +31,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A - [multi_galaxy package: new regime package in autolens_workspace](draft/docs/autolens/multi_galaxy_package.md) — autolens · large · supervised · high - [Tune the JAX multi-start optimizers into a standard option (MGE](draft/experiment/autolens_profiling/jax_optimizer_settings_tuning.md) — autolens_profiling · large · supervised · high - [Profile and speed up JAX likelihood-function compile times (all use](draft/feature/autolens_profiling/jax_compile_time_profiling.md) — autolens_profiling · large · supervised · high -- [Optimize MultiStartProdigy for pixelized meshes on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu.md) — autolens_workspace_developer · large · human-required · high **Quick wins** (small enough, and safe enough to run unattended) @@ -94,7 +94,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**136** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**137** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
bug — 40 @@ -201,8 +201,9 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-research — 20 +research — 21 +- [Find what kills MGE multi-start lanes — it is not](draft/research/autolens_profiling/mge_lane_death.md) — autolens_profiling · medium · supervised · high - [Optimize pixelized Prodigy settings on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu_phase_2_settings.md) — autolens_workspace_developer · medium · human-required · high - [Optimize MultiStartProdigy for pixelized meshes on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu.md) — autolens_workspace_developer · large · human-required · high - [Deep research: Can we speed up Delaunay in PyAutoArray?](draft/research/autoarray/delaunay_research.md) — autoarray · too-large · supervised · high diff --git a/draft/research/autolens_profiling/mge_lane_death.md b/draft/research/autolens_profiling/mge_lane_death.md new file mode 100644 index 00000000..445c4ddf --- /dev/null +++ b/draft/research/autolens_profiling/mge_lane_death.md @@ -0,0 +1,74 @@ +# Find what kills MGE multi-start lanes — it is not the ell_comps plateau + +Type: research +Target: autolens_profiling +Repos: +- autolens_profiling +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +# Find what kills MGE multi-start lanes — it is not the ell_comps plateau + +@autolens_profiling + +The frozen-lane counter shipped (PyAutoFit #1475, PyAutoGalaxy #572) and its +first run cleared the ell_comps plateau as a suspect for the MGE cell — and +surfaced a much larger effect nobody has characterised. + +Real `imaging/mge` cell (production dataset, model and analysis via +`build_for_cell`; 16 starts x 150 steps, cloud CPU, 352s): + +| counter | lane-steps | share | +|---|---:|---:| +| `n_value_nan_lane_steps` | 1498 | **62.42%** | +| `n_grad_nan_lane_steps` | 9 | 0.38% | +| `n_constrained_lane_steps` | 0 | 0.00% | + +The population fell from `alive 16/16` to `alive 2/16`. **Roughly seven of every +eight starts die, and the best-fit is being found by two survivors.** + +## Why this is worth a task + +It contradicts a documented assumption. `AbstractMultiStartGradient`'s +`resurrect` docstring states the default `resurrect=False` is safe because "the +parametric (MGE-class) cell has only the measure-zero singularity, so the +`apply_if_finite` guard suffices". The measurement says otherwise: value-NaN, +not gradient-NaN, dominates at 62% of lane-steps, and `apply_if_finite` does not +rescue a value-NaN lane — it only zeroes the step, so the lane stays dead and +`resurrect=False` never redraws it. + +If that holds at production budget, the MGE cell is running at a small fraction +of its nominal start count, and `n_starts` is not buying what it appears to. + +## What to establish + +- **Where the NaNs come from.** Which parameters/regions produce a non-finite + likelihood in the MGE cell — the mask edge, a Gaussian `sigma` collapsing, the + linear inversion, the `sqrt` at r=0, something else. The hazard index in this + repo is the natural place to look first and to record the answer. +- **When they happen.** Deaths concentrated in the first steps (bad draws that + `_broad_starts` should have filtered) mean something different from deaths + accumulating throughout (trajectories walking into a wall). +- **Whether `resurrect=True` recovers the budget** on this cell, and what it + costs. The docstring says it is for pixelized sources; this evidence suggests + the parametric cell may need it too. +- **Whether it reproduces at production budget and on GPU**, and across seeds. + The measurement above is one seed at reduced budget. + +## Boundary + +Investigation and measurement, in `@autolens_profiling`. Do not change the +`resurrect` default or any search behaviour as part of this — if the evidence +supports a change, that is a separate PyAutoFit task with its own benchmark +impact, since it would shift every existing multi-start result. + +## Provenance + +Measured 2026-08-15 with the frozen-lane counter; see +`complete/2026/08/frozen-lane-counter.md` for the full run, the positive control +that validates the zero, and the environment notes (`jaxnnls` is a required +extra; the runner writes into `dataset/`). + + From d935c97c37a53e16a41a5845541fe7f3673e3675 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 20:09:21 +0000 Subject: [PATCH 11/20] prompt: file mge_lane_death research task (autolens_profiling) Reproduce-first research prompt for the reported MGE-cell lane death: is the parametric MGE likelihood only measure-zero singular, as the resurrect docstring claims? Records a provenance gap rather than papering over it. The launch context cites complete/2026/08/frozen-lane-counter.md and PyAutoFit#1475 / PyAutoGalaxy#572; none exist at origin/main 1f7cca8. The counters shipped as multistart-nan-step-diagnostics.md (PyAutoFit#1472 -> #1473, autolens_profiling#127, no PyAutoGalaxy leg), and no record mentions the 62% rate, alive 2/16, or the ell_comps clearance. Step 1 therefore establishes the number rather than assuming it. Phased: only step 1 (production-budget GPU reproduction across seeds, with a descent-path control) is issued from this prompt. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- .../autolens_profiling/mge_lane_death.md | 168 ++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 draft/research/autolens_profiling/mge_lane_death.md diff --git a/draft/research/autolens_profiling/mge_lane_death.md b/draft/research/autolens_profiling/mge_lane_death.md new file mode 100644 index 00000000..d3e7eb1e --- /dev/null +++ b/draft/research/autolens_profiling/mge_lane_death.md @@ -0,0 +1,168 @@ +# MGE-cell lane death: is the parametric MGE likelihood only measure-zero singular? + +Type: research +Target: autolens_profiling +Repos: +- autolens_profiling +- PyAutoFit +Difficulty: large +Autonomy: supervised +Priority: normal +Status: formalised + +## Phasing (one prompt = one task = one PR) + +Only **step 1 is issued from this prompt** — reproduce the rate at production +budget on GPU across seeds, with the control, and write up the number. That is +one task and one autolens_profiling PR. + +Steps 2 and 3 are gated on step 1's result and are spun off as their own prompts +when it lands: if the rate comes back near zero, step 2 never happens and the +task closes as "the docstring was right, the reported 62% was a +reduced-budget/CPU artefact". Do not open all three as one PR. + +## The claim under test + +The `resurrect` docstring in +`@PyAutoFit/autofit/non_linear/search/mle/multi_start_gradient/search.py` +characterises the parametric MGE-class cell as having only a **measure-zero +singularity** — lane death should therefore be rare and incidental, not a +structural property of the likelihood surface. + +The human reports a first run of the newly shipped value-NaN / gradient-NaN +lane-step counters (PyAutoFit#1472 → #1473, `n_value_nan_lane_steps` / +`n_grad_nan_lane_steps`) on the `imaging/mge` profiling cell that contradicts +this: **~62% of lane-steps died to value-NaN, and the population fell to alive +2/16.** The same run is reported to have cleared the `ell_comps` plateau as a +suspect for that cell, and to have carried a positive control validating the +zero. + +**These figures are reported, not yet reproduced, and are not present in the +PyAutoMind record** — see "Provenance gap" below. Reproducing them is step 1 of +this task, not an assumption of it. + +If the 62% holds at production budget, it is not a measurement curiosity: it +means the MGE cell's likelihood is undefined over a *set of positive measure* +along the descent path, every benchmark number produced on that cell was +produced by a search running on 2 of 16 lanes, and the resurrection policy +deliberately deferred in #1472 ("decide AFTER the counters show how often frozen +lanes actually occur") now has its answer. + +## Provenance gap — resolve before or during step 1 + +The launch context cites a completion record `complete/2026/08/frozen-lane-counter.md` +and PRs **PyAutoFit#1475 / PyAutoGalaxy#572**. None of these exist in PyAutoMind +as of `origin/main` @ `1f7cca8`: + +- The frozen-lane (gradient-NaN) counter shipped as + `complete/2026/08/multistart-nan-step-diagnostics.md` — issue PyAutoFit#1472, + library PR **PyAutoFit#1473** (merged `fbfcece3`), profiling PR + **autolens_profiling#127** (merged `a34d6191`). There is **no PyAutoGalaxy leg**: + the change was confined to `search.py` and `autofit/text/text_util.py`. +- No record anywhere in the repo mentions a 62% rate, `alive 2/16`, an + `ell_comps` plateau clearance for the MGE cell, or a positive control for this + run. The most recent counter work + (`complete/2026/08/multistart-gradient-resume-fom-sanity-check.md`, updated + 2026-08-15) is **resume-accumulation** verification on a clean Gaussian fit + with synthetic NaN traps — not a production MGE run. + +So either the run happened in a session whose write-up was never pushed, or the +identifiers are misremembered. Recover the actual run artefacts +(`search.summary`, `samples_info`, the `results/searches/**` JSON) before +treating 62% as a baseline; if they cannot be recovered, step 1 *establishes* +the number rather than reproducing it. + +## Step 1 — reproduce at production budget on GPU (do this first) + +The reported 62% is a reduced-budget CPU number. Do not let it generalise +untested. + +- Run `imaging/mge` at **production** `SEARCHES_N_STARTS` / `SEARCHES_N_STEPS` + on GPU, across **at least two seeds**, and record + `n_value_nan_lane_steps`, `n_grad_nan_lane_steps`, `n_resurrections` and the + alive-lane trajectory for each. +- Report the counters as **rates** normalised by `n_starts * total_steps` — + raw counts are not comparable across budgets (this is the normalisation + `search.summary` already applies). +- Include the reduced-budget CPU configuration as one arm, so the + budget-dependence of the rate is measured rather than assumed. If the rate is + strongly budget-dependent, that is itself the finding. +- Positive control: an analysis with a known-zero NaN rate must report zero + through the same path, on the same hardware. `_broad_starts` rejects + non-finite draws, so lanes always *begin* healthy — a control that only proves + the counter can read zero is weak; pair it with a trap **on the descent path** + (the `where`/`sqrt` pattern in the `Fitness.call` docstring gives a finite + value with a NaN gradient) so the counters are proven to fire on the same run + shape. + +## Step 2 — locate the deaths on the likelihood surface + +Only if step 1 confirms a materially non-zero rate. + +- Which parameters are the dying lanes in when the value goes non-finite? + The counters are per-step aggregates; getting from a rate to a *cause* needs + the lane parameter vectors at the death step. +- Distinguish the candidate mechanisms: MGE sigma range degeneracy (cf. + `complete/2026/08/mge-sigma-min-workspace-sweep.md` and + `complete/2026/05/mge-cse-fallback.md`), NNLS solver failure on the JAX path, + underflow in the likelihood normalisation, and genuine model-space + singularities. These have different fixes and only one of them is + "measure-zero". +- The `ell_comps` plateau is reported as already cleared for this cell; confirm + that from the run artefacts rather than inheriting it + (cf. `complete/2026/08/circular-ell-comps-image-gradient.md`, + `complete/2026/08/resolve-sersic-ell-comps-gradient.md`). + +## Step 3 — what the answer changes + +- **The docstring.** If the singularity is not measure-zero, the `resurrect` + docstring is wrong and misleads every future reader about which cells are safe. +- **The resurrection policy.** #1472 deferred the decision to make `resurrect` + trigger on non-finite *gradients* until the counters spoke. A 62% value-NaN + rate with alive 2/16 is a much stronger signal than that deferral anticipated — + but note that value-NaN is *already* today's resurrection trigger, so a high + value-NaN rate means resurrection is firing and failing to keep the population + alive, which is a different problem from the frozen-lane one and needs stating + separately. +- **Every MGE benchmark number to date.** If the population is routinely 2/16, + the wsdev #117/#125 comparisons and the sampler benchmark rows are measuring a + crippled search. Scope the re-run implications; do not silently invalidate. + +Deliberately out of scope: changing resurrection behaviour. This prompt is +research — it produces the evidence and a recommendation. A behaviour change is +a separate feature/bug prompt so the benchmark comparability argument from #1472 +gets made explicitly rather than by accident. + +## Environment (human-supplied; cost real time last session) + +- **Python 3.12+** required (autonerves). +- **`jaxnnls` is a required extra** for the JAX NNLS solver path — not pulled in + by default. +- Install `autolens` with **`--no-deps`** when running editable local + autofit/autogalaxy, or the released wheels clobber them. +- **`build_for_cell` writes into `dataset/`** — it rewrites the HST FITS, adds + `positions.json`, and emits `results/simulators/*`. Not read-only. +- Cell scripts honour `SEARCHES_N_STARTS`, `SEARCHES_N_STEPS`, + `SEARCHES_BATCH_SIZE`, `SEARCHES_DISABLE_VIZ`. +- `imaging/mge` runs on **CPU in ~6 min at 16x150**. The **pixelized mesh cells + do not** — two attempts timed out emitting zero steps. It is **JIT compile, + not memory** (`batch_size=1` clears the OOM). Those need the GPU. +- **Shrinking the source mesh is the wrong cost lever** — the image-plane grid + at `mask_radius 3.5` dominates, not the mesh. +- On A100, set `jax_enable_x64` **explicitly** — it is not inherited under + `sbatch`, and float32 would understate the quantity under test (carried + forward from the #1472 "Still owed" note). + +## Prior art to read first + +- `complete/2026/08/multistart-nan-step-diagnostics.md` — the counters, what + they mean, the normalisation, and the `_broad_starts` trap. +- `complete/2026/08/multistart-gradient-resume-fom-sanity-check.md` — descent-path + NaN injection technique and the equality-vs-`>=` assertion lesson. +- `complete/2026/07/pixelized-multistart-prodigy-cpu.md` and the DelaunayNN + free-AdaptSplit open question in `active.md` (109 resurrections, NaN death vs + over-regularized-floor death) — the same question, different cell. + + From 6fa0498915e8d17f48b01c4f572b146d55f072af Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 20:11:04 +0000 Subject: [PATCH 12/20] prompt: record that autolens_profiling has no lane-death run either MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Checked the profiling repo, not just Mind. Its main is at a34d6191 — the same #127 merge the Mind record cites, so nothing has landed since, and no remote branch carries lane-death work. The only MGE NaN-accounting artefact is results/searches/multi_start_nan_accounting/local_cpu.json, which is the overhead benchmark (16 starts x 5 steps x 3 reps, local_cpu, verdict "fused accounting costs 4.1us on a 1.027s step") and reports no NaN counts at all. Also notes that a 5-step budget cannot resolve a rate that accumulates along a descent path. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- .../autolens_profiling/mge_lane_death.md | 26 +++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/draft/research/autolens_profiling/mge_lane_death.md b/draft/research/autolens_profiling/mge_lane_death.md index d3e7eb1e..ae418cb4 100644 --- a/draft/research/autolens_profiling/mge_lane_death.md +++ b/draft/research/autolens_profiling/mge_lane_death.md @@ -66,11 +66,27 @@ as of `origin/main` @ `1f7cca8`: 2026-08-15) is **resume-accumulation** verification on a clean Gaussian fit with synthetic NaN traps — not a production MGE run. -So either the run happened in a session whose write-up was never pushed, or the -identifiers are misremembered. Recover the actual run artefacts -(`search.summary`, `samples_info`, the `results/searches/**` JSON) before -treating 62% as a baseline; if they cannot be recovered, step 1 *establishes* -the number rather than reproducing it. +`autolens_profiling` was checked too, and does not have the run either. Its +`main` is at `a34d6191` — exactly the #127 merge the Mind record cites, so +nothing has landed since. The only MGE NaN-accounting artefact in the repo is +`results/searches/multi_start_nan_accounting/local_cpu.json`, and that is the +**overhead benchmark**, not a lane-death rate: `imaging`/`mge`/`hst`, +`n_starts: 16`, `n_steps: 5`, `reps: 3` on `local_cpu`, whose only verdict is +`"fused accounting costs 4.1us on a 1.027s step = 0.00039% of run time"`. It +reports no NaN counts and no alive-lane trajectory. No remote branch in +autolens_profiling carries lane-death work. + +So the 62% run exists in no pushed artefact in either repo. Either it was run in +a session whose results were never pushed, or the identifiers and figures are +misremembered. Recover the actual run artefacts (`search.summary`, +`samples_info`, the `results/searches/**` JSON) before treating 62% as a +baseline; if they cannot be recovered, step 1 *establishes* the number rather +than reproducing it. + +Note also that the 5-step, 3-rep budget of the one real artefact is far too +short to say anything about a rate that accumulates along a descent path — which +is consistent with the launch context's own warning not to let a reduced-budget +CPU number generalise. ## Step 1 — reproduce at production budget on GPU (do this first) From 1b7c42a38649bf04258619558c5c0991d7bfde04 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 20:55:36 +0000 Subject: [PATCH 13/20] prompt: embed the mge lane-death reproducers so another session can rerun them The 62% measurement and its positive control lived only in a session scratchpad, which is not visible to any other session. Both drivers and the verbatim result JSON are now in the prompt itself, so a cold session picking this up can reproduce the number rather than take it on trust. Includes the invocation, the environment prerequisites (Python 3.12+, jaxnnls, autolens --no-deps), and the note that the positive control must probe with jnp arrays because concrete floats raise from validate_ell_comps before the constraint is reached. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012GAFoogitLceTsgA7bfB4k --- .../autolens_profiling/mge_lane_death.md | 244 ++++++++++++++++++ 1 file changed, 244 insertions(+) diff --git a/draft/research/autolens_profiling/mge_lane_death.md b/draft/research/autolens_profiling/mge_lane_death.md index 445c4ddf..17c06d06 100644 --- a/draft/research/autolens_profiling/mge_lane_death.md +++ b/draft/research/autolens_profiling/mge_lane_death.md @@ -72,3 +72,247 @@ that validates the zero, and the environment notes (`jaxnnls` is a required extra; the runner writes into `dataset/`). + +## Reproducer + +Both scripts below are self-contained and were used to produce the numbers in +this prompt. They live here rather than in `@autolens_profiling` because they +are throwaway measurement drivers, not part of that repo's script tiers — copy +them out to run, do not commit them there. + +### The measured result (verbatim) + +```json +{ + "cell": "imaging/mge", + "instrument": "hst", + "hardware": "cloud_cpu", + "n_starts": 16, + "n_steps": 150, + "batch_size": null, + "lane_steps": 2400, + "wall_s": 352.21100759506226, + "free_parameters": 15, + "counters": { + "n_value_nan_lane_steps": 1498, + "n_grad_nan_lane_steps": 9, + "n_constrained_lane_steps": 0, + "n_resurrections": 0 + } +}``` + +### `rerun_cell.py` — runs one production cell at reduced budget + +Invoke as `SEARCHES_DISABLE_VIZ=1 N_STARTS=16 N_STEPS=150 python rerun_cell.py mge`. +Environment: Python 3.12+, `pip install jaxnnls`, and install `autolens` with +`--no-deps` if using editable local `autofit`/`autogalaxy`. + +```python +"""Re-run one autolens_profiling search cell on CPU at reduced budget, to read +the new constrained-lane counter. + +Uses the repo's own `build_for_cell` so the dataset, model and analysis are +exactly the production ones. Only the search budget is reduced (n_starts, +n_steps, batch_size) — that changes how thoroughly the space is explored, not +the shape of the likelihood surface, which is what the counter reports on. +""" + +import os, sys, time, json +from pathlib import Path + +ROOT = Path("/workspace/pyautolabs/autolens_profiling") +sys.path.insert(0, str(ROOT / "scripts" / "misc")) +sys.path.insert(0, str(ROOT)) + +# Dataset paths in `_setup.py` are relative to the repo root. +os.chdir(ROOT) + +import numpy as np +import autofit as af + +from searches import _setup +from searches._setup import build_for_cell + +# Optional mesh shrink. The production fiducial is a (39, 39) = 1521-pixel mesh +# with 1500 Hilbert pixels, whose JIT compile alone exceeds an hour on CPU. A +# shrunken mesh is NOT the production cell — the landscape differs — but it +# keeps the same clamp, the same unbounded stepping, and the same mesh-shaped +# likelihood, so it can still say whether lanes reach the plateau at all. +MESH = os.environ.get("MESH_SHAPE") +HILBERT = os.environ.get("HILBERT_PIXELS") +if MESH: + n = int(MESH) + _setup._PIXELIZATION_MESH_SHAPE = (n, n) + print(f" [shrunk] pixelization mesh {(n, n)} (production is (39, 39))") +if HILBERT: + _setup._HILBERT_PIXELS = int(HILBERT) + print(f" [shrunk] hilbert pixels {HILBERT} (production is 1500)") + +MODEL_TYPE = sys.argv[1] if len(sys.argv) > 1 else "mge" +N_STARTS = int(os.environ.get("N_STARTS", "16")) +N_STEPS = int(os.environ.get("N_STEPS", "150")) +BATCH = os.environ.get("BATCH_SIZE") +INSTRUMENT = os.environ.get("INSTRUMENT", "hst") + +print(f"=== cell: imaging/{MODEL_TYPE} [{INSTRUMENT}] " + f"starts={N_STARTS} steps={N_STEPS} batch={BATCH or 'all'} ===") + +t0 = time.time() +dataset, model, analysis = build_for_cell( + dataset_class="imaging", + model_type=MODEL_TYPE, + instrument=INSTRUMENT, + use_jax=True, + use_mixed_precision=False, +) +print(f" build: {time.time() - t0:.1f}s free parameters: {model.total_free_parameters}") + +constrained = model.constrained_model_tuples() +print(f" components declaring a model constraint: {len(constrained)}") +for path, sub in constrained[:8]: + print(f" {'.'.join(p for p in path if p) or ''} -> {sub.cls.__name__}") + +kwargs = dict( + name=f"rerun_{MODEL_TYPE}", + n_starts=N_STARTS, + n_steps=N_STEPS, + iterations_per_log=25, + convergence=af.MultiStartGradientConvergence(check_for_convergence=False), +) +if BATCH: + kwargs["batch_size"] = int(BATCH) + +search = af.MultiStartProdigy(**kwargs) + +t0 = time.time() +result = search.fit(model=model, analysis=analysis) +wall = time.time() - t0 + +primary = result[0] if isinstance(result, list) else result +si = primary.search_internal +if not isinstance(si, dict): + si = getattr(si, "__dict__", {}) or {} + +counters = { + k: si.get(k) + for k in ( + "n_value_nan_lane_steps", + "n_grad_nan_lane_steps", + "n_constrained_lane_steps", + "n_resurrections", + ) +} +lane_steps = N_STARTS * N_STEPS + +print(f"\n=== imaging/{MODEL_TYPE} — {wall:.1f}s wall, {lane_steps} lane-steps ===") +for k, v in counters.items(): + pct = f"({100.0 * v / lane_steps:.2f}% of lane-steps)" if isinstance(v, int) and lane_steps else "" + print(f" {k:<28} {v} {pct}") + +out = { + "cell": f"imaging/{MODEL_TYPE}", + "instrument": INSTRUMENT, + "hardware": "cloud_cpu", + "n_starts": N_STARTS, + "n_steps": N_STEPS, + "batch_size": int(BATCH) if BATCH else None, + "lane_steps": lane_steps, + "wall_s": wall, + "free_parameters": model.total_free_parameters, + "counters": counters, +} +dest = Path(f"/tmp/claude-0/-home-user/ef0adef1-5fcd-5111-9cdf-bcb1014fc23d/scratchpad/rerun_{MODEL_TYPE}.json") +dest.write_text(json.dumps(out, indent=2)) +print(f"\nwrote {dest}") +``` + +### `validate_zero.py` — the positive control that makes the zero meaningful + +Proves the counter was watching that exact model and would have fired. Note it +probes with `jnp` arrays: concrete Python floats trip `validate_ell_comps` and +raise before the constraint is ever reached. + +```python +"""Positive control: prove the zero from imaging/mge is a real zero. + +A count of 0 is only evidence if the counter would have fired had a lane been +trapped. This builds the SAME production model the mge cell used, then checks: + + 1. the constraint is discovered on it at all; + 2. a vector inside the valid region reports zero violation; + 3. a vector placed beyond the clamp reports a positive violation; + 4. the lane-count predicate turns that into a counted lane. + +If (1) failed, the zero would mean "nothing was watching", not "nothing happened". +""" + +import os, sys +from pathlib import Path + +ROOT = Path("/workspace/pyautolabs/autolens_profiling") +sys.path.insert(0, str(ROOT / "scripts" / "misc")) +sys.path.insert(0, str(ROOT)) +os.chdir(ROOT) + +import numpy as np +import jax.numpy as jnp +import autofit as af +from autofit.non_linear.search.mle.multi_start_gradient.search import ( + AbstractMultiStartGradient, +) +from searches._setup import build_for_cell + +dataset, model, analysis = build_for_cell( + dataset_class="imaging", + model_type="mge", + instrument="hst", + use_jax=True, + use_mixed_precision=False, +) + +# --- 1. discovery on the real production model +constrained = model.constrained_model_tuples() +print(f"1. constrained components discovered: {len(constrained)}") +for path, sub in constrained: + print(f" {'.'.join(p for p in path if p) or ''} -> {sub.cls.__name__}") +assert constrained, "NOTHING WAS WATCHING — the zero would be meaningless" + +# JAX arrays, not Python floats: concrete scalars trip the component's own +# `validate_ell_comps` guard before the constraint is ever reached (exactly as +# the method's docstring warns). The real search path is traced, where that +# guard returns early — jnp arrays reproduce that. +vector = [jnp.asarray(float(v)) for v in model.physical_values_from_prior_medians] + +# --- 2. valid region -> zero +inside = float(model.model_constraint_from_vector(vector, xp=jnp)) +print(f"\n2. violation at prior medians: {inside}") + +# --- 3. beyond the clamp -> positive. +# Brute-force which parameters drive the constraint rather than guessing names: +# push each one past the clamp in turn and see which move the violation. +responders = [] +for i in range(len(vector)): + probe = list(vector) + probe[i] = jnp.asarray(3.0) + v = float(model.model_constraint_from_vector(probe, xp=jnp)) + if v > 0.0: + responders.append((i, v)) + +print(f"3. parameters that drive the constraint when pushed to 3.0: " + f"{len(responders)} of {len(vector)}") +for i, v in responders[:6]: + print(f" index {i:>3} -> violation {v:.4f}") +outside = responders[0][1] if responders else 0.0 + +# --- 4. the predicate counts it +counted = AbstractMultiStartGradient._constrained_lane_count( + alive=np.array([True, True]), + grad_finite=np.array([True, True]), + constraint_violation=np.array([inside, outside]), +) +print(f"4. lanes counted from [valid, trapped]: {counted}") + +ok = bool(constrained) and inside == 0.0 and outside > 0.0 and counted == 1 +print(f"\nVERDICT: the mge zero is {'a REAL zero' if ok else 'NOT TRUSTWORTHY'}") +sys.exit(0 if ok else 1) +``` From c47cbc1f4163ab91a86806a7ae4dd49ce45d239b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 21:04:38 +0000 Subject: [PATCH 14/20] =?UTF-8?q?prompt:=20correct=20mge=5Flane=5Fdeath=20?= =?UTF-8?q?=E2=80=94=20#1475=20and=20#572=20are=20real=20and=20merged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both PRs exist and merged 2026-08-15 ~15:52 UTC, after the Mind main this branch was cut from (1f7cca8), which is why the earlier revision could not find them: PyAutoFit#1475 (004f798) adds _constrained_lane_count, PyAutoGalaxy#572 (695b27c) declares EllProfile.__model_constraint__. The "frozen-lane counter" is that trapped-lane pair, not the #1473 gradient-NaN counter. Verified the measure-zero claim verbatim at search.py:119 and quotes it. Adds a three-counter table so value-NaN / gradient-NaN / constrained are not conflated, and sharpens step 3: a 62% VALUE-NaN rate does not bear on the #1472 gradient-trigger deferral (value-NaN is already the trigger) — it bears on the resurrect=False default, whose stated justification is the measure-zero claim. Also flags that #572 declares a constraint on EllProfile only, so a zero constrained reading may mean nothing declared it rather than nothing trapped. Still not pushed anywhere: the completion record and the run artefacts behind the 62% / alive 2/16. Step 1 now begins by recovering that log. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- .../autolens_profiling/mge_lane_death.md | 166 +++++++++++------- 1 file changed, 102 insertions(+), 64 deletions(-) diff --git a/draft/research/autolens_profiling/mge_lane_death.md b/draft/research/autolens_profiling/mge_lane_death.md index ae418cb4..06b3a7b1 100644 --- a/draft/research/autolens_profiling/mge_lane_death.md +++ b/draft/research/autolens_profiling/mge_lane_death.md @@ -29,17 +29,32 @@ characterises the parametric MGE-class cell as having only a **measure-zero singularity** — lane death should therefore be rare and incidental, not a structural property of the likelihood surface. -The human reports a first run of the newly shipped value-NaN / gradient-NaN -lane-step counters (PyAutoFit#1472 → #1473, `n_value_nan_lane_steps` / -`n_grad_nan_lane_steps`) on the `imaging/mge` profiling cell that contradicts -this: **~62% of lane-steps died to value-NaN, and the population fell to alive -2/16.** The same run is reported to have cleared the `ell_comps` plateau as a -suspect for that cell, and to have carried a positive control validating the -zero. - -**These figures are reported, not yet reproduced, and are not present in the -PyAutoMind record** — see "Provenance gap" below. Reproducing them is step 1 of -this task, not an assumption of it. +The claim is verbatim, at `search.py:119` under the `resurrect` parameter: + +> Default ``False`` — the parametric (MGE-class) cell has only the measure-zero +> singularity, so the ``apply_if_finite`` guard suffices and behaviour/results +> are unchanged. + +The human reports that the first run of the **trapped-lane counter** — shipped +today as PyAutoFit#1475 (`_constrained_lane_count`, merged `004f798`) with its +PyAutoGalaxy leg #572 (`EllProfile.__model_constraint__`, merged `695b27c`) — +contradicts this on the `imaging/mge` cell: **~62% of lane-steps died to +value-NaN, and the population fell to alive 2/16.** + +The same run reportedly **cleared the `ell_comps` plateau as a suspect** for +this cell, which is coherent with what #1475/#572 do: #572 declares the +`ell_comps` saturation constraint, #1475 counts lanes sitting outside it, and a +near-zero `constrained N/16` reading on the MGE cell exonerates the plateau. The +"positive control validating the zero" is the load-bearing part of that — a zero +reading only means something if the detector is proven able to fire, which is +exactly the point of #1475's note that a component-wise gradient test *"caught 0 +of 17 genuinely trapped lanes in a JAX reproduction"* while the declared- +constraint detector caught them. + +So the plateau is cleared and the value-NaN rate is the finding. **The 62% and +alive 2/16 figures themselves are reported from a local run log and are not in +any pushed artefact** — see "Provenance gap" below. Reproducing them is step 1 +of this task, not an assumption of it. If the 62% holds at production budget, it is not a measurement curiosity: it means the MGE cell's likelihood is undefined over a *set of positive measure* @@ -48,45 +63,50 @@ produced by a search running on 2 of 16 lanes, and the resurrection policy deliberately deferred in #1472 ("decide AFTER the counters show how often frozen lanes actually occur") now has its answer. +## Three counters, not one — keep them straight + +The lane accounting shipped in two waves, and the second is a day old. Anything +written about "the counter" needs to say which: + +| Counter | Failure mode | Shipped as | +|---|---|---| +| `n_value_nan_lane_steps` | likelihood **undefined** (today's `resurrect` trigger) | PyAutoFit#1472 → **#1473** (`fbfcece`) | +| `n_grad_nan_lane_steps` | likelihood defined but **not differentiable** — the frozen zombie lane | same PR | +| `_constrained_lane_count` | finite **and** differentiable, but **trapped** on a saturating plateau with no restoring force | PyAutoFit **#1475** (`004f798`) + PyAutoGalaxy **#572** (`695b27c`) | + +All three are disjoint by construction ("one lane, one bucket") and all three are +**measurement only** — none gates a redraw or changes stepping. The reported 62% +is in the **first** bucket, which is the one whose resurrection policy already +exists. That matters for step 3. + ## Provenance gap — resolve before or during step 1 -The launch context cites a completion record `complete/2026/08/frozen-lane-counter.md` -and PRs **PyAutoFit#1475 / PyAutoGalaxy#572**. None of these exist in PyAutoMind -as of `origin/main` @ `1f7cca8`: - -- The frozen-lane (gradient-NaN) counter shipped as - `complete/2026/08/multistart-nan-step-diagnostics.md` — issue PyAutoFit#1472, - library PR **PyAutoFit#1473** (merged `fbfcece3`), profiling PR - **autolens_profiling#127** (merged `a34d6191`). There is **no PyAutoGalaxy leg**: - the change was confined to `search.py` and `autofit/text/text_util.py`. -- No record anywhere in the repo mentions a 62% rate, `alive 2/16`, an - `ell_comps` plateau clearance for the MGE cell, or a positive control for this - run. The most recent counter work - (`complete/2026/08/multistart-gradient-resume-fom-sanity-check.md`, updated - 2026-08-15) is **resume-accumulation** verification on a clean Gaussian fit - with synthetic NaN traps — not a production MGE run. - -`autolens_profiling` was checked too, and does not have the run either. Its -`main` is at `a34d6191` — exactly the #127 merge the Mind record cites, so -nothing has landed since. The only MGE NaN-accounting artefact in the repo is -`results/searches/multi_start_nan_accounting/local_cpu.json`, and that is the -**overhead benchmark**, not a lane-death rate: `imaging`/`mge`/`hst`, -`n_starts: 16`, `n_steps: 5`, `reps: 3` on `local_cpu`, whose only verdict is -`"fused accounting costs 4.1us on a 1.027s step = 0.00039% of run time"`. It -reports no NaN counts and no alive-lane trajectory. No remote branch in -autolens_profiling carries lane-death work. - -So the 62% run exists in no pushed artefact in either repo. Either it was run in -a session whose results were never pushed, or the identifiers and figures are -misremembered. Recover the actual run artefacts (`search.summary`, -`samples_info`, the `results/searches/**` JSON) before treating 62% as a -baseline; if they cannot be recovered, step 1 *establishes* the number rather -than reproducing it. - -Note also that the 5-step, 3-rep budget of the one real artefact is far too -short to say anything about a rate that accumulates along a descent path — which -is consistent with the launch context's own warning not to let a reduced-budget -CPU number generalise. +The PRs are real and merged; the **write-up and the run artefacts are not +pushed**. + +- `complete/2026/08/frozen-lane-counter.md` does not exist. PyAutoMind + `origin/main` is at `1f7cca8`, which predates both #1475 and #572 (merged + 2026-08-15 ~15:52 UTC), so no completion record for that wave has been written + yet. The nearest existing records are + `complete/2026/08/multistart-nan-step-diagnostics.md` (the #1473 wave) and + `complete/2026/08/multistart-gradient-resume-fom-sanity-check.md` (resume + accumulation, verified on a clean Gaussian fit with synthetic NaN traps — not + a production MGE run). +- No pushed artefact anywhere carries the 62% or the `alive 2/16`. + `autolens_profiling` `main` is at `a34d6191` (the #127 merge) with nothing + since and no lane-death branch; PyAutoFit has no such branch either. The only + MGE NaN artefact in the profiling repo is + `results/searches/multi_start_nan_accounting/local_cpu.json`, and that is the + **overhead benchmark** — `imaging`/`mge`/`hst`, `n_starts: 16`, `n_steps: 5`, + `reps: 3`, `local_cpu`, verdict `"fused accounting costs 4.1us on a 1.027s + step"`. It reports no NaN counts and no alive-lane trajectory, and at 5 steps + it could not resolve a rate that accumulates along a descent path anyway. + +So the figures live in a local run log. **Recover and commit that log (or the +`search.summary` / `samples_info` / `results/searches/**` JSON behind it) as the +first act of step 1** — otherwise the reproduction has nothing to be graded +against, and a disagreement between the GPU run and the remembered number will +be unresolvable. ## Step 1 — reproduce at production budget on GPU (do this first) @@ -95,8 +115,12 @@ untested. - Run `imaging/mge` at **production** `SEARCHES_N_STARTS` / `SEARCHES_N_STEPS` on GPU, across **at least two seeds**, and record - `n_value_nan_lane_steps`, `n_grad_nan_lane_steps`, `n_resurrections` and the - alive-lane trajectory for each. + `n_value_nan_lane_steps`, `n_grad_nan_lane_steps`, the `_constrained_lane_count` + reading, `n_resurrections`, and the `alive N/16` + `constrained N/16` + trajectory for each. Record **all three** counters even though only the first + is expected to be large — a step-3 argument that "the plateau is cleared" + needs the constrained column present and near zero on the production run, not + inherited from the reduced-budget one. - Report the counters as **rates** normalised by `n_starts * total_steps` — raw counts are not comparable across budgets (this is the normalisation `search.summary` already applies). @@ -124,22 +148,31 @@ Only if step 1 confirms a materially non-zero rate. underflow in the likelihood normalisation, and genuine model-space singularities. These have different fixes and only one of them is "measure-zero". -- The `ell_comps` plateau is reported as already cleared for this cell; confirm - that from the run artefacts rather than inheriting it - (cf. `complete/2026/08/circular-ell-comps-image-gradient.md`, +- The `ell_comps` plateau is reported as already cleared for this cell by the + #1475/#572 constrained counter; confirm that from the production run artefacts + rather than inheriting it (cf. + `complete/2026/08/circular-ell-comps-image-gradient.md`, `complete/2026/08/resolve-sersic-ell-comps-gradient.md`). +- **A cleared plateau does not clear the constraint mechanism generally.** #572 + declares `__model_constraint__` on `EllProfile` only. If a *different* + saturating clamp is in play on this cell, the constrained counter reads zero + because nothing declared it — not because nothing is trapped. Enumerate which + classes in the MGE cell's model declare a constraint before reading a zero as + an all-clear. ## Step 3 — what the answer changes - **The docstring.** If the singularity is not measure-zero, the `resurrect` docstring is wrong and misleads every future reader about which cells are safe. -- **The resurrection policy.** #1472 deferred the decision to make `resurrect` - trigger on non-finite *gradients* until the counters spoke. A 62% value-NaN - rate with alive 2/16 is a much stronger signal than that deferral anticipated — - but note that value-NaN is *already* today's resurrection trigger, so a high - value-NaN rate means resurrection is firing and failing to keep the population - alive, which is a different problem from the frozen-lane one and needs stating - separately. +- **The `resurrect` default, not the resurrection trigger.** This is the + distinction to get right. #1472 deferred whether `resurrect` should *also* + trigger on non-finite gradients. But a 62% **value**-NaN rate says nothing + about that deferral — value-NaN is already the trigger. What it says is that + the MGE cell's `resurrect=False` **default** is wrong: the docstring justifies + that default by asserting the cell has only a measure-zero singularity, and if + 62% holds, the justification is false and the cell has been running with + resurrection off through a landscape that needs it. Recommend on the default; + leave the gradient-trigger question where #1472 left it. - **Every MGE benchmark number to date.** If the population is routinely 2/16, the wsdev #117/#125 comparisons and the sampler benchmark rows are measuring a crippled search. Scope the re-run implications; do not silently invalidate. @@ -178,7 +211,12 @@ gets made explicitly rather than by accident. - `complete/2026/07/pixelized-multistart-prodigy-cpu.md` and the DelaunayNN free-AdaptSplit open question in `active.md` (109 resurrections, NaN death vs over-regularized-floor death) — the same question, different cell. - - +- PyAutoFit `004f798` and PyAutoGalaxy `695b27c` themselves — the constrained + counter and the `EllProfile` constraint have no completion record yet, so the + commit messages are currently the only write-up. + + From 0b8895ae4e84ff24e2d86021da7b544ac26ae417 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 21:57:39 +0000 Subject: [PATCH 15/20] prompt: order the lane-death work by information per unit time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cause-finding moves ahead of the GPU reproduction. The reproduction confirms an effect already measured once; the cause is unanswered, and it does not need production budget or a GPU — the existing ~6-minute 16x150 CPU run already produces the 1498 deaths, so dumping each lane's parameter vector at its death step is a minutes-long loop rather than a queued one. Grades the reproduction on the alive-versus-step curve rather than on recovering "62%", since the scalar is budget-dependent by construction (same landscape, ~75% at 300 steps) and two runs at different budgets cannot be compared on it. Marks "when they happen" as partly answered by the survival-curve inversion (mean death ~step 43 of 150, so mid-descent rather than bad draws), and makes a disagreement with that inversion a finding in its own right rather than a re-derivation. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- .../autolens_profiling/mge_lane_death.md | 45 ++++++++++++++----- 1 file changed, 33 insertions(+), 12 deletions(-) diff --git a/draft/research/autolens_profiling/mge_lane_death.md b/draft/research/autolens_profiling/mge_lane_death.md index 5b22acfa..7ea77813 100644 --- a/draft/research/autolens_profiling/mge_lane_death.md +++ b/draft/research/autolens_profiling/mge_lane_death.md @@ -79,18 +79,39 @@ scalar percentage only within a fixed budget.** ## What to establish -- **Where the NaNs come from.** Which parameters/regions produce a non-finite - likelihood in the MGE cell — the mask edge, a Gaussian `sigma` collapsing, the - linear inversion, the `sqrt` at r=0, something else. The hazard index in this - repo is the natural place to look first and to record the answer. -- **When they happen.** Deaths concentrated in the first steps (bad draws that - `_broad_starts` should have filtered) mean something different from deaths - accumulating throughout (trajectories walking into a wall). -- **Whether `resurrect=True` recovers the budget** on this cell, and what it - costs. The docstring says it is for pixelized sources; this evidence suggests - the parametric cell may need it too. -- **Whether it reproduces at production budget and on GPU**, and across seeds. - The measurement above is one seed at reduced budget. +Ordered by information per unit time, which is **not** the order they were +originally listed in. The cause is the unanswered question and it is the cheap +one; the GPU reproduction confirms an effect that has already been measured once. +Do not queue for a GPU before step 1 has been attempted. + +1. **Where the NaNs come from** — the actual question. Which parameters/regions + produce a non-finite likelihood in the MGE cell: the mask edge, a Gaussian + `sigma` collapsing, the linear inversion / NNLS solve, the `sqrt` at r=0, + something else. **This does not need production budget or a GPU.** The + existing ~6-minute 16x150 CPU run already produces 1498 deaths; instrument it + to dump each lane's parameter vector at the step its value first goes + non-finite, and the distribution of those vectors is the answer. That is a + minutes-long iteration loop, not a queued one. Record the finding in this + repo's hazard index. +2. **Whether `resurrect=True` recovers the budget** on this cell, and what it + costs. The docstring says resurrection is for pixelized sources; this evidence + suggests the parametric cell may need it too. Measurement of the candidate + remedy — not adoption of it (see Boundary). +3. **Whether it holds at production budget, on GPU, across seeds.** The + measurement above is one seed at reduced budget on CPU, and the float64 GPU + path is not the same numerics. **Grade this on the alive-versus-step curve, + not on recovering "62%"** — per the survival-curve section above, the scalar + is budget-dependent by construction and the same landscape reports ~75% at 300 + steps. Log `alive N/16` per step so the curves can be overlaid across budgets; + a run that only reports the final percentage cannot be compared to this one. + +**When they happen** is already partly answered and should not be re-derived from +scratch: inverting the 1498 gives a mean death step of ~43 of 150, so deaths +concentrate in the first third — trajectories walking into a wall mid-descent, +not bad initial draws that `_broad_starts` failed to filter. Step 1's per-lane +death steps should confirm or overturn that inversion; if they disagree with a +mean of ~43, the disagreement is itself a finding (it would mean lanes are +recovering and re-dying, which `resurrect=False` says they cannot). ## Boundary From 3cc94642d0a51969a3999d6335c0ac5b3f69ab3a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 22:01:23 +0000 Subject: [PATCH 16/20] prompt: file issue for mge-lane-death (autolens_profiling#128) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Advances the prompt draft/ -> active/ and registers the task, per the lifecycle. The active.md entry records what is settled (62.42% value-NaN, alive 2/16, ell_comps cleared by a validated zero, resurrect=False the reason dead lanes stay dead) separately from what the task actually owes — the cause, which nothing so far identifies. Carries two things forward that are easy to lose: 62% is a survival integral so re-runs must be graded on the alive-versus-step curve rather than the scalar, and the step ordering puts cause-finding on the existing 6-minute CPU run ahead of any GPU queue. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- active.md | 14 ++++++++++++++ .../mge_lane_death.md | 0 2 files changed, 14 insertions(+) rename {draft/research/autolens_profiling => active}/mge_lane_death.md (100%) diff --git a/active.md b/active.md index 5f046002..bc9a14b0 100644 --- a/active.md +++ b/active.md @@ -1,5 +1,19 @@ # Active Tasks +## mge-lane-death +- issue: https://github.com/PyAutoLabs/autolens_profiling/issues/128 +- prompt: active/mge_lane_death.md +- status: planned — research/measurement in autolens_profiling; no worktree claimed yet +- worktree: ~/Code/PyAutoLabs-wt/mge-lane-death (not yet created) +- repos: + - autolens_profiling: research/mge-lane-death (not yet created) +- finding so far: the `imaging/mge` cell at 16x150 on CPU reports 1498/2400 value-NaN lane-steps (62.42%), 9 gradient-NaN, 0 constrained, 0 resurrections; population falls `alive 16/16` -> `alive 2/16`. The `ell_comps` plateau is CLEARED as a suspect (constrained count is a validated zero — see the positive control in the prompt). +- what is NOT yet known: the CAUSE. Which parameters/regions and which operation produce the non-finite likelihood. That is the whole task. +- reading the number: 62% is a survival integral, not a hazard rate — a frozen lane keeps counting every subsequent step, so the same death curve reports ~75% at 300 steps. Inverting it gives a mean death step of ~43 of 150 (mid-descent, not bad initial draws). Grade any re-run on the alive-versus-step CURVE, not on recovering the scalar. +- ordering (deliberate, do not revert): cause-finding FIRST on the existing ~6-min CPU run, then the `resurrect=True` budget-recovery measurement, then the production/GPU/seed confirmation. Do not queue for a GPU before the cause step has been attempted. +- boundary: investigation only. Changing the `resurrect` default is a separate PyAutoFit task — it would shift every existing multi-start benchmark. +- upstream: PyAutoFit#1475 (`004f798`) + PyAutoGalaxy#572 (`695b27c`) shipped the trapped-lane counter; record in `complete/2026/08/frozen-lane-counter.md`. + ## pix-prodigy-gpu-compat - issue: https://github.com/PyAutoLabs/autolens_workspace_developer/issues/125 - prompt: active/pixelized_prodigy_laptop_gpu_phase_1_compatibility.md diff --git a/draft/research/autolens_profiling/mge_lane_death.md b/active/mge_lane_death.md similarity index 100% rename from draft/research/autolens_profiling/mge_lane_death.md rename to active/mge_lane_death.md From ed86189bb514d9983b34ea8750f43258869dce34 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 22:31:46 +0000 Subject: [PATCH 17/20] =?UTF-8?q?prompt:=20record=20the=20mge=20lane-death?= =?UTF-8?q?=20cause=20=E2=80=94=20it=20is=20the=20prior,=20not=20the=20lik?= =?UTF-8?q?elihood?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root-caused on cloud CPU and written up on autolens_profiling#128. The objective is -2 * (log_likelihood + sum(log_prior_list)). A UniformPrior is -inf outside its box and MultiStartGradient steps in physical space with no projection back onto it, so a lane crossing a hard prior edge reads non-finite and resurrect=False never redraws it. The likelihood never went non-finite in ~7200 lane-steps across three arms. Decisive arm: neutering log_prior_list_from_vector drops value-NaN from 1446 to 215 and survivors from 2 to 13, with all residual deaths being NaN-params. A narrower hypothesis — widen the shear box, which accounted for 10 of 11 exits — was refuted: deaths moved later and got marginally worse, because widening one box only moves the wall. Counter-finding that corrects the earlier framing: the ell_comps plateau was masked, not cleared. The zero was correctly measured and the positive control was sound, but it meant "nothing got that far" — with the prior deaths removed the constrained count is 667 (27.79%). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- active.md | 10 ++++-- active/mge_lane_death.md | 70 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 3 deletions(-) diff --git a/active.md b/active.md index bc9a14b0..421eb944 100644 --- a/active.md +++ b/active.md @@ -3,12 +3,16 @@ ## mge-lane-death - issue: https://github.com/PyAutoLabs/autolens_profiling/issues/128 - prompt: active/mge_lane_death.md -- status: planned — research/measurement in autolens_profiling; no worktree claimed yet +- status: CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed confirmation, and the two follow-ups below. - worktree: ~/Code/PyAutoLabs-wt/mge-lane-death (not yet created) - repos: - autolens_profiling: research/mge-lane-death (not yet created) -- finding so far: the `imaging/mge` cell at 16x150 on CPU reports 1498/2400 value-NaN lane-steps (62.42%), 9 gradient-NaN, 0 constrained, 0 resurrections; population falls `alive 16/16` -> `alive 2/16`. The `ell_comps` plateau is CLEARED as a suspect (constrained count is a validated zero — see the positive control in the prompt). -- what is NOT yet known: the CAUSE. Which parameters/regions and which operation produce the non-finite likelihood. That is the whole task. +- CAUSE: the deaths are in the PRIOR term, not the likelihood. The objective is `fom = -2 * (log_likelihood + sum(log_prior_list))` (`Fitness(fom_is_log_likelihood=False)`); a `UniformPrior` is `-inf` outside its box; `MultiStartGradient` steps in PHYSICAL space with no projection back onto that box. A lane crossing a hard prior edge reads as non-finite, and `resurrect=False` never redraws it, so it stays dead for every remaining step — that accumulation IS the 62%. **The likelihood never went non-finite in ~7200 lane-steps across three arms.** +- evidence: per-lane autopsy at the death vectors — 11/14 have finite likelihood at every pipeline stage and `sum(log_prior) = -inf` with 1-2 params outside a `UniformPrior`; 2/14 have NaN params (the gradient path); 1/14 unexplained. Decisive arm: neutering `log_prior_list_from_vector` -> zeros drops value-NaN 1446 -> 215 (60.25% -> 8.96%) and survivors 2 -> 13, with all 3 residual deaths being NaN-params. A narrower hypothesis (widen the shear box, which was 10 of the 11 exits) was REFUTED — deaths moved later and got marginally worse, because widening one box only moves the wall. +- reproduction: 16x150 cloud CPU gave 1446/18/0/0 and `alive 2/16` against the filed 1498/9/0/0 and the same 2/16. The survival identity is exact: `sum(150 - k_i) = 14*150 - 654 = 1446` = `n_value_nan_lane_steps`. +- COUNTER-FINDING, corrects the framing: the `ell_comps` plateau was MASKED, not cleared. The baseline's `n_constrained_lane_steps = 0` was a correctly-measured zero (the positive control was sound) but it meant "nothing got that far" — lanes died of prior-exit first. With the prior deaths removed the constrained count is 667 (27.79%). #1475's trapped-lane counter is measuring a live failure mode on this cell, hidden behind a larger one. Lanes stop being dead and start being STUCK. +- follow-ups owed (both out of this task's boundary): (1) PyAutoFit — bounded stepping (projection/clipping onto prior support) or soft-walled priors; `resurrect=True` is NOT the fix, it redraws a lane that then walks out again. (2) the ell_comps trapping at 27.79%, now that it is visible. +- caveat: one baseline death (lane 9, step 39) re-evaluates finite in every term with all params inside their boxes — the jitted/vmapped float32 path differs from the eager recompute there, unexplained. Single seed per arm, CPU, x64 off. - reading the number: 62% is a survival integral, not a hazard rate — a frozen lane keeps counting every subsequent step, so the same death curve reports ~75% at 300 steps. Inverting it gives a mean death step of ~43 of 150 (mid-descent, not bad initial draws). Grade any re-run on the alive-versus-step CURVE, not on recovering the scalar. - ordering (deliberate, do not revert): cause-finding FIRST on the existing ~6-min CPU run, then the `resurrect=True` budget-recovery measurement, then the production/GPU/seed confirmation. Do not queue for a GPU before the cause step has been attempted. - boundary: investigation only. Changing the `resurrect` default is a separate PyAutoFit task — it would shift every existing multi-start benchmark. diff --git a/active/mge_lane_death.md b/active/mge_lane_death.md index 7ea77813..4255163a 100644 --- a/active/mge_lane_death.md +++ b/active/mge_lane_death.md @@ -129,6 +129,76 @@ extra; the runner writes into `dataset/`). +## CAUSE FOUND (2026-08-15, cloud CPU) — it is the prior, not the likelihood + +Written up in full on autolens_profiling#128. Summary, so the file stands alone: + +`_fit` builds its objective as `Fitness(..., fom_is_log_likelihood=False, +convert_to_chi_squared=True)`: + +``` +fom = -2 * (log_likelihood + sum(log_prior_list)) +``` + +A `UniformPrior` is `-inf` outside its box, and `MultiStartGradient` steps in +**physical** parameter space with **no projection back onto that box**. A lane +crossing a hard prior edge reads as non-finite; `resurrect=False` never redraws +it; it stays dead for every remaining step. That accumulation is the 62%. + +**The likelihood never went non-finite** — not once in ~7200 lane-steps across +three arms. Every pipeline stage (positions penalty, deflections, convergence, +model data, residuals, chi-squared, NNLS reconstruction, log-determinants, +`figure_of_merit`) was finite at every death vector. + +| arm (16x150, cloud CPU) | value-NaN | grad-NaN | constrained | dead | alive end | +|---|---:|---:|---:|---:|---:| +| reproduction | 1446 (60.25%) | 18 | 0 | 14/16 | 2 | +| shear box widened to ±1 | 1422 (59.25%) | 36 | 13 | 15/16 | 1 | +| **prior term neutered** | **215 (8.96%)** | 27 | **667 (27.79%)** | **3/16** | **13** | + +Per-lane autopsy at the death vectors: 11/14 finite likelihood with +`sum(log_prior) = -inf`; 2/14 NaN params; 1/14 unexplained. All three residual +deaths in the neutered arm are NaN-params, not likelihood deaths. + +The survival identity is exact: `sum(150 - k_i) = 14*150 - 654 = 1446`, which is +`n_value_nan_lane_steps` to the unit — the counter *is* the area under the death +curve. + +**A narrower hypothesis was refuted.** 10 of the 11 box exits were the shear +`UniformPrior(-0.3, 0.3)`, but widening only those two priors did not collapse +the deaths — they moved later and got marginally worse. Widening one box only +moves the wall. Test the mechanism, not the parameter. + +### The docstring claim is correct, and irrelevant + +The parametric MGE likelihood does appear to have only measure-zero +singularities. What is wrong is using that to justify `resurrect=False`, since +the deaths come from the prior, which the claim never covered. And this is not +MGE-specific: any model with a hard-box prior and unbounded stepping has it. + +### Counter-finding: the ell_comps plateau was MASKED, not cleared + +The baseline's `n_constrained_lane_steps = 0` was correctly measured and the +positive control was sound — but it meant *"nothing got that far"*, not +*"nothing gets trapped there"*. Lanes died of prior-exit before reaching the +saturation plateau. Remove the prior deaths and the constrained count is **667 +(27.79%)**. #1475's counter is measuring a live failure mode on this cell that +was hidden behind a larger one. Lanes stop being dead and start being **stuck**. + +### Out of this task's boundary, now owed elsewhere + +1. **PyAutoFit** — bounded stepping (projection/clipping onto prior support) or + soft-walled priors. `resurrect=True` is *not* the fix: it redraws a lane that + then walks out again. +2. **The ell_comps trapping at 27.79%**, now that it is visible. + +### Still owed here + +GPU / float64 / multi-seed confirmation, graded on the alive-versus-step curve. +One caveat: baseline lane 9 (step 39) re-evaluates finite in every term with all +params inside their boxes — the jitted/vmapped float32 path differs from the +eager recompute there, unexplained. + ## Reproducer Both scripts below are self-contained and were used to produce the numbers in From 53fdedce919ad0336c472241a754cb76d4295217 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 23:38:16 +0000 Subject: [PATCH 18/20] prompt: file the search-agnostic prior-support Clipper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix for the MGE lane-death cause found in autolens_profiling#128: the search steps in physical space while UniformPrior is -inf outside its box, so lanes that overstep an edge read non-finite and are never redrawn. Scoped search-agnostic because the exposure is not MultiStart-only. LBFGS builds the same log-posterior Fitness, steps in physical space, and calls optimize.minimize with no bounds= at all, despite L-BFGS-B supporting them. Design departs from the Initializer analogy in one place: Clipper has two structurally different consumers — MultiStartGradient enforces per step and wants project(), LBFGS delegates to scipy and wants bounds — so the base class serves both from one source of truth. project() returns the clipped-coordinate mask, because the prototype left 5/16 lanes pinned to a bound with Prodigy state still pushing outward, and the Clipper cannot reset momentum it does not own. Default is ClipperNone and PR 1 is bit-identical, per the #1475 precedent; flipping the default carries the benchmark re-baseline and is PR 2. Also records two incidental bugs that only appear when lanes survive: float32 breaking save_json, and a crashed run poisoning the next same-named run into a 4-second no-op that reads as a clean result. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- .../feature/autofit/prior_support_clipper.md | 214 ++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 draft/feature/autofit/prior_support_clipper.md diff --git a/draft/feature/autofit/prior_support_clipper.md b/draft/feature/autofit/prior_support_clipper.md new file mode 100644 index 00000000..04954d4d --- /dev/null +++ b/draft/feature/autofit/prior_support_clipper.md @@ -0,0 +1,214 @@ +# Search-agnostic prior-support enforcement: a Clipper class + +Type: feature +Target: PyAutoFit +Repos: +- PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +## Why + +`@PyAutoFit/autofit/non_linear/search/mle/multi_start_gradient/search.py` builds +its objective as + +``` +fom = -2 * (log_likelihood + sum(log_prior_list)) +``` + +A `UniformPrior` returns `log_prior = -inf` outside its box, and the search steps +in **physical** parameter space with nothing constraining it to that box. A lane +that oversteps a hard prior edge reads as non-finite, is marked dead, and with +`resurrect=False` is never redrawn. + +Measured on the real `imaging/mge` profiling cell (16 starts x 150 steps, cloud +CPU) — full investigation and evidence in autolens_profiling#128: + +| arm | value-NaN lane-steps | lanes dead | alive at end | +|---|---:|---:|---:| +| baseline | 1446 (60.25%) | 14/16 | 2 | +| shear box widened to ±1 | 1422 (59.25%) | 15/16 | 1 | +| prior term neutered (diagnostic) | 215 (8.96%) | 3/16 | 13 | +| **clip to prior box (prototype)** | **425 (17.71%)** | **5/16** | **11** | + +**The likelihood never went non-finite** in ~7200 lane-steps. This is entirely a +prior-support problem. + +The behaviour is worse than "frozen": the overshoot is tiny (median 3% of box +width, min 0.21%), and because `log_prior = -inf` is *constant* outside the box +its derivative is zero, so the total gradient is the finite **likelihood** +gradient. `optax.apply_if_finite` therefore never fires and the dead lane **keeps +stepping forever** — full likelihood-and-gradient cost every step, output +discarded, wandering far (one parameter went `0.30 -> -1.76`). 0/16 lanes ever +revive. + +## The exposure is not MultiStart-only + +This is why the fix should not live inside one search: + +- **`MultiStartGradient`** (`MultiStartAdam` / `MultiStartADABelief` / + `MultiStartLion` / `MultiStartProdigy` all share one `_fit`) — measured above. +- **`@PyAutoFit/autofit/non_linear/search/mle/bfgs/search.py`** — same + `Fitness(fom_is_log_likelihood=False, resample_figure_of_merit=-np.inf, + convert_to_chi_squared=True)`, steps in physical space, and calls + `optimize.minimize(fun=..., x0=..., method=self.method, options=..., tol=...)` + with **no `bounds=` argument**. `L-BFGS-B` supports box bounds natively; they + are simply not passed. Being single-start, this presents as a failed fit rather + than a population collapse, so it is easier to misattribute. +- **NUTS** (`@PyAutoFit/autofit/non_linear/search/mcmc/blackjax/nuts/search.py`) + also targets the log posterior from a physical `initial_position`. HMC entering + a `-inf` region *diverges* rather than freezing. **Out of scope here** — different + mechanism, needs its own investigation. See "Deliberately out of scope". + +Not exposed, and correctly so: the nested samplers already work in unit-cube +coordinates, and the MCMC samplers reject `-inf` proposals so the walker stays +put. **Rejection is the restoring mechanism that gradient methods lack.** + +## The design + +A `Clipper`, modelled on `@PyAutoFit/autofit/non_linear/initializer.py` — a +pluggable, per-search strategy object with a config-resolved default. + +**One place the `Initializer` analogy does not carry.** `Initializer` has a single +consumption pattern (`samples_from_model`). `Clipper` has **two structurally +different consumers** and must serve both from one source of truth: + +- `MultiStartGradient` enforces the constraint itself, every step → wants an + imperative `project(...)`. +- `LBFGS` hands bounds to scipy and lets *scipy* enforce → wants a declarative + `bounds`. + +Proposed contract: + +```python +class AbstractClipper(ABC): + @abstractmethod + def bounds_from_model(self, model) -> tuple[np.ndarray, np.ndarray]: + """(lower, upper) in PHYSICAL parameter order. Unbounded -> -inf/+inf.""" + + @abstractmethod + def project(self, vector, model, xp=np): + """Return (projected_vector, clipped_mask). Identity where unbounded.""" + + +class ClipperNone(AbstractClipper): + """No-op. Bounds are ±inf, project is the identity. THE DEFAULT (see below).""" + + +class ClipperPriorBox(AbstractClipper): + """Hard projection onto the prior support, inset by a margin.""" +``` + +`project` **must return which coordinates it clipped**, not just the new vector. +That mask is what lets a caller zero the optimiser momentum along clipped +directions. It is needed: the prototype left 5 of 16 lanes pinned to a bound at +the end of the run because the parameters were projected while Prodigy's +accumulated state kept pushing outward. The `Clipper` cannot fix that itself — it +does not own `opt_state` — so it must expose enough for the search to. + +Later strategies (`ClipperReflect`, a soft-wall variant) drop in without touching +callers. A soft wall must be a *Clipper* (search-local), **never** a change to the +`Prior` classes — that would silently alter the objective for the nested samplers, +where the hard box currently works correctly. + +## Scope — PR 1 (this task) + +1. `AbstractClipper` + `ClipperNone` + `ClipperPriorBox` in a new + `@PyAutoFit/autofit/non_linear/clipper.py`. +2. Bounds extraction covering **every** prior type. Confirmed present in the + reference model: `UniformPrior` (finite both sides), `TruncatedGaussianPrior` + (finite both sides, e.g. `(-1, 1)` for `ell_comps`), `GaussianPrior` + (`±inf` — must pass through untouched). Audit the rest (`LogUniformPrior`, + `LogGaussianPrior`, any `Constant`/deterministic entries). +3. Wire into `AbstractMultiStartGradient._fit`, applied after + `optax.apply_updates`, **opt-in**. +4. Wire into `LBFGS`, passing `bounds=` through to `optimize.minimize`, + **opt-in**. Only valid for bound-supporting methods (`L-BFGS-B`, `TNC`, + `SLSQP`) — guard or warn for plain `BFGS`. +5. `clipper: AbstractClipper = None` constructor arg on the searches, resolved + like `initializer`. + +**Default is `ClipperNone`, and PR 1 must be bit-identical with it.** Follow the +precedent set by PyAutoFit#1475, whose models declaring no constraint +short-circuit to bit-identical behaviour. Flipping the default is a real +behaviour change that shifts every stored multi-start benchmark, which is exactly +the comparability argument PyAutoFit#1472 made when it deferred its own policy +change. + +## Scope — PR 2 (separate prompt, file after PR 1 lands) + +Flip `MultiStartGradient`'s default to `ClipperPriorBox`, **with** the benchmark +re-baseline, plus the momentum-reset-on-clip decision informed by how bad the +pinning actually is at production budget. + +## Traps, measured + +- **Parameter ordering is load-bearing and silent if wrong.** + `model.priors_ordered_by_id` was used for the prototype and lined up correctly + with `model.instance_from_vector`, but a mismatch would clip the *wrong + parameter* with no error. Assert the correspondence in a test rather than + trusting it. +- **Boundary semantics.** Decide and document whether `log_prior` at *exactly* the + limit is finite. The prototype inset by `1e-6` of the box width to stay strictly + inside; that margin is a guess and should be a justified constant. +- **Pinning is correct behaviour, not a bug.** Where the likelihood genuinely + prefers a value outside the prior, a clipped lane sitting on the bound is the + correct MAP answer under the declared prior. It is worth surfacing (it says the + prior is fighting the data) rather than hiding. In the reference cell the shear + escapes were mixed-sign (`+0.353`, `-0.341`, `+0.301`, `-0.312`), which reads + more like a poorly-constrained parameter diffusing out than a true value sitting + outside. +- **Clipping does not fix every death.** 5/16 lanes still died in the prototype; + those are the NaN-gradient population (likelihood NaN in the *jitted* path, + which the `Fitness` guard maps to `-inf` and whose `where` makes the gradient + NaN). Separate mechanism, do not expect this task to remove it. + +## Two incidental bugs found while investigating — do not lose these + +Both surfaced only because clipping let lanes *survive*, i.e. on a code path this +cell had apparently never taken: + +1. **`float32` is not JSON serializable in result output.** + `@PyAutoFit/autofit/non_linear/paths/directory.py:80` `save_json` raises + `TypeError: Object of type float32 is not JSON serializable` at the end of a + successful clipped run. Did not fire on the baseline runs, where 14/16 lanes + were dead. File separately if confirmed. +2. **A crashed run poisons the next run of the same name.** The half-written + output left by (1) caused the next search with the same `name` to fail with + `JSONDecodeError` while trying to resume — a 4-second no-op run that *looked + like* a clean result (zero deaths, because zero steps). This is a new form of + the cached-result hazard already recorded in + `complete/2026/08/multistart-nan-step-diagnostics.md`. + +## Deliberately out of scope + +- **NUTS.** Divergence, not lane death; may need a transform or a soft wall rather + than projection. Its own task. +- **Unit-cube stepping.** The more principled long-term fix — PyAutoFit's prior + machinery is already unit-cube and the nested samplers work that way, and it + would also normalise parameter scales (`einstein_radius ∈ [0,8]` alongside + `ell_comps ∈ [-1,1]`). Rejected *for now* on three grounds: a logit + reparameterisation sends the optimum to infinity when it genuinely sits on a + boundary, which this cell demonstrably has; the inverse-CDF transform for + non-uniform priors has `∂θ/∂u -> ∞` at the cube faces, trading one numerical + hazard for another; and it invalidates every stored benchmark. If pursued, note + that reparameterising the *search path* does not move the optimum **provided the + objective is still the physical-space posterior evaluated at `θ(u)`** — optimise + the density *of u* instead and the Jacobian makes the MAP non-invariant, which + fails silently. +- **Changing `resurrect` defaults.** Not the fix: a redrawn lane walks out again. + +## Testing + +- Bounds extraction per prior type, including `±inf` passthrough for `GaussianPrior`. +- Ordering assertion (see traps). +- `ClipperNone` is bit-identical: same seed, same final parameters, on both + `MultiStartGradient` and `LBFGS`. +- A lane deliberately stepped across a boundary is projected back inside, and the + returned mask names exactly the crossed coordinates. +- `LBFGS` passes bounds through and rejects/warns for non-bound-supporting methods. +- Regression: with `ClipperPriorBox` on a model with a tight `UniformPrior`, the + value-NaN rate falls substantially. The reference numbers above are CPU/float32, + single seed — assert a direction and a large margin, not an exact figure. From 8a64883e9951ccce9b8854a077b01159c554ab78 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 23:47:34 +0000 Subject: [PATCH 19/20] prompt: file the Clipper validation campaign (phase 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sequences the prior-support fix as three PRs: PyAutoFit Clipper (opt-in, bit-identical) -> autolens_profiling validation -> PyAutoFit default flip. This files the middle one, which is where the fix is either justified or is not. Anchors validation on a truth bar that already exists: the Nautilus run on the same imaging/mge cell (max_log_likelihood 31786.78, A100 fp64). Nautilus samples in unit-cube coordinates so it is structurally immune to this bug, which makes it a reference answer rather than another data point. The load-bearing test is therefore whether clipped MultiStartProdigy moves TOWARD it — "fewer lanes die" would be satisfied by a change that keeps lanes alive and useless. States the falsification conditions up front, and marks pinning at a bound as a possible science finding about the shear prior rather than a clipping artefact. Carries the traps this investigation already paid for: grade on the alive curve not the scalar (the counter is a survival integral), delete output between arms or a crashed run resumes into a 4-second no-op that reads as clean, and 0 is not null. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- .../autofit/clipper_validation_campaign.md | 172 ++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 draft/feature/autofit/clipper_validation_campaign.md diff --git a/draft/feature/autofit/clipper_validation_campaign.md b/draft/feature/autofit/clipper_validation_campaign.md new file mode 100644 index 00000000..efdacdcd --- /dev/null +++ b/draft/feature/autofit/clipper_validation_campaign.md @@ -0,0 +1,172 @@ +# Clipper: demonstrate and validate on the profiling search cells + +Type: feature +Target: autolens_profiling +Repos: +- autolens_profiling +- PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised + +## What this is + +The **validation phase** of the prior-support fix. It runs after +`draft/feature/autofit/prior_support_clipper.md` lands the `Clipper` in +`@PyAutoFit`, and it produces the evidence that justifies flipping the default. + +Three phases, three PRs, in this order — do not merge them out of order: + +| phase | repo | what | +|---|---|---| +| 1 | `@PyAutoFit` | `Clipper` class, opt-in, bit-identical by default (`prior_support_clipper.md`) | +| **2** | **`@autolens_profiling`** | **this task — demonstrate and validate across the search cells** | +| 3 | `@PyAutoFit` | flip the default to `ClipperPriorBox`, carrying the phase-2 re-baseline | + +Phase 2 is where the claim "clipping recovers the lost lanes without changing the +answer" either survives or does not. Phase 3 must not be written until phase 2 +has run. + +Background and the full root-cause investigation: **autolens_profiling#128**. +One-line version: the objective is `-2 * (log_likelihood + sum(log_prior))`, a +`UniformPrior` is `-inf` outside its box, the search steps in physical space with +nothing holding it there, and a lane that oversteps by ~3% of a box width is +marked dead and never redrawn — while continuing to step and consume full +likelihood-and-gradient cost with its output discarded. + +## The truth bar already exists — use it + +`results/searches/nautilus/imaging/mge/hst/hpc_a100_fp64.json` is a Nautilus run +on **the same cell, the same 15-parameter model**: + +``` +max_log_likelihood = 31786.782462488976 +log_evidence = 31690.47079355404 +posterior_samples = 63800 (A100, fp64, n_live 200) +``` + +Nautilus samples in **unit-cube coordinates**, so it is structurally immune to +this bug. That makes it the reference answer, not merely another data point. + +**This is the load-bearing validation.** "Fewer lanes die" is a weak claim on its +own — a change that keeps lanes alive by making them useless would satisfy it. +The claim worth testing is: **does clipped MultiStartProdigy get closer to the +Nautilus maximum log-likelihood than unclipped does?** If lane deaths fall and +best-fit logL does *not* improve toward 31786.8, clipping is cosmetic and phase 3 +should not happen. + +## Arms + +Per cell, at minimum: + +1. `clipper=None` / `ClipperNone` — control, must reproduce today's numbers. +2. `ClipperPriorBox` — the candidate. +3. `ClipperPriorBox` **+ momentum reset** on clipped coordinates, if phase 1 + shipped it. The prototype left 5/16 lanes *pinned* to a bound with Prodigy's + state still pushing outward; this arm is what says whether that matters. + +Record for every arm: `n_value_nan_lane_steps`, `n_grad_nan_lane_steps`, +`n_constrained_lane_steps`, `n_resurrections`, **`alive N/n_starts` per step**, +best-fit log-likelihood, wall time, and the count of lanes ending pinned to a +bound. + +**At least two seeds per arm.** Single-seed CPU numbers are what this whole +investigation had to go back and re-derive. + +## Cells + +Start with the characterised one, then widen: + +- **`imaging/mge` (hst)** — the reference. ~250s at 16x150 on cloud CPU, so it + iterates fast. Known baseline: 1446/2400 value-NaN (60.25%), `alive 16/16 -> + 2/16`, 14 lanes dead at steps `[34,36,36,37,38,39,39,40,40,41,41,52,56,125]`. + Prototype clipping gave 425 (17.71%), 5 dead, `alive -> 11`. +- **The pixelized mesh cells** (`delaunay`, `pixelization`, DelaunayNN) — **GPU + required**. These are the cells `resurrect=True` was introduced for, so they are + the strongest test of whether clipping changes the resurrection story. Two prior + attempts timed out on CPU with zero steps emitted; that is **JIT compile, not + memory** (`batch_size=1` clears the OOM). Do not shrink the source mesh to make + them cheaper — the image-plane grid at `mask_radius 3.5` dominates, not the mesh. +- **`point_source`** cells — different model family and different prior structure; + confirms the fix is not MGE-shaped. +- **A negative control**: a model whose priors are all unbounded (`GaussianPrior`). + `ClipperPriorBox` must be a *no-op* there. If it changes anything, the bounds + extraction is wrong. + +## What would falsify the fix + +Write these down before running, and report them honestly if they happen: + +- Lane deaths fall but best-fit logL does **not** move toward the Nautilus + reference → clipping keeps lanes alive without making them useful. +- Most surviving lanes end pinned to a bound → the wall is absorbing the + population; momentum reset is mandatory, or projection is the wrong strategy. +- Wall time per step rises materially → a clip on `(n_starts, ndim)` should be + unmeasurable; if it is not, something is wrong with where it was inserted. +- The pixelized cells get *worse* → clipping and `resurrect=True` interact badly, + and phase 3 must be scoped per-search rather than globally. + +## Pinning is a result, not a failure + +Where the likelihood genuinely prefers a value outside the prior, a clipped lane +sitting on the bound is the **correct MAP answer under the declared prior**. In +the reference cell the shear escapes were mixed-sign (`+0.353`, `-0.341`, +`+0.301`, `-0.312`), which reads more like a poorly-constrained parameter +diffusing out than a true value sitting outside — but that is a hypothesis, not a +finding. If clipped runs pin `gamma` at `±0.3` reproducibly, that is evidence the +shear prior is fighting the data and belongs in the write-up as a science finding, +not swept up as a clipping artefact. + +## Deliverables + +- Results JSONs under `results/searches/` alongside the existing NaN-accounting + artefacts, following the conventions already there. +- A note under `results/notes/` — the comparison table, the Nautilus-reference + verdict, and an explicit recommendation for or against phase 3. +- The hazard-index entry for the prior-exit failure mode (owed from #128). + +## Environment + +- **Python 3.12+** (autonerves). `pip install jaxnnls` is **required** for the JAX + NNLS solver path and is not pulled in by default. `optax` likewise. +- Install `autolens` with **`--no-deps`** when running editable local + `autofit`/`autogalaxy`, or the released wheels clobber them. Phase 1 is + unreleased, so this task **must** run against `@PyAutoFit` `main` (or the phase-1 + branch), not a PyPI wheel — verify `autofit.__file__` resolves to the checkout + before trusting any number. +- `build_for_cell` **writes into `dataset/`** (rewrites the HST FITS, adds + `positions.json`, emits `results/simulators/*`). Not read-only. +- Cell scripts honour `SEARCHES_N_STARTS` / `SEARCHES_N_STEPS` / + `SEARCHES_BATCH_SIZE` / `SEARCHES_DISABLE_VIZ`. +- On A100 set `jax_enable_x64` **explicitly** — it is not inherited under `sbatch`, + and float32 would understate the quantity under test. All #128 numbers are + float32 CPU; expect them to move on fp64 and do not treat a difference as a + regression without checking precision first. + +## Traps, all paid for already + +- **Grade on the alive-versus-step curve, not the percentage.** `n_value_nan_lane_steps` + is a *survival integral*: a dead lane keeps counting every subsequent step, so + the same death curve reports 60% at 150 steps and ~75% at 300. Verified exactly: + `sum(150 - k_i) = 14*150 - 654 = 1446` = the counter, to the unit. Two arms at + different budgets cannot be compared on the scalar. +- **A crashed run poisons the next run of the same `name`.** A half-written output + JSON makes the next search try to *resume* and fail — a 4-second no-op that reads + as a clean result (zero deaths, because zero steps). Delete `output//` + between arms, or use unique names, and **assert the recorded step count equals + `n_steps`** before believing any counter. +- **`float32` breaks `save_json`.** `autofit/non_linear/paths/directory.py:80` + raises `TypeError: Object of type float32 is not JSON serializable` at the end of + a *successful* run. It does not fire when most lanes are dead, so it will start + firing exactly when the fix works. If phase 1 has not fixed it, capture counters + independently of the result object. +- **`0` and `null` are different findings.** Read counters with `.get()` and a + `null` means the search never wrote the key — broken plumbing, not a clean cell. + +## Deliberately out of scope + +- Flipping any default (phase 3). +- NUTS. It targets the log posterior from a physical start and *diverges* rather + than dying — a different mechanism needing its own investigation. +- Unit-cube stepping. Rejected for now in `prior_support_clipper.md`, with reasons. From 506b14df2ed45152fdc073ee84442f3f4d89c98c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 23:54:21 +0000 Subject: [PATCH 20/20] mind: regenerate dashboard.md for the lane-death and Clipper prompts The refresh check failed on the branch: two new draft prompts and the draft/ -> active/ move for mge_lane_death left dashboard.md stale. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011vKZhT7hoMMnAGziPk24y1 --- dashboard.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/dashboard.md b/dashboard.md index 14970a4a..1ba7b353 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,21 +8,22 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 7 | +| [In flight](#in-flight) (`active/`) | 8 | | [Parked](#parked) (`parked.md`) | 6 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 137 | +| [Backlog](#backlog) (`draft/`) | 138 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) ## Start here -**Highest priority** (filed as `high`) — showing 12 of 31 +**Highest priority** (filed as `high`) — showing 12 of 32 - [pre_build stages untracked files, publishing uncommitted human work](draft/bug/pyautohands/pre_build_stages_untracked_wip.md) — pyautohands · small · supervised · high - [TRIAGE: needs manual review before routing](draft/triage/jax_zero_contour.md) — medium · safe · high +- [Clipper: demonstrate and validate on the profiling search cells](draft/feature/autofit/clipper_validation_campaign.md) — autofit · medium · supervised · high +- [Search-agnostic prior-support enforcement: a Clipper class](draft/feature/autofit/prior_support_clipper.md) — autofit · medium · supervised · high - [PyAutoLens RTD docs: three-regime restructure (multi_galaxy / group / cluster)](draft/docs/autolens/docs_three_regime_restructure.md) — autolens · medium · supervised · high -- [Find what kills MGE multi-start lanes — it is not](draft/research/autolens_profiling/mge_lane_death.md) — autolens_profiling · medium · supervised · high - [Optimize pixelized Prodigy settings on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu_phase_2_settings.md) — autolens_workspace_developer · medium · human-required · high - [Release does not sync __version__ stamps and workspace pins back](draft/bug/pyautobuild/release_version_sync_back_to_main.md) — pyautobuild · medium · supervised · high - [Make draft/ staleness detectable — `intake reconcile` measured, and the](draft/feature/pyautomind/draft_staleness_detection_signals.md) — pyautomind · medium · supervised · high @@ -30,7 +31,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A - [Cluster package: point-source-default narrative + extended-source follow-up feature](draft/docs/workspaces/cluster_regime_narrative.md) — workspaces · medium · supervised · high - [multi_galaxy package: new regime package in autolens_workspace](draft/docs/autolens/multi_galaxy_package.md) — autolens · large · supervised · high - [Tune the JAX multi-start optimizers into a standard option (MGE](draft/experiment/autolens_profiling/jax_optimizer_settings_tuning.md) — autolens_profiling · large · supervised · high -- [Profile and speed up JAX likelihood-function compile times (all use](draft/feature/autolens_profiling/jax_compile_time_profiling.md) — autolens_profiling · large · supervised · high **Quick wins** (small enough, and safe enough to run unattended) @@ -55,6 +55,7 @@ Issued — each has an open GitHub issue and usually a branch. The full record f - [Address ECEB editorial comments on ECLIPSE-C](active/euclid_eceb_editorial_revision.md) - [JAX-native posterior sampler wave — ranked shortlist from the 2026-07-16](active/jax_native_posterior_sampler_wave.md) — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius… - [Remove standalone matplotlib-inline comments](active/matplotlib_inline_standalones.md) +- [Find what kills MGE multi-start lanes — it is not](active/mge_lane_death.md) — [issue #128](https://github.com/PyAutoLabs/autolens_profiling/issues/128) — CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed… - [Pixelized Prodigy laptop-GPU compatibility across four meshes](active/pixelized_prodigy_laptop_gpu_phase_1_compatibility.md) — [issue #125](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/125) — workspace-dev — phase 1 and phase 2 COMPLETE 2026-08-13, all 13 cells landed, PR #126 ready for review - [PyAutoReduce validation: slacs1430+4105 ACS reduction vs trusted legacy dataset](active/pyautoreduce_slacs1430_acs_comparison.md) - [Research profiling experiment in the autolens_profiling repo](active/research_profiling_experiment_in_the_autolens_pr.md) — [issue #82](https://github.com/PyAutoLabs/autolens_profiling/issues/82) @@ -94,7 +95,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**137** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**138** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
bug — 40 @@ -143,8 +144,10 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-feature — 25 +feature — 27 +- [Clipper: demonstrate and validate on the profiling search cells](draft/feature/autofit/clipper_validation_campaign.md) — autofit · medium · supervised · high +- [Search-agnostic prior-support enforcement: a Clipper class](draft/feature/autofit/prior_support_clipper.md) — autofit · medium · supervised · high - [Make draft/ staleness detectable — `intake reconcile` measured, and the](draft/feature/pyautomind/draft_staleness_detection_signals.md) — pyautomind · medium · supervised · high - [LACosmic per-frame CR masking option + decouple PSF-star pass from](draft/feature/pyautoreduce/lacosmic_cr_option_and_star_pass_decoupling.md) — pyautoreduce · medium · supervised · high - [Profile and speed up JAX likelihood-function compile times (all use](draft/feature/autolens_profiling/jax_compile_time_profiling.md) — autolens_profiling · large · supervised · high @@ -201,9 +204,8 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-research — 21 +research — 20 -- [Find what kills MGE multi-start lanes — it is not](draft/research/autolens_profiling/mge_lane_death.md) — autolens_profiling · medium · supervised · high - [Optimize pixelized Prodigy settings on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu_phase_2_settings.md) — autolens_workspace_developer · medium · human-required · high - [Optimize MultiStartProdigy for pixelized meshes on the laptop GPU](draft/research/autolens_workspace_developer/pixelized_prodigy_laptop_gpu.md) — autolens_workspace_developer · large · human-required · high - [Deep research: Can we speed up Delaunay in PyAutoArray?](draft/research/autoarray/delaunay_research.md) — autoarray · too-large · supervised · high