diff --git a/active.md b/active.md index cd9c39b9..c5f91ea8 100644 --- a/active.md +++ b/active.md @@ -8,7 +8,9 @@ - autolens_profiling: feature/inference-programme-ledger - prompt: active/inference_programme_ledger.md - note: docs-only (results/notes/inference/ PROGRAMME.md + DECISIONS.md + LITERATURE.md). - Disjoint from the registered mge-lane-death claim (research/mge-lane-death, never created). + The mge-lane-death claim it was recorded as disjoint from is gone — archived + 2026-08-18 as superseded (`complete/2026/08/mge-lane-death.md`); its + `research/mge-lane-death` branch was never created. This is the canonical copy of the 2026-08-17 human-approved inference programme. ## positions-lh-penalty-accumulation @@ -69,26 +71,6 @@ - do-not: do NOT weaken PyAutoGalaxy `validate_ell_comps`; do NOT edit the tutorial. Test mode is NOT implicated (`ENV: real_search` releases `PYAUTO_TEST_MODE`) — verified, do not re-open. -## mge-lane-death -- issue: https://github.com/PyAutoLabs/autolens_profiling/issues/128 -- prompt: active/mge_lane_death.md -- status: CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed confirmation, and the two follow-ups below. -- worktree: ~/Code/PyAutoLabs-wt/mge-lane-death (not yet created) -- repos: - - autolens_profiling: research/mge-lane-death (not yet created) -- CAUSE: the deaths are in the PRIOR term, not the likelihood. The objective is `fom = -2 * (log_likelihood + sum(log_prior_list))` (`Fitness(fom_is_log_likelihood=False)`); a `UniformPrior` is `-inf` outside its box; `MultiStartGradient` steps in PHYSICAL space with no projection back onto that box. A lane crossing a hard prior edge reads as non-finite, and `resurrect=False` never redraws it, so it stays dead for every remaining step — that accumulation IS the 62%. **The likelihood never went non-finite in ~7200 lane-steps across three arms.** -- evidence: per-lane autopsy at the death vectors — 11/14 have finite likelihood at every pipeline stage and `sum(log_prior) = -inf` with 1-2 params outside a `UniformPrior`; 2/14 have NaN params (the gradient path); 1/14 unexplained. Decisive arm: neutering `log_prior_list_from_vector` -> zeros drops value-NaN 1446 -> 215 (60.25% -> 8.96%) and survivors 2 -> 13, with all 3 residual deaths being NaN-params. A narrower hypothesis (widen the shear box, which was 10 of the 11 exits) was REFUTED — deaths moved later and got marginally worse, because widening one box only moves the wall. -- reproduction: 16x150 cloud CPU gave 1446/18/0/0 and `alive 2/16` against the filed 1498/9/0/0 and the same 2/16. The survival identity is exact: `sum(150 - k_i) = 14*150 - 654 = 1446` = `n_value_nan_lane_steps`. -- COUNTER-FINDING, corrects the framing: the `ell_comps` plateau was MASKED, not cleared. The baseline's `n_constrained_lane_steps = 0` was a correctly-measured zero (the positive control was sound) but it meant "nothing got that far" — lanes died of prior-exit first. With the prior deaths removed the constrained count is 667 (27.79%). #1475's trapped-lane counter is measuring a live failure mode on this cell, hidden behind a larger one. Lanes stop being dead and start being STUCK. -- follow-ups owed (both out of this task's boundary), ALL NOW FILED 2026-08-16 — nothing from this investigation lives only inside another prompt: (1) PyAutoFit — bounded stepping (projection/clipping onto prior support) or soft-walled priors; `resurrect=True` is NOT the fix, it redraws a lane that then walks out again. **PHASE 1 SHIPPED 2026-08-16** as PyAutoFit#1477 (`1f4b66a`) → record `complete/2026/08/prior-support-clipper.md`; **PHASE 2 RAN AND SHIPPED 2026-08-16** (autolens_profiling#132 + PyAutoFit#1482, issues #129/#131 closed) → record `complete/2026/08/clipper-validation-campaign.md` — VERDICT: do NOT flip the clipper default on accuracy grounds (clipping eliminates the deaths and does not move the answer; seed dependence swings 171,272 nats and dwarfs it), and the momentum-reset arm is a clean negative. Phase 3, if ever written, must argue hygiene, not accuracy — it remains undrafted, and `draft/feature/autofit/clipper_in_search_identifier.md` is its prerequisite decision. (2) the ell_comps trapping at 27.79%, now that it is visible → `draft/research/autolens_profiling/ell_comps_trapping_unmasked.md` — note that 27.79% came from the prior-neutered DIAGNOSTIC arm and is not a citable production number; it runs on top of phase 1 and can share phase 2's arms. -- clipper-reporting: SHIPPED 2026-08-16 as PyAutoFit#1478 (`bbceff6`) → record `complete/2026/08/clipper-usage-in-search-summary.md`. `search.summary` now reports `Clipper`, `Clipped Lane-Steps`, `Clipped Lane-Step Rate` and `Constrained Lane-Steps`. CORRECTS an earlier note here that claimed `search.summary` had no search-specific channel at all — false; `search_summary_from_samples` already read `samples_info` and already emitted the NaN counters. The real gap was two omissions, not a missing mechanism. Phase 2 reads the clip count straight from `search.summary` now, and a `ClipperPriorBox` arm reporting ZERO clips has not exercised the clipper — a broken arm, not a null result. -- incidental PyAutoFit bugs found while investigating: BOTH SHIPPED 2026-08-16. `save_json` float32 crash → PyAutoFit#1479 (`b6e89cd`), record `complete/2026/08/save-json-numpy-scalar-typeerror.md`; fixing it turned up a second unguarded writer the prompt never named, `Samples.info_to_json`, which is the more dangerous since `samples_info` gains a counter every time a search does. Crashed-run-poisons-resume → PyAutoFit#1480 (`5c9244b`), record `complete/2026/08/crashed-run-poisons-resume.md`. CORRECTION carried in that record: the "zero-step no-op reported as a clean result" symptom did NOT reproduce; what reproduces is a hard `JSONDecodeError` on every rerun of the same search name. `JSONDecodeError` subclasses `ValueError`, which is why it slipped past every guard on the resume path. Both traps the phase-2 campaign worked around are now closed. -- caveat: one baseline death (lane 9, step 39) re-evaluates finite in every term with all params inside their boxes — the jitted/vmapped float32 path differs from the eager recompute there, unexplained. Single seed per arm, CPU, x64 off. -- reading the number: 62% is a survival integral, not a hazard rate — a frozen lane keeps counting every subsequent step, so the same death curve reports ~75% at 300 steps. Inverting it gives a mean death step of ~43 of 150 (mid-descent, not bad initial draws). Grade any re-run on the alive-versus-step CURVE, not on recovering the scalar. -- ordering (deliberate, do not revert): cause-finding FIRST on the existing ~6-min CPU run, then the `resurrect=True` budget-recovery measurement, then the production/GPU/seed confirmation. Do not queue for a GPU before the cause step has been attempted. -- boundary: investigation only. Changing the `resurrect` default is a separate PyAutoFit task — it would shift every existing multi-start benchmark. -- upstream: PyAutoFit#1475 (`004f798`) + PyAutoGalaxy#572 (`695b27c`) shipped the trapped-lane counter; record in `complete/2026/08/frozen-lane-counter.md`. - ## heart-green-validation-ingest - issue: https://github.com/PyAutoLabs/PyAutoGalaxy/issues/567 (open, reopened 2026-08-11T00:22Z) - session: none — the Codex session that did the work ran out of credits after merging, before any bookkeeping. Registered 2026-08-11 by a cloud session reconstructing its state from GitHub. diff --git a/active/mge_lane_death.md b/complete/2026/08/mge-lane-death.md similarity index 74% rename from active/mge_lane_death.md rename to complete/2026/08/mge-lane-death.md index 4255163a..3500273a 100644 --- a/active/mge_lane_death.md +++ b/complete/2026/08/mge-lane-death.md @@ -1,3 +1,128 @@ +MGE multi-start lanes die on the PRIOR term, not the likelihood (lanes walk out of UniformPrior support and `resurrect=False` never redraws them — that accumulation IS the 62%) + +Filed against the real `imaging/mge` cell, where the frozen-lane counter reported +`n_value_nan_lane_steps` at 62.42% and a population collapse from `alive 16/16` to +`alive 2/16`. + +**Closed out 2026-08-18: cause found, written up, and every follow-up it owed +already shipped. SUPERSEDED** — the remaining confirmation legs (GPU / float64 / +multi-seed re-measurement of the 62%) are not carried forward as scoped; a larger +issue now owns this direction. Tracking issue autolens_profiling#128 is still open +and wants closing by hand. + +## Cause: the deaths are in the PRIOR term, not the likelihood + +The objective is `fom = -2 * (log_likelihood + sum(log_prior_list))` +(`Fitness(fom_is_log_likelihood=False)`). A `UniformPrior` is `-inf` outside its +box, and `MultiStartGradient` steps in PHYSICAL space with no projection back onto +that box. A lane that crosses a hard prior edge reads as non-finite; `resurrect=False` +never redraws it, so it stays dead for every remaining step. That accumulation **is** +the 62%. + +**The likelihood never went non-finite in ~7200 lane-steps across three arms.** + +This contradicts the documented assumption it was filed against: +`AbstractMultiStartGradient.resurrect`'s docstring held that `resurrect=False` is +safe on the parametric (MGE-class) cell because only a measure-zero singularity is +in play and `apply_if_finite` guards it. `apply_if_finite` does not rescue a +value-NaN lane — it only zeroes the step. Nothing was broken; the default is simply +wrong for this cell. + +## Evidence + +- **Per-lane autopsy at the death vectors.** 11/14 have finite likelihood at every + pipeline stage and `sum(log_prior) = -inf` with 1-2 params outside a + `UniformPrior`; 2/14 have NaN params (the gradient path); 1/14 unexplained. +- **Decisive arm.** Neutering `log_prior_list_from_vector` to zeros drops value-NaN + 1446 -> 215 (60.25% -> 8.96%) and survivors 2 -> 13, with all 3 residual deaths + being NaN-params. +- **Refuted narrower hypothesis.** Widening the shear box (10 of the 11 exits) made + it marginally *worse* — deaths moved later. Widening one box only moves the wall. +- **Reproduction.** 16x150 cloud CPU gave 1446/18/0/0 and `alive 2/16` against the + filed 1498/9/0/0 and the same 2/16. The survival identity is exact: + `sum(150 - k_i) = 14*150 - 654 = 1446` = `n_value_nan_lane_steps`. + +## Counter-finding: the `ell_comps` plateau was MASKED, not cleared + +The baseline's `n_constrained_lane_steps = 0` was a correctly-measured zero — the +positive control was sound — but it meant "nothing got that far": lanes died of +prior-exit first. With the prior deaths removed the constrained count is 667 +(**27.79%**). PyAutoFit#1475's trapped-lane counter is measuring a live failure mode +on this cell, hidden behind a larger one. Lanes stop being dead and start being +STUCK. + +Caveat for anyone citing it: 27.79% came from the prior-neutered DIAGNOSTIC arm and +is **not** a citable production number. + +## How to read the 62% (trap) + +It is a **survival integral, not a hazard rate**. A frozen lane keeps counting every +subsequent step, so the same death curve reports ~75% at 300 steps. Inverting it +gives a mean death step of ~43 of 150 — mid-descent, not bad initial draws. **Grade +any re-run on the alive-versus-step CURVE, not on recovering the scalar.** + +## What shipped out of this investigation + +Upstream (the instrument that made it visible): + +- PyAutoFit#1475 (`004f798`) + PyAutoGalaxy#572 (`695b27c`) — the trapped-lane + counter. Record: `complete/2026/08/frozen-lane-counter.md`. + +Downstream, all 2026-08-16: + +- **PyAutoFit#1477** (`1f4b66a`) — bounded stepping, phase 1: the prior-support + clipper. Record: `complete/2026/08/prior-support-clipper.md`. (`resurrect=True` is + NOT the fix — it redraws a lane that then walks out again.) +- **autolens_profiling#132 + PyAutoFit#1482** — phase 2 validation (issues #129/#131 + closed). Record: `complete/2026/08/clipper-validation-campaign.md`. VERDICT: do + **not** flip the clipper default on accuracy grounds — clipping eliminates the + deaths and does not move the answer; seed dependence swings 171,272 nats and + dwarfs it. The momentum-reset arm is a clean negative. +- **PyAutoFit#1478** (`bbceff6`) — clipper reporting in `search.summary` + (`Clipper`, `Clipped Lane-Steps`, `Clipped Lane-Step Rate`, `Constrained + Lane-Steps`). Record: `complete/2026/08/clipper-usage-in-search-summary.md`. A + `ClipperPriorBox` arm reporting ZERO clips has not exercised the clipper — that is + a broken arm, not a null result. +- **PyAutoFit#1479** (`b6e89cd`) — `save_json` float32 crash, incidental. Record: + `complete/2026/08/save-json-numpy-scalar-typeerror.md`. Fixing it turned up a + second unguarded writer the prompt never named, `Samples.info_to_json`, which is + the more dangerous one since `samples_info` gains a counter every time a search + does. +- **PyAutoFit#1480** (`5c9244b`) — crashed-run-poisons-resume, incidental. Record: + `complete/2026/08/crashed-run-poisons-resume.md`. What reproduces is a hard + `JSONDecodeError` on every rerun of the same search name; `JSONDecodeError` + subclasses `ValueError`, which is why it slipped past every guard on the resume + path. + +## Not carried forward + +- **GPU / float64 / multi-seed confirmation** of the 62% at production budget. The + filed measurement stands on a single seed per arm, cloud CPU, x64 off. +- **The `ell_comps` trapping at 27.79%**, now unmasked — drafted at + `draft/research/autolens_profiling/ell_comps_trapping_unmasked.md`. It runs on top + of phase 1 and can share phase 2's arms. +- **Clipper phase 3**, which was never drafted. If it is ever written it must argue + hygiene, not accuracy; `draft/feature/autofit/clipper_in_search_identifier.md` is + its prerequisite decision. + +## Unexplained, left open + +One baseline death (lane 9, step 39) re-evaluates finite in every term with all +params inside their boxes. The jitted/vmapped float32 path differs from the eager +recompute there. Single seed per arm, CPU, x64 off. + +## Notes + +- No worktree was ever created and the `autolens_profiling` branch + `research/mge-lane-death` was never cut — the whole investigation ran in cloud + sessions against the existing ~6-min CPU run, per the task's deliberate ordering + (cause-finding first, GPU confirmation last). +- The task's boundary was investigation only. Changing the `resurrect` default + remains a separate PyAutoFit task — it would shift every existing multi-start + benchmark. + +## Original prompt + # Find what kills MGE multi-start lanes — it is not the ell_comps plateau Type: research diff --git a/complete/index.md b/complete/index.md index 6ff26619..c4db9b33 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1002 records across 7 buckets. +1003 records across 7 buckets. ## Highlights @@ -75,6 +75,7 @@ _(curate hard-won records here — survives regeneration.)_ - [llms-txt-census-fixes](2026/08/llms-txt-census-fixes.md) - [message-prior-xp-jax-trace](2026/08/message-prior-xp-jax-trace.md) — made the remaining message and compound-prior array-namespace paths - [messages-xp-stack-jax-trace](2026/08/messages-xp-stack-jax-trace.md) — replaced ten fixed-shape `xp.array([...])` message constructors +- [mge-lane-death](2026/08/mge-lane-death.md) — lanes walk out of UniformPrior support and `resurrect=False` never redraws them — that accumulation IS the 62% - [mge-sigma-min-workspace-sweep](2026/08/mge-sigma-min-workspace-sweep.md) - [missing-auto-simulate-guards](2026/08/missing-auto-simulate-guards.md) - [multi-start-auto-convergence-real-search](2026/08/multi-start-auto-convergence-real-search.md) — `scripts/jax_assertions/multi_start_gradient_auto_convergence.py` failed diff --git a/dashboard.md b/dashboard.md index af95fc09..4a25537f 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,7 +8,7 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 8 | +| [In flight](#in-flight) (`active/`) | 7 | | [Parked](#parked) (`parked.md`) | 2 | | [Planned](#planned) (`planned.md`) | 7 | | [Backlog](#backlog) (`draft/`) | 139 | @@ -50,7 +50,6 @@ Issued — each has an open GitHub issue and usually a branch. The full record f - [Commit the inference-methods programme plan + knowledge ledger into autolens_profiling](active/inference_programme_ledger.md) — [issue #134](https://github.com/PyAutoLabs/autolens_profiling/issues/134) — pr-open (https://github.com/PyAutoLabs/autolens_profiling/pull/135) - [Remove standalone matplotlib-inline comments](active/matplotlib_inline_standalones.md) -- [Find what kills MGE multi-start lanes — it is not](active/mge_lane_death.md) — [issue #128](https://github.com/PyAutoLabs/autolens_profiling/issues/128) — CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed… - [PositionsLH penalty accumulation returns 2x the LAST penalty, discarding the](active/positions_lh_penalty_accumulation_bug.md) — [issue #699](https://github.com/PyAutoLabs/PyAutoLens/issues/699) — pr-open (https://github.com/PyAutoLabs/PyAutoLens/pull/700) - [PyAutoReduce validation: slacs1430+4105 ACS reduction vs trusted legacy dataset](active/pyautoreduce_slacs1430_acs_comparison.md) - [Tenant-firewall drift: clear the 9 right + gate recurrence (Aug](active/tenant-firewall-drift-aug.md) — [issue #198](https://github.com/PyAutoLabs/PyAutoMind/issues/198) — MERGED 2026-08-17T23:13Z — all four PRs on the canonical mains, each green at merge (order held: PyAutoMind#199 → PyAutoBrain#229…