Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 3 additions & 21 deletions active.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@
- autolens_profiling: feature/inference-programme-ledger
- prompt: active/inference_programme_ledger.md
- note: docs-only (results/notes/inference/ PROGRAMME.md + DECISIONS.md + LITERATURE.md).
Disjoint from the registered mge-lane-death claim (research/mge-lane-death, never created).
The mge-lane-death claim it was recorded as disjoint from is gone — archived
2026-08-18 as superseded (`complete/2026/08/mge-lane-death.md`); its
`research/mge-lane-death` branch was never created.
This is the canonical copy of the 2026-08-17 human-approved inference programme.

## positions-lh-penalty-accumulation
Expand Down Expand Up @@ -69,26 +71,6 @@
- do-not: do NOT weaken PyAutoGalaxy `validate_ell_comps`; do NOT edit the tutorial. Test mode is NOT
implicated (`ENV: real_search` releases `PYAUTO_TEST_MODE`) — verified, do not re-open.

## mge-lane-death
- issue: https://github.com/PyAutoLabs/autolens_profiling/issues/128
- prompt: active/mge_lane_death.md
- status: CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed confirmation, and the two follow-ups below.
- worktree: ~/Code/PyAutoLabs-wt/mge-lane-death (not yet created)
- repos:
- autolens_profiling: research/mge-lane-death (not yet created)
- CAUSE: the deaths are in the PRIOR term, not the likelihood. The objective is `fom = -2 * (log_likelihood + sum(log_prior_list))` (`Fitness(fom_is_log_likelihood=False)`); a `UniformPrior` is `-inf` outside its box; `MultiStartGradient` steps in PHYSICAL space with no projection back onto that box. A lane crossing a hard prior edge reads as non-finite, and `resurrect=False` never redraws it, so it stays dead for every remaining step — that accumulation IS the 62%. **The likelihood never went non-finite in ~7200 lane-steps across three arms.**
- evidence: per-lane autopsy at the death vectors — 11/14 have finite likelihood at every pipeline stage and `sum(log_prior) = -inf` with 1-2 params outside a `UniformPrior`; 2/14 have NaN params (the gradient path); 1/14 unexplained. Decisive arm: neutering `log_prior_list_from_vector` -> zeros drops value-NaN 1446 -> 215 (60.25% -> 8.96%) and survivors 2 -> 13, with all 3 residual deaths being NaN-params. A narrower hypothesis (widen the shear box, which was 10 of the 11 exits) was REFUTED — deaths moved later and got marginally worse, because widening one box only moves the wall.
- reproduction: 16x150 cloud CPU gave 1446/18/0/0 and `alive 2/16` against the filed 1498/9/0/0 and the same 2/16. The survival identity is exact: `sum(150 - k_i) = 14*150 - 654 = 1446` = `n_value_nan_lane_steps`.
- COUNTER-FINDING, corrects the framing: the `ell_comps` plateau was MASKED, not cleared. The baseline's `n_constrained_lane_steps = 0` was a correctly-measured zero (the positive control was sound) but it meant "nothing got that far" — lanes died of prior-exit first. With the prior deaths removed the constrained count is 667 (27.79%). #1475's trapped-lane counter is measuring a live failure mode on this cell, hidden behind a larger one. Lanes stop being dead and start being STUCK.
- follow-ups owed (both out of this task's boundary), ALL NOW FILED 2026-08-16 — nothing from this investigation lives only inside another prompt: (1) PyAutoFit — bounded stepping (projection/clipping onto prior support) or soft-walled priors; `resurrect=True` is NOT the fix, it redraws a lane that then walks out again. **PHASE 1 SHIPPED 2026-08-16** as PyAutoFit#1477 (`1f4b66a`) → record `complete/2026/08/prior-support-clipper.md`; **PHASE 2 RAN AND SHIPPED 2026-08-16** (autolens_profiling#132 + PyAutoFit#1482, issues #129/#131 closed) → record `complete/2026/08/clipper-validation-campaign.md` — VERDICT: do NOT flip the clipper default on accuracy grounds (clipping eliminates the deaths and does not move the answer; seed dependence swings 171,272 nats and dwarfs it), and the momentum-reset arm is a clean negative. Phase 3, if ever written, must argue hygiene, not accuracy — it remains undrafted, and `draft/feature/autofit/clipper_in_search_identifier.md` is its prerequisite decision. (2) the ell_comps trapping at 27.79%, now that it is visible → `draft/research/autolens_profiling/ell_comps_trapping_unmasked.md` — note that 27.79% came from the prior-neutered DIAGNOSTIC arm and is not a citable production number; it runs on top of phase 1 and can share phase 2's arms.
- clipper-reporting: SHIPPED 2026-08-16 as PyAutoFit#1478 (`bbceff6`) → record `complete/2026/08/clipper-usage-in-search-summary.md`. `search.summary` now reports `Clipper`, `Clipped Lane-Steps`, `Clipped Lane-Step Rate` and `Constrained Lane-Steps`. CORRECTS an earlier note here that claimed `search.summary` had no search-specific channel at all — false; `search_summary_from_samples` already read `samples_info` and already emitted the NaN counters. The real gap was two omissions, not a missing mechanism. Phase 2 reads the clip count straight from `search.summary` now, and a `ClipperPriorBox` arm reporting ZERO clips has not exercised the clipper — a broken arm, not a null result.
- incidental PyAutoFit bugs found while investigating: BOTH SHIPPED 2026-08-16. `save_json` float32 crash → PyAutoFit#1479 (`b6e89cd`), record `complete/2026/08/save-json-numpy-scalar-typeerror.md`; fixing it turned up a second unguarded writer the prompt never named, `Samples.info_to_json`, which is the more dangerous since `samples_info` gains a counter every time a search does. Crashed-run-poisons-resume → PyAutoFit#1480 (`5c9244b`), record `complete/2026/08/crashed-run-poisons-resume.md`. CORRECTION carried in that record: the "zero-step no-op reported as a clean result" symptom did NOT reproduce; what reproduces is a hard `JSONDecodeError` on every rerun of the same search name. `JSONDecodeError` subclasses `ValueError`, which is why it slipped past every guard on the resume path. Both traps the phase-2 campaign worked around are now closed.
- caveat: one baseline death (lane 9, step 39) re-evaluates finite in every term with all params inside their boxes — the jitted/vmapped float32 path differs from the eager recompute there, unexplained. Single seed per arm, CPU, x64 off.
- reading the number: 62% is a survival integral, not a hazard rate — a frozen lane keeps counting every subsequent step, so the same death curve reports ~75% at 300 steps. Inverting it gives a mean death step of ~43 of 150 (mid-descent, not bad initial draws). Grade any re-run on the alive-versus-step CURVE, not on recovering the scalar.
- ordering (deliberate, do not revert): cause-finding FIRST on the existing ~6-min CPU run, then the `resurrect=True` budget-recovery measurement, then the production/GPU/seed confirmation. Do not queue for a GPU before the cause step has been attempted.
- boundary: investigation only. Changing the `resurrect` default is a separate PyAutoFit task — it would shift every existing multi-start benchmark.
- upstream: PyAutoFit#1475 (`004f798`) + PyAutoGalaxy#572 (`695b27c`) shipped the trapped-lane counter; record in `complete/2026/08/frozen-lane-counter.md`.

## heart-green-validation-ingest
- issue: https://github.com/PyAutoLabs/PyAutoGalaxy/issues/567 (open, reopened 2026-08-11T00:22Z)
- session: none — the Codex session that did the work ran out of credits after merging, before any bookkeeping. Registered 2026-08-11 by a cloud session reconstructing its state from GitHub.
Expand Down
125 changes: 125 additions & 0 deletions active/mge_lane_death.md → complete/2026/08/mge-lane-death.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,128 @@
MGE multi-start lanes die on the PRIOR term, not the likelihood (lanes walk out of UniformPrior support and `resurrect=False` never redraws them — that accumulation IS the 62%)

Filed against the real `imaging/mge` cell, where the frozen-lane counter reported
`n_value_nan_lane_steps` at 62.42% and a population collapse from `alive 16/16` to
`alive 2/16`.

**Closed out 2026-08-18: cause found, written up, and every follow-up it owed
already shipped. SUPERSEDED** — the remaining confirmation legs (GPU / float64 /
multi-seed re-measurement of the 62%) are not carried forward as scoped; a larger
issue now owns this direction. Tracking issue autolens_profiling#128 is still open
and wants closing by hand.

## Cause: the deaths are in the PRIOR term, not the likelihood

The objective is `fom = -2 * (log_likelihood + sum(log_prior_list))`
(`Fitness(fom_is_log_likelihood=False)`). A `UniformPrior` is `-inf` outside its
box, and `MultiStartGradient` steps in PHYSICAL space with no projection back onto
that box. A lane that crosses a hard prior edge reads as non-finite; `resurrect=False`
never redraws it, so it stays dead for every remaining step. That accumulation **is**
the 62%.

**The likelihood never went non-finite in ~7200 lane-steps across three arms.**

This contradicts the documented assumption it was filed against:
`AbstractMultiStartGradient.resurrect`'s docstring held that `resurrect=False` is
safe on the parametric (MGE-class) cell because only a measure-zero singularity is
in play and `apply_if_finite` guards it. `apply_if_finite` does not rescue a
value-NaN lane — it only zeroes the step. Nothing was broken; the default is simply
wrong for this cell.

## Evidence

- **Per-lane autopsy at the death vectors.** 11/14 have finite likelihood at every
pipeline stage and `sum(log_prior) = -inf` with 1-2 params outside a
`UniformPrior`; 2/14 have NaN params (the gradient path); 1/14 unexplained.
- **Decisive arm.** Neutering `log_prior_list_from_vector` to zeros drops value-NaN
1446 -> 215 (60.25% -> 8.96%) and survivors 2 -> 13, with all 3 residual deaths
being NaN-params.
- **Refuted narrower hypothesis.** Widening the shear box (10 of the 11 exits) made
it marginally *worse* — deaths moved later. Widening one box only moves the wall.
- **Reproduction.** 16x150 cloud CPU gave 1446/18/0/0 and `alive 2/16` against the
filed 1498/9/0/0 and the same 2/16. The survival identity is exact:
`sum(150 - k_i) = 14*150 - 654 = 1446` = `n_value_nan_lane_steps`.

## Counter-finding: the `ell_comps` plateau was MASKED, not cleared

The baseline's `n_constrained_lane_steps = 0` was a correctly-measured zero — the
positive control was sound — but it meant "nothing got that far": lanes died of
prior-exit first. With the prior deaths removed the constrained count is 667
(**27.79%**). PyAutoFit#1475's trapped-lane counter is measuring a live failure mode
on this cell, hidden behind a larger one. Lanes stop being dead and start being
STUCK.

Caveat for anyone citing it: 27.79% came from the prior-neutered DIAGNOSTIC arm and
is **not** a citable production number.

## How to read the 62% (trap)

It is a **survival integral, not a hazard rate**. A frozen lane keeps counting every
subsequent step, so the same death curve reports ~75% at 300 steps. Inverting it
gives a mean death step of ~43 of 150 — mid-descent, not bad initial draws. **Grade
any re-run on the alive-versus-step CURVE, not on recovering the scalar.**

## What shipped out of this investigation

Upstream (the instrument that made it visible):

- PyAutoFit#1475 (`004f798`) + PyAutoGalaxy#572 (`695b27c`) — the trapped-lane
counter. Record: `complete/2026/08/frozen-lane-counter.md`.

Downstream, all 2026-08-16:

- **PyAutoFit#1477** (`1f4b66a`) — bounded stepping, phase 1: the prior-support
clipper. Record: `complete/2026/08/prior-support-clipper.md`. (`resurrect=True` is
NOT the fix — it redraws a lane that then walks out again.)
- **autolens_profiling#132 + PyAutoFit#1482** — phase 2 validation (issues #129/#131
closed). Record: `complete/2026/08/clipper-validation-campaign.md`. VERDICT: do
**not** flip the clipper default on accuracy grounds — clipping eliminates the
deaths and does not move the answer; seed dependence swings 171,272 nats and
dwarfs it. The momentum-reset arm is a clean negative.
- **PyAutoFit#1478** (`bbceff6`) — clipper reporting in `search.summary`
(`Clipper`, `Clipped Lane-Steps`, `Clipped Lane-Step Rate`, `Constrained
Lane-Steps`). Record: `complete/2026/08/clipper-usage-in-search-summary.md`. A
`ClipperPriorBox` arm reporting ZERO clips has not exercised the clipper — that is
a broken arm, not a null result.
- **PyAutoFit#1479** (`b6e89cd`) — `save_json` float32 crash, incidental. Record:
`complete/2026/08/save-json-numpy-scalar-typeerror.md`. Fixing it turned up a
second unguarded writer the prompt never named, `Samples.info_to_json`, which is
the more dangerous one since `samples_info` gains a counter every time a search
does.
- **PyAutoFit#1480** (`5c9244b`) — crashed-run-poisons-resume, incidental. Record:
`complete/2026/08/crashed-run-poisons-resume.md`. What reproduces is a hard
`JSONDecodeError` on every rerun of the same search name; `JSONDecodeError`
subclasses `ValueError`, which is why it slipped past every guard on the resume
path.

## Not carried forward

- **GPU / float64 / multi-seed confirmation** of the 62% at production budget. The
filed measurement stands on a single seed per arm, cloud CPU, x64 off.
- **The `ell_comps` trapping at 27.79%**, now unmasked — drafted at
`draft/research/autolens_profiling/ell_comps_trapping_unmasked.md`. It runs on top
of phase 1 and can share phase 2's arms.
- **Clipper phase 3**, which was never drafted. If it is ever written it must argue
hygiene, not accuracy; `draft/feature/autofit/clipper_in_search_identifier.md` is
its prerequisite decision.

## Unexplained, left open

One baseline death (lane 9, step 39) re-evaluates finite in every term with all
params inside their boxes. The jitted/vmapped float32 path differs from the eager
recompute there. Single seed per arm, CPU, x64 off.

## Notes

- No worktree was ever created and the `autolens_profiling` branch
`research/mge-lane-death` was never cut — the whole investigation ran in cloud
sessions against the existing ~6-min CPU run, per the task's deliberate ordering
(cause-finding first, GPU confirmation last).
- The task's boundary was investigation only. Changing the `resurrect` default
remains a separate PyAutoFit task — it would shift every existing multi-start
benchmark.

## Original prompt

# Find what kills MGE multi-start lanes — it is not the ell_comps plateau

Type: research
Expand Down
3 changes: 2 additions & 1 deletion complete/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema:
only then grep a dated bucket. Curators: edit the band between the CURATED
markers; everything below GENERATED is rebuilt.

1002 records across 7 buckets.
1003 records across 7 buckets.

<!-- CURATED:START -->
## Highlights
Expand Down Expand Up @@ -75,6 +75,7 @@ _(curate hard-won records here — survives regeneration.)_
- [llms-txt-census-fixes](2026/08/llms-txt-census-fixes.md)
- [message-prior-xp-jax-trace](2026/08/message-prior-xp-jax-trace.md) — made the remaining message and compound-prior array-namespace paths
- [messages-xp-stack-jax-trace](2026/08/messages-xp-stack-jax-trace.md) — replaced ten fixed-shape `xp.array([...])` message constructors
- [mge-lane-death](2026/08/mge-lane-death.md) — lanes walk out of UniformPrior support and `resurrect=False` never redraws them — that accumulation IS the 62%
- [mge-sigma-min-workspace-sweep](2026/08/mge-sigma-min-workspace-sweep.md)
- [missing-auto-simulate-guards](2026/08/missing-auto-simulate-guards.md)
- [multi-start-auto-convergence-real-search](2026/08/multi-start-auto-convergence-real-search.md) — `scripts/jax_assertions/multi_start_gradient_auto_convergence.py` failed
Expand Down
3 changes: 1 addition & 2 deletions dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here.

| Where | Count |
|-------|------:|
| [In flight](#in-flight) (`active/`) | 8 |
| [In flight](#in-flight) (`active/`) | 7 |
| [Parked](#parked) (`parked.md`) | 2 |
| [Planned](#planned) (`planned.md`) | 7 |
| [Backlog](#backlog) (`draft/`) | 139 |
Expand Down Expand Up @@ -50,7 +50,6 @@ Issued — each has an open GitHub issue and usually a branch. The full record f

- [Commit the inference-methods programme plan + knowledge ledger into autolens_profiling](active/inference_programme_ledger.md) — [issue #134](https://github.com/PyAutoLabs/autolens_profiling/issues/134) — pr-open (https://github.com/PyAutoLabs/autolens_profiling/pull/135)
- [Remove standalone matplotlib-inline comments](active/matplotlib_inline_standalones.md)
- [Find what kills MGE multi-start lanes — it is not](active/mge_lane_death.md) — [issue #128](https://github.com/PyAutoLabs/autolens_profiling/issues/128) — CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed…
- [PositionsLH penalty accumulation returns 2x the LAST penalty, discarding the](active/positions_lh_penalty_accumulation_bug.md) — [issue #699](https://github.com/PyAutoLabs/PyAutoLens/issues/699) — pr-open (https://github.com/PyAutoLabs/PyAutoLens/pull/700)
- [PyAutoReduce validation: slacs1430+4105 ACS reduction vs trusted legacy dataset](active/pyautoreduce_slacs1430_acs_comparison.md)
- [Tenant-firewall drift: clear the 9 right + gate recurrence (Aug](active/tenant-firewall-drift-aug.md) — [issue #198](https://github.com/PyAutoLabs/PyAutoMind/issues/198) — MERGED 2026-08-17T23:13Z — all four PRs on the canonical mains, each green at merge (order held: PyAutoMind#199 → PyAutoBrain#229…
Expand Down
Loading