Skip to content

prompt: close out the Clipper handover — retire shipped phase 1, file what was owed - #190

Merged
Jammy2211 merged 5 commits into
mainfrom
claude/clipper-validation-handover-ltin4j
Aug 16, 2026
Merged

prompt: close out the Clipper handover — retire shipped phase 1, file what was owed#190
Jammy2211 merged 5 commits into
mainfrom
claude/clipper-validation-handover-ltin4j

Conversation

@Jammy2211

@Jammy2211 Jammy2211 commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Closes out the Clipper handover. Nothing from the autolens_profiling#128 investigation is left living only inside another prompt file, and the Mind's view of the Clipper work now matches reality.

Why this needs to be on main: the same reason #188 gave. A cold session gets whatever is in its checkout, which comes from main. Until this merges, the phase-1 prompt still reads as unstarted high-priority backlog on the dashboard people pick work from, and the four filed follow-ups do not exist.

Phase 1 had already shipped and the Mind had not noticed

PyAutoFit#1477 merged as 1f4b66a — the current tip of PyAutoFit main — shipping AbstractClipper / ClipperNone / ClipperPriorBox, wired into both AbstractMultiStartGradient and AbstractBFGS, opt-in, default ClipperNone, bit-identical under it, 22 tests.

prior_support_clipper.md was still sitting in draft/ as a high-priority backlog item. Retired to complete/2026/08/prior-support-clipper.md via lifecycle.py record, prompt folded in, with the findings worth keeping written up: the inset keyed on bound kind rather than unguarded upper - lower; LogGaussianPrior reporting (-inf, inf) for a support of (0, inf); scipy reading a bounds tuple as (min, max) pairs and silently mis-fitting two-parameter models.

Four prompts filed, three of them previously homeless

Two incidental PyAutoFit bugs that prior_support_clipper.md flagged under "do not lose these — file separately if confirmed". Both confirmed by reading PyAutoFit main, and both still present at 1f4b66a — phase 1 did not fix either. Both are now implemented and in flight:

  • save_json_numpy_scalar_typeerror.mdPyAutoFit#1479. paths/directory.py:80 is a bare json.dump with no default= hook, and there is no JSON encoder anywhere in autofit/. float32 raises at the end of a successful run. It did not fire while 14/16 lanes were dead, so it starts firing exactly when lane survival improves. Implementing it turned up a second unguarded site the prompt never named — Samples.info_to_json — which is the more dangerous of the two, since samples_info is a search's own diagnostic channel.

  • crashed_run_poisons_resume.mdPyAutoFit#1480. directory.py:79 opens "w+" (truncates, no atomic rename) → is_complete tests only the .completed marker → abstract_search.py:582 dispatches to resume → the resume reads the truncated JSON → and JSONDecodeError subclasses ValueError, so the guard at multi_start_gradient/search.py:720 catching (FileNotFoundError, TypeError, KeyError) never falls through to the fresh-start branch.

    Correction, from reproducing it. The prompt as filed described the result as "a zero-step no-op reported as a clean run". That did not reproduce. What reproduces against main is a hard JSONDecodeError on every rerun of that search name, naming no file and offering no remedy. The no-op variant presumably needs a surviving search_internal whose restored total_steps short-circuits the loop — the crash path deletes it first. Same root cause, and #1480 covers both paths, but only the crash is evidenced, and the prompt now says so.

One research task, follow-up (2) owed by active/mge_lane_death.md:

  • ell_comps_trapping_unmasked.md — the plateau was masked, not cleared. The baseline n_constrained_lane_steps = 0 meant "nothing got that far"; lanes died of prior-exit first. Records that the 27.79% came from the prior-neutered diagnostic arm and is not a citable production number, so the task sequences after phase 1 and measures the clipped run instead.

And one new feature task:

  • clipper_usage_in_search_summary.mdPyAutoFit#1478. Surface the clip count in search.summary.

Phase 2 unblocked

clipper_validation_campaign.md marked ready, with what phase 1 changed for it: record n_clipped_lane_steps as a fourth counter from search_internal; a ClipperPriorBox arm reporting zero clips has not exercised the clipper and its "no change" result is meaningless; the momentum-reset arm needs writing since nothing yet consumes the mask; and the save_json trap stands until #1479 lands.

Bookkeeping

All seven references to the moved prompt repointed. lifecycle.py check and index --check both OK; dashboard.md regenerated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FzF2XmKQaqZRWZfZMxvTNR

claude added 5 commits August 16, 2026 16:18
Closes out the Clipper handover. Three findings from autolens_profiling#128
were living only inside other prompt files, where nothing indexes them and
nobody can pick them up. All three are now filed prompts.

Two incidental PyAutoFit bugs, split out of prior_support_clipper.md's
"do not lose these" section, which asked for them to be filed separately
once confirmed. Both confirmed against PyAutoFit main 1f4b66a93:

- save_json_numpy_scalar_typeerror.md — paths/directory.py:80 is a bare
  json.dump with no default= hook, and there is no JSON encoder anywhere in
  autofit/. numpy float32 raises TypeError at the END of a successful run.
  It did not fire while 14/16 lanes were dead, so it fires exactly when lane
  survival improves — any fix to the prior-support problem will surface it.

- crashed_run_poisons_resume.md — the chain, each link read on main:
  directory.py:79 opens "w+" (truncates, no atomic rename), so the crash above
  leaves a partial file; is_complete (directory.py:180) tests only the
  .completed marker, so abstract_search.py:582 dispatches to resume; the
  resume reads the truncated JSON; and JSONDecodeError subclasses ValueError,
  so the guard at multi_start_gradient/search.py:720 catching
  (FileNotFoundError, TypeError, KeyError) does not fall through to the
  fresh-start branch. Result is a zero-step no-op reported as a clean run —
  the phase-2 campaign currently documents this as a trap to work around.

One research task, follow-up (2) owed by active/mge_lane_death.md:

- ell_comps_trapping_unmasked.md — the plateau was masked, not cleared. The
  baseline n_constrained_lane_steps = 0 meant "nothing got that far"; lanes
  died of prior-exit first. Records that the 27.79% came from the
  prior-neutered diagnostic arm and is NOT a citable production number, so
  the task sequences after phase 1 and measures the clipped run instead.

active.md now points at all five filed prompts (these three plus the two
Clipper phases) so the thread is followable from the ledger. dashboard.md
regenerated; lifecycle check OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzF2XmKQaqZRWZfZMxvTNR
…ting gap

Phase 1 has landed. PyAutoFit#1477 merged as 1f4b66a — the current tip of
PyAutoFit main — shipping AbstractClipper / ClipperNone / ClipperPriorBox in
autofit/non_linear/clipper.py, wired into both AbstractMultiStartGradient and
AbstractBFGS, opt-in, default ClipperNone, bit-identical under it, 22 tests.

The Mind had not noticed: prior_support_clipper.md was still sitting in
draft/ as a high-priority backlog item on a dashboard people pick work from.
Retired to complete/2026/08/prior-support-clipper.md via lifecycle.py record,
with the original prompt folded in and the findings worth keeping written up
(the inset keyed on bound kind rather than unguarded upper-lower; the
LogGaussianPrior support mismatch; scipy reading a bounds tuple as (min, max)
pairs and silently mis-fitting two-parameter models).

Clip-count reporting, filed as clipper_usage_in_search_summary.md. The
counting half already exists — n_clipped_lane_steps is accumulated per-lane
(not per-coordinate) at multi_start_gradient/search.py:863, written to
search_internal and restored on resume as a lifetime total. The storing half
does not, and the gap is not where it looks:

- search.summary is written by text_util.search_summary_to_file, which only
  ever sees samples and timings. There is no search-specific channel at all,
  for any counter — save_summary never sees search_internal. Routing is the
  work, not the arithmetic, which is why the prompt asks for a general hook
  serving the other multi-start counters rather than one threaded parameter.
- LBFGS produces no count whatsoever. It hands optimize.Bounds to scipy and
  never calls project, so there is no mask to count. Emitting 0 there would
  read as "never fired" when it means "cannot know".

Phase 2 (clipper_validation_campaign.md) marked UNBLOCKED with what phase 1
changed for it: record n_clipped_lane_steps as a fourth counter from
search_internal; a ClipperPriorBox arm reporting zero clips has not exercised
the clipper and its "no change" result is meaningless; the momentum-reset arm
needs writing since nothing yet consumes the mask; and the float32 save_json
crash was NOT fixed by phase 1 (verified at 1f4b66a), so that trap stands and
will start firing precisely because clipping keeps lanes alive.

Repointed all seven references to the moved prompt. lifecycle check + index
check OK; dashboard regenerated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzF2XmKQaqZRWZfZMxvTNR
…mentation

The clip-count work is written and tested against PyAutoFit 1f4b66a, but push
access to PyAutoFit was denied in this session, so it lands as a patch rather
than a PR. Prompt updated to say so and to carry the verification evidence.

Corrects a wrong premise in the prompt as first filed. It claimed search.summary
had "no search-specific channel at all". That is false: search_summary_from_samples
(text_util.py:115-161) already reads samples.samples_info and already emits
Resurrections, Value-NaN Lane-Steps, Gradient-NaN Lane-Steps and both rates,
key-guarded so other searches are unaffected. I had read search_summary_to_file
and not the function it calls. Left uncorrected it would have sent an
implementer off to build a routing mechanism that already exists.

The real gap is narrow: n_clipped_lane_steps never reaches samples_info, and
n_constrained_lane_steps reaches it but is never emitted — so PyAutoFit#1475's
trapped-lane counter has been invisible in search.summary since it shipped.

As built (tmp/clipper-search-summary.patch, gitignored, 4 files +243/-0):
multistart publishes clipper name + n_clipped_lane_steps into samples_info;
bfgs publishes the clipper name and deliberately no count, since it is
declarative and scipy enforces, so a 0 would read as "never fired" when it
means "cannot know"; text_util grows _clipper_summary_from handling the three
cases plus the owed Constrained Lane-Steps line; 6 new tests.

Verified on 3.13: non_linear + text = 501 passed, 3 skipped, 1 failed, the
failure (nautilus single-core pool) reproduced identically on a stashed clean
tree, as was the astropy collection error — both already recorded under
complete/2026/08/frozen-lane-counter.md. Also verified end-to-end against the
search.summary files four real searches actually wrote, giving
Clipped Lane-Steps = 414 at rate 0.958 on the clipped multistart arm.

Incidental, worth keeping: on that toy 3-parameter Gaussian the unclipped arm
showed Value-NaN Lane-Steps = 378 (94.5%) and the clipped arm 0 — the #128
mechanism reproducing on a model with nothing astrophysical in it, which makes
a far cheaper regression fixture than the imaging/mge cell.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzF2XmKQaqZRWZfZMxvTNR
…1478

Advance to complete/ on merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzF2XmKQaqZRWZfZMxvTNR
save_json_numpy_scalar_typeerror -> PyAutoFit#1479. Reproduced against the
unfixed tree before fixing. Fixed with a NumpyEncoder wired into BOTH output
writers -- save_json and, which this prompt did not name, Samples.info_to_json.
That second site is the more dangerous: samples_info is a search's own
diagnostic channel, so every counter added to it could reintroduce the crash.

crashed_run_poisons_resume -> PyAutoFit#1480, and carries a CORRECTION. The
prompt claimed the poisoned rerun is "a 4-second no-op that reads as a clean
result". That did not reproduce. What reproduces is a hard JSONDecodeError on
every rerun of the same search name, naming no file and offering no remedy.
The no-op variant presumably needs a surviving search_internal whose restored
total_steps short-circuits the loop; the crash path deletes it first. Same root
cause, fix covers both, but only the crash is evidenced -- recorded so the
unobserved symptom is not cited as fact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzF2XmKQaqZRWZfZMxvTNR
@Jammy2211
Jammy2211 merged commit 726be74 into main Aug 16, 2026
2 checks passed
Jammy2211 added a commit that referenced this pull request Aug 16, 2026
…llow-ups filed (#194)

Two sessions independently wrote a completion record for the prior-support
Clipper. #189 was opened first, by the session that actually shipped
PyAutoFit#1477; #190 (mine) landed later because I never listed open PRs before
starting. This consolidates them so main holds one record, not two.

complete/2026/08/prior-support-clipper.md is now the union. The detailed body
comes from #189 and is materially richer than what I wrote: the bound-kind
design decision (why one relative margin is wrong in two silent ways), eight
traps measured against a running install, the verification log (bit-identity
10/10, core promise 8/8, guards verified by inversion), and the process lesson
-- a first commit shipped an undefined `optimize` in LBFGS._fit and the full
1790-test suite passed against it, because nothing in the suite executes an
LBFGS fit. My "what shipped after" section is appended, since #189 predates
#1478/#1479/#1480 and so records follow-ups 1 and 2 as open when they are now
fixed.

#189's phase-2 harness traps salvaged into the campaign prompt -- the
.completed short-circuit, fit() rebuilding search.paths so instance-level
patches are discarded, search_internal being deleted on success, seeding both
random AND numpy, and a box containing the optimum never exercising the
clipper. Also that arm 3 does not exist yet (phase 1 ships the mask, no reset)
and two phase-1 measurements to carry in as priors.

Follow-ups 3 and 4 filed as prompts rather than left in a record:
loggaussian_prior_declares_own_support.md and clipper_in_search_identifier.md.
The second is Autonomy: human-required -- both answers orphan or collide
someone's stored results, and it is best decided BEFORE phase 3 so the
re-baseline and any re-keying are not entangled.

One correction absorbed from #189: test_nautilus single_core_builds_no_pool
PASSES in CI and fails only in local venvs (both 3.12 and 3.13, two independent
sessions). "Not caused by this work" is right; "pre-existing on clean main" --
the looser phrasing used in #1479 and #1480's bodies -- is wrong. Recorded.


Claude-Session: https://claude.ai/code/session_01FzF2XmKQaqZRWZfZMxvTNR

Co-authored-by: Claude <noreply@anthropic.com>
@Jammy2211
Jammy2211 deleted the claude/clipper-validation-handover-ltin4j branch August 17, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants