fix: MultiStart gradient step loop crashes on a real iterations_per_full_update cadence

[Jammy2211]

Overview

AbstractSearch.__init__ unconditionally coerces iterations_per_full_update to
float (abstract_search.py:219), so the MultiStart gradient step loop builds
its per-chunk step count as a float and range(float) raises
TypeError: 'float' object cannot be interpreted as an integer. The crash fires
only when the checkpoint cadence is below the remaining step budget — the
packaged config default is 1e99, so min(1e99, steps_remaining) always returns
the int operand and every default run took the safe branch. Any user who passes a
real cadence (e.g. iterations_per_full_update=50 with n_steps=3000) crashes on
step-loop entry. Six RAL chain jobs (331182–331190) died on it back-to-back during
the wsdev#117 Pix-Prodigy CPU campaign.

Reproduced on main (2026-07-27):

>>> s = af.MultiStartAdam(n_steps=3000, iterations_per_full_update=50)
>>> s.iterations_per_full_update
50.0
>>> range(min(s.iterations_per_full_update or s.n_steps, 3000))
TypeError: 'float' object cannot be interpreted as an integer

Plan

• Cast the per-chunk step count to int at the consumer in the MultiStart
  gradient _fit, rather than changing the shared float coercion in
  abstract_search.py — that coercion exists to accept the inf-like 1e99
  config default and is shared by every other search, so changing it there is the
  regression-prone option.
• Put the computation behind a tiny named helper on the abstract MultiStart class
  so the defect is unit-testable without JAX — _fit itself requires jax,
  optax and a JAX-traceable Analysis, and the library unit suite is NumPy-only
  by standing rule.
• Add NumPy-only unit tests: a real cadence below the budget returns an int
  usable by range(); the 1e99 default still yields the remaining-steps branch;
  the iterations_per_full_update=None fallback to n_steps is preserved.
• Run the full PyAutoFit suite (Python 3.12/3.13 on CI) — no public API change,
  so there is no downstream workspace impact.
• Two follow-ups are recorded below rather than bundled into this PR (Mind rule:
  one prompt = one task = one PR).

Detailed implementation plan

Affected Repositories

• PyAutoFit (primary, and only)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean

Worktree claim on PyAutoFit by task testmode-env-drift was verified stale
(PyAutoFit#1417 merged, PyAutoCTI#96 merged, PyAutoCTI#95 closed) and released
into complete/2026/07/testmode-env-drift.md (PyAutoMind 3a99904) before this
task started. No live conflict.

Suggested branch: feature/multistart-cadence-int-cast

Implementation Steps

1. In autofit/non_linear/search/mle/multi_start_gradient/search.py, add a
   method to AbstractMultiStartGradient:

   def _steps_in_chunk(self, steps_remaining: int) -> int:
       """
       Number of gradient steps to run before the next ``perform_update``
       checkpoint boundary.
   
       ``iterations_per_full_update`` is stored as a **float** by
       ``AbstractSearch.__init__`` so the inf-like ``1e99`` config default is
       representable; ``range`` needs an ``int``, so the chunk size is cast
       here at the consumer. Without the cast the loop only survives because
       ``min(1e99, steps_remaining)`` returns the int operand — a real cadence
       below the remaining budget raises ``TypeError``.
       """
       return int(min(self.iterations_per_full_update or self.n_steps, steps_remaining))

2. Replace the inline computation in _fit (currently search.py:284-286) with
   iterations = self._steps_in_chunk(steps_remaining).

3. Add unit tests to
   test_autofit/non_linear/search/mle/test_multi_start_gradient.py (NumPy-only,
   consistent with that file's stated JAX-free contract):

   • iterations_per_full_update=50, n_steps=3000 → _steps_in_chunk(3000) is
     50 and isinstance(..., int); range() over it yields 50 items (the
     assertion that actually pins the reported crash).
   • the same search near the end of its budget → chunk clamps to
     steps_remaining.
   • default (config 1e99) → returns steps_remaining as an int.
   • iterations_per_full_update=None → falls back to n_steps.

4. Run python -m pytest test_autofit/ in the task worktree.

Key Files

• autofit/non_linear/search/mle/multi_start_gradient/search.py — the crash site
  (_fit step loop) and the new _steps_in_chunk helper.
• autofit/non_linear/search/abstract_search.py:219 — the shared float()
  coercion; deliberately unchanged.
• test_autofit/non_linear/search/mle/test_multi_start_gradient.py — the
  NumPy-only test surface.

Explicitly out of scope (follow-ups, not this PR)

1. Sibling searches share the same latent float-cadence class. A sweep of
   every consumer of self.iterations_per_full_update found five more sites that
   feed the float straight into an iteration budget, each masked today by the
   same 1e99 default:

   • autofit/non_linear/search/mcmc/emcee/search.py:203-206 → float iterations
     into EnsembleSampler.sample(iterations=...)
   • autofit/non_linear/search/mcmc/zeus/search.py:239-242 → same shape
   • autofit/non_linear/search/mcmc/blackjax/nuts/search.py:291 →
     chunk_n = min(float, int) into the chunked NUTS runner
   • autofit/non_linear/search/mle/bfgs/search.py:171 → float options["maxiter"]
     into scipy.optimize.minimize
   • autofit/non_linear/search/nest/nautilus/search.py:477 and
     nest/dynesty/search/abstract.py:365 → float iteration budgets returned

   None of these is reproduced yet — each needs its own repro before a fix, and
   bundling six unverified changes into a hotfix PR is the wrong trade. Filed as
   a separate Mind prompt.

2. The workspace hotfix must be removed once this merges. The post-construction
   int overwrite in
   autolens_workspace_developer/searches_minimal/pix_prodigy.py (branch
   feature/pix-prodigy-cpu, task pix-prodigy-cpu, wsdev#117) exists only to
   work around this bug. That repo is claimed by a live task, so the removal
   belongs to it — noted on wsdev#117 rather than done here.

Original Prompt

Click to expand starting prompt

# MultiStart gradient step loop crashes when iterations_per_full_update < remaining steps

Type: bug
Target: autofit
Repos:
- PyAutoFit
Difficulty: small
Autonomy: safe
Priority: high
Status: draft

## The bug (hit live on RAL, 2026-07-27, wsdev#117 campaign)

`abstract_search.py:219` unconditionally coerces `iterations_per_full_update`
to **float** (`float(iterations_per_full_update or conf...)`). The multi-start
gradient `_fit` (`autofit/non_linear/search/mle/multi_start_gradient/search.py:284`)
then computes

    iterations = min(self.iterations_per_full_update or self.n_steps, steps_remaining)
    ...
    for _ in range(iterations):   # line 298

`range(float)` → `TypeError: 'float' object cannot be interpreted as an
integer` — but **only when the cadence is below the remaining budget**, because
`min(float, int)` returns the int operand otherwise. The config default is huge,
so every existing run took the `steps_remaining` branch and the crash never
fired; any user who passes a real checkpoint cadence (e.g.
`iterations_per_full_update=50` with `n_steps=3000`) crashes on step-loop entry.
Six RAL chain jobs (331182-331190) died on it back-to-back.

Evidence: `/mnt/ral/jnightin/pixgrad_logs/pix_prod_*-33118[5-9].err`; workspace
hotfix (post-construction int overwrite) in
`autolens_workspace_developer/searches_minimal/pix_prodigy.py` (branch
`feature/pix-prodigy-cpu`) — remove it when this ships.

## Fix

`iterations = int(min(...))` in `_fit` (the float coercion in abstract_search
serves inf-like config values and is shared by other searches — casting at the
consumer is the minimal, non-regressing change). Add a unit test that runs a
MultiStart search with `iterations_per_full_update` smaller than `n_steps`
(numpy objective — library unit tests stay JAX-free) and asserts the step loop
executes + checkpoints.

---

Autonomy: launched with --auto. Prompt header is Autonomy: safe, but
AUTONOMY.md caps work-type bug at supervised, so the effective level is
min(safe, supervised) = supervised. This run proceeds through
implementation and tests, then parks at ship sign-off with a question on this
issue rather than opening the PR unattended.

[Jammy2211]

Parked at ship sign-off — awaiting input

The fix is implemented and tested on branch feature/multistart-cadence-int-cast
(worktree ~/Code/PyAutoLabs-wt/multistart-cadence-int-cast), uncommitted.
Two gates stop this run short of commit/push/PR:

1. Effective autonomy is supervised, not safe. The prompt header says
   Autonomy: safe, but AUTONOMY.md caps work-type bug at supervised, and
   the effective level is min(header, cap). At supervised, ship sign-off is a
   checkpoint: the run parks and asks rather than opening the PR.
2. Heart is YELLOW and was not acknowledged at launch. A launch-time
   acknowledgement binds to an exact reason set for that launch only; today's
   earlier acknowledgement (for purge-autogalaxy-database) does not carry over.

What was done

autofit/non_linear/search/mle/multi_start_gradient/search.py

• New AbstractMultiStartGradient._steps_in_chunk(steps_remaining) -> int —
  computes the per-chunk step count and casts to int. The docstring records
  why the cast lives at the consumer (the shared float() coercion in
  abstract_search.py:219 exists for the inf-like 1e99 default and is used by
  every other search).
• _fit's step loop now calls it instead of computing min(...) inline.

abstract_search.py is unchanged, as planned.

test_autofit/non_linear/search/mle/test_multi_start_gradient.py — four
NumPy-only tests (that module is JAX-free by contract, and _fit itself needs
jax, optax and a JAX-traceable Analysis, so it cannot be driven from the
library suite):

• a real cadence below the budget returns an int that range() consumes — the
  test that pins the reported crash;
• the chunk clamps to steps_remaining near the end of the budget;
• the 1e99 default still yields a single whole-budget chunk;
• a falsy cadence falls back to n_steps rather than a zero-length chunk.

Gate status

Leg	Status
Tests	✅ pytest test_autofit/ -x → 1534 passed, 1 skipped in the task worktree. No public API change (the new method is private), so no downstream-dependent suites are required.
Regression pinned	✅ Removing the int() cast makes exactly one new test fail (test__steps_in_chunk__real_cadence_is_an_int_range_can_consume); restored afterwards.
Reproduced on main	✅ range(min(50.0, 3000)) → TypeError: 'float' object cannot be interpreted as an integer.
Smoke	⏸ not run — gated behind sign-off.
Review faculty	⏸ not run — needs a commit to diff against main.
Heart	⚠️ YELLOW, score 65, 2026-07-27T12:11:07Z. No RED reasons.

Heart's current reason set, verbatim:

• workspace validation not passing (13 failed, 2026-07-21T19-05-22Z)
• 33 stale parked script(s)
• manifest drift: tenant firewall (organ code) — 5 mismatch(es) vs PyAutoMind/repos.yaml
• (stale tier) release validation stale: source moved since rehearsal (PyAutoNerves, PyAutoFit, PyAutoArray, PyAutoGalaxy, PyAutoLens)

None of these touches autofit/non_linear/search/mle/; they are organism-scope,
not branch-scope.

The question

Acknowledge that exact YELLOW reason set and sign off on shipping, and this
becomes a commit + push + pending-release PR (merge stays human, always).

• Answer "ship it" → smoke + review-faculty legs run, then the PR opens
  against main, active.md moves to library-shipped, awaiting-merge, and a
  calibration row is appended.
• Answer "hold" → the branch stays uncommitted in the worktree and this issue
  stays awaiting-input.
• Any new Heart reason, or a RED verdict, re-parks the run regardless of the
  answer — the acknowledgement binds only to the list above.

Two follow-ups deliberately not in this change

1. Five sibling searches share the same latent float-cadence class, each
   masked today by the same 1e99 default: mcmc/emcee/search.py:203-206,
   mcmc/zeus/search.py:239-242, mcmc/blackjax/nuts/search.py:291,
   mle/bfgs/search.py:171, and the float budgets returned by
   nest/nautilus/search.py:477 / nest/dynesty/search/abstract.py:365. None is
   reproduced yet; each needs its own repro before a fix, and bundling six
   unverified changes into a hotfix PR is the wrong trade. A separate Mind prompt
   should carry it.
2. The workspace hotfix must be removed once this merges — the
   post-construction int overwrite in
   autolens_workspace_developer/searches_minimal/pix_prodigy.py (branch
   feature/pix-prodigy-cpu). That repo is claimed by the live pix-prodigy-cpu
   task (wsdev#117), so the removal belongs to it.

[Jammy2211]

Resumed and shipped — PR #1421

The parked question was answered in session: ship it, with the YELLOW reason
set acknowledged verbatim for this launch. Unparked and ran the remaining legs.

Library PR: #1421 (pending-release, 2 commits). Merge stays human.

The review pass caught a defect in my own first commit

int() truncates towards zero, so a fractional cadence below 1 (e.g.
iterations_per_full_update=0.5) would give range(0): no steps run,
total_steps never advances, and the enclosing while loop spins forever
re-running perform_update. The first commit therefore traded a loud TypeError
for a silent hang — strictly worse on HPC, where it burns the whole
allocation. The second commit floors the chunk at 1 (the slowest cadence that
still progresses; it can never overshoot steps_remaining, which is >= 1
whenever the loop is entered) and adds a test. Review is CLEAN on the second pass.

Gate results

Leg	Result
Tests	✅ 1535 passed, 1 skipped. No public API change (new method is private), so downstream dependent suites are n/a.
Smoke	✅ no regression — 50 passed, 7 failed, 3 skipped/missing across all six workspaces.
Review	✅ CLEAN (pass 2; pass 1 = FINDINGS, resolved above).
Heart	⚠️ YELLOW score 65, human-acknowledged at launch for that exact reason set.

On the smoke failures — none is caused by this branch. All 7 reproduce
identically when re-run against main under the same conditions; they are the
jax_likelihood parity scripts, which the smoke profile runs with
PYAUTO_DISABLE_JAX=1. A further 5 scripts failed only in the parallel sweep
(imaging/features/pixelization/delaunay.py,
interferometer/jax_likelihood/potential_correction.py,
imaging/subhalo_recovery.py, interferometer/subhalo_recovery_interferometer.py,
multi_galaxy/composition_mge.py) and pass on the branch when re-run
sequentially, matching main — contention over shared output//dataset/
state between 65 concurrently launched scripts in the runner, not a regression.
Worth noting as a smoke-harness sharp edge in its own right.

Workspace impact: option (iii), none

No public API changed, so no workspace migration and no new demos are needed.
The one workspace consequence is a deletion: the post-construction int
overwrite in autolens_workspace_developer/searches_minimal/pix_prodigy.py
(branch feature/pix-prodigy-cpu) becomes dead once this merges. That repo is
claimed by the live pix-prodigy-cpu task, so the removal belongs there —
noted on wsdev#117 rather than done here.

The sibling-search follow-up (five more searches sharing the latent float-cadence
class) is still unfiled as a prompt; it is recorded in the PR body and in the
opening comment above.

[Jammy2211]

Correction: the sibling follow-up list was wrong (3 of 5 are safe)

An independent adversarial review (Codex, gpt-5.6-sol) checked every consumer of
self.iterations_per_full_update empirically rather than by reading the call
shape, and falsified part of what I wrote above. Re-verified here:

Consumer	Earlier claim	Verified
mcmc/emcee/search.py:206	latent	❌ genuinely crashes — EnsembleSampler.sample does range(iterations) with no cast
mcmc/blackjax/nuts/search.py:291	latent	❌ genuinely crashes — jax.random.split(key, 50.0) raises the same TypeError
mcmc/zeus/search.py:242	latent	✅ safe — zeus casts internally (self.nsteps = int(iterations))
mle/bfgs/search.py:171	latent	✅ tolerated — SciPy uses maxiter in comparisons only
nest/dynesty/.../abstract.py:365, nest/nautilus/search.py:477	latent	✅ tolerated — used as comparison limits

So the real follow-up is two searches, both confirmed reproducible
(af.Emcee(nsteps=100, iterations_per_full_update=50) and the equivalent
BlackJAX config), not five unverified ones. That is a smaller and much more
actionable job than I described, and it no longer needs a "reproduce first"
caveat for those two.

It also undercuts my stated rationale for leaving abstract_search.py:219
alone. I argued the shared float() coercion is load-bearing because 1e99
needs to be representable — but Python ints represent 1e99 fine. The coercion
is not protecting anything; it is the actual source of the defect class. Casting
at the consumer is still correct for this PR (narrow, non-regressing), but
"fix the producer" is now the better framing for the follow-up, in line with this
workspace's crash-loudly / delete-the-trap doctrine.

Two further points from that review, on this diff

1. My "can never overshoot steps_remaining" claim is conditional, not
   absolute. It holds only because n_steps is an integer. With a float
   n_steps=2.5, the loop's guard proves steps_remaining > 0, not >= 1, so
   max(1, int(0.5)) returns 1 and total_steps reaches 3 — past the documented
   ceiling. n_steps is annotated int, so this is not reachable through the
   documented API, but the invariant is annotated rather than validated and I
   overstated it.

2. max(1, ...) silently launders invalid input. iterations_per_full_update
   of -5, 0.5 or 50.9 all become a valid-looking cadence instead of an
   error. That is exactly the silent-guard pattern this workspace removes rather
   than adds; a ValueError naming the bad value would be the doctrinal fix.
   Flagging rather than changing unilaterally, since it widens this PR's scope.

Two pre-existing issues that review surfaced (not caused by this branch)

• The final perform_update appears to run twice. MultiStart performs its
  final update at search.py:395 with during_analysis=False, and
  start_resume_fit then unconditionally performs the same final update at
  abstract_search.py:704 — duplicating final sample output, latent computation,
  visualization and profiling.
• A completed stop_reason="max_steps" search resumed with a larger n_steps
  keeps the stale stop reason in intermediate checkpoints, because
  search.py:375 only resets it on convergence or on reaching the new ceiling.

Both pre-date this branch and belong in their own prompts.

[Jammy2211]

Acted on the review — commit 3 pushed, three follow-up prompts filed

Commit 3: reject an unusable cadence instead of clamping it

max(1, int(...)) is gone. Both the cadence and n_steps are now validated
as whole numbers of at least 1, and an unusable value raises a ValueError
naming it:

MultiStartAdam: `iterations_per_full_update` must be a whole number of gradient
steps and at least 1, but was 0.5. A fractional or sub-1 value gives a
zero-length step chunk, which never advances the search and would hang it rather
than fail.

Validating n_steps is what let the floor be removed rather than kept
alongside the check: once n_steps is a positive whole number, steps_remaining
is an int of at least 1 whenever the loop is entered, so the chunk is
guaranteed >= 1 and <= steps_remaining without a clamp. That closes the
overshoot gap properly instead of papering over it.

Tests: unusable cadence (0.5, 50.9, -5) and unusable n_steps (2.5, 0,
-10) each raise; plus a wiring guard — every other test drives
_steps_in_chunk directly, so all of them would still pass if _fit went back
to computing the chunk inline, which is the exact regression this PR is about.

Full suite: 1541 passed, 1 skipped.

The two pre-existing bugs — confirmed, and the first is sharper than reported

Verified by reading the call sites. The duplicate-update one is not a framework
quirk, it is MultiStart being an outlier:

Search	during_analysis inside _fit
mcmc/emcee/search.py:238	True (unconditional)
mle/bfgs/search.py:219	True (unconditional)
nest/nautilus/search.py:438	True (unconditional)
multi_start_gradient/search.py:432	not is_final — so False on the last chunk

Every other search leaves the single during_analysis=False update to
start_resume_fit. MultiStart emits its own, so the final sample output, latent
computation, visualization and profiling all run twice at the end of every run.

Prompts filed (PyAutoMind 8a3a4d1)

• draft/bug/autofit/emcee_blackjax_iterations_per_full_update_float_crash.md —
  the two confirmed sibling crashes, with the three verified-safe searches called
  out explicitly so nobody "fixes" zeus. Recommends fixing the producer
  (abstract_search.py:219) this time, since the float coercion protects nothing.
• draft/bug/autofit/multistart_duplicate_final_perform_update.md
• draft/bug/autofit/multistart_stale_stop_reason_on_resume.md

All three are drafts, not issues — no bulk issue queue.

[Jammy2211]

Shipped — PR #1421 merged as e217292 (CI green on Docs + both Tests matrix legs).

Three follow-up prompts are filed in PyAutoMind (8a3a4d1) and are being taken up now as a single combined task: the emcee/BlackJAX sibling crashes, the duplicate final perform_update, and the stale stop_reason on resume.

Closing.