Skip to content

fix: three cadence/update bugs from the PR#1421 review (+2 found in review) - #1423

Merged
Jammy2211 merged 3 commits into
mainfrom
feature/multistart-cadence-followups
Jul 27, 2026
Merged

fix: three cadence/update bugs from the PR#1421 review (+2 found in review)#1423
Jammy2211 merged 3 commits into
mainfrom
feature/multistart-cadence-followups

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Closes #1422.

Three bugs surfaced by the adversarial review of #1420 / PR#1421, fixed in one PR
at the human's request. Two more were found during this PR's own review and are
fixed here too.

Summary

Part 1 — Emcee and BlackJAX NUTS crashed on a real iterations_per_full_update
cadence
, the same defect PR#1421 fixed for MultiStart. emcee's sample() does
range(iterations) with no cast of its own; jax.random.split rejects a float.
Both verified by probing the callee, not by reading the call shape.

The issue plan said to fix the producer (the float() coercion at
abstract_search.py:219), on the grounds that Python ints hold 1e99 fine.
Rejected after auditing the consumers: int(1e99) is a 99-digit integer,
and writing that into every saved search.json in place of a readable inf-like
sentinel is a bad trade for a conversion each caller can do at the point of use.

Instead the conversion moves up, into one shared, validated
AbstractSearch._steps_until_full_update. Deriving it once beats re-deriving it
per search, because re-deriving it is precisely how this bug class shipped three
times. AbstractMultiStartGradient._steps_in_chunk (added in PR#1421) is folded
into it.

Part 2 — MultiStart ran the expensive final pass twice. _fit emitted
during_analysis=False on its last chunk while start_resume_fit performs the
final update unconditionally anyway.

The first attempt here just flipped the flag to True, matching every sibling
search. Review showed that was cosmetic: SearchUpdater.update rebuilds the
samples, recomputes the summary and re-runs likelihood profiling on every call
regardless of the flag, and the visualization gate keys off paths.is_complete,
which isn't written until after _fit returns. The in-loop update is now
skipped outright at a terminal boundary.

Part 3 — a resumed run inherited the previous run's stop_reason. Raising
n_steps to extend a finished search left a stale "max_steps" in every
intermediate checkpoint, so a still-running search reported itself finished to
the aggregator and any results inspector. Cleared on entry, with "converged"
preserved because the loop guard uses it to refuse resuming a converged search.

Two extra bugs found by this PR's own review

Zeus was not safe — just not crashing. I had cleared it on #1420 because it
casts internally (self.nsteps = int(iterations)). But PyAutoFit adds the
uncast float to its own total_iterations (zeus/search.py:261), so a
fractional cadence drifts the bookkeeping away from the samples zeus actually
drew: nsteps=100, iterations_per_full_update=50.9 runs 50 then 49 steps — 99
samples — while the bookkeeping reaches 100. BFGS also bypassed the validation
the helper advertises for maxiter. Both now use the shared helper. This
supersedes the "do not touch zeus/bfgs" note on #1422
, which rested on my own
earlier, wrong finding.

The falsy-cadence branch reintroduced the silent behaviour the validation
exists to remove
: a stored 0 meant "never checkpoint". 1e99 is already that
sentinel and flows through the min without a special case, so 0 is a
misconfiguration — reachable via the HPC override, which assigns the config value
with no or fallback — and now raises.

API Changes

None. Three new private methods (AbstractSearch._steps_until_full_update /
._check_step_count, AbstractMultiStartGradient._is_final_boundary /
._stop_reason_on_resume); AbstractMultiStartGradient._steps_in_chunk, private
and added in the previous PR, is removed. No public class, signature, argument,
default or config key changed, and the serialized iterations_per_full_update
stays the float 1e99 — deliberately.

Behaviour changes are all corrective:

  • inputs that previously crashed (Emcee/BlackJAX with a real cadence) now work;
  • a cadence or budget that is fractional, zero or negative now raises ValueError
    instead of hanging, drifting, or silently disabling checkpointing;
  • MultiStart no longer performs a duplicate final update, so runs get faster
    and figures/summaries are written once instead of twice;
  • a resumed MultiStart run reports no stop reason until it actually stops.

Testing

  • pytest test_autofit/1557 passed, 1 skipped.
  • New test_autofit/non_linear/search/test_steps_until_full_update.py — the
    shared cadence seam: a real cadence returns an int range() consumes; clamps
    to the remaining budget; the 1e99 default is one chunk; unusable cadence
    (0.5, 50.9, -5), unusable remaining budget (2.5, 0, -10) and a
    stored 0 each raise; plus a wiring guard across all five chunked searches.
  • _is_final_boundary and _stop_reason_on_resume are parametrised
    exhaustively, so Parts 2 and 3 are pinned by behaviour, not by source
    strings.
  • All tests NumPy-only, per the library-suite rule — the _fit bodies need
    jax/optax/emcee and are exercised in the workspace test repos.

Review

Two adversarial passes with Codex gpt-5.6-sol (xhigh), per the instruction on
#1422.

  • Pass 1 → FINDINGS, all three confirmed and fixed: the Part 2 flag flip was
    cosmetic; zeus/bfgs still bypassed the helper; the falsy branch was silent.
  • Pass 2 → production code CLEAN, verified by running real JAX fits: a
    one-chunk run produces one checkpoint, zero in-loop updates and exactly one
    outer during_analysis=False update; a two-chunk run produces [True, False]
    and two checkpoints. It also confirmed all five helper callers accept the
    normal integral-float configs and that no packaged, HPC or test config stores a
    value that now raises.
  • Pass 2's remaining findings were about test strength — source-string assertions
    that a semantically-equivalent regression could slip past. Addressed by
    extracting the two rules into seams and testing those directly.

Known residual, stated rather than papered over: the cross-search wiring
guard cannot prove a search uses the value the helper returns, so a contrived
regression that calls it and discards the result would pass. Closing that needs
the _fit bodies, which need jax/optax/emcee — out of scope for the NumPy-only
library suite, and covered where those bodies actually run, in the workspace test
repos.

🤖 Generated with Claude Code

Jammy2211 and others added 3 commits July 27, 2026 16:08
Part 1 - Emcee and BlackJAX NUTS crashed on a real cadence, the same
defect PR#1421 fixed for MultiStart: emcee's sample() does
range(iterations) with no cast, and jax.random.split rejects a float.
Both verified by probing the callee rather than reading the call shape.

The plan said fix the producer (the float() coercion in
AbstractSearch.__init__), on the grounds that Python ints hold 1e99 fine.
Rejected after auditing the consumers: int(1e99) is a 99-digit integer,
and writing that into every saved search.json in place of a readable
inf-like sentinel is a bad trade. Instead the conversion moves up to one
shared, validated AbstractSearch._steps_until_full_update, so it is
derived once rather than re-derived (and forgotten) per search - which is
exactly how this class of bug shipped three times. MultiStart's local
_steps_in_chunk is folded into it.

Part 2 - MultiStart emitted during_analysis=False on its last chunk while
start_resume_fit emits the final update unconditionally anyway, so the
whole final pass (sample output, latents, visualization, profiling) ran
twice on every search. Emcee, BFGS and Nautilus all pass True
unconditionally; MultiStart was the outlier. Now it matches them.

Part 3 - a resumed run inherited the previous run's stop_reason, so
raising n_steps to extend a finished search left a stale max_steps in
every intermediate checkpoint and a still-running search reported itself
finished. Cleared on entry, with converged preserved because the loop
guard uses it to refuse resuming a converged search.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three findings, all confirmed before acting.

1. The Part 2 fix was cosmetic. Flipping during_analysis to True does
   not deduplicate the final pass: SearchUpdater.update rebuilds the
   samples, recomputes the summary and re-runs likelihood profiling on
   every call regardless of the flag, and the visualize gate keys off
   paths.is_complete, which is not written until after _fit returns. The
   in-loop update is now skipped outright at a terminal boundary. On the
   default single-chunk cadence _fit emits no update at all, leaving
   exactly one final pass.

2. Zeus was NOT safe, just not crashing. It casts iterations internally,
   but PyAutoFit adds the uncast float to total_iterations, so a
   fractional cadence drifts the bookkeeping away from the samples zeus
   actually drew (nsteps=100, cadence=50.9 finishes with 99). BFGS also
   bypassed the validation the helper advertises for maxiter. Both now
   use the shared helper, so every chunked search derives its chunk one
   way. This supersedes the issue's "do not touch zeus/bfgs" note, which
   rested on my own earlier finding that they were fine.

3. The falsy-cadence branch reintroduced the silent behaviour the
   validation exists to remove: a stored 0 meant "never checkpoint".
   1e99 is already that sentinel and needs no special case, so 0 is a
   misconfiguration - reachable through the HPC override, which assigns
   the config value with no "or" fallback - and now raises.

Also fixes the test guards the review showed were brittle or tautological:
BlackJAX, Zeus and BFGS added to the wiring guard; the loop-guard
assertion now pins the while line rather than being satisfied by the
clearing if; source assertions normalise whitespace.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second review pass confirmed the production fixes (it ran real JAX fits:
a one-chunk run now emits zero in-loop updates and exactly one final
update; a two-chunk run emits [True, False]). What it did not accept were
the regression tests: source-level string assertions can pass while a
semantically-equivalent regression is reintroduced - assigning is_final
False, or adding a second unconditional update after the guarded one.

So extract the two rules the loop was applying inline into named seams -
_is_final_boundary and _stop_reason_on_resume - and test those directly
and exhaustively, the same move that made the cadence fix testable
without JAX. The source assertion shrinks to a wiring guard over the call
sites, with the behaviour pinned by the parametrised tests instead.

Known residual, accepted rather than papered over: the cross-search
wiring guard still cannot prove a search *uses* the value the helper
returns, so a contrived regression that calls the helper and discards the
result would pass it. Closing that needs the searches' _fit bodies, which
need jax/optax/emcee and are out of scope for the NumPy-only library
suite; it is covered where those bodies actually run, in the workspace
test repos.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release PR queued for the next release build label Jul 27, 2026
@Jammy2211
Jammy2211 merged commit c042d72 into main Jul 27, 2026
5 checks passed
@Jammy2211
Jammy2211 deleted the feature/multistart-cadence-followups branch July 27, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: three cadence/update bugs from the PR#1421 review (emcee+blackjax crash, double final update, stale stop_reason)

1 participant