fix: stop a crashed run poisoning every later run of the same search

[Jammy2211]

A run interrupted while writing its output left a half-written JSON file on disk, and every subsequent run of that search name then died on it — from inside an optional sanity check — until the output directory was deleted by hand.

Reproduced against main

Using the real trigger, a float32 killing save_json at the end of a successful fit:

run 1:  TypeError: Object of type float32 is not JSON serializable
run 2:  JSONDecodeError: Expecting value: line 1 column 13 (char 12)

The message names no file and suggests no remedy. And run 2 is not a one-off: nothing rewrites the corrupt file until a run gets far enough to finish, so the search stays wedged.

Note on the original report. This was filed as producing "a 4-second no-op run that reads as a clean result". That is not what reproduces here — what reproduces is the hard crash above. The silent-no-op variant presumably needs a surviving search_internal (whose total_steps short-circuits the loop). Both are the same root cause, and the fix covers both paths, but I could only reproduce the crash and would rather say so than claim otherwise.

Three causes, three legs

1. Writes were not atomic. open(path, "w+") truncates first and writes second, so a failure partway destroys the file that was there. Adds open_atomic, which writes a sibling temp file and os.replaces it into place — atomic on POSIX and Windows, and same-directory so it stays a rename within one filesystem. On failure the temp file is removed and the original is untouched. It catches BaseException, since a KeyboardInterrupt mid-write leaves exactly the same debris.

Used by save_json and by save_search_internal — the latter matters more, since search_internal is what a resumed run restores its step count and counters from.

2. Fitness.check_log_likelihood aborted the run on an unreadable summary. It already returns early on FileNotFoundError — no previous run, nothing to check — and a corrupt summary is the same situation, since there is no trustworthy old likelihood either way. It now returns early there too, with a warning naming the cause and the remedy rather than silence: an unreadable file is a real event, unlike a missing one.

3. The multi-start resume guard had the same narrow shape. It caught (FileNotFoundError, TypeError, KeyError), so a corrupt search_internal raised instead of falling into the fresh-start branch directly below it. Widened to treat truncated or unpicklable state as "no state to resume from", warning when it does.

The one fact behind all three

json.JSONDecodeError subclasses ValueError. So it is neither a FileNotFoundError, a TypeError nor a KeyError, and it fell through every guard on the resume path. Asserted directly in a test so it cannot silently stop being true.

After

The same sequence warns and completes a full 30-step run instead of dying:

WARNING - Could not read the previous samples summary while resuming
(JSONDecodeError: ...). It is missing or corrupt, most likely because an
earlier run of this search was interrupted while writing its output. The
likelihood-function sanity check is being SKIPPED for this run, and results
are being recomputed. Delete the search's output directory if you want a
guaranteed-clean start.

total_steps: 30    stop_reason: max_steps

Testing

9 new tests: open_atomic success/failure/interrupt/temp-file-cleanup/binary, save_json preserving the previous file when a write fails, the JSONDecodeError-is-a-ValueError assertion, and an end-to-end two-run test that asserts the second run takes the resume path (rather than being short-circuited as already-complete, which would make it assert nothing) and does not raise.

Full suite: 1800 passed, 4 skipped, 1 failed. The failure (test_nautilus.py::test__single_core_builds_no_pool) is pre-existing and reproduces identically on a clean tree.

Related

Branched from main, independent of #1478 and #1479. #1479 removes the most common trigger (the float32 crash); this PR removes the consequence, so a crash from any cause — full disk, killed process, interrupt — can no longer wedge a search. They are worth landing together but neither depends on the other.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FzF2XmKQaqZRWZfZMxvTNR

---

Generated by Claude Code