fix: triage the release-run Emcee NaN + database regressions

[Jammy2211]

Overview

Release run 28784914443
(PyAutoHeart #27, the first real release-profile validation) reported 42 script failures, split
across seven sibling prompts in PyAutoMind/draft/bug/health_fixes/. This task owns nine of
them: two autofit_workspace cookbooks failing with emcee's ValueError: Probability function returned NaN on a bounded LogUniformPrior, and seven autofit_workspace_test database/output
scripts reporting empty aggregators, stale/mismatched scraped search metadata and changed
likelihood assertions.

The premise is 6+ weeks stale and independently suspect, so this issue is deliberately
reproduction-gated: Phase 1 establishes what still fails on current main from a genuinely clean
state under the real release profile, and only the survivors get fix PRs.

Plan

• Phase 1 (gate). Re-run all nine scripts on current main from a clean output/ under each
  workspace's config/build/profile_release.yaml, resolving each script's env with PyAutoHands'
  own resolver. Post a nine-row pass/fail/exception table here.
• Decide from the gate. 0/9 failing → park the prompt like its sibling and close with the
  evidence, no PR. Failures matching the prompt → Phases 2–4 for those only. Failures with a
  different shape → re-diagnose from the real error, never apply the prompt's prescribed fix.
• Phase 2. Fix the Emcee / bounded-prior NaN in PyAutoFit source (not in the cookbooks).
• Phase 3. Audit and fix the database directory/scrape/grid/sensitivity/minimal-output paths in
  PyAutoFit; change a script assertion only where the documented contract intentionally changed.
• Phase 4. Add regression coverage for each real defect, run PyAutoFit pytest, re-run all nine
  scripts clean, then ship library-first.

Detailed implementation plan

Work Classification

Both (library + workspace) → /start_library first; workspace follow-up only if Phase 3 needs
a script change.

Affected Repositories

• PyAutoFit (primary)
• autofit_workspace
• autofit_workspace_test

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean
./autofit_workspace	main	clean
./autofit_workspace_test	main	clean

No worktree conflicts — the live active.md claims cover PyAutoGalaxy,
autolens_workspace_developer, autolens_profiling and PyAutoHands only.

Suggested branch: feature/autofit-sampler-database
Worktree root: ~/Code/PyAutoLabs-wt/autofit-sampler-database/ (created by /start_library)

Why this is reproduction-gated

1. The sibling prompt from this exact cluster was parked as not-reproducing.
   samples_parameter_paths.md (9 scripts, fix(autofit): resolve sample parameter-path KeyError under release profile #1327) passed every leg on current main — in-memory
   and via true from-disk reload. Root cause was judged stale cached output/ in the release
   run, not a library defect. Its planned.md entry ends: "Sibling health_fixes/ prompts from
   the same run are suspect too."
2. A second sibling already shipped — aggregator_output_contracts (2026-07-07, fix: restore aggregator output contracts #1324 plus
   three workspace PRs) touched result/aggregator prerequisites and generated paths, i.e. machinery
   adjacent to several scripts here.
3. PyAutoFit has merged a wall of directly-relevant fixes since the run: fix: db scrape crashes on empty PrimaryHDU; latent guard vacuous #1413 (db scrape
   crashes on empty PrimaryHDU), 9f887a9b1 (tolerate data-less HDUs in the scrape), 8f6f4ef7d
   (sqlite slicing + direct-write samples_summary), fix: exercise + assess the sqlite results database (scrape + direct-write) — aggregator Phase D #1377 (exercise/assess the sqlite results
   database, scrape + direct-write), fix: Aggregator.from_directory is not test-mode-aware (interpolate tutorial IndexError) #1401 (Aggregator.from_directory not test-mode-aware), fix: three cadence/update bugs from the PR#1421 review (emcee+blackjax crash, double final update, stale stop_reason) #1422
   (emcee crash + cadence bugs), fix: expose valid stored sample pairs #1470/fix(autofit): reconstructing a stored sample raises through ignore_assertions=True #1486 (stored-sample recovery), docs: pin Fitness's NaN guard contract — value-only, never gradient #1391 (pin Fitness's
   NaN-guard contract: value-only, never gradient). main is at Release 2026.8.20.1.
4. None of the nine scripts is parked in either workspace's config/build/no_run.yaml, so they
   are live in validation. The currently-green per-PR "Smoke Tests" gate runs the smoke profile
   over a curated subset and is not evidence about release-profile behaviour.

Implementation Steps

Phase 1 — clean-state reproduction under the release profile (the gate)

1. Resolve each script's environment with PyAutoHands/autobuild → build_env_for_script, run
   with the workspace as CWD, so both the profile defaults and the in-file __Env__ /
   # ENV: real_search declarations are honoured. Do not hand-roll env vars.

   Workspace	profile_release.yaml defaults
   autofit_workspace (2 cookbooks)	PYAUTO_TEST_MODE=1 (reduced real sampler), PYAUTO_SKIP_{FIT_OUTPUT,VISUALIZATION,CHECKS}=0, PYAUTO_DISABLE_JAX=0, JAX_ENABLE_X64=True
   autofit_workspace_test (7 scripts)	PYAUTO_TEST_MODE=0 (full real searches), PYAUTO_DISABLE_JAX=0, JAX_ENABLE_X64=True

   Five of the seven test scripts carry an in-file ENV: real_search declaration
   (database/{directory,scrape}/general.py, database/{directory,scrape}/multi_analysis.py,
   features/minimal_output.py); database/scrape/grid_search.py and
   database/scrape/sensitivity.py do not — check what they resolve to before running.

2. Clean-state rules (these are precisely what made the sibling prompt's failures evaporate):

   • Move output/ aside per workspace before each run — keep it for comparison, don't delete.
   • Database scripts need all three paths cleared: output/database/,
     output/database.sqlite, output/database.info. database.sqlite is a sibling of
     output/database/, so clearing only the directory leaves the aggregator reading the previous
     run's rows — a false PASS, or inverted, a false stale-metadata FAIL.
   • Run the directory and scrape scripts in the documented order, one at a time, never against a
     prior run's products.

3. Record per script: pass/fail, exact exception + file:line, and whether the failure shape
   matches the prompt's claim. Post the table to this issue.

Phase 2 — Emcee / bounded-prior NaN (only if reproduced)

4. Locus is PyAutoFit source. emcee's stretch move proposes points whose LogUniformPrior
   (1e-2 … 1e2, autofit_workspace/scripts/cookbooks/result.py:289) transform yields a
   non-finite log-probability; emcee raises rather than treating it as zero-probability.
   Investigate autofit/non_linear/search/mcmc/emcee/ plus the shared Fitness NaN guard (docs: pin Fitness's NaN guard contract — value-only, never gradient #1391
   pinned the contract: value-only, never gradient). The fix is to return the resample sentinel
   emcee understands for out-of-bounds/NaN evaluations, and to confirm the initializer never seeds
   walkers outside the bounded prior support.
5. Fix the library, not the cookbook — cookbooks/{result,samples}.py are user-facing docs and
   must not gain injected env vars, hard-coded paths or silent guards.

Phase 3 — database directory/scrape/grid/sensitivity (only if reproduced)

6. For each surviving script, audit against current PyAutoFit: output paths and unique identifiers,
   session lifecycle, Aggregator.from_directory vs from_database + add_directory, grid-search
   and sensitivity metadata writes, minimal_output expectations.
7. Fix the library where the contract is broken. Change a script assertion only where the
   documented output contract intentionally changed, citing the changing commit/PR inline here.

Phase 4 — coverage, tests, re-run

8. Add a PyAutoFit unit/integration test per defect actually fixed — one that fails on the parent
   commit. Run PyAutoFit pytest.
9. Re-run all nine scripts from clean state under the release profile. No new no_run.yaml
   parkings may be introduced to reach green.
10. /ship_library (PyAutoFit) → /ship_workspace for any workspace edits.

Key Files

• autofit_workspace/scripts/cookbooks/result.py — Emcee at L104/L258, LogUniformPrior at L289
• autofit_workspace/scripts/cookbooks/samples.py — second Emcee NaN cookbook
• autofit_workspace_test/scripts/database/directory/{general,multi_analysis}.py
• autofit_workspace_test/scripts/database/scrape/{general,grid_search,multi_analysis,sensitivity}.py
• autofit_workspace_test/scripts/features/minimal_output.py
• autofit/non_linear/search/mcmc/emcee/ — Emcee search + fitness wiring
• autofit/database/ — scrape, session lifecycle, aggregator queries
• */config/build/profile_release.yaml — the release-fidelity env profiles
• */config/build/no_run.yaml — confirm none of the nine is parked (currently none is)

Cost warning

autofit_workspace_test runs at PYAUTO_TEST_MODE=0 — full searches, release cap 1800s/script.
Seven of those (including multi_analysis, grid_search, sensitivity, each running many fits)
plausibly means multiple hours for one clean pass. Phase 1 should run detached and sequentially.

Verification

• Phase 1: the nine-row reproduction table posted here, produced from a moved-aside output/ and a
  cleared database.sqlite/.info.
• Phases 2–3: each fix has a PyAutoFit test that fails on the parent commit.
• Phase 4: pytest green; all nine scripts green from clean state under profile_release.yaml.

Original Prompt

Click to expand starting prompt

Fix Autofit release sampler and database regressions

Type: bug
Target: health_fixes
Difficulty: too-large
Autonomy: supervised
Priority: high
Status: formalised

Context

Release run 28784914443 failed two @autofit_workspace cookbooks and seven
@autofit_workspace_test scripts. The cookbook failures reproduce on current main:
Emcee proposes invalid LogUniform values and raises ValueError: Probability function returned NaN. Database failures include empty aggregators, stale/mismatched scraped
search metadata, and changed likelihood assertions.

Primary library: @PyAutoFit.

Scripts

• autofit_workspace/scripts/cookbooks/result.py
• autofit_workspace/scripts/cookbooks/samples.py
• autofit_workspace_test/scripts/database/directory/general.py
• autofit_workspace_test/scripts/database/directory/multi_analysis.py
• autofit_workspace_test/scripts/database/scrape/general.py
• autofit_workspace_test/scripts/database/scrape/grid_search.py
• autofit_workspace_test/scripts/database/scrape/multi_analysis.py
• autofit_workspace_test/scripts/database/scrape/sensitivity.py
• autofit_workspace_test/scripts/features/minimal_output.py

Required work

1. Reproduce each script from a clean output/database state using the exact release
   profiles; directory and scrape scripts must not consume prior local runs.
2. Fix Emcee initialization/proposal handling so bounded priors cannot emit NaN
   probabilities during reduced release-mode sampling.
3. Audit database directory and scrape paths, identifiers, session lifecycle, grid and
   sensitivity metadata, and minimal-output expectations against current PyAutoFit.
4. Fix library defects in PyAutoFit. Change script assertions only when the documented
   output contract intentionally changed, with an explicit explanation.
5. Add unit/integration coverage, run PyAutoFit pytest, and rerun all nine scripts.

[Jammy2211]

Phase 1 gate result: 0 / 9 reproduce on current main

Ran all nine scripts on feature/autofit-sampler-database (PyAutoFit @ 248ca971f, Release
2026.8.20.1; autofit_workspace @ b8a0c86; autofit_workspace_test @ d5d73ab), each from a
moved-aside output/ under its workspace's config/build/profile_release.yaml, with env
resolved by autohands.env_config.build_env_for_script at workspace CWD.

#	Script	Result	Secs
1	autofit_workspace/scripts/cookbooks/result.py	✅ PASS	8
2	autofit_workspace/scripts/cookbooks/samples.py	✅ PASS	4
3	autofit_workspace_test/scripts/database/directory/general.py	✅ PASS	5
4	autofit_workspace_test/scripts/database/directory/multi_analysis.py	✅ PASS	4
5	autofit_workspace_test/scripts/database/scrape/general.py	✅ PASS	4
6	autofit_workspace_test/scripts/database/scrape/grid_search.py	✅ PASS	10
7	autofit_workspace_test/scripts/database/scrape/multi_analysis.py	✅ PASS	5
8	autofit_workspace_test/scripts/database/scrape/sensitivity.py	✅ PASS	21
9	autofit_workspace_test/scripts/features/minimal_output.py	✅ PASS	6

The passes are not vacuous

Audited every log rather than trusting exit 0:

• 20 searches started fresh across the nine scripts (Starting new {Dynesty,Emcee} non-linear search); zero stale-output resumes, zero Failed to load samples at the script level.
  The one Fit Already Completed (in minimal_output.py) is that script's own deliberate second
  call to the same search, after a fresh first run — not a stale-cache artifact.
• The database scripts' assert len(agg) > 0 executed and held, and the aggregator loop bodies
  ran (assert type(samples) == af.SamplesNest, assert search.paths.name == "general", the
  instance/data/noise_map asserts). Aggregators were populated, not empty.
• 4 Emcee searches ran (2 in each cookbook). No Probability function returned NaN, no
  ValueError, anywhere in the nine logs.
• Env resolution verified before the run: the five scripts carrying ENV: real_search correctly
  had PYAUTO_TEST_MODE released (absent == off == "0" per ENV_DECLARATION_TOKENS), and
  scrape/grid_search.py / scrape/sensitivity.py took "0" from the profile defaults — i.e. all
  seven autofit_workspace_test scripts ran real full searches.

Two honest limits on this evidence

1. The cookbooks ran at PYAUTO_TEST_MODE=1 — that is exactly what
   autofit_workspace/config/build/profile_release.yaml pins, and matches the prompt's own
   "reduced release-mode sampling" framing. But reduced iterations mean fewer emcee proposals, so a
   NaN needing a long walk to hit could be missed. The Emcee leg's refutation is therefore weaker
   than the database legs'.
2. This ran against source checkouts, not the TestPyPI wheels the release run installed. A
   wheel-only packaging defect would not show here.

Incidental findings (noise, not regressions — filed separately, not fixed here)

• autofit.database.aggregator.scrape logs Failed to load latent variables for essentially
  every scraped fit (11 occurrences). Probing one fit's zip shows it contains
  files/samples.csv, samples_info.json, samples_summary.json and no latent file at all —
  so _add_files_fit's bare except AttributeError (autofit/database/aggregator/scrape.py:150-158)
  is warning about legitimate absence. CLI noise on the happy path.
• Same guard emits Failed to load samples for the multi-analysis child fits (…_0, …_1),
  which have no own samples.csv. Nothing asserts on them, so it is silent.
• 6 × Could not save covariance matrix because of the following error: from
  autofit.non_linear.paths.directory.

Recommendation

Park this the way its sibling was parked. samples_parameter_paths (#1327) from this same
release run was investigated to the same conclusion — root cause judged stale cached output/
in the run, not a library defect — and a third sibling (aggregator_output_contracts) shipped on
2026-07-07. Since the run, PyAutoFit merged #1413, 9f887a9b1, 8f6f4ef7d, #1377, #1401, #1422,
#1470, #1486 and #1391, all in the exact scrape/aggregator/emcee machinery this prompt blames.

No library fix is warranted on this evidence, and no PR is opened. The prompt returns to
draft/ and the task registers in planned.md as parked, blocked on a clean-output release
re-validation (wheel-based, mode=release) — that run, not a source-tree repro, is what can
still distinguish "already fixed" from "wheel-only defect".

[Jammy2211]

Closing — no library defect on this evidence

Phase 1 settled it: 0/9 of the scripts reproduce on current main, with the passes audited
rather than assumed (20 fresh searches, zero stale resumes, assert len(agg) > 0 executed and
held, 4 Emcee searches with no Probability function returned NaN). Full table and the two
evidence limits are in the comment above.

Why close rather than leave parked-open like the sibling #1327: the re-validation this was
blocked on happens automatically. None of the nine scripts is in either workspace's
config/build/no_run.yaml, so every mode=release validation run executes all nine under exactly
the profile used here. If a real defect survives in the wheel path, the next release run fails
loudly on these same scripts and earns a fresh issue with fresh evidence — which is worth more than
this one carrying a 6-week-old diagnosis forward. There is no human reminder to lose.

The prompt returns to PyAutoMind/draft/bug/health_fixes/autofit_sampler_database.md with a parked
entry in planned.md recording this result, so the cluster's own tracking still reflects it.

No PR was opened and no source file was changed.