fix: db scrape crashes on empty PrimaryHDU; latent guard vacuous

[Jammy2211]

Overview

Filed from the 2026-07-21 census as four test-mode NoneType failures needing env_vars.yaml overrides. Reproduction shows that premise is wrong for three of the four. Only one script fails for a real reason, and that reason is a PyAutoFit library bug, not env config. A second script passes but vacuously — a genuine env gap, though not the crash the census reported. The remaining two are clean.

The census ran against a warm output/ tree. PYAUTO_TEST_MODE=2 writes no samples, so a second run of a bypass script resolves as Fit Already Completed: skipping non-linear search and returns result.samples = None — which is where all three NoneType tracebacks came from.

Findings

Script	Census verdict	Reproduced (clean output/)
autolens_workspace_test imaging/model_fit.py	NoneType.parameter_lists	PASSES — stale-output artifact, no change needed
autolens_workspace_test latent/latent_variables_smoke.py	NoneType.model	PASSES — stale-output artifact; docstring explicitly designs for TEST_MODE=2
autolens_workspace_test latent/latent_nan_robustness.py	NoneType.sample_list	PASSES but vacuously — real env gap
autofit_workspace_test profiling/aggregator/profile_database.py	NoneType.dtype	REAL bug — in PyAutoFit

1. Library bug: data-less PrimaryHDU crashes the sqlite scrape

autofit/database/model/array.py:187 — the HDU.hdu setter does self.array = hdu.data, and the Array.array setter dereferences array.dtype unguarded. Any PrimaryHDU without data raises AttributeError: 'NoneType' object has no attribute 'dtype'.

Minimal repro, env-independent:

FAIL data-less PrimaryHDU: AttributeError: 'NoneType' object has no attribute 'dtype'
OK   PrimaryHDU with data: round-trip -> PrimaryHDU, data=(2, 2)

This is user-facing, not a test artifact: autofit/aggregator/summary/aggregate_fits.py:107 emits fits.PrimaryHDU() as the first HDU of every aggregated HDUList, and its own docstring (line 94) states "The first HDU in each list is an empty PrimaryHDU". The database path therefore cannot ingest the library's own standard multi-extension FITS output.

2. Env gap: latent_nan_robustness.py never exercises its branch

The guard exists to catch a JAX-only latent column-masking bug. Under the smoke profile it is doubly vacuous:

• PYAUTO_TEST_MODE=2 yields 4 bypass samples, barely clearing its own > LATENT_BATCH_SIZE (3) assertion;
• PYAUTO_DISABLE_JAX=1 silently flips its deliberate AnalysisImaging(use_jax=True) to False (PyAutoLens/autolens/analysis/analysis/dataset.py:89), so the JAX masking branch under test is never taken.

Same "passes vacuously" pattern as the existing searches/MultiStartAdam and searches/BlackJAXNUTS overrides in autofit_workspace_test.

Unsetting both makes it genuinely pass (100 samples, JAX path live) — but costs 412s against the 300s per-script cap. PYAUTO_TEST_MODE=1 does not help (455s); Nautilus is not the bottleneck (~136s post-fit results update + ~56s latent compute). Both latent scripts sit in the curated smoke_tests.txt, which does read env_vars.yaml, so this lands in the per-PR gate.

Plan

• Fix the PyAutoFit database scrape so a data-less PrimaryHDU round-trips instead of crashing.
• Cover it with a unit test, since the library's own AggregateFITS emits exactly that shape.
• Unset PYAUTO_TEST_MODE and PYAUTO_DISABLE_JAX for latent_nan_robustness so it exercises the branch it guards.
• Trim that script so the honest run lands under the 300s cap.
• Record that imaging/model_fit.py and latent_variables_smoke.py are not bugs.

Detailed implementation plan

Work Classification

Both (library first, workspace follows).

Affected Repositories

• PyAutoFit (primary)
• autolens_workspace_test
• autofit_workspace_test (verification only — no edits expected)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean
./autolens_workspace_test	main	25 modified
./autofit_workspace_test	main	9 modified

The dirty files in both workspaces are regenerated simulator datasets timestamped 2026-07-21 21:33–21:49 (yesterday's census run), not from this task. They must be excluded at commit time — ship_workspace binary-leak pre-flight.

Suggested branch: feature/empty-hdu-scrape-and-latent-guard

Worktree root: ~/Code/PyAutoLabs-wt/empty-hdu-scrape-and-latent-guard/

Conflict note: worktree_check_conflict flags PyAutoFit as claimed by interpolator-stale-needs-fix (worktree: none, in-place branch, status awaiting-merge behind PR #1412). Human-approved to proceed in a dedicated worktree: the files are disjoint (interpolator/abstract.py vs database/model/array.py) and a worktree never touches the main checkout that task's branch lives on.

Implementation Steps

1. autofit/database/model/array.py — in the HDU.hdu setter (~L187), guard hdu.data is None and record an empty payload (null dtype/shape/bytes). In the hdu getter (~L175), reconstruct with data=None when that marker is set, so PrimaryHDU() → database → PrimaryHDU() is lossless. Keep Array.array's unguarded dtype path intact — the fix belongs in HDU, which is where a None payload is legitimate.
2. test_autofit/database/ — add a test round-tripping fits.HDUList([fits.PrimaryHDU(), fits.ImageHDU(data=...)]), asserting the primary returns data-less and the image HDU intact. Numpy-only, no JAX.
3. autolens_workspace_test/config/build/env_vars.yaml — add an override for latent/latent_nan_robustness with unset: [PYAUTO_TEST_MODE, PYAUTO_DISABLE_JAX], commented with the double-vacuity rationale, on the searches/MultiStartAdam model.
4. autolens_workspace_test/scripts/latent/latent_nan_robustness.py — reduce post-fit cost to fit the 300s cap. Primary lever is sample count (100 samples drives both the ~136s results update and the ~56s latent compute; the guard only needs enough samples to span several batch_size=3 batches). Re-time after each change; target ≤240s for headroom.
5. Re-verify imaging/model_fit.py and latent_variables_smoke.py pass from a clean output tree — no edits.

Testing

• pytest on the new PyAutoFit database test.
• profile_database.py end-to-end under the smoke profile.
• latent_nan_robustness.py timed under the new override.
• The two no-change scripts re-run from clean output.

Key Files

• PyAutoFit/autofit/database/model/array.py — the HDU setter/getter carrying the bug
• PyAutoFit/autofit/aggregator/summary/aggregate_fits.py — proves the empty-primary shape is library-produced
• autolens_workspace_test/config/build/env_vars.yaml — per-script env overrides
• autolens_workspace_test/scripts/latent/latent_nan_robustness.py — the vacuous guard

Out of scope — file separately

Bypass-mode runs leave an output dir that resolves as "Fit Already Completed" with samples=None, so any repeat run against a warm tree fails. This is the root cause of the census's three false positives and a genuine trap, but it is a PyAutoFit resume-path question well beyond this task.

Original Prompt

Click to expand starting prompt

# Test-mode NoneType failures: scripts that need real samples/fits (env_vars overrides)

Type: bug
Target: workspaces
Repos:
- autolens_workspace_test
- autofit_workspace_test
Difficulty: small
Autonomy: safe
Priority: normal
Status: formalised

From the 2026-07-21 census. These are NOT code bugs — they are the known "script needs real samples /
a real fit, but the smoke profile bypasses the sampler" env-config gap. Each fails with a NoneType
attribute error because the aggregator/samples object is empty under `PYAUTO_TEST_MODE=2`.

- autolens_workspace_test `imaging/model_fit.py` — `AttributeError: 'NoneType' object has no attribute 'parameter_lists'`
- autolens_workspace_test `latent/latent_nan_robustness.py` — `'NoneType' has no attribute 'sample_list'`
- autolens_workspace_test `latent/latent_variables_smoke.py` — `'NoneType' has no attribute 'model'`
- autofit_workspace_test `profiling/aggregator/profile_database.py` — database array HDU is None -> `'NoneType' has no attribute 'dtype'`

Fix: add per-script overrides in each repo's `config/build/env_vars.yaml` unsetting `PYAUTO_TEST_MODE`
(and `PYAUTO_SKIP_FIT_OUTPUT` where the script reads outputs), mirroring the existing
`guides/results/` and `database/scrape/` precedents. Keep them fast — prefer the minimum unset that
produces real samples. Verify each via `run_python.py <project> <scripts/dir>`; confirm no sibling
regressions and that runtimes stay within the per-script cap.

[Jammy2211]

Library PR Created

PR: #1415 (pending-release, OPEN / MERGEABLE, commit 9f887a9b1)

The library half — the only real defect of the four reported — is done.

Fix: HDU.hdu setter records a null payload when hdu.data is None; the getter reconstructs with data=None via a new HDU.has_data property. An empty PrimaryHDU now round-trips losslessly instead of aborting the sqlite scrape.

Verification

• Both new tests confirmed to fail without the fix, with exactly the reported AttributeError at array.py:56 — not vacuous.
• test_autofit/ full suite: 1527 passed, 1 skipped.
• autofit_workspace_test profiling/aggregator/profile_database.py: exit 0 (previously the reported crash).
• autolens_workspace_test database/scrape/general.py re-smoked on the widened path: exit 0, no regression.
• black --check clean.

API impact: additive property + a crash converted into correct behaviour. No signature changed, nothing removed or renamed; grep across all six workspace script trees found no direct callers of the changed symbols. No migration needed.

Heart: YELLOW score 52 at ship time, red_reasons empty — acknowledged by the human for these four pre-existing reasons only (workspace validation 2026-07-20; 58 stale parked scripts; 10 slow scripts; PyAutoCTI open PR age). None caused by this diff.

Workspace half — queued, not abandoned

latent/latent_nan_robustness.py needs the env_vars.yaml override plus a trim under the 300s cap. It is registered in PyAutoMind/planned.md as latent-nan-guard-honest-run, blocked-by slow-skip-timeout-cap-doc, which holds a live worktree on autolens_workspace_test (its PR #194 still OPEN). Measurements already banked for that work: honest run 412s; PYAUTO_TEST_MODE=1 does not help (455s) because Nautilus is not the bottleneck (~136s post-fit results update + ~56s latent compute on 100 samples), so sample count is the lever.

imaging/model_fit.py and latent/latent_variables_smoke.py need no change — both verified passing from a clean output tree.

[Jammy2211]

Shipped

PR: #1415 — MERGED 2026-07-22T10:11:37Z (CI green: docs + unittest 3.12/3.13).

This issue was auto-closed by the PR's Closes line. That is correct for the library half, which was the only real defect of the four reported.

What shipped

autofit/database/model/array.py — data-less HDUs now store a null payload and are reconstructed with data=None, distinguished by a new HDU.has_data property. An empty PrimaryHDU round-trips losslessly instead of aborting the sqlite scrape. Two regression tests in test_autofit/database/test_file_types.py.

What did not need fixing

Three of the four reported scripts were not bugs. PYAUTO_TEST_MODE=2 writes no samples, so a second run against a warm output tree takes the Fit Already Completed: skipping non-linear search path and returns result.samples = None — which produced every one of the reported NoneType tracebacks. imaging/model_fit.py and latent/latent_variables_smoke.py were verified passing untouched from a clean tree and need no change.

Still open — tracked elsewhere

latent/latent_nan_robustness.py passes but vacuously: 4 bypass samples, and PYAUTO_DISABLE_JAX=1 silently flips its deliberate AnalysisImaging(use_jax=True) to False, so the JAX branch the guard exists to catch never runs. That work is registered as latent-nan-guard-honest-run in PyAutoMind/planned.md, blocked by slow-skip-timeout-cap-doc's live worktree on autolens_workspace_test.

It will need a new issue — this one is closed. Measurements are already banked: honest run 412s vs the 300s cap, and PYAUTO_TEST_MODE=1 does not help (455s) because the cost is post-fit processing (~136s results update + ~56s latent compute on 100 samples), not Nautilus — so sample count is the lever.

Completion record: PyAutoMind/complete/2026/07/empty-hdu-scrape-and-latent-guard.md.