test_full_hierachical flaky — hierarchical EP lands at wrong mu_logt

[Jammy2211]

Overview

test_autofit/graphical/hierarchical/test_hierarchical.py::test_full_hierachical fails on clean main (mu_logt → −0.69 vs expected 2.87 ± 2.9) — reproduced 3× single-test on 2026-07-10, yet the identical content passed inside two full-suite runs the same morning. Stash-proven unrelated to the #1350 batch. Data fixture is seeded; runs were serial — suspect timing/thread sensitivity in the Laplace/EP loop or hidden cross-test state. Not a marginal miss (rel=1.0 tolerance): a qualitatively different fixed point.

Prompt: PyAutoMind/bug/autofit/hierarchical_ep_test_flaky.md.

Plan

1. Re-reproduce on clean main; establish flake rate (N single runs + N suite runs).
2. Instrument failing vs passing runs with the feat(graphical): EP diagnostics and monitoring tooling #1349 diagnostics (EPDiagnostics, per-factor evidence/KL via graph_factors.png, check_sigma_collapse).
3. Classify: multiple EP fixed points (damping/update-order sensitivity) vs state leak vs optimiser nondeterminism.
4. Fix at source; never widen the tolerance.

Autonomy: supervised (--auto chain, human directed 2026-07-10).

[Jammy2211]

Triage result: not a flake — deterministic test-order dependence

Probe (6× fresh single-test subprocesses on clean main): 0/6 pass, bit-identical mu_logt = -0.693511011300846 every run. Whole-file run (test_hierarchical.py, 3 tests): all pass. So test_full_hierachical deterministically fails standalone and deterministically passes when the two earlier tests in the same file run first — and full-suite runs pass for the same reason. (My earlier 'passed twice / failed once in suites today' observation resolves consistently: the suite failure was in a run whose collection included new tests earlier in the graphical tree, shifting the pre-test state.)

Leading hypothesis: the global Prior._ids / Variable id counter. Earlier tests consume ids, shifting id-ordered structures — prior_tuples_ordered_by_id and (critically) the EP factor-update order in EPOptimiser. A different update order reaches a different fixed point within the test's max_steps=10 budget. That mechanism produces exactly what we observe: bit-determinism within a context, divergence across contexts, and a qualitatively different mu_logt (−0.69 vs 2.87) rather than a marginal miss.

Next steps (this issue's follow-on work):

1. Confirm: log the factor update order + variable ids in standalone vs whole-file runs (one print or the feat(graphical): EP diagnostics and monitoring tooling #1349 EPDiagnostics per-update snapshots).
2. If confirmed, decide the real fix: (a) make the test's EP setup order-robust (explicit factor ordering / more damping / higher max_steps with an assertion that it converged rather than budget-exhausted), and/or (b) treat update-order sensitivity as an EP robustness finding — EPHistory should be able to report non-convergence at budget exhaustion instead of silently returning the last iterate (relates to research: EP review phase 1 — statistics audit of autofit/graphical #1332 F10's guard philosophy).
3. Never widen the tolerance.

Classification for CI: the test is trustworthy in full-suite order today but will bite anyone debugging it standalone (and any future collection-order change — as it did today). Priority normal, but the fix should land before the next EP-heavy campaign.

[Jammy2211]

Update — hard CI regression, not a flake; bisects to #1351

A /health check green-light sweep today caught this test failing deterministically, and the CI history contradicts the "unrelated to the #1350/#1351 batch" read.

Tests workflow on main (linear merges):

commit	PR	Tests
2e33b175	#1347	✅ success (last green)
1a06bbc1	#1349	✅ success
c5c50cca	#1351	❌ failure (first red)
ffc04d83	#1354	❌ failure

The full-suite CI flipped pass→fail exactly at #1351 (fix(graphical): EP statistics fix batch — F1/F2/F4/F8) and is still red at HEAD ffc04d83 — so PyAutoFit's Tests workflow is currently red on main.

Reproduction is now deterministic. On ffc04d83 the test fails both in the full suite (1 failed / 1474 passed) and in isolation (pytest …::test_full_hierachical, ~22 s; seeded np.random.seed(1)): mu_logt → −0.30 vs 2.87 ± 2.9.

Tension to settle during the fix: the batch commit message stash-proved the failure unrelated to #1350 by reverting each fix in isolation — but those single-test isolation runs are exactly the order/thread-sensitive regime this issue flags, so they're the least authoritative signal. The full-suite CI (green at #1347/#1349 → red at #1351) is the one that matters, and it points at #1351's changes:

• autofit/graphical/laplace/newton.py — 66 lines, F8 dead-quasi-Newton deletion (diag_sr1_update_, diag_sr1_bfgs_update, bfgs1_update)
• autofit/graphical/mean_field.py — F1 log_norm → plates ctor slot fix
• autofit/messages/{abstract,beta,gamma}.py — F2 KL direction

Root-causing starts at newton.py/mean_field.py. Raising priority to high — this blocks the Tests workflow on main.

[Jammy2211]

Root cause: order-dependent EP factor schedule (global id → set iteration). #1351 was an innocent trigger.

Deep research settles the #1351 tension — and corrects my previous comment's "deterministic in isolation" wording.

The isolation failure is pre-existing at every commit

In-tree, test_full_hierachical fails in isolation at BOTH 2e33b175 (#1347, CI-green) and c5c50cca (#1351, CI-red). So the isolation failure the batch commit "stash-proved" is real but is not what CI measures. The regression is in full-suite behaviour:

scope	#1347 (2e33b175)	#1351 (c5c50cca)
test_autofit/graphical/ suite	200 passed (hier ✅ in-suite)	1 failed / 208 passed (hier ❌ in-suite)
isolation (::test_full_hierachical)	❌ FAIL	❌ FAIL

Proof it's order-dependence

At a fixed commit with identical seeded data, advancing the global object-id counter by N before the fit flips the outcome non-monotonically:

N=0 FAIL  N=1 FAIL  N=2 PASS  N=3 FAIL  N=4 PASS  N=5 PASS
N=6 FAIL  N=7 PASS  N=8 PASS  N=9 PASS  N=10 PASS

The fit lands on a good or a sigma-collapsed fixed point (F10 warns std 0 below floor 1e-08 on the bad runs) purely as a function of how many objects were created earlier in the process.

Mechanism

• Variable.__hash__ / AbstractNode.id come from a process-global itertools.count() (autofit/mapper/variable.py:19,30; autofit/graphical/factor_graphs/abstract.py:61,85).
• EPOptimiser derives its per-step factor update schedule from self.factors = factor_graph.factors, a set — iterated at expectation_propagation/optimiser.py:337 (for factor, optimiser in self.factor_optimisers.items(), dict built from that set at :203–206; set(self.factors) at :194).
• Set iteration order of Variable/Factor = id-hash order = global-counter order. So the EP update order — and thus which fixed point a marginal fit converges to — depends on process history.
• fix(graphical): EP statistics fix batch — F1/F2/F4/F8 from #1332 #1351 added test_autofit/graphical/functionality/test_ep_statistics_fixes.py (~8 objects) which runs before hierarchical/ and shifts the id offset from a passing region into a failing one. Nothing in fix(graphical): EP statistics fix batch — F1/F2/F4/F8 from #1332 #1351's math (F1/F2/F4/F8) is implicated.

Fix direction (for review — this touches the core EP optimiser, blast radius = all graphical/EP fits)

1. Deterministic factor schedule — order self.factors by a stable structural key (or preserve insertion order) so a fit is reproducible regardless of process id state. Correct-in-principle; must verify the chosen order yields the good fixed point here.
2. Robust convergence / damping — the bad fixed point is a sigma-collapse; the F10 warning itself suggests delta < 1 damping. Makes convergence order-robust at the numerical level.
3. Test-level determinism — reset the id counters in a fixture. Minimal/low-risk, but masks the real (reproducibility) fragility rather than fixing it.

Recommend (1) [+ possibly (2)] as the real fix, since a model giving different answers across runs depending on process history is a reproducibility bug beyond this test. Seeking a steer on approach before implementing.

[Jammy2211]

Correction — real root cause is ambient np.random, not factor ordering

My previous comment fingered set-iteration factor ordering. Instrumenting the actual fit disproves that: the factor update order is already a deterministic insertion-order tuple (normal_0..4, prior_*, mean_loglike, logt_loglike) on every run. The order-dependence is stochastic, via global np.random:

Chain

1. NormalMessage.sample() draws from global np.random.randn (autofit/messages/normal.py:274,278).
2. LaplaceOptimiser has n_refine = 3 (graphical/laplace/optimiser.py:28); optimise_approx → refine_state(next_state, mean_field.sample, ...) (:131,140) calls mean_field.sample() 3× per factor optimisation, consuming global RNG.
3. So every EP fit's trajectory — and which fixed point a marginal hierarchical fit lands on (good vs sigma-collapse) — depends on the ambient np.random state at fit start.
4. The test seeds only before data generation, never before the fit.
5. fix(graphical): EP statistics fix batch — F1/F2/F4/F8 from #1332 #1351 added test_autofit/graphical/functionality/test_ep_statistics_fixes.py, whose Monte-Carlo KL-direction property tests consume np.random heavily, shifting the ambient state into a failing region before hierarchical/ runs. (fix(graphical): EP statistics fix batch — F1/F2/F4/F8 from #1332 #1351's F1/F2/F4/F8 math remains exonerated; the id-offset correlation in my first probe was a red herring — the carrier is RNG state drift between successive fits, not object ids.)

Proof

Same commit, same seeded data, varying only the ambient RNG state just before the fit:

reseed seed=1 before each fit:   PASS PASS PASS PASS PASS   (deterministic given state)
sweep pre-fit seed 0..7:         PASS PASS PASS PASS PASS PASS PASS FAIL   (seed 7 -> bad fixed point)

The fit result is a pure function of the pre-fit RNG state — seed=7 reproduces the sigma-collapse.

Implication (bigger than this test)

EP/Laplace fits in PyAutoFit are not reproducible across process states — the same model fit twice with different ambient np.random can converge differently. This is a latent reproducibility bug that this test happened to expose.

Corrected fix direction

• A — seed the refinement sampler: give LaplaceOptimiser (or the refine sampling) a controlled RNG so a fit is deterministic regardless of ambient global state. Real reproducibility fix; blast radius = all EP fits (numeric values become independent of ambient state — a strict improvement, though exact numbers may shift from today's ambient-dependent ones).
• B — deterministic refinement: replace random n_refine sampling with deterministic quadrature/fixed evaluation points. Larger change; removes the stochasticity entirely.
• C — test-level seed: np.random.seed(...) before the fit in the test. Minimal, unblocks CI now, but leaves the reproducibility bug for real users.

Recommend A as the real fix (optionally C as an immediate CI unblock while A lands). Seeking a steer before implementing — this supersedes the earlier factor-ordering direction.

[Jammy2211]

Decision: seed the test (it's an inherently stochastic test)

Maintainer steer: this is inherent randomness — the EP fit genuinely samples during Laplace refinement — so seeding is the right, feasible fix rather than re-architecting the optimiser.

Shipping: a one-line np.random.seed(0) immediately before the fit in test_full_hierachical (+ an explanatory comment). Full test_autofit/graphical/ suite: 216 passed, 0 failed.

Why not the optimiser fix (A): I built the controlled-RNG change to LaplaceOptimiser.refine_state and then reverted it — it's insufficient on its own. Proven: with A active and the test seed removed, the fit still fails; and an id-offset sweep with A+C still flips PASS/FAIL. The fit's fixed point depends on both ambient np.random and variable-id ordering (set iteration of model.variables), so seeding RNG alone isn't fully deterministic. Chasing full determinism would mean pinning id-order across the EP layer — out of proportion for a marginal stochastic test.

Known limitation (documented, by choice): the seed makes CI green and the fit reproducible in the current suite, but the test remains inherently stochastic; a future test addition that shifts variable ids could in principle re-perturb it (the same mechanism by which #1351 surfaced this). Accepted per "keep it feasible."

The broader observation stands as a note, not a fix here: EP/Laplace fits draw from global np.random, so results aren't reproducible across arbitrary process states — worth a dedicated controlled-RNG design if reproducibility becomes a requirement.

[Jammy2211]

Shipped — PR #1355 (pending-release)

Test-seed fix opened: #1355. Gate green:

• test_autofit/graphical/ — 216 passed, 0 failed
• Full test_autofit/ — 1475 passed, 14 skipped, 0 failed (was 1474 pass / 1 fail at HEAD ffc04d83)

No workspace impact (test-only, no API change). Awaiting merge — the red Tests workflow on main clears once this lands.

[Jammy2211]

Merged via #1355 (main de6534dfb). Full PyAutoFit suite 1475 pass / 0 fail; the red Tests workflow on main clears with this. Closing.