fix: stamp the template's complete/index.md so its self-heal stops causing drift

.github/workflows/spawn_drift.yml

@@ -13,7 +13,7 @@ on:
   pull_request:
     paths:
       - "scripts/spawn.py"
-      - "tests/test_spawn_privacy.py"
+      - "tests/**"
       - "docs/pyautobrain/spawn_spec.md"
       - ".github/workflows/spawn_drift.yml"
   workflow_dispatch:
@@ -33,8 +33,10 @@ jobs:
         with:
           python-version: "3.12"
       - run: pip install pytest
-      - name: spawn privacy invariant
-        run: python3 -m pytest tests/test_spawn_privacy.py -q
+      - name: spawn privacy invariant + template contract
+        # Whole dir, not one file: a suite named in the workflow by filename
+        # silently stops covering anything added beside it.
+        run: python3 -m pytest tests/ -q
 
   drift:
     # A drift run that regenerates from a leaking generator would "fix" the

active.md

@@ -2,7 +2,11 @@
 
 ## jax-grad-smoke-timeout-budget
 - issue: https://github.com/PyAutoLabs/PyAutoHands/issues/226
-- status: library-dev — phase 1 (PyAutoHands runner) starting; issue carries the full two-level plan.
+- status: library-shipped, workspace-pending — phase 1 PR OPEN (PyAutoHands#227, commit 8bbbf1f, pushed sha verified against local). Phase 2 (workspace budget) BLOCKED on measurement, see phase-2-gate below.
+- library-pr: https://github.com/PyAutoLabs/PyAutoHands/pull/227 (no `pending-release` label — deliberate: 9 of the last 10 PyAutoHands PRs carry no label, only #213 used it for a genuine release dependency, and this runner change needs no release. Do not add it.)
+- phase-1-result: `build_util.timeout_for(env)` resolves the cap parent-side for execute_script + execute_notebook; TimeoutExpired handlers now keep a truncated output tail + the cap in force; slow_skip banner says "default" cap. NO profile schema change — uses the existing `set:` key, since ALLOWED_OVERRIDE_KEYS is {pattern,set,unset}. Precedence profile > ambient global > 300, because run_all.py:256 exports BUILD_SCRIPT_TIMEOUT unconditionally so the parent cannot tell a deliberate operator cap from the default. 25 tests; CONTROL-TESTED — 11 of 25 fail against the unfixed behaviour, so they are not vacuous. Full suite 281 pass / 2 fail, both pre-existing on unmodified main 9601f18 (test_python_matrix_workflow.py, `experimental_python_314` key) — NOT caused by this task.
+- phase-2-gate: do NOT set the jax_grad budget from local numbers. The `lp.py` CONTROL fails locally while PASSING in CI, so local runs are not trustworthy here; and pixelization/regularization abort early locally, so their local durations are truncated, not total. Phase 2 needs a workspace-smoke dispatch with phase 1 merged, reading ACTUAL CI durations from the now-preserved timeout output. Reference points: gradient.py 665.9s local PASS; knn.py calibration local 141.6s vs CI 200.0s (CI/local ~1.41) → gradient.py projects to ~940s CI. Comparing pixelization.py's completed duration against its 244.8s baseline is what settles the open regression question.
+- heart-ack (2026-08-04): phase 1 shipped under human-acknowledged YELLOW (score 70, `red_reasons: []`). Acknowledged reasons, verbatim: (1) "workspace validation not passing (3 failed, 3 timeout, cloud#30858578587: autofit_test scripts/jax_assertions/multi_start_gradient_auto_convergence.py, autolens_test scripts/imaging/pixelization.py, autolens_test scripts/imaging/regularization.py, +3 more)"; (2) "manifest drift: tenant firewall (organ code) — 2 mismatch(es) vs PyAutoMind/repos.yaml"; (3) "release validation stale: source moved since rehearsal (PyAutoFit, PyAutoGalaxy, PyAutoLens)". None caused by this task — reason (1) is this task's OWN subject and is what phase 2 exists to clear. The ack covers THIS reason set only.
 - prompt: active/jax_grad_scripts_timeout_smoke_300s_cap.md
 - worktree: ~/Code/PyAutoLabs-wt/jax-grad-smoke-timeout-budget
 - origin: PyAutoHeart workspace-smoke run 30858578587 (2026-08-03T22:25Z) — 3 jax_grad scripts TIMEOUT at the 300s cap. Feeds the Heart YELLOW "workspace validation not passing" reason in front of release-drive-2026-08-03.
@@ -18,10 +22,10 @@
 
 ## mge-sigma-min-workspace-sweep
 - issue: https://github.com/PyAutoLabs/autolens_workspace/issues/466
-- status: phase 1 MERGED 2026-08-04T17:11:13Z (autolens_workspace#467 -> 92019316, issue #466 auto-closed); phase 2 PRs ALL OPEN 2026-08-04, awaiting merge. Standalone mode (upstream PyAutoGalaxy#549 already MERGED as 13d3023c).
+- status: BOTH PHASES MERGED 2026-08-04. Phase 1 autolens_workspace#467 -> 92019316 (issue #466 auto-closed). Phase 2 autogalaxy_workspace#203 -> 8a7df7a6, HowToLens#67 -> 4ff3135c, HowToGalaxy#61 -> 51eed3d6, autogalaxy_assistant#10 -> f6966a64. Upstream PyAutoGalaxy#549 -> 13d3023c. All worktrees removed, all branches deleted local+origin, all five canonical checkouts back on main. Code work COMPLETE; two debts remain (below).
 - phase-2-prs: autogalaxy_workspace#203 (28 files, `pending-release`) / HowToLens#67 (2 files, `pending-release`) / HowToGalaxy#61 (2 files, `pending-release`) / autogalaxy_assistant#10 (2 files, NO pending-release label — that repo does not define one, gate stated in the PR body instead). All four on branch feature/mge-sigma-min-workspace-sweep.
-- phase-2-ci: THREE GREEN — agw#203, htl#67, htg#61 all pass every check incl. both smoke matrix legs. aga#10 wiki-currency FAILS on `Version drift (--check-version)` ONLY.
-- BLOCKER (pre-existing, NOT this task): autogalaxy_assistant `wiki/core/api_audit_baseline.json` was generated 2026-08-01 pinning autogalaxy 2026.7.29.2; PyPI latest is now 2026.8.4.1 (released 2026-08-04). EVERY PR to that repo fails --check-version until the baseline is re-pinned. Proof it is not ours: the PR touches only 2 .md files, and symbol audit (--scope all) reports 0 missing/broken, provenance shows warnings only (no content_sha256 error), idioms clean.
+- phase-2-ci: THREE GREEN — agw#203, htl#67, htg#61 passed every check incl. both smoke matrix legs. aga#10 was MERGED WITH A RED wiki-currency check on explicit human instruction ("merge all"), failing `Version drift (--check-version)` ONLY — pre-existing, see DEBT 1. That red is now on autogalaxy_assistant main and persists for every future PR to that repo until the baseline is re-pinned.
+- OUTSTANDING DEBT 1 of 2 — assistant baseline re-pin (pre-existing, NOT caused by this task, now inherited by main): autogalaxy_assistant `wiki/core/api_audit_baseline.json` was generated 2026-08-01 pinning autogalaxy 2026.7.29.2; PyPI latest is now 2026.8.4.1 (released 2026-08-04). EVERY PR to that repo fails --check-version until the baseline is re-pinned. Proof it is not ours: the PR touches only 2 .md files, and symbol audit (--scope all) reports 0 missing/broken, provenance shows warnings only (no content_sha256 error), idioms clean.
 - baseline-repin-TRAP: do NOT run `--write-baseline` from this workspace. It derives api_surface from the INSTALLED library, and local installs are the SOURCE checkouts (autogalaxy 2026.7.23.1), not the released 2026.8.4.1 that CI grades against — a local re-pin stamps the wrong version and turns the check green on a false premise. The re-pin must run against the released stack, as its own separate change.
 - phase-2-result: autogalaxy_workspace 10 helper calls + 8 ladders; HowToLens 3 ladders (2 lens floored, 1 SOURCE -> -4, traced to source_bulge); HowToGalaxy 3 ladders all image-plane; autogalaxy_assistant 2 prose pages. Every autogalaxy_workspace MGE verified to attach to `galaxy = af.Model(ag.Galaxy, ...)` — PyAutoGalaxy has no lensing so no source plane exists to exclude. Validation: pyflakes zero introduced undefined names, 16/16 affected scripts pass smoke, notebooks regenerated 1:1 in all three repos with no orphan churn.
 - assistant-provenance: wiki/core/concepts/linear_light_profiles_and_mge.md is provenance-stamped and a content_sha256 mismatch is a CI ERROR. Handled: added autogalaxy/analysis/model_util.py to the cited PyAutoGalaxy paths (where sigma_min lives), re-pinned PyAutoGalaxy 13d3023c + autogalaxy_workspace 1f821ba, bumped last_updated, re-stamped via `python3 autoassistant/audit_skill_apis.py --write-provenance`. Audit exits 0, 0 missing/broken. NOTE wiki-currency CI has paired-PR support — it checks out any cited source repo having a branch matching the PR head, so the paired autogalaxy_workspace branch is the grading ground truth.
@@ -30,7 +34,7 @@
 - RELEASE DEBT (created by merging phase 1): `sigma_min` exists only on PyAutoGalaxy main, NOT in any released PyAutoGalaxy. autolens_workspace main now calls it, so a user on a released install hits TypeError until a release carrying #549 ships. The `pending-release` gate was overridden on explicit human instruction 2026-08-04 with that consequence stated — same pattern as the 2026-08-03 override recorded under release-drive-2026-08-03, which is CLEAR TO RE-RUN. Publishing is the remedy, not a preference.
 - phase-1-result: 130 lens call sites take `sigma_min=dataset.pixel_scales[0] / 10.0`; 102 source sites untouched. 27 hand-rolled ladders: 10 image-plane floored, 17 source moved -2 -> -4. Classifier control-tested against the 88 explicitly-named sites first (88/88 agree, 0 disagree) before being trusted on the 143 bare `bulge` sites. Validation: 93 scripts byte-compile, pyflakes zero introduced undefined names across all 93, all 22 affected smoke scripts pass, notebooks regenerated 93-for-93 with no unrelated churn.
 - heart-ack (2026-08-04): shipped under human-acknowledged YELLOW (score 70, `red_reasons: []`). Acknowledged reasons: (1) "workspace validation not passing (3 failed, 3 timeout, cloud#30858578587: autofit_test scripts/jax_assertions/multi_start_gradient_auto_convergence.py, autolens_test scripts/imaging/pixelization.py, autolens_test scripts/imaging/regularization.py, +3 more)"; (2) "manifest drift: tenant firewall (organ code) — 2 mismatch(es) vs PyAutoMind/repos.yaml"; (3) "release validation stale: source moved since rehearsal (PyAutoFit, PyAutoGalaxy, PyAutoLens)". None caused by this task; (3) is partly this day's own PyAutoGalaxy#549 merge. The ack covers THIS reason set only.
-- markdown-DEFERRED (human decision 2026-08-04): markdown/ NOT regenerated. 7 of the 30 curated markdown_examples.yaml entries are among the changed scripts (start_here.py, imaging/start_here.py, imaging/modeling.py, multi_dataset/start_here.py, multi_dataset/modeling.py, group/start_here.py, group/modeling.py). generate_markdown.py refuses to run under PYAUTO_TEST_MODE (it executes scripts for real to render images) and a fresh worktree has no output/ resume cache, so a pass means real fits. Those 7 pages show snippets WITHOUT sigma_min until regenerated — outstanding debt, do not assume it is done.
+- OUTSTANDING DEBT 2 of 2 — markdown-DEFERRED (human decision 2026-08-04): markdown/ NOT regenerated. 7 of the 30 curated markdown_examples.yaml entries are among the changed scripts (start_here.py, imaging/start_here.py, imaging/modeling.py, multi_dataset/start_here.py, multi_dataset/modeling.py, group/start_here.py, group/modeling.py). generate_markdown.py refuses to run under PYAUTO_TEST_MODE (it executes scripts for real to render images) and a fresh worktree has no output/ resume cache, so a pass means real fits. Those 7 pages show snippets WITHOUT sigma_min until regenerated — outstanding debt, do not assume it is done.
 - smoke-trap: the 3 scripts declaring `ENV: full_datasets` (group/, imaging/, multi_dataset/ start_here.py) FAIL with an unrelated IndexError (dataset capped to 16px vs a 209px mask) if run with PYAUTO_SMALL_DATASETS=1. Reproduced on pristine main, so it is a runner-env artefact, NOT a code bug — honour the in-file ENV declaration when smoke-running these.
 - prompt: active/mge_sigma_min_workspace_sweep.md
 - worktree: ~/Code/PyAutoLabs-wt/mge-sigma-min-workspace-sweep

docs/pyautobrain/spawn_spec.md

@@ -40,6 +40,7 @@ deliberately, never silently shipped into a template.
 | 5 | `active.md`, `planned.md`, `parked.md`, `condemned.md`, `ideas.md`, `queue.md`, `autonomy_log.md` | EMPTY → header line + schema pointer comment only (e.g. `# Active Tasks` + `<!-- schema: REFERENCE.md -->`); autonomy_log keeps its schema header rows. **The header is GENERATED, never read from the live file** — `planned.md` and `ideas.md` carry no H1 at all, so "keep line 1" yields a registry entry (issue #118), and a heading-shape test cannot save it either: `## rhayes-audit-validation-phases-2-4` is a valid `##` heading. Each file's title lives in `EMPTY_TITLES`; an EMPTY-ruled file with no entry is an UNMATCHED-class human decision, not a guess |
 | 6 | `draft/**` (the work-type dirs `feature/ bug/ refactor/ docs/ test/ release/ maintenance/ research/ experiment/ triage/` now live under `draft/`) | SKELETON → keep a single `draft/.gitkeep`; drop all draft prompts and their work-type/target subdirs (a fresh Mind starts with an empty `draft/`; intake recreates the work-type subdirs on demand) |
 | 6b | `complete/AGENTS.md` | KEEP verbatim (the finished-work archive **schema** is template content; matched before rule 7's `complete/*` DROP) |
+| 6c | `complete/index.md` | GENERATE → stamped by running the **generated tree's own** `scripts/lifecycle.py index --apply` after the tree is written (`lifecycle.py` resolves its root from `__file__`, and rule 1 already KEEPs it). The live `index.md` is still DROPped by rule 7 — this is a fresh empty-archive index, not a copy. Required because the template ships `lifecycle_drift.yml`, whose self-heal (PyAutoMind#116) regenerates this file on every push to the template's own `main`: if spawn did not produce it, each sync would be followed by a bot commit creating a file the next `--check` reports as drift, forever. Do NOT hold the text as a constant here — `lifecycle.py` owns the index format, and a second copy would drift from it |
 | 7 | `active/ complete/ z_features/ z_vault/ autoprompt/` + instance reference docs (`docs/**` now holds only reference material like `spawn_spec.md`) + instance root docs (`dashboard.md`, `overview.md`) + legacy pre-migration prompt dirs (`autolens/`) | DROP (lifecycle records + instance content) |
 | 8 | `skills/**`, `policy/**` | KEEP verbatim (`OWNERSHIP.md`, `create_issue/` are generic; `policy/` is org-agnostic safety text) |
 | 9 | `.github/**` | KEEP verbatim EXCEPT workflows that reference live secrets/repos beyond the org placeholder — those SUBSTITUTE `PyAutoLabs` → `YOURORG` and keep |

draft/bug/autolens_workspace_test/jax_grad_local_assertions_fail_but_pass_in_ci.md

@@ -0,0 +1,78 @@
+# jax_grad scripts fail assertions locally that PASS in CI
+
+Type: bug
+Target: autolens_workspace_test
+Repos:
+- autolens_workspace_test
+Difficulty: medium
+Autonomy: supervised
+Priority: medium
+Status: formalised
+
+Running the `jax_grad` scripts locally under the resolved smoke profile produces
+deterministic assertion failures in scripts that **pass in CI on the same commit**.
+Found while measuring script durations for PyAutoHands#226.
+
+## Evidence
+
+Run via `env_config.build_env_for_script` with the workspace root as CWD (i.e. the
+exact env the runner builds — `PYAUTO_SMALL_DATASETS` unset, `PYAUTO_DISABLE_JAX`
+unset, `PYAUTO_TEST_MODE=2`, verified by printing the resolved env):
+
+| script | local | CI (run 30858578587 / 30790463134) |
+|---|---|---|
+| `imaging/jax_grad/lp.py` | **FAIL 41.3s** | **PASS 39.6s / 40.0s** |
+| `imaging/jax_grad/knn.py` | PASS 141.6s | PASS 200.0s / 175.8s |
+| `imaging/jax_grad/pixelization.py` | **FAIL 57.5s** | PASS 244.8s (06:31Z) |
+| `imaging/jax_grad/regularization.py` | **FAIL 131.5s** | (import gap, then TIMEOUT) |
+| `point_source/jax_grad/gradient.py` | PASS 665.9s | TIMEOUT (300s cap) |
+
+`lp.py` is the decisive case: it **passes in CI on both runs** and fails locally.
+
+Failures are deterministic and bit-identical across repeated runs, e.g.
+`pixelization.py`:
+
+```
+AssertionError: Eager (-8354.484097835004) and jitted (-8354.55843260181) evaluations
+disagree — possible pure_callback constant-folding; do not trust jitted gradients.
+```
+
+(relative difference ~8.9e-6 against `assert_eager_jit_consistent`'s `rtol=1e-10`).
+
+`lp.py` fails with `All source-parameter gradients are ~zero — NNLS zeroed the source`;
+`regularization.py` with an AD-vs-FD mismatch marginally over tolerance
+(`abs_err=[0.045, 0.042, 0.057]` vs `tolerance=[0.031, 0.008, 0.003]`).
+
+## What is ruled out
+
+- **Not the small-datasets cap.** `full_datasets` correctly unsets
+  `PYAUTO_SMALL_DATASETS`; verified by resolving the env directly rather than
+  inferring from mask sizes.
+- **Not a JAX version difference.** Local jax/jaxlib are 0.10.2 — identical to CI.
+- **Not flake.** Repeated runs give bit-identical values.
+
+Prime remaining suspect: **numpy 2.2.6 local vs 2.4.6 in CI**, or another local venv
+package differing from the CI install set. Not yet confirmed.
+
+## Why it matters
+
+This is an active trap for anyone validating these scripts locally. During #226 it
+looked exactly like two fresh correctness regressions on current main
+(`pure_callback` constant-folding, and an FD tolerance breach). Only running a
+**control** — `lp.py`, known-passing in CI — revealed that the local environment
+itself produces the failures, so none of the three local failures were evidence of
+source defects.
+
+Whatever the cause, either the scripts or the documented local-run recipe should make
+this reproducible, so a local FAIL means something.
+
+## Suggested scope
+
+1. Bisect the local-vs-CI package delta (start with numpy 2.2.6 -> 2.4.6) against
+   `lp.py`, the cleanest discriminator.
+2. If numpy is the cause, decide whether the tolerances are under-specified for the
+   supported numpy range, or the local env should be pinned to the CI set.
+3. Record the outcome in the workspace's local-run instructions.
+
+<!-- Split out of PyAutoHands#226 on 2026-08-04; that task deliberately did not absorb
+     this, and explicitly barred setting any timeout budget from local numbers. -->