From 1ec77fd01d88648a84a6e7fc44888ecd5152929c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 02:19:28 +0000 Subject: [PATCH] Shelve superseded JAX-native posterior sampler wave The wave (parked.md :: blackjax-smc-gradient-kernel, wsdev#113) is superseded by the 2026-08-17 inference-methods programme (active/inference_programme_ledger.md, autolens_profiling#134), which now owns sampler direction. Stage (a) shipped its findings; stages (b)-(e) were never issued and are not carried forward as scoped. - move active/jax_native_posterior_sampler_wave.md -> complete/archive/shelved/, with a SUPERSEDED header that preserves the full parked context (branch tip, findings doc, RAL job, resume threads) that the deleted parked.md entry carried - drop the parked.md entry - ideas.md: repoint the "warm-started sampler wave on meshes" idea at the shelved shortlist rather than a parked task that no longer exists - draft/maintenance/.../rectangular_experiments_gut_stash.md: the blackjax-smc-gradient-kernel claim it waited on is released, so the blocker is discharged - regenerate dashboard.md (in flight 10 -> 9, parked 4 -> 3) lifecycle.py check: OK Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016X8zrRQuau8VoGwx3i45tf --- active/jax_native_posterior_sampler_wave.md | 14 ------- .../jax_native_posterior_sampler_wave.md | 40 +++++++++++++++++++ dashboard.md | 8 ++-- .../rectangular_experiments_gut_stash.md | 5 ++- ideas.md | 2 +- parked.md | 10 ----- 6 files changed, 47 insertions(+), 32 deletions(-) delete mode 100644 active/jax_native_posterior_sampler_wave.md create mode 100644 complete/archive/shelved/jax_native_posterior_sampler_wave.md diff --git a/active/jax_native_posterior_sampler_wave.md b/active/jax_native_posterior_sampler_wave.md deleted file mode 100644 index 83f85897..00000000 --- a/active/jax_native_posterior_sampler_wave.md +++ /dev/null @@ -1,14 +0,0 @@ -# JAX-native posterior sampler wave — ranked shortlist from the 2026-07-16 deep-research session - -Type: research -Target: autolens_workspace_developer -Repos: -- autolens_workspace_developer -Difficulty: large -Autonomy: supervised -Priority: normal -Status: formalised - -Research: JAX-native posterior sampler wave — implement and benchmark the ranked sampler shortlist from the 2026-07-16 deep-research session on the standard searches_minimal problem in @autolens_workspace_developer, via the sampler_pipeline ingest-prototype-profile-promote flow. Ranked stages: (a) blackjax adaptive tempered SMC upgraded to a gradient inner kernel (MALA/HMC + inner_kernel_tuning; extends the existing blackjax_smc.py RWM smoke script; gives logZ from tempering increments), (b) ChEES-HMC many-chain adaptation (cheapest first-class addition; vmap-friendly fixed-length trajectories vs NUTS variable-length trees), (c) MCLMC + adjusted MCLMC warm-started from the multi-start Adam basin, paired with harmonic (learned harmonic mean) for evidence, (d) flowMC (normalizing-flow global jumps; multimodality insurance), (e) jaxns cameo (independent GPU-native nested-sampling cross-check vs Nautilus; inversion-heavy vmap OOM caution applies, NSS history). Constraints from prior campaigns: reverse-mode-only gradients, NaN-gradient degenerate points need masking, vmap fan-out needs lax.map batch_size tiling (Fit#1374 lever), RAL A100 float32. Deliverable per stage: findings doc + benchmark vs Nautilus baseline. Issue stages one at a time as predecessors ship — do not bulk-issue. - - diff --git a/complete/archive/shelved/jax_native_posterior_sampler_wave.md b/complete/archive/shelved/jax_native_posterior_sampler_wave.md new file mode 100644 index 00000000..92dc0139 --- /dev/null +++ b/complete/archive/shelved/jax_native_posterior_sampler_wave.md @@ -0,0 +1,40 @@ +# JAX-native posterior sampler wave — ranked shortlist from the 2026-07-16 deep-research session — SUPERSEDED 2026-08-18 + +> **SUPERSEDED / SHELVED 2026-08-18.** Removed from the registry (was +> `parked.md :: blackjax-smc-gradient-kernel`, issue +> [autolens_workspace_developer#113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113)) +> — the wave as scoped here is superseded by the 2026-08-17 human-approved +> inference programme (`active/inference_programme_ledger.md`, autolens_profiling#134), +> which owns sampler direction going forward. Stage (a) shipped its findings; +> stages (b) ChEES-HMC, (c) MCLMC+harmonic, (d) flowMC, (e) jaxns were never +> issued and are not carried forward as-is. +> +> **Preserved parked context (from the deleted `parked.md` entry):** stage (a) +> POSITIVE — warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, +> `einstein_radius` 1.5998 vs truth 1.6, max logL ~31781 vs Prodigy MLE +> 31787.93). Parked 2026-07-24 to clear the deck for the autolens_profiling +> refactor. All work PRESERVED on pushed branch +> `feature/blackjax-smc-gradient-kernel` (origin autolens_workspace_developer, +> tip `6867762`); local worktree removed. Full write-up: +> `searches_minimal/smc_gradient_findings.md` on that branch. Science memory: +> `project_gradient_smc_warm_start_sampler_wave`. Gradient path certified +> OK_HMC_VIABLE (`probe_grad.py`); baseline `nss_grad` row = logZ -31.47. Open +> resume threads at park time: RAL job 331058 +> (`/mnt/ral/jnightin/smc_grad_logs/smc_warm_ok-331058.out`, 3 warm arms @128p); +> logZ (~31690-31798) vs Nautilus comparison row; A100 rep-timing; +> `warm_start.json` is a regenerable cache (re-`--refresh` if it predates the +> `cov` field). MGE only; pix deferred. + + +Type: research +Target: autolens_workspace_developer +Repos: +- autolens_workspace_developer +Difficulty: large +Autonomy: supervised +Priority: normal +Status: formalised + +Research: JAX-native posterior sampler wave — implement and benchmark the ranked sampler shortlist from the 2026-07-16 deep-research session on the standard searches_minimal problem in @autolens_workspace_developer, via the sampler_pipeline ingest-prototype-profile-promote flow. Ranked stages: (a) blackjax adaptive tempered SMC upgraded to a gradient inner kernel (MALA/HMC + inner_kernel_tuning; extends the existing blackjax_smc.py RWM smoke script; gives logZ from tempering increments), (b) ChEES-HMC many-chain adaptation (cheapest first-class addition; vmap-friendly fixed-length trajectories vs NUTS variable-length trees), (c) MCLMC + adjusted MCLMC warm-started from the multi-start Adam basin, paired with harmonic (learned harmonic mean) for evidence, (d) flowMC (normalizing-flow global jumps; multimodality insurance), (e) jaxns cameo (independent GPU-native nested-sampling cross-check vs Nautilus; inversion-heavy vmap OOM caution applies, NSS history). Constraints from prior campaigns: reverse-mode-only gradients, NaN-gradient degenerate points need masking, vmap fan-out needs lax.map batch_size tiling (Fit#1374 lever), RAL A100 float32. Deliverable per stage: findings doc + benchmark vs Nautilus baseline. Issue stages one at a time as predecessors ship — do not bulk-issue. + + diff --git a/dashboard.md b/dashboard.md index 3d9652c4..e99cd35f 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,8 +8,8 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 10 | -| [Parked](#parked) (`parked.md`) | 4 | +| [In flight](#in-flight) (`active/`) | 9 | +| [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 7 | | [Backlog](#backlog) (`draft/`) | 139 | @@ -49,7 +49,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A Issued — each has an open GitHub issue and usually a branch. The full record for each is in [`active.md`](active.md). - [Commit the inference-methods programme plan + knowledge ledger into autolens_profiling](active/inference_programme_ledger.md) — [issue #134](https://github.com/PyAutoLabs/autolens_profiling/issues/134) — pr-open (https://github.com/PyAutoLabs/autolens_profiling/pull/135) -- [JAX-native posterior sampler wave — ranked shortlist from the 2026-07-16](active/jax_native_posterior_sampler_wave.md) — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius… - [Remove standalone matplotlib-inline comments](active/matplotlib_inline_standalones.md) - [Find what kills MGE multi-start lanes — it is not](active/mge_lane_death.md) — [issue #128](https://github.com/PyAutoLabs/autolens_profiling/issues/128) — CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed… - [PositionsLH penalty accumulation returns 2x the LAST penalty, discarding the](active/positions_lh_penalty_accumulation_bug.md) — [issue #699](https://github.com/PyAutoLabs/PyAutoLens/issues/699) — pr-open (https://github.com/PyAutoLabs/PyAutoLens/pull/700) @@ -64,9 +63,8 @@ Issued — each has an open GitHub issue and usually a branch. The full record f Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md).
-4 task(s) +3 task(s) -- **blackjax-smc-gradient-kernel** — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius… - **group4-mge-search-benchmark** — [issue #82](https://github.com/PyAutoLabs/autolens_profiling/issues/82) - **matplotlib-inline-standalones** - **pyautoreduce-slacs1430-acs-comparison** diff --git a/draft/maintenance/autolens_workspace_developer/rectangular_experiments_gut_stash.md b/draft/maintenance/autolens_workspace_developer/rectangular_experiments_gut_stash.md index 2ef6c775..6dea4e2c 100644 --- a/draft/maintenance/autolens_workspace_developer/rectangular_experiments_gut_stash.md +++ b/draft/maintenance/autolens_workspace_developer/rectangular_experiments_gut_stash.md @@ -43,8 +43,9 @@ clean. ## Constraints -- Wait for the `blackjax-smc-gradient-kernel` claim to release; verify the repo - is clean before starting (it had 23 dirty files at wrap-up). +- The `blackjax-smc-gradient-kernel` claim is RELEASED (worktree removed 2026-07-24; + the parked task was shelved as superseded 2026-08-18), so this is no longer + blocked — but verify the repo is clean before starting (23 dirty files at wrap-up). - Developer repo (doc-light, not CI-gated like the workspaces); still verify renamed scripts import cleanly against merged main. - Copy any gitignored products out before any worktree teardown. diff --git a/ideas.md b/ideas.md index bf1babc0..5eab8a0a 100644 --- a/ideas.md +++ b/ideas.md @@ -29,7 +29,7 @@ - [formalised -> feature/pyautogut/repo_skeleton.md] [from: research pyautogut-organ-decision · research/pyautobrain/pyautogut_organ_decision.md] Stand up the PyAutoGut repo skeleton — the new peer organ repo owning condemned self-material: the attic remote, the `refs/archive/condemned/` archive-ref convention (stays out of `git branch -a`), and the `condemned.md` Mind manifest schema (type/locator/confidence/reason/merged?/condemned/sweep-after/breaks-if-wrong/archive-ref+SHA). Work-type feature/pyautogut; also adds PyAutoGut to repos.yaml + ORGANISM.md as a peer organ. - [formalised -> feature/pyautobrain/hygiene_pyautogut_drive_seam.md] [from: research pyautogut-organ-decision · research/pyautobrain/pyautogut_organ_decision.md] Add the hygiene→PyAutoGut drive seam in the Brain — a PyAutoGut-aware hygiene `tidy` mode that files condemned candidates into `condemned.md` async (no synchronous per-item gate, replacing the mid-sweep repo_cleanup interrogation), plus a batch `sweep` mode that runs the existing repo_cleanup safety gates against the manifest at a user-chosen time. Mirrors the Heart↔vitals template: the organ holds+voids, the conductor drives. Work-type feature/pyautobrain. - [formalised -> research/autofit/multi_band_factorgraph_compile_deeper_dig.md] [from: research multiband-compile · Experiment A · jax_compile/README.md multi-band section + project_multiband_factorgraph_compile_cliff memory] Multi-band FactorGraphModel value_and_grad compile — deeper dig + productize the fix. Experiment A (autolens_profiling branch research/multiband-compile-ab) proved the heterogeneous-per-band-shape cold compile is a 5.9× superlinear XLA-fusion cliff (homogeneous 4-band == single-band 120s; two distinct shapes = 704s CPU), that trace/eval are unchanged, and that the persistent cache rescues both arms (warm 2–7s). Follow-up, ONE task, profiling/research: (1) sub-investigation B — does a per-factor jit boundary inside PyAutoFit FactorGraphModel.log_likelihood_function bound cold to N×single-band + a linear combine, removing the superlinearity WITHOUT padding (the potential source lever, touches PyAutoFit); (2) sub-investigation C — add A100 multi-band vag rows (datacube_img / datacube_img_hetero) to the jax_compile census, quantify the far-lower absolute A100 cliff, and ship the final N-band verdict; (3) productize the immediate user workaround — pad short-wavelength bands to a common grid so all factors share one fused kernel — as a documented recipe and/or helper. Does NOT re-open the #71 single-band settings verdict. Repos: PyAutoFit, autolens_profiling. Difficulty large, supervised. -- [from: research wsdev#117 pix-prodigy-cpu · pix_prodigy_findings.md] Warm-started sampler wave on meshes — revive the parked jax_native_posterior_sampler_wave with the MultiStartProdigy endpoint population (16 basin-labelled points) as initializer; mesh↔kernel mapping: ChEES-HMC/MCLMC on kernel-CDF+knn, tempered SMC (logZ) on delaunay. +- [from: research wsdev#117 pix-prodigy-cpu · pix_prodigy_findings.md] Warm-started sampler wave on meshes — revive the shelved jax_native_posterior_sampler_wave shortlist (complete/archive/shelved/, superseded 2026-08-18) with the MultiStartProdigy endpoint population (16 basin-labelled points) as initializer; mesh↔kernel mapping: ChEES-HMC/MCLMC on kernel-CDF+knn, tempered SMC (logZ) on delaunay. - [from: research wsdev#117 · resurrection diagnostics] Validate resurrection-rate → HMC-divergence-rate prediction per mesh — cheap; would give a principled mesh/kernel selection rule for Hamiltonian sampling on pixelized sources. - [from: research wsdev#117 · thin-baseline caveat] Heavier-settings Nautilus reference runs (n_live >= 500) on the knn/delaunay pix objectives — the n_live=100 baselines missed the truth basin; needed to claim "Prodigy matches a CONVERGED sampler" and to quantify the prior-volume search the gradient stage saves. - [from: research wsdev#117 · matern arm] Matérn kernel regularization as a production option — does its evidence ranking track AdaptSplit across real datasets? If yes the gradient-safe scheme can be the SLaM default, not just the search-time choice. diff --git a/parked.md b/parked.md index dc9a9f9a..73a25be7 100644 --- a/parked.md +++ b/parked.md @@ -6,16 +6,6 @@ unindexed worktrees or stashes. Move an entry back to `active.md` (or to `planned.md` if re-scoping is needed) when work resumes; on shipping, write the dated `complete///.md` record instead. -## blackjax-smc-gradient-kernel -- issue: https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113 -- status: PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius 1.5998 vs truth 1.6, max logL ~31781 vs Prodigy MLE 31787.93). Parked to clear the deck for the autolens_profiling refactor (scripts live in autolens_workspace_developer/searches_minimal/, which the refactor churns). All work PRESERVED on pushed branch feature/blackjax-smc-gradient-kernel (origin autolens_workspace_developer, tip 6867762); local worktree removed. Full write-up: searches_minimal/smc_gradient_findings.md on that branch. Science memory: project_gradient_smc_warm_start_sampler_wave. RESUME AFTER REFACTOR: (1) check RAL job 331058 (/mnt/ral/jnightin/smc_grad_logs/smc_warm_ok-331058.out) — 3 warm arms @128p (MALA auto-step, MALA+--tune, HMC): does --tune HOLD acceptance ~0.57 as lambda->1 (fixed step decays 0.80->0.17)? does HMC beat MALA? (2) compare logZ (~31690-31798) vs Nautilus, write comparison.txt row; (3) A100 rep-timing (GPUs full at park); (4) stage (b) ChEES-HMC reusing _warm_start. warm_start.json is a regenerable cache — re-`--refresh` if it predates the cov field (else silent diagonal fallback). MGE only; pix deferred. -- worktree: none (pushed to origin; local worktree removed 2026-07-24) -- autonomy: supervised -- prompt: active/jax_native_posterior_sampler_wave.md -- note: WAVE TRACKER — this parks the whole JAX-native posterior sampler wave (stages b ChEES-HMC, c MCLMC+harmonic, d flowMC, e jaxns still remain). Do NOT move the prompt to complete/ — stage (a) shipped its findings but the wave continues as a separate run after the refactor. Gradient path certified OK_HMC_VIABLE (probe_grad.py); baseline nss_grad row = logZ -31.47. -- repos: - - autolens_workspace_developer: feature/blackjax-smc-gradient-kernel (pushed, tip 6867762) - ## group4-mge-search-benchmark - issue: https://github.com/PyAutoLabs/autolens_profiling/issues/82 - prompt: active/research_profiling_experiment_in_the_autolens_pr.md