Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 0 additions & 14 deletions active/jax_native_posterior_sampler_wave.md

This file was deleted.

40 changes: 40 additions & 0 deletions complete/archive/shelved/jax_native_posterior_sampler_wave.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# JAX-native posterior sampler wave — ranked shortlist from the 2026-07-16 deep-research session — SUPERSEDED 2026-08-18

> **SUPERSEDED / SHELVED 2026-08-18.** Removed from the registry (was
> `parked.md :: blackjax-smc-gradient-kernel`, issue
> [autolens_workspace_developer#113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113))
> — the wave as scoped here is superseded by the 2026-08-17 human-approved
> inference programme (`active/inference_programme_ledger.md`, autolens_profiling#134),
> which owns sampler direction going forward. Stage (a) shipped its findings;
> stages (b) ChEES-HMC, (c) MCLMC+harmonic, (d) flowMC, (e) jaxns were never
> issued and are not carried forward as-is.
>
> **Preserved parked context (from the deleted `parked.md` entry):** stage (a)
> POSITIVE — warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering,
> `einstein_radius` 1.5998 vs truth 1.6, max logL ~31781 vs Prodigy MLE
> 31787.93). Parked 2026-07-24 to clear the deck for the autolens_profiling
> refactor. All work PRESERVED on pushed branch
> `feature/blackjax-smc-gradient-kernel` (origin autolens_workspace_developer,
> tip `6867762`); local worktree removed. Full write-up:
> `searches_minimal/smc_gradient_findings.md` on that branch. Science memory:
> `project_gradient_smc_warm_start_sampler_wave`. Gradient path certified
> OK_HMC_VIABLE (`probe_grad.py`); baseline `nss_grad` row = logZ -31.47. Open
> resume threads at park time: RAL job 331058
> (`/mnt/ral/jnightin/smc_grad_logs/smc_warm_ok-331058.out`, 3 warm arms @128p);
> logZ (~31690-31798) vs Nautilus comparison row; A100 rep-timing;
> `warm_start.json` is a regenerable cache (re-`--refresh` if it predates the
> `cov` field). MGE only; pix deferred.


Type: research
Target: autolens_workspace_developer
Repos:
- autolens_workspace_developer
Difficulty: large
Autonomy: supervised
Priority: normal
Status: formalised

Research: JAX-native posterior sampler wave — implement and benchmark the ranked sampler shortlist from the 2026-07-16 deep-research session on the standard searches_minimal problem in @autolens_workspace_developer, via the sampler_pipeline ingest-prototype-profile-promote flow. Ranked stages: (a) blackjax adaptive tempered SMC upgraded to a gradient inner kernel (MALA/HMC + inner_kernel_tuning; extends the existing blackjax_smc.py RWM smoke script; gives logZ from tempering increments), (b) ChEES-HMC many-chain adaptation (cheapest first-class addition; vmap-friendly fixed-length trajectories vs NUTS variable-length trees), (c) MCLMC + adjusted MCLMC warm-started from the multi-start Adam basin, paired with harmonic (learned harmonic mean) for evidence, (d) flowMC (normalizing-flow global jumps; multimodality insurance), (e) jaxns cameo (independent GPU-native nested-sampling cross-check vs Nautilus; inversion-heavy vmap OOM caution applies, NSS history). Constraints from prior campaigns: reverse-mode-only gradients, NaN-gradient degenerate points need masking, vmap fan-out needs lax.map batch_size tiling (Fit#1374 lever), RAL A100 float32. Deliverable per stage: findings doc + benchmark vs Nautilus baseline. Issue stages one at a time as predecessors ship — do not bulk-issue.

<!-- formalised by the Intake (Conception) Agent on 2026-07-16 from user-intake -->
8 changes: 3 additions & 5 deletions dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here.

| Where | Count |
|-------|------:|
| [In flight](#in-flight) (`active/`) | 10 |
| [Parked](#parked) (`parked.md`) | 4 |
| [In flight](#in-flight) (`active/`) | 9 |
| [Parked](#parked) (`parked.md`) | 3 |
| [Planned](#planned) (`planned.md`) | 7 |
| [Backlog](#backlog) (`draft/`) | 139 |

Expand Down Expand Up @@ -49,7 +49,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A
Issued — each has an open GitHub issue and usually a branch. The full record for each is in [`active.md`](active.md).

- [Commit the inference-methods programme plan + knowledge ledger into autolens_profiling](active/inference_programme_ledger.md) — [issue #134](https://github.com/PyAutoLabs/autolens_profiling/issues/134) — pr-open (https://github.com/PyAutoLabs/autolens_profiling/pull/135)
- [JAX-native posterior sampler wave — ranked shortlist from the 2026-07-16](active/jax_native_posterior_sampler_wave.md) — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius…
- [Remove standalone matplotlib-inline comments](active/matplotlib_inline_standalones.md)
- [Find what kills MGE multi-start lanes — it is not](active/mge_lane_death.md) — [issue #128](https://github.com/PyAutoLabs/autolens_profiling/issues/128) — CAUSE FOUND 2026-08-15 (cloud CPU session) — written up on autolens_profiling#128. Remaining: GPU/float64/multi-seed…
- [PositionsLH penalty accumulation returns 2x the LAST penalty, discarding the](active/positions_lh_penalty_accumulation_bug.md) — [issue #699](https://github.com/PyAutoLabs/PyAutoLens/issues/699) — pr-open (https://github.com/PyAutoLabs/PyAutoLens/pull/700)
Expand All @@ -64,9 +63,8 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md).

<details>
<summary><b>4</b> task(s)</summary>
<summary><b>3</b> task(s)</summary>

- **blackjax-smc-gradient-kernel** — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius…
- **group4-mge-search-benchmark** — [issue #82](https://github.com/PyAutoLabs/autolens_profiling/issues/82)
- **matplotlib-inline-standalones**
- **pyautoreduce-slacs1430-acs-comparison**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,9 @@ clean.

## Constraints

- Wait for the `blackjax-smc-gradient-kernel` claim to release; verify the repo
is clean before starting (it had 23 dirty files at wrap-up).
- The `blackjax-smc-gradient-kernel` claim is RELEASED (worktree removed 2026-07-24;
the parked task was shelved as superseded 2026-08-18), so this is no longer
blocked — but verify the repo is clean before starting (23 dirty files at wrap-up).
- Developer repo (doc-light, not CI-gated like the workspaces); still verify
renamed scripts import cleanly against merged main.
- Copy any gitignored products out before any worktree teardown.
2 changes: 1 addition & 1 deletion ideas.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
- [formalised -> feature/pyautogut/repo_skeleton.md] [from: research pyautogut-organ-decision · research/pyautobrain/pyautogut_organ_decision.md] Stand up the PyAutoGut repo skeleton — the new peer organ repo owning condemned self-material: the attic remote, the `refs/archive/condemned/<name>` archive-ref convention (stays out of `git branch -a`), and the `condemned.md` Mind manifest schema (type/locator/confidence/reason/merged?/condemned/sweep-after/breaks-if-wrong/archive-ref+SHA). Work-type feature/pyautogut; also adds PyAutoGut to repos.yaml + ORGANISM.md as a peer organ.
- [formalised -> feature/pyautobrain/hygiene_pyautogut_drive_seam.md] [from: research pyautogut-organ-decision · research/pyautobrain/pyautogut_organ_decision.md] Add the hygiene→PyAutoGut drive seam in the Brain — a PyAutoGut-aware hygiene `tidy` mode that files condemned candidates into `condemned.md` async (no synchronous per-item gate, replacing the mid-sweep repo_cleanup interrogation), plus a batch `sweep` mode that runs the existing repo_cleanup safety gates against the manifest at a user-chosen time. Mirrors the Heart↔vitals template: the organ holds+voids, the conductor drives. Work-type feature/pyautobrain.
- [formalised -> research/autofit/multi_band_factorgraph_compile_deeper_dig.md] [from: research multiband-compile · Experiment A · jax_compile/README.md multi-band section + project_multiband_factorgraph_compile_cliff memory] Multi-band FactorGraphModel value_and_grad compile — deeper dig + productize the fix. Experiment A (autolens_profiling branch research/multiband-compile-ab) proved the heterogeneous-per-band-shape cold compile is a 5.9× superlinear XLA-fusion cliff (homogeneous 4-band == single-band 120s; two distinct shapes = 704s CPU), that trace/eval are unchanged, and that the persistent cache rescues both arms (warm 2–7s). Follow-up, ONE task, profiling/research: (1) sub-investigation B — does a per-factor jit boundary inside PyAutoFit FactorGraphModel.log_likelihood_function bound cold to N×single-band + a linear combine, removing the superlinearity WITHOUT padding (the potential source lever, touches PyAutoFit); (2) sub-investigation C — add A100 multi-band vag rows (datacube_img / datacube_img_hetero) to the jax_compile census, quantify the far-lower absolute A100 cliff, and ship the final N-band verdict; (3) productize the immediate user workaround — pad short-wavelength bands to a common grid so all factors share one fused kernel — as a documented recipe and/or helper. Does NOT re-open the #71 single-band settings verdict. Repos: PyAutoFit, autolens_profiling. Difficulty large, supervised.
- [from: research wsdev#117 pix-prodigy-cpu · pix_prodigy_findings.md] Warm-started sampler wave on meshes — revive the parked jax_native_posterior_sampler_wave with the MultiStartProdigy endpoint population (16 basin-labelled points) as initializer; mesh↔kernel mapping: ChEES-HMC/MCLMC on kernel-CDF+knn, tempered SMC (logZ) on delaunay.
- [from: research wsdev#117 pix-prodigy-cpu · pix_prodigy_findings.md] Warm-started sampler wave on meshes — revive the shelved jax_native_posterior_sampler_wave shortlist (complete/archive/shelved/, superseded 2026-08-18) with the MultiStartProdigy endpoint population (16 basin-labelled points) as initializer; mesh↔kernel mapping: ChEES-HMC/MCLMC on kernel-CDF+knn, tempered SMC (logZ) on delaunay.
- [from: research wsdev#117 · resurrection diagnostics] Validate resurrection-rate → HMC-divergence-rate prediction per mesh — cheap; would give a principled mesh/kernel selection rule for Hamiltonian sampling on pixelized sources.
- [from: research wsdev#117 · thin-baseline caveat] Heavier-settings Nautilus reference runs (n_live >= 500) on the knn/delaunay pix objectives — the n_live=100 baselines missed the truth basin; needed to claim "Prodigy matches a CONVERGED sampler" and to quantify the prior-volume search the gradient stage saves.
- [from: research wsdev#117 · matern arm] Matérn kernel regularization as a production option — does its evidence ranking track AdaptSplit across real datasets? If yes the gradient-safe scheme can be the SLaM default, not just the search-time choice.
Expand Down
10 changes: 0 additions & 10 deletions parked.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,6 @@ unindexed worktrees or stashes. Move an entry back to `active.md` (or to
`planned.md` if re-scoping is needed) when work resumes; on shipping,
write the dated `complete/<YYYY>/<MM>/<slug>.md` record instead.

## blackjax-smc-gradient-kernel
- issue: https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113
- status: PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius 1.5998 vs truth 1.6, max logL ~31781 vs Prodigy MLE 31787.93). Parked to clear the deck for the autolens_profiling refactor (scripts live in autolens_workspace_developer/searches_minimal/, which the refactor churns). All work PRESERVED on pushed branch feature/blackjax-smc-gradient-kernel (origin autolens_workspace_developer, tip 6867762); local worktree removed. Full write-up: searches_minimal/smc_gradient_findings.md on that branch. Science memory: project_gradient_smc_warm_start_sampler_wave. RESUME AFTER REFACTOR: (1) check RAL job 331058 (/mnt/ral/jnightin/smc_grad_logs/smc_warm_ok-331058.out) — 3 warm arms @128p (MALA auto-step, MALA+--tune, HMC): does --tune HOLD acceptance ~0.57 as lambda->1 (fixed step decays 0.80->0.17)? does HMC beat MALA? (2) compare logZ (~31690-31798) vs Nautilus, write comparison.txt row; (3) A100 rep-timing (GPUs full at park); (4) stage (b) ChEES-HMC reusing _warm_start. warm_start.json is a regenerable cache — re-`--refresh` if it predates the cov field (else silent diagonal fallback). MGE only; pix deferred.
- worktree: none (pushed to origin; local worktree removed 2026-07-24)
- autonomy: supervised
- prompt: active/jax_native_posterior_sampler_wave.md
- note: WAVE TRACKER — this parks the whole JAX-native posterior sampler wave (stages b ChEES-HMC, c MCLMC+harmonic, d flowMC, e jaxns still remain). Do NOT move the prompt to complete/ — stage (a) shipped its findings but the wave continues as a separate run after the refactor. Gradient path certified OK_HMC_VIABLE (probe_grad.py); baseline nss_grad row = logZ -31.47.
- repos:
- autolens_workspace_developer: feature/blackjax-smc-gradient-kernel (pushed, tip 6867762)

## group4-mge-search-benchmark
- issue: https://github.com/PyAutoLabs/autolens_profiling/issues/82
- prompt: active/research_profiling_experiment_in_the_autolens_pr.md
Expand Down
Loading