prompt: shelve benchmark-calibration-runs — campaign abandoned

complete/2026/07/pyautolens-assistant-joss-paper.md

@@ -11,6 +11,7 @@
 - trap avoided: `audit_skill_apis.py` writes a dated report to `autoassistant/audit/` as a side effect, which `git add -A` would sweep up — already gitignored (.gitignore:30), commit stayed clean. See [[feedback_ship_workspace_binary_leak]].
 - heart: shipped at YELLOW score 55 (no RED), author-acknowledged; both reasons ("workspace validation not passing (3 failed, 2026-07-09)"; "58 stale parked script(s)") pre-date the branch and are unrelated to a docs-only change. Ack recorded verbatim on the issue + in active.md.
 - OPEN / needs author: `State of the field` + `Research impact statement` are commented stubs (JOSS requires both; neither draftable without author judgement). Benchmark-results prose stays FUTURE-TENSE because benchmarks/RESULTS.md records ZERO runs for all 4 benchmarks — despite #59 (first calibration campaign) being CLOSED. Either results exist un-ingested or the campaign closed without producing them; worth a look before submission.
+- RESOLVED 2026-08-18: the campaign closed without producing them. Verified against autolens_assistant main — `benchmarks/runs/` holds only `.gitkeep`, so nothing was un-ingested. The campaign prompt has been shelved (`complete/archive/shelved/benchmark_calibration_runs.md`); future-tense benchmark prose in the paper is correct as written.
 
 ## Original prompt
 

complete/archive/shelved/benchmark_calibration_runs.md

@@ -0,0 +1,94 @@
+# First benchmark calibration campaign — run the 4 assistant benchmarks across models
+
+Type: research
+Target: autolens_assistant
+Difficulty: medium
+Autonomy: supervised
+Priority: normal
+Status: filed
+
+Follow-up to feature/autolens_assistant assistant_benchmarks (issue
+https://github.com/PyAutoLabs/autolens_assistant/issues/57): the benchmark
+package ships four frozen prompt cards (`benchmarks/prompts/`) and the
+run/track harness (`autoassistant/benchmark.py`), but `benchmarks/runs/` is
+empty. Run the first calibration campaign so the comparison tables have real
+rows and the rubrics get validated against real transcripts.
+
+Scope:
+
+- Run at least the two cheap benchmarks (`teacher-basic-workflow`,
+  `assistant-easy-cosmos-web-ring`) on 2+ model×harness combinations (e.g.
+  Claude Code with two different models; add Codex/Gemini if convenient), per
+  the protocol in `benchmarks/README.md` — fresh sessions, prompts verbatim,
+  operator behaves like a real user, no coaching.
+- Record every run (transcript, meta.yaml with hardware/duration, artifacts),
+  score with evidence, regenerate `RESULTS.md`, commit + push.
+- The medium/hard benchmarks are hours-scale: run at most one of them once,
+  or explicitly defer them with a runtime note.
+- Deliverable beyond the run records: a short calibration verdict on the
+  rubrics themselves — were any rows unscoreable, ambiguous, or trivially
+  gamed? File rubric fixes as `version`-bump proposals rather than editing
+  cards in place.
+
+Blocked until the assistant-benchmarks PRs are merged (the cards/harness must
+be on main so runs execute against the published, frozen prompt versions).
+
+## SHELVED 2026-08-18 — campaign abandoned, not deferred
+
+Author decision: the calibration campaign will not be run. Shelved here rather
+than lost; pull back into `draft/research/autolens_assistant/` if that changes.
+
+**The dedicated issue was already closed.** `autolens_assistant#59` (this
+campaign) was closed on/before 2026-07-15 to release the repo claim that was
+blocking the JOSS paper task — see `complete/2026/07/pyautolens-assistant-joss-paper.md`,
+which flagged the discrepancy as needing a look. That question is now answered:
+verified against autolens_assistant main on 2026-08-18, `benchmarks/runs/`
+contains only `.gitkeep`, so **#59 closed without ever producing runs**. Nothing
+is open on GitHub. Only the Mind-side residue (this prompt, the `parked.md`
+entry, the dashboard row) survived, and it is removed with this commit.
+
+**State of the benchmark package as of 2026-08-18** (in case anyone revives it):
+the four frozen cards (`benchmarks/prompts/`) and the harness
+(`autoassistant/benchmark.py`) are on main and working; `RESULTS.md` records
+zero runs for all four benchmarks. The blocker this prompt described had
+cleared — the campaign was simply never a cost the author wanted to pay.
+
+**Pre-run rubric findings, preserved.** A read-only audit of the cards and
+harness surfaced issues that would bite any future campaign — worth reading
+before re-filing rather than rediscovering:
+
+- The frozen-prompt rule is enforced by nothing.
+  `autoassistant/tests/test_benchmark.py:170-176` asserts only
+  `prompt.strip() in readme`, for three of the four cards; no test binds a
+  card's prompt text to its `version`. A coordinated README+card edit passes CI
+  with `version` unchanged, so `meta.yaml` keeps recording the same
+  `prompt_version` across a changed prompt. `hard_group_multi.md` has no freeze
+  check at all.
+- The "Machine-checkable" band is never machine-checked. `parse_score`
+  (`benchmark.py:215-244`) sums whatever the operator typed; nothing inspects
+  `scripts/`, `output/` or figures, and "Machine rows (M*) need verifiable
+  evidence" (`benchmark.py:151`) is prose in the generated `score.md`, not a
+  check — `score_run` rejects only unfilled *Awarded* cells, never empty
+  Evidence.
+- Rows satisfiable without the work: `easy M4/M5` reward *showing a path*
+  rather than the figure existing; `medium M4` gives 5 points for printing two
+  numbers a fabricating run could invent, while `J5` separately penalises
+  fabrication; `teacher M3` sets no agreement threshold, so a comparison
+  showing recovery failed scores full marks.
+- Evidence double-counted on the hard card: `M1`/`J1` both score the quad
+  verification, and `M4`/`J4` both score the follow-up composition — a run that
+  completes M4 cannot fail J4. Note `test_repo_prompt_cards_parse` asserts
+  `machine + judged == 100`, so any fix needs an explicit re-split.
+- Compound rows bundle orthogonal failures: `J5 "Conduct"` (10 pts, all three
+  assistant cards) mixes concision, fabrication and API-gate discipline. The
+  real-data gate is a 15-point row on `easy` but folded into a shared 10-point
+  row on `medium`, distorting cross-card comparison of the same behaviour.
+- `medium J2` (15 pts) says the HPC option may be "set up **or offered**",
+  letting one sentence carry most of the row.
+- Undefined discriminators: `easy M2`'s "not test-mode", `hard M3`'s proof of a
+  simultaneous fit, and `teacher J4`'s Euclid tolerance (stated in card prose,
+  not the rubric row).
+
+Sequencing note for any revival: scores are comparable only within a card
+`version`, so rubric fixes must land *before* a campaign — fixing them
+afterwards splits the comparison tables and forces a re-run.

dashboard.md

@@ -1,15 +1,15 @@
 # PyAutoMind task dashboard
 
-<!-- generated by `pyauto-brain intake dashboard --apply` on 2026-08-17 — regenerate, do not hand-edit -->
+<!-- generated by `pyauto-brain intake dashboard --apply` on 2026-08-18 — regenerate, do not hand-edit -->
 
 Every task the Mind is holding, on one page: what is in flight, what is parked, and the whole backlog to pick from. Pick a line, then run `/start_dev <prompt-path>` to start it.
 
 Tasks only — the organism's health lives with the Heart (`/health`), not here.
 
 | Where | Count |
 |-------|------:|
-| [In flight](#in-flight) (`active/`) | 13 |
-| [Parked](#parked) (`parked.md`) | 6 |
+| [In flight](#in-flight) (`active/`) | 12 |
+| [Parked](#parked) (`parked.md`) | 5 |
 | [Planned](#planned) (`planned.md`) | 7 |
 | [Backlog](#backlog) (`draft/`) | 139 |
 
@@ -48,7 +48,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A
 
 Issued — each has an open GitHub issue and usually a branch. The full record for each is in [`active.md`](active.md).
 
-- [First benchmark calibration campaign — run the 4 assistant benchmarks](active/benchmark_calibration_runs.md)
 - [ConstantZeroth regularization is broken twice over — dead code presenting](active/constant_zeroth_broken_dead_code.md) — [issue #448](https://github.com/PyAutoLabs/PyAutoArray/issues/448) — remote-handoff (prepared 2026-08-17 for a cloud/phone session; NO local worktree claim)
 - [Address ECEB editorial comments on ECLIPSE-C](active/euclid_eceb_editorial_revision.md)
 - [Commit the inference-methods programme plan + knowledge ledger into autolens_profiling](active/inference_programme_ledger.md) — [issue #134](https://github.com/PyAutoLabs/autolens_profiling/issues/134) — pr-open (https://github.com/PyAutoLabs/autolens_profiling/pull/135)
@@ -67,12 +66,11 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
 Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md).
 
 <details>
-<summary><b>6</b> task(s)</summary>
+<summary><b>5</b> task(s)</summary>
 
 - **blackjax-smc-gradient-kernel** — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius…
 - **group4-mge-search-benchmark** — [issue #82](https://github.com/PyAutoLabs/autolens_profiling/issues/82)
 - **matplotlib-inline-standalones**
-- **benchmark-calibration-runs**
 - **euclid-eceb-editorial-revision**
 - **pyautoreduce-slacs1430-acs-comparison**
 

parked.md

@@ -36,16 +36,6 @@ write the dated `complete/<YYYY>/<MM>/<slug>.md` record instead.
 - note: do NOT broaden into the old `pyprojroot` bootstrap sweep — the prompt is explicit
   that the dependent AutoCTI follow-up owns that.
 
-## benchmark-calibration-runs
-- prompt: active/benchmark_calibration_runs.md
-- parked: 2026-08-08 — surfaced by the orphaned-prompt triage; VERIFIED NOT STARTED
-- parent-issue: https://github.com/PyAutoLabs/autolens_assistant/issues/57 (the benchmark package itself)
-- classification: research (autolens_assistant)
-- evidence: `benchmarks/runs/` on autolens_assistant main still contains only `.gitkeep`, exactly
-  the state the prompt describes. The four frozen prompt cards (`benchmarks/prompts/`) and the
-  harness (`autoassistant/benchmark.py`) are in place; only the campaign is missing.
-- note: needs real model runs, so it is a cost decision rather than a coding task.
-
 ## euclid-eceb-editorial-revision
 - prompt: active/euclid_eceb_editorial_revision.md
 - parked: 2026-08-08 — surfaced by the orphaned-prompt triage; STATE UNVERIFIED