diff --git a/active/benchmark_calibration_runs.md b/active/benchmark_calibration_runs.md deleted file mode 100644 index 223036ec..00000000 --- a/active/benchmark_calibration_runs.md +++ /dev/null @@ -1,34 +0,0 @@ -# First benchmark calibration campaign — run the 4 assistant benchmarks across models - -Type: research -Target: autolens_assistant -Difficulty: medium -Autonomy: supervised -Priority: normal -Status: filed - -Follow-up to feature/autolens_assistant assistant_benchmarks (issue -https://github.com/PyAutoLabs/autolens_assistant/issues/57): the benchmark -package ships four frozen prompt cards (`benchmarks/prompts/`) and the -run/track harness (`autoassistant/benchmark.py`), but `benchmarks/runs/` is -empty. Run the first calibration campaign so the comparison tables have real -rows and the rubrics get validated against real transcripts. - -Scope: - -- Run at least the two cheap benchmarks (`teacher-basic-workflow`, - `assistant-easy-cosmos-web-ring`) on 2+ model×harness combinations (e.g. - Claude Code with two different models; add Codex/Gemini if convenient), per - the protocol in `benchmarks/README.md` — fresh sessions, prompts verbatim, - operator behaves like a real user, no coaching. -- Record every run (transcript, meta.yaml with hardware/duration, artifacts), - score with evidence, regenerate `RESULTS.md`, commit + push. -- The medium/hard benchmarks are hours-scale: run at most one of them once, - or explicitly defer them with a runtime note. -- Deliverable beyond the run records: a short calibration verdict on the - rubrics themselves — were any rows unscoreable, ambiguous, or trivially - gamed? File rubric fixes as `version`-bump proposals rather than editing - cards in place. - -Blocked until the assistant-benchmarks PRs are merged (the cards/harness must -be on main so runs execute against the published, frozen prompt versions). diff --git a/complete/2026/07/pyautolens-assistant-joss-paper.md b/complete/2026/07/pyautolens-assistant-joss-paper.md index be862a44..4764143b 100644 --- a/complete/2026/07/pyautolens-assistant-joss-paper.md +++ b/complete/2026/07/pyautolens-assistant-joss-paper.md @@ -11,6 +11,7 @@ - trap avoided: `audit_skill_apis.py` writes a dated report to `autoassistant/audit/` as a side effect, which `git add -A` would sweep up — already gitignored (.gitignore:30), commit stayed clean. See [[feedback_ship_workspace_binary_leak]]. - heart: shipped at YELLOW score 55 (no RED), author-acknowledged; both reasons ("workspace validation not passing (3 failed, 2026-07-09)"; "58 stale parked script(s)") pre-date the branch and are unrelated to a docs-only change. Ack recorded verbatim on the issue + in active.md. - OPEN / needs author: `State of the field` + `Research impact statement` are commented stubs (JOSS requires both; neither draftable without author judgement). Benchmark-results prose stays FUTURE-TENSE because benchmarks/RESULTS.md records ZERO runs for all 4 benchmarks — despite #59 (first calibration campaign) being CLOSED. Either results exist un-ingested or the campaign closed without producing them; worth a look before submission. +- RESOLVED 2026-08-18: the campaign closed without producing them. Verified against autolens_assistant main — `benchmarks/runs/` holds only `.gitkeep`, so nothing was un-ingested. The campaign prompt has been shelved (`complete/archive/shelved/benchmark_calibration_runs.md`); future-tense benchmark prose in the paper is correct as written. ## Original prompt diff --git a/complete/archive/shelved/benchmark_calibration_runs.md b/complete/archive/shelved/benchmark_calibration_runs.md new file mode 100644 index 00000000..2decdd66 --- /dev/null +++ b/complete/archive/shelved/benchmark_calibration_runs.md @@ -0,0 +1,94 @@ +# First benchmark calibration campaign — run the 4 assistant benchmarks across models + +Type: research +Target: autolens_assistant +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: filed + +Follow-up to feature/autolens_assistant assistant_benchmarks (issue +https://github.com/PyAutoLabs/autolens_assistant/issues/57): the benchmark +package ships four frozen prompt cards (`benchmarks/prompts/`) and the +run/track harness (`autoassistant/benchmark.py`), but `benchmarks/runs/` is +empty. Run the first calibration campaign so the comparison tables have real +rows and the rubrics get validated against real transcripts. + +Scope: + +- Run at least the two cheap benchmarks (`teacher-basic-workflow`, + `assistant-easy-cosmos-web-ring`) on 2+ model×harness combinations (e.g. + Claude Code with two different models; add Codex/Gemini if convenient), per + the protocol in `benchmarks/README.md` — fresh sessions, prompts verbatim, + operator behaves like a real user, no coaching. +- Record every run (transcript, meta.yaml with hardware/duration, artifacts), + score with evidence, regenerate `RESULTS.md`, commit + push. +- The medium/hard benchmarks are hours-scale: run at most one of them once, + or explicitly defer them with a runtime note. +- Deliverable beyond the run records: a short calibration verdict on the + rubrics themselves — were any rows unscoreable, ambiguous, or trivially + gamed? File rubric fixes as `version`-bump proposals rather than editing + cards in place. + +Blocked until the assistant-benchmarks PRs are merged (the cards/harness must +be on main so runs execute against the published, frozen prompt versions). + +## SHELVED 2026-08-18 — campaign abandoned, not deferred + +Author decision: the calibration campaign will not be run. Shelved here rather +than lost; pull back into `draft/research/autolens_assistant/` if that changes. + +**The dedicated issue was already closed.** `autolens_assistant#59` (this +campaign) was closed on/before 2026-07-15 to release the repo claim that was +blocking the JOSS paper task — see `complete/2026/07/pyautolens-assistant-joss-paper.md`, +which flagged the discrepancy as needing a look. That question is now answered: +verified against autolens_assistant main on 2026-08-18, `benchmarks/runs/` +contains only `.gitkeep`, so **#59 closed without ever producing runs**. Nothing +is open on GitHub. Only the Mind-side residue (this prompt, the `parked.md` +entry, the dashboard row) survived, and it is removed with this commit. + +**State of the benchmark package as of 2026-08-18** (in case anyone revives it): +the four frozen cards (`benchmarks/prompts/`) and the harness +(`autoassistant/benchmark.py`) are on main and working; `RESULTS.md` records +zero runs for all four benchmarks. The blocker this prompt described had +cleared — the campaign was simply never a cost the author wanted to pay. + +**Pre-run rubric findings, preserved.** A read-only audit of the cards and +harness surfaced issues that would bite any future campaign — worth reading +before re-filing rather than rediscovering: + +- The frozen-prompt rule is enforced by nothing. + `autoassistant/tests/test_benchmark.py:170-176` asserts only + `prompt.strip() in readme`, for three of the four cards; no test binds a + card's prompt text to its `version`. A coordinated README+card edit passes CI + with `version` unchanged, so `meta.yaml` keeps recording the same + `prompt_version` across a changed prompt. `hard_group_multi.md` has no freeze + check at all. +- The "Machine-checkable" band is never machine-checked. `parse_score` + (`benchmark.py:215-244`) sums whatever the operator typed; nothing inspects + `scripts/`, `output/` or figures, and "Machine rows (M*) need verifiable + evidence" (`benchmark.py:151`) is prose in the generated `score.md`, not a + check — `score_run` rejects only unfilled *Awarded* cells, never empty + Evidence. +- Rows satisfiable without the work: `easy M4/M5` reward *showing a path* + rather than the figure existing; `medium M4` gives 5 points for printing two + numbers a fabricating run could invent, while `J5` separately penalises + fabrication; `teacher M3` sets no agreement threshold, so a comparison + showing recovery failed scores full marks. +- Evidence double-counted on the hard card: `M1`/`J1` both score the quad + verification, and `M4`/`J4` both score the follow-up composition — a run that + completes M4 cannot fail J4. Note `test_repo_prompt_cards_parse` asserts + `machine + judged == 100`, so any fix needs an explicit re-split. +- Compound rows bundle orthogonal failures: `J5 "Conduct"` (10 pts, all three + assistant cards) mixes concision, fabrication and API-gate discipline. The + real-data gate is a 15-point row on `easy` but folded into a shared 10-point + row on `medium`, distorting cross-card comparison of the same behaviour. +- `medium J2` (15 pts) says the HPC option may be "set up **or offered**", + letting one sentence carry most of the row. +- Undefined discriminators: `easy M2`'s "not test-mode", `hard M3`'s proof of a + simultaneous fit, and `teacher J4`'s Euclid tolerance (stated in card prose, + not the rubric row). + +Sequencing note for any revival: scores are comparable only within a card +`version`, so rubric fixes must land *before* a campaign — fixing them +afterwards splits the comparison tables and forces a re-run. diff --git a/dashboard.md b/dashboard.md index 5b41eba2..07fe108f 100644 --- a/dashboard.md +++ b/dashboard.md @@ -1,6 +1,6 @@ # PyAutoMind task dashboard - + Every task the Mind is holding, on one page: what is in flight, what is parked, and the whole backlog to pick from. Pick a line, then run `/start_dev ` to start it. @@ -8,8 +8,8 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 13 | -| [Parked](#parked) (`parked.md`) | 6 | +| [In flight](#in-flight) (`active/`) | 12 | +| [Parked](#parked) (`parked.md`) | 5 | | [Planned](#planned) (`planned.md`) | 7 | | [Backlog](#backlog) (`draft/`) | 139 | @@ -48,7 +48,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A Issued — each has an open GitHub issue and usually a branch. The full record for each is in [`active.md`](active.md). -- [First benchmark calibration campaign — run the 4 assistant benchmarks](active/benchmark_calibration_runs.md) - [ConstantZeroth regularization is broken twice over — dead code presenting](active/constant_zeroth_broken_dead_code.md) — [issue #448](https://github.com/PyAutoLabs/PyAutoArray/issues/448) — remote-handoff (prepared 2026-08-17 for a cloud/phone session; NO local worktree claim) - [Address ECEB editorial comments on ECLIPSE-C](active/euclid_eceb_editorial_revision.md) - [Commit the inference-methods programme plan + knowledge ledger into autolens_profiling](active/inference_programme_ledger.md) — [issue #134](https://github.com/PyAutoLabs/autolens_profiling/issues/134) — pr-open (https://github.com/PyAutoLabs/autolens_profiling/pull/135) @@ -67,12 +66,11 @@ Issued — each has an open GitHub issue and usually a branch. The full record f Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md).
-6 task(s) +5 task(s) - **blackjax-smc-gradient-kernel** — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius… - **group4-mge-search-benchmark** — [issue #82](https://github.com/PyAutoLabs/autolens_profiling/issues/82) - **matplotlib-inline-standalones** -- **benchmark-calibration-runs** - **euclid-eceb-editorial-revision** - **pyautoreduce-slacs1430-acs-comparison** diff --git a/parked.md b/parked.md index 88dda799..e88fb8ce 100644 --- a/parked.md +++ b/parked.md @@ -36,16 +36,6 @@ write the dated `complete///.md` record instead. - note: do NOT broaden into the old `pyprojroot` bootstrap sweep — the prompt is explicit that the dependent AutoCTI follow-up owns that. -## benchmark-calibration-runs -- prompt: active/benchmark_calibration_runs.md -- parked: 2026-08-08 — surfaced by the orphaned-prompt triage; VERIFIED NOT STARTED -- parent-issue: https://github.com/PyAutoLabs/autolens_assistant/issues/57 (the benchmark package itself) -- classification: research (autolens_assistant) -- evidence: `benchmarks/runs/` on autolens_assistant main still contains only `.gitkeep`, exactly - the state the prompt describes. The four frozen prompt cards (`benchmarks/prompts/`) and the - harness (`autoassistant/benchmark.py`) are in place; only the campaign is missing. -- note: needs real model runs, so it is a cost decision rather than a coding task. - ## euclid-eceb-editorial-revision - prompt: active/euclid_eceb_editorial_revision.md - parked: 2026-08-08 — surfaced by the orphaned-prompt triage; STATE UNVERIFIED