From 78d0a16cfc1d2dce1e248887b174ce701ab7a4da Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 01:44:54 +0000 Subject: [PATCH 1/2] prompt: pre-campaign rubric calibration audit for the assistant benchmarks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified against autolens_assistant main that the blocker on active/benchmark_calibration_runs.md is clear (the four frozen cards and autoassistant/benchmark.py are on main) and that benchmarks/runs/ is still empty, so the campaign is genuinely un-started. Took the prompt's second deliverable — the calibration verdict on the rubrics — early, since every finding was reachable by reading the cards and harness at no run cost. Filed as a version-bump proposal rather than editing cards in place, per the prompt. Findings include an unenforced prompt-freeze rule (no test binds a card's prompt text to its version, so a coordinated README+card edit keeps recording the same prompt_version across a changed prompt), a "machine-checkable" band the harness never actually checks, rows satisfiable without the underlying work, and evidence double-counted across bands on the hard card. Resequences the campaign: scores are comparable only within a card version, so landing rubric fixes after the runs would split the comparison tables and force a re-run. Fix first, run once against v2. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GFgC9kZLtHGcbbdyodgqCx --- active/benchmark_calibration_runs.md | 31 +++++ dashboard.md | 9 +- ...enchmark_rubric_freeze_and_scoring_gaps.md | 130 ++++++++++++++++++ parked.md | 10 ++ 4 files changed, 176 insertions(+), 4 deletions(-) create mode 100644 draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md diff --git a/active/benchmark_calibration_runs.md b/active/benchmark_calibration_runs.md index 223036ec..e2480e9a 100644 --- a/active/benchmark_calibration_runs.md +++ b/active/benchmark_calibration_runs.md @@ -32,3 +32,34 @@ Scope: Blocked until the assistant-benchmarks PRs are merged (the cards/harness must be on main so runs execute against the published, frozen prompt versions). + +## 2026-08-18 — blocker cleared; rubric verdict taken early + +Verified against a read-only clone of autolens_assistant main: the four frozen +cards (`benchmarks/prompts/`) and `autoassistant/benchmark.py` are present, and +`benchmarks/runs/` still holds only `.gitkeep`. The blocking condition above is +therefore **cleared** and the campaign is un-started, exactly as `parked.md` +records. + +The final deliverable — the calibration verdict on the rubrics — was taken +early and filed as +`draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md` +(9 findings: an unenforced prompt-freeze rule, a "machine-checkable" band the +harness never checks, gameable and double-counted rows, and several rows whose +discriminators the operator must invent). + +**Resequence this prompt accordingly.** `benchmarks/README.md` makes scores +comparable only within one card `version`, so landing rubric fixes after the +campaign bumps the cards and splits the comparison tables — the runs would have +to be repeated. Land the harness fixes and the `version: 2` card bumps first, +then run the campaign once against v2. The remaining scope here is the runs +themselves. + +Outstanding before the campaign can start: + +- a human cost decision on which model x harness combinations to buy; +- a session with push rights to autolens_assistant, the bundled datasets and a + GPU — a cloud Mind/Brain session cannot execute the protocol; +- an operator/judge who has **not** read the rubrics (the protocol's + "no benchmark-aware behaviour" rule); the agent that produced the audit above + is disqualified as an agent-under-test for these cards. diff --git a/dashboard.md b/dashboard.md index 5b41eba2..456bfe37 100644 --- a/dashboard.md +++ b/dashboard.md @@ -1,6 +1,6 @@ # PyAutoMind task dashboard - + Every task the Mind is holding, on one page: what is in flight, what is parked, and the whole backlog to pick from. Pick a line, then run `/start_dev ` to start it. @@ -11,7 +11,7 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | [In flight](#in-flight) (`active/`) | 13 | | [Parked](#parked) (`parked.md`) | 6 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 139 | +| [Backlog](#backlog) (`draft/`) | 140 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) @@ -97,10 +97,10 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**139** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**140** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
-bug — 40 +bug — 41 - [pre_build stages untracked files, publishing uncommitted human work](draft/bug/pyautohands/pre_build_stages_untracked_wip.md) — pyautohands · small · supervised · high - [UniformPrior bounds are not enforced in the objective on the](draft/bug/autofit/uniform_prior_bounds_unenforced_on_numpy_path.md) — autofit · medium · human-required · high @@ -133,6 +133,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. - [interferometer Delaunay pixelization — non-PD FitException in test-mode bypass](draft/bug/autolens/interferometer_delaunay_nonpd_fitexception.md) — autolens · medium · supervised · normal - [JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38](draft/bug/autolens/jax_point_source_point_smoke_sentinel.md) — autolens · medium · supervised · normal - [JIT cache not hit in modeling_visualization delaunay/rectangular scripts](draft/bug/autolens/jit_cache_not_hit_modeling_visualization.md) — autolens · medium · supervised · normal +- [Benchmark rubric calibration — freeze integrity + scoring gaps (pre-campaign)](draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md) — autolens_assistant · medium · supervised · normal - [`@PyAutoFit` Add property-based correctness tests for every `Prior` subclass](draft/bug/priors/09_prior_property_tests.md) — priors · large · supervised · normal - [`@PyAutoFit` `TransformedMessage` reversal convention is undocumented foot-gun](draft/bug/priors/11_transformed_message_semantics_doc.md) — priors · large · supervised · normal - [Resolve release-profile timeout scripts deliberately](draft/bug/health_fixes/release_timeout_policy.md) — health_fixes · too-large · supervised · normal diff --git a/draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md b/draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md new file mode 100644 index 00000000..a4267be8 --- /dev/null +++ b/draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md @@ -0,0 +1,130 @@ +# Benchmark rubric calibration — freeze integrity + scoring gaps (pre-campaign) + +Type: bug +Target: autolens_assistant +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: filed + +Pre-run calibration audit of the four benchmark cards and the scoring harness +shipped by autolens_assistant#57. Filed as `version`-bump proposals, per +`active/benchmark_calibration_runs.md` — no card is edited in place. + +**Why this is filed before the campaign, not after.** The calibration prompt +puts the rubric verdict *after* the runs. That order is the expensive one: +`benchmarks/README.md` makes scores comparable only within one card `version`, +so any fix landed after the campaign bumps the card and splits the tables — +the runs would have to be repeated to populate v2. Every finding below was +reachable by reading the cards and harness, at no run cost. Recommend landing +F1/F2 (harness, no version bump) and the F3–F8 card bumps first, then running +the campaign once, against v2. + +## F1 — the frozen-prompt rule is not enforced by anything (highest consequence) + +`autoassistant/tests/test_benchmark.py:170-176` asserts only +`prompt.strip() in readme`, for three cards. **No test binds a card's prompt +text to its `version`.** A coordinated edit to `README.md` and the card passes +CI with `version` unchanged, so every run's `meta.yaml` keeps recording the same +`prompt_version` across a changed prompt — silently violating the comparability +rule the whole benchmark rests on. `hard_group_multi.md` is not a README example +and so has no freeze check at all. + +Fix: record a checksum of the prompt block in card frontmatter +(`prompt_sha256:`) and assert it in CI; changing the text without bumping both +`version` and the checksum fails. Cheap, and it must precede the first campaign +or those records inherit the ambiguity. + +## F2 — "Machine-checkable" is a promise the harness does not keep + +`benchmark.py:215-244` (`parse_score`) splits rows on the `M` prefix and sums +whatever the operator typed. Nothing inspects `scripts/`, `output/` or the +figures. The "Machine rows (M*) need verifiable evidence" line +(`benchmark.py:151`) is prose emitted into `score.md`, not a check — the +Evidence column may be left empty and `score_run` still passes, since it rejects +only unfilled *Awarded* cells. + +Fix (either, ideally both): enforce non-empty Evidence on `M*` rows in +`score_run`; and/or rename the band to "artifact-checkable" so `RESULTS.md` +stops implying an objectivity the harness does not provide. + +## F3 — rows satisfiable without the underlying work + +- `easy M4` (10) / `M5` (5): "figure was produced **and its path shown**" — the + reward attaches to showing a path. Reword to require the file to exist at the + shown path, operator-confirmed. +- `medium M4` (5): "Both evidence comparisons reported as numbers in the final + answer" is pure output formatting, awardable to a run that fabricated the + numbers — while `J5` separately penalises fabrication. Make it conditional on + M1–M3 having been awarded. +- `teacher M3` (10): "Recovered parameters compared against input truths + explicitly" carries no agreement threshold, so a comparison demonstrating that + recovery *failed* scores full marks. That may be intended (it scores the act + of comparing) — the card must say which, or two judges will differ. + +## F4 — the same evidence scored twice (hard card) + +- `M1` (10, "quad morphology visible/verified") vs `J1` (15, "the quad verified + rather than assumed") — one behaviour, both bands. +- `M4` (5, follow-up "retaining MGE lens light and the SIE mass profiles") vs + `J4` (5, "pixelized reconstruction replaces the MGE source while the MGE lens + light and SIE masses carry over") — the same composition twice; a run that + completes M4 cannot fail J4. + +Fix: narrow M1 to dataset existence and leave verification to J1; collapse +M4/J4 into one row. Note `test_repo_prompt_cards_parse` asserts +`machine + judged == 100`, so the freed points need an explicit re-split. + +## F5 — compound rows bundling orthogonal failure modes + +- `J5 "Conduct"` (10, all three assistant cards) bundles concision + no + fabricated numbers + API-gate discipline. Fabrication is an integrity failure; + concision is style. One number cannot express "fabricated a result but wrote + concisely". Split fabrication out — arguably as a gate rather than points. +- The real-data gate is worth 15 as its own row on `easy` (J1) but is folded + into a shared 10-point conduct row on `medium` (J5). Either weighting is + defensible; the inconsistency is not, since it distorts cross-card comparison + of the same behaviour. +- `teacher J1` (20) enumerates five separate explanations in one row. + +## F6 — a 15-point row with a near-zero floor + +`medium J2`: "…HPC option **set up or offered** if the local estimate is slow." +"or offered" lets one sentence carry most of a 15-point row, and the operator +note reinforces it ("a run that stops at a well-set-up HPC handoff … can still +score highly"). Split explicitly: runtime estimate (x) / setup-or-offer quality +(y). + +## F7 — discriminators the operator has to invent + +- `easy M2`: "a completed non-linear search result exists under `output/` (**not + test-mode**)" — no stated test for "not test-mode". Name the artifact (the + `autonerves` `test_mode` flag, or a minimum `nlive`/sample count). +- `hard M3`: "one search over a shared model, not two independent fits" — name + the proof (a single summed `Analysis`, one search output directory). +- `teacher J4`: scored against "VIS-like pixel scale ~0.1"", which appears only + in the card's prose, not the rubric row. Move the tolerance into the row. + +## F8 — partial credit is allowed but unguided + +`benchmark.py:149` tells the scorer "fractions allowed" and `parse_score` +accepts any float in `0..max`. Combined with the compound rows in F5 and no +per-row allocation convention, two honest judges diverge exactly on the +heaviest rows. Decomposing the compound rows (F5) largely subsumes this; +otherwise state a convention (compound rows score in equal sub-parts). + +## F9 — the comparability rules cover the judged band only + +`benchmarks/README.md` says "Same judge for judged rows". Because the machine +band is operator-adjudicated (F2), operator identity matters there too, and +`meta.yaml` records `operator` and `score.judge` separately. Until F2 is fixed, +the rule should say both travel with the whole score. + +## Scope + +Harness/CI (no version bump): F1, F2, and the F4 re-split arithmetic. +Card `version: 2` bumps: F3, F5, F6, F7, F8, and the F4 row surgery. +Docs: F9 (`benchmarks/README.md`). + +Not in scope: running the benchmarks — that remains +`active/benchmark_calibration_runs.md`, and should follow this. diff --git a/parked.md b/parked.md index 88dda799..954f6eda 100644 --- a/parked.md +++ b/parked.md @@ -45,6 +45,16 @@ write the dated `complete///.md` record instead. the state the prompt describes. The four frozen prompt cards (`benchmarks/prompts/`) and the harness (`autoassistant/benchmark.py`) are in place; only the campaign is missing. - note: needs real model runs, so it is a cost decision rather than a coding task. +- 2026-08-18 (cloud session, read-only clone of autolens_assistant): blocker RE-VERIFIED CLEAR — + the four frozen cards and `autoassistant/benchmark.py` are on main; `benchmarks/runs/` is still + `.gitkeep` only. The campaign itself remains un-runnable from a cloud session (no push rights to + autolens_assistant, no bundled data/GPU, and the protocol requires fresh uncontaminated harness + sessions), so it stays parked pending a local run + a cost decision on model x harness combos. +- 2026-08-18: the prompt's second deliverable (the rubric calibration verdict) was taken early and + filed as `draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md` — 9 findings, + incl. an unenforced prompt-freeze rule. RESEQUENCING RECOMMENDED: land those card `version: 2` + bumps BEFORE the campaign. Scores are comparable only within a card version, so fixing the + rubrics after the runs (the order the prompt assumes) would split the tables and force a re-run. ## euclid-eceb-editorial-revision - prompt: active/euclid_eceb_editorial_revision.md From 359293ce646fe54d7f12b3e26f643b9a9eda78be Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 01:48:50 +0000 Subject: [PATCH 2/2] =?UTF-8?q?prompt:=20shelve=20benchmark-calibration-ru?= =?UTF-8?q?ns=20=E2=80=94=20campaign=20abandoned?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Author decision: the assistant benchmark calibration campaign will not be run. The dedicated issue was already closed: autolens_assistant#59 was closed on/before 2026-07-15 to release the repo claim blocking the JOSS paper task. Nothing is open on GitHub; only Mind-side residue survived, and this removes it — the prompt moves to complete/archive/shelved/, the parked.md entry and the dashboard row go. Also resolves the open question the JOSS completion record was carrying ("either results exist un-ingested or the campaign closed without producing them; worth a look before submission"). Verified against autolens_assistant main: benchmarks/runs/ holds only .gitkeep, so #59 closed without ever producing runs and nothing was un-ingested. The paper's future-tense benchmark prose is correct as written. The pre-run rubric audit filed in the previous commit is dropped as a task and preserved as evidence inside the shelved record instead, so a future revival does not rediscover it — notably that the frozen-prompt rule is enforced by no test, and that the "machine-checkable" band is never machine-checked. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GFgC9kZLtHGcbbdyodgqCx --- active/benchmark_calibration_runs.md | 65 --------- .../07/pyautolens-assistant-joss-paper.md | 1 + .../shelved/benchmark_calibration_runs.md | 94 +++++++++++++ dashboard.md | 15 +- ...enchmark_rubric_freeze_and_scoring_gaps.md | 130 ------------------ parked.md | 20 --- 6 files changed, 101 insertions(+), 224 deletions(-) delete mode 100644 active/benchmark_calibration_runs.md create mode 100644 complete/archive/shelved/benchmark_calibration_runs.md delete mode 100644 draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md diff --git a/active/benchmark_calibration_runs.md b/active/benchmark_calibration_runs.md deleted file mode 100644 index e2480e9a..00000000 --- a/active/benchmark_calibration_runs.md +++ /dev/null @@ -1,65 +0,0 @@ -# First benchmark calibration campaign — run the 4 assistant benchmarks across models - -Type: research -Target: autolens_assistant -Difficulty: medium -Autonomy: supervised -Priority: normal -Status: filed - -Follow-up to feature/autolens_assistant assistant_benchmarks (issue -https://github.com/PyAutoLabs/autolens_assistant/issues/57): the benchmark -package ships four frozen prompt cards (`benchmarks/prompts/`) and the -run/track harness (`autoassistant/benchmark.py`), but `benchmarks/runs/` is -empty. Run the first calibration campaign so the comparison tables have real -rows and the rubrics get validated against real transcripts. - -Scope: - -- Run at least the two cheap benchmarks (`teacher-basic-workflow`, - `assistant-easy-cosmos-web-ring`) on 2+ model×harness combinations (e.g. - Claude Code with two different models; add Codex/Gemini if convenient), per - the protocol in `benchmarks/README.md` — fresh sessions, prompts verbatim, - operator behaves like a real user, no coaching. -- Record every run (transcript, meta.yaml with hardware/duration, artifacts), - score with evidence, regenerate `RESULTS.md`, commit + push. -- The medium/hard benchmarks are hours-scale: run at most one of them once, - or explicitly defer them with a runtime note. -- Deliverable beyond the run records: a short calibration verdict on the - rubrics themselves — were any rows unscoreable, ambiguous, or trivially - gamed? File rubric fixes as `version`-bump proposals rather than editing - cards in place. - -Blocked until the assistant-benchmarks PRs are merged (the cards/harness must -be on main so runs execute against the published, frozen prompt versions). - -## 2026-08-18 — blocker cleared; rubric verdict taken early - -Verified against a read-only clone of autolens_assistant main: the four frozen -cards (`benchmarks/prompts/`) and `autoassistant/benchmark.py` are present, and -`benchmarks/runs/` still holds only `.gitkeep`. The blocking condition above is -therefore **cleared** and the campaign is un-started, exactly as `parked.md` -records. - -The final deliverable — the calibration verdict on the rubrics — was taken -early and filed as -`draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md` -(9 findings: an unenforced prompt-freeze rule, a "machine-checkable" band the -harness never checks, gameable and double-counted rows, and several rows whose -discriminators the operator must invent). - -**Resequence this prompt accordingly.** `benchmarks/README.md` makes scores -comparable only within one card `version`, so landing rubric fixes after the -campaign bumps the cards and splits the comparison tables — the runs would have -to be repeated. Land the harness fixes and the `version: 2` card bumps first, -then run the campaign once against v2. The remaining scope here is the runs -themselves. - -Outstanding before the campaign can start: - -- a human cost decision on which model x harness combinations to buy; -- a session with push rights to autolens_assistant, the bundled datasets and a - GPU — a cloud Mind/Brain session cannot execute the protocol; -- an operator/judge who has **not** read the rubrics (the protocol's - "no benchmark-aware behaviour" rule); the agent that produced the audit above - is disqualified as an agent-under-test for these cards. diff --git a/complete/2026/07/pyautolens-assistant-joss-paper.md b/complete/2026/07/pyautolens-assistant-joss-paper.md index be862a44..4764143b 100644 --- a/complete/2026/07/pyautolens-assistant-joss-paper.md +++ b/complete/2026/07/pyautolens-assistant-joss-paper.md @@ -11,6 +11,7 @@ - trap avoided: `audit_skill_apis.py` writes a dated report to `autoassistant/audit/` as a side effect, which `git add -A` would sweep up — already gitignored (.gitignore:30), commit stayed clean. See [[feedback_ship_workspace_binary_leak]]. - heart: shipped at YELLOW score 55 (no RED), author-acknowledged; both reasons ("workspace validation not passing (3 failed, 2026-07-09)"; "58 stale parked script(s)") pre-date the branch and are unrelated to a docs-only change. Ack recorded verbatim on the issue + in active.md. - OPEN / needs author: `State of the field` + `Research impact statement` are commented stubs (JOSS requires both; neither draftable without author judgement). Benchmark-results prose stays FUTURE-TENSE because benchmarks/RESULTS.md records ZERO runs for all 4 benchmarks — despite #59 (first calibration campaign) being CLOSED. Either results exist un-ingested or the campaign closed without producing them; worth a look before submission. +- RESOLVED 2026-08-18: the campaign closed without producing them. Verified against autolens_assistant main — `benchmarks/runs/` holds only `.gitkeep`, so nothing was un-ingested. The campaign prompt has been shelved (`complete/archive/shelved/benchmark_calibration_runs.md`); future-tense benchmark prose in the paper is correct as written. ## Original prompt diff --git a/complete/archive/shelved/benchmark_calibration_runs.md b/complete/archive/shelved/benchmark_calibration_runs.md new file mode 100644 index 00000000..2decdd66 --- /dev/null +++ b/complete/archive/shelved/benchmark_calibration_runs.md @@ -0,0 +1,94 @@ +# First benchmark calibration campaign — run the 4 assistant benchmarks across models + +Type: research +Target: autolens_assistant +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: filed + +Follow-up to feature/autolens_assistant assistant_benchmarks (issue +https://github.com/PyAutoLabs/autolens_assistant/issues/57): the benchmark +package ships four frozen prompt cards (`benchmarks/prompts/`) and the +run/track harness (`autoassistant/benchmark.py`), but `benchmarks/runs/` is +empty. Run the first calibration campaign so the comparison tables have real +rows and the rubrics get validated against real transcripts. + +Scope: + +- Run at least the two cheap benchmarks (`teacher-basic-workflow`, + `assistant-easy-cosmos-web-ring`) on 2+ model×harness combinations (e.g. + Claude Code with two different models; add Codex/Gemini if convenient), per + the protocol in `benchmarks/README.md` — fresh sessions, prompts verbatim, + operator behaves like a real user, no coaching. +- Record every run (transcript, meta.yaml with hardware/duration, artifacts), + score with evidence, regenerate `RESULTS.md`, commit + push. +- The medium/hard benchmarks are hours-scale: run at most one of them once, + or explicitly defer them with a runtime note. +- Deliverable beyond the run records: a short calibration verdict on the + rubrics themselves — were any rows unscoreable, ambiguous, or trivially + gamed? File rubric fixes as `version`-bump proposals rather than editing + cards in place. + +Blocked until the assistant-benchmarks PRs are merged (the cards/harness must +be on main so runs execute against the published, frozen prompt versions). + +## SHELVED 2026-08-18 — campaign abandoned, not deferred + +Author decision: the calibration campaign will not be run. Shelved here rather +than lost; pull back into `draft/research/autolens_assistant/` if that changes. + +**The dedicated issue was already closed.** `autolens_assistant#59` (this +campaign) was closed on/before 2026-07-15 to release the repo claim that was +blocking the JOSS paper task — see `complete/2026/07/pyautolens-assistant-joss-paper.md`, +which flagged the discrepancy as needing a look. That question is now answered: +verified against autolens_assistant main on 2026-08-18, `benchmarks/runs/` +contains only `.gitkeep`, so **#59 closed without ever producing runs**. Nothing +is open on GitHub. Only the Mind-side residue (this prompt, the `parked.md` +entry, the dashboard row) survived, and it is removed with this commit. + +**State of the benchmark package as of 2026-08-18** (in case anyone revives it): +the four frozen cards (`benchmarks/prompts/`) and the harness +(`autoassistant/benchmark.py`) are on main and working; `RESULTS.md` records +zero runs for all four benchmarks. The blocker this prompt described had +cleared — the campaign was simply never a cost the author wanted to pay. + +**Pre-run rubric findings, preserved.** A read-only audit of the cards and +harness surfaced issues that would bite any future campaign — worth reading +before re-filing rather than rediscovering: + +- The frozen-prompt rule is enforced by nothing. + `autoassistant/tests/test_benchmark.py:170-176` asserts only + `prompt.strip() in readme`, for three of the four cards; no test binds a + card's prompt text to its `version`. A coordinated README+card edit passes CI + with `version` unchanged, so `meta.yaml` keeps recording the same + `prompt_version` across a changed prompt. `hard_group_multi.md` has no freeze + check at all. +- The "Machine-checkable" band is never machine-checked. `parse_score` + (`benchmark.py:215-244`) sums whatever the operator typed; nothing inspects + `scripts/`, `output/` or figures, and "Machine rows (M*) need verifiable + evidence" (`benchmark.py:151`) is prose in the generated `score.md`, not a + check — `score_run` rejects only unfilled *Awarded* cells, never empty + Evidence. +- Rows satisfiable without the work: `easy M4/M5` reward *showing a path* + rather than the figure existing; `medium M4` gives 5 points for printing two + numbers a fabricating run could invent, while `J5` separately penalises + fabrication; `teacher M3` sets no agreement threshold, so a comparison + showing recovery failed scores full marks. +- Evidence double-counted on the hard card: `M1`/`J1` both score the quad + verification, and `M4`/`J4` both score the follow-up composition — a run that + completes M4 cannot fail J4. Note `test_repo_prompt_cards_parse` asserts + `machine + judged == 100`, so any fix needs an explicit re-split. +- Compound rows bundle orthogonal failures: `J5 "Conduct"` (10 pts, all three + assistant cards) mixes concision, fabrication and API-gate discipline. The + real-data gate is a 15-point row on `easy` but folded into a shared 10-point + row on `medium`, distorting cross-card comparison of the same behaviour. +- `medium J2` (15 pts) says the HPC option may be "set up **or offered**", + letting one sentence carry most of the row. +- Undefined discriminators: `easy M2`'s "not test-mode", `hard M3`'s proof of a + simultaneous fit, and `teacher J4`'s Euclid tolerance (stated in card prose, + not the rubric row). + +Sequencing note for any revival: scores are comparable only within a card +`version`, so rubric fixes must land *before* a campaign — fixing them +afterwards splits the comparison tables and forces a re-run. diff --git a/dashboard.md b/dashboard.md index 456bfe37..07fe108f 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,10 +8,10 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 13 | -| [Parked](#parked) (`parked.md`) | 6 | +| [In flight](#in-flight) (`active/`) | 12 | +| [Parked](#parked) (`parked.md`) | 5 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 140 | +| [Backlog](#backlog) (`draft/`) | 139 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) @@ -48,7 +48,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A Issued — each has an open GitHub issue and usually a branch. The full record for each is in [`active.md`](active.md). -- [First benchmark calibration campaign — run the 4 assistant benchmarks](active/benchmark_calibration_runs.md) - [ConstantZeroth regularization is broken twice over — dead code presenting](active/constant_zeroth_broken_dead_code.md) — [issue #448](https://github.com/PyAutoLabs/PyAutoArray/issues/448) — remote-handoff (prepared 2026-08-17 for a cloud/phone session; NO local worktree claim) - [Address ECEB editorial comments on ECLIPSE-C](active/euclid_eceb_editorial_revision.md) - [Commit the inference-methods programme plan + knowledge ledger into autolens_profiling](active/inference_programme_ledger.md) — [issue #134](https://github.com/PyAutoLabs/autolens_profiling/issues/134) — pr-open (https://github.com/PyAutoLabs/autolens_profiling/pull/135) @@ -67,12 +66,11 @@ Issued — each has an open GitHub issue and usually a branch. The full record f Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md).
-6 task(s) +5 task(s) - **blackjax-smc-gradient-kernel** — [issue #113](https://github.com/PyAutoLabs/autolens_workspace_developer/issues/113) — PARKED 2026-07-24 — stage (a) POSITIVE: warm-started gradient SMC SAMPLES (acc 0.80->0.17 across tempering, einstein_radius… - **group4-mge-search-benchmark** — [issue #82](https://github.com/PyAutoLabs/autolens_profiling/issues/82) - **matplotlib-inline-standalones** -- **benchmark-calibration-runs** - **euclid-eceb-editorial-revision** - **pyautoreduce-slacs1430-acs-comparison** @@ -97,10 +95,10 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**140** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**139** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
-bug — 41 +bug — 40 - [pre_build stages untracked files, publishing uncommitted human work](draft/bug/pyautohands/pre_build_stages_untracked_wip.md) — pyautohands · small · supervised · high - [UniformPrior bounds are not enforced in the objective on the](draft/bug/autofit/uniform_prior_bounds_unenforced_on_numpy_path.md) — autofit · medium · human-required · high @@ -133,7 +131,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. - [interferometer Delaunay pixelization — non-PD FitException in test-mode bypass](draft/bug/autolens/interferometer_delaunay_nonpd_fitexception.md) — autolens · medium · supervised · normal - [JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38](draft/bug/autolens/jax_point_source_point_smoke_sentinel.md) — autolens · medium · supervised · normal - [JIT cache not hit in modeling_visualization delaunay/rectangular scripts](draft/bug/autolens/jit_cache_not_hit_modeling_visualization.md) — autolens · medium · supervised · normal -- [Benchmark rubric calibration — freeze integrity + scoring gaps (pre-campaign)](draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md) — autolens_assistant · medium · supervised · normal - [`@PyAutoFit` Add property-based correctness tests for every `Prior` subclass](draft/bug/priors/09_prior_property_tests.md) — priors · large · supervised · normal - [`@PyAutoFit` `TransformedMessage` reversal convention is undocumented foot-gun](draft/bug/priors/11_transformed_message_semantics_doc.md) — priors · large · supervised · normal - [Resolve release-profile timeout scripts deliberately](draft/bug/health_fixes/release_timeout_policy.md) — health_fixes · too-large · supervised · normal diff --git a/draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md b/draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md deleted file mode 100644 index a4267be8..00000000 --- a/draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md +++ /dev/null @@ -1,130 +0,0 @@ -# Benchmark rubric calibration — freeze integrity + scoring gaps (pre-campaign) - -Type: bug -Target: autolens_assistant -Difficulty: medium -Autonomy: supervised -Priority: normal -Status: filed - -Pre-run calibration audit of the four benchmark cards and the scoring harness -shipped by autolens_assistant#57. Filed as `version`-bump proposals, per -`active/benchmark_calibration_runs.md` — no card is edited in place. - -**Why this is filed before the campaign, not after.** The calibration prompt -puts the rubric verdict *after* the runs. That order is the expensive one: -`benchmarks/README.md` makes scores comparable only within one card `version`, -so any fix landed after the campaign bumps the card and splits the tables — -the runs would have to be repeated to populate v2. Every finding below was -reachable by reading the cards and harness, at no run cost. Recommend landing -F1/F2 (harness, no version bump) and the F3–F8 card bumps first, then running -the campaign once, against v2. - -## F1 — the frozen-prompt rule is not enforced by anything (highest consequence) - -`autoassistant/tests/test_benchmark.py:170-176` asserts only -`prompt.strip() in readme`, for three cards. **No test binds a card's prompt -text to its `version`.** A coordinated edit to `README.md` and the card passes -CI with `version` unchanged, so every run's `meta.yaml` keeps recording the same -`prompt_version` across a changed prompt — silently violating the comparability -rule the whole benchmark rests on. `hard_group_multi.md` is not a README example -and so has no freeze check at all. - -Fix: record a checksum of the prompt block in card frontmatter -(`prompt_sha256:`) and assert it in CI; changing the text without bumping both -`version` and the checksum fails. Cheap, and it must precede the first campaign -or those records inherit the ambiguity. - -## F2 — "Machine-checkable" is a promise the harness does not keep - -`benchmark.py:215-244` (`parse_score`) splits rows on the `M` prefix and sums -whatever the operator typed. Nothing inspects `scripts/`, `output/` or the -figures. The "Machine rows (M*) need verifiable evidence" line -(`benchmark.py:151`) is prose emitted into `score.md`, not a check — the -Evidence column may be left empty and `score_run` still passes, since it rejects -only unfilled *Awarded* cells. - -Fix (either, ideally both): enforce non-empty Evidence on `M*` rows in -`score_run`; and/or rename the band to "artifact-checkable" so `RESULTS.md` -stops implying an objectivity the harness does not provide. - -## F3 — rows satisfiable without the underlying work - -- `easy M4` (10) / `M5` (5): "figure was produced **and its path shown**" — the - reward attaches to showing a path. Reword to require the file to exist at the - shown path, operator-confirmed. -- `medium M4` (5): "Both evidence comparisons reported as numbers in the final - answer" is pure output formatting, awardable to a run that fabricated the - numbers — while `J5` separately penalises fabrication. Make it conditional on - M1–M3 having been awarded. -- `teacher M3` (10): "Recovered parameters compared against input truths - explicitly" carries no agreement threshold, so a comparison demonstrating that - recovery *failed* scores full marks. That may be intended (it scores the act - of comparing) — the card must say which, or two judges will differ. - -## F4 — the same evidence scored twice (hard card) - -- `M1` (10, "quad morphology visible/verified") vs `J1` (15, "the quad verified - rather than assumed") — one behaviour, both bands. -- `M4` (5, follow-up "retaining MGE lens light and the SIE mass profiles") vs - `J4` (5, "pixelized reconstruction replaces the MGE source while the MGE lens - light and SIE masses carry over") — the same composition twice; a run that - completes M4 cannot fail J4. - -Fix: narrow M1 to dataset existence and leave verification to J1; collapse -M4/J4 into one row. Note `test_repo_prompt_cards_parse` asserts -`machine + judged == 100`, so the freed points need an explicit re-split. - -## F5 — compound rows bundling orthogonal failure modes - -- `J5 "Conduct"` (10, all three assistant cards) bundles concision + no - fabricated numbers + API-gate discipline. Fabrication is an integrity failure; - concision is style. One number cannot express "fabricated a result but wrote - concisely". Split fabrication out — arguably as a gate rather than points. -- The real-data gate is worth 15 as its own row on `easy` (J1) but is folded - into a shared 10-point conduct row on `medium` (J5). Either weighting is - defensible; the inconsistency is not, since it distorts cross-card comparison - of the same behaviour. -- `teacher J1` (20) enumerates five separate explanations in one row. - -## F6 — a 15-point row with a near-zero floor - -`medium J2`: "…HPC option **set up or offered** if the local estimate is slow." -"or offered" lets one sentence carry most of a 15-point row, and the operator -note reinforces it ("a run that stops at a well-set-up HPC handoff … can still -score highly"). Split explicitly: runtime estimate (x) / setup-or-offer quality -(y). - -## F7 — discriminators the operator has to invent - -- `easy M2`: "a completed non-linear search result exists under `output/` (**not - test-mode**)" — no stated test for "not test-mode". Name the artifact (the - `autonerves` `test_mode` flag, or a minimum `nlive`/sample count). -- `hard M3`: "one search over a shared model, not two independent fits" — name - the proof (a single summed `Analysis`, one search output directory). -- `teacher J4`: scored against "VIS-like pixel scale ~0.1"", which appears only - in the card's prose, not the rubric row. Move the tolerance into the row. - -## F8 — partial credit is allowed but unguided - -`benchmark.py:149` tells the scorer "fractions allowed" and `parse_score` -accepts any float in `0..max`. Combined with the compound rows in F5 and no -per-row allocation convention, two honest judges diverge exactly on the -heaviest rows. Decomposing the compound rows (F5) largely subsumes this; -otherwise state a convention (compound rows score in equal sub-parts). - -## F9 — the comparability rules cover the judged band only - -`benchmarks/README.md` says "Same judge for judged rows". Because the machine -band is operator-adjudicated (F2), operator identity matters there too, and -`meta.yaml` records `operator` and `score.judge` separately. Until F2 is fixed, -the rule should say both travel with the whole score. - -## Scope - -Harness/CI (no version bump): F1, F2, and the F4 re-split arithmetic. -Card `version: 2` bumps: F3, F5, F6, F7, F8, and the F4 row surgery. -Docs: F9 (`benchmarks/README.md`). - -Not in scope: running the benchmarks — that remains -`active/benchmark_calibration_runs.md`, and should follow this. diff --git a/parked.md b/parked.md index 954f6eda..e88fb8ce 100644 --- a/parked.md +++ b/parked.md @@ -36,26 +36,6 @@ write the dated `complete///.md` record instead. - note: do NOT broaden into the old `pyprojroot` bootstrap sweep — the prompt is explicit that the dependent AutoCTI follow-up owns that. -## benchmark-calibration-runs -- prompt: active/benchmark_calibration_runs.md -- parked: 2026-08-08 — surfaced by the orphaned-prompt triage; VERIFIED NOT STARTED -- parent-issue: https://github.com/PyAutoLabs/autolens_assistant/issues/57 (the benchmark package itself) -- classification: research (autolens_assistant) -- evidence: `benchmarks/runs/` on autolens_assistant main still contains only `.gitkeep`, exactly - the state the prompt describes. The four frozen prompt cards (`benchmarks/prompts/`) and the - harness (`autoassistant/benchmark.py`) are in place; only the campaign is missing. -- note: needs real model runs, so it is a cost decision rather than a coding task. -- 2026-08-18 (cloud session, read-only clone of autolens_assistant): blocker RE-VERIFIED CLEAR — - the four frozen cards and `autoassistant/benchmark.py` are on main; `benchmarks/runs/` is still - `.gitkeep` only. The campaign itself remains un-runnable from a cloud session (no push rights to - autolens_assistant, no bundled data/GPU, and the protocol requires fresh uncontaminated harness - sessions), so it stays parked pending a local run + a cost decision on model x harness combos. -- 2026-08-18: the prompt's second deliverable (the rubric calibration verdict) was taken early and - filed as `draft/bug/autolens_assistant/benchmark_rubric_freeze_and_scoring_gaps.md` — 9 findings, - incl. an unenforced prompt-freeze rule. RESEQUENCING RECOMMENDED: land those card `version: 2` - bumps BEFORE the campaign. Scores are comparable only within a card version, so fixing the - rubrics after the runs (the order the prompt assumes) would split the tables and force a re-run. - ## euclid-eceb-editorial-revision - prompt: active/euclid_eceb_editorial_revision.md - parked: 2026-08-08 — surfaced by the orphaned-prompt triage; STATE UNVERIFIED