prompt: shelve benchmark-calibration-runs — campaign abandoned - #201
Merged
Conversation
…marks Verified against autolens_assistant main that the blocker on active/benchmark_calibration_runs.md is clear (the four frozen cards and autoassistant/benchmark.py are on main) and that benchmarks/runs/ is still empty, so the campaign is genuinely un-started. Took the prompt's second deliverable — the calibration verdict on the rubrics — early, since every finding was reachable by reading the cards and harness at no run cost. Filed as a version-bump proposal rather than editing cards in place, per the prompt. Findings include an unenforced prompt-freeze rule (no test binds a card's prompt text to its version, so a coordinated README+card edit keeps recording the same prompt_version across a changed prompt), a "machine-checkable" band the harness never actually checks, rows satisfiable without the underlying work, and evidence double-counted across bands on the hard card. Resequences the campaign: scores are comparable only within a card version, so landing rubric fixes after the runs would split the comparison tables and force a re-run. Fix first, run once against v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GFgC9kZLtHGcbbdyodgqCx
Author decision: the assistant benchmark calibration campaign will not be run.
The dedicated issue was already closed: autolens_assistant#59 was closed
on/before 2026-07-15 to release the repo claim blocking the JOSS paper task.
Nothing is open on GitHub; only Mind-side residue survived, and this removes it
— the prompt moves to complete/archive/shelved/, the parked.md entry and the
dashboard row go.
Also resolves the open question the JOSS completion record was carrying
("either results exist un-ingested or the campaign closed without producing
them; worth a look before submission"). Verified against autolens_assistant
main: benchmarks/runs/ holds only .gitkeep, so #59 closed without ever
producing runs and nothing was un-ingested. The paper's future-tense benchmark
prose is correct as written.
The pre-run rubric audit filed in the previous commit is dropped as a task and
preserved as evidence inside the shelved record instead, so a future revival
does not rediscover it — notably that the frozen-prompt rule is enforced by no
test, and that the "machine-checkable" band is never machine-checked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GFgC9kZLtHGcbbdyodgqCx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Author decision: the assistant benchmark calibration campaign will not be run. This removes its Mind-side residue.
The issue was already closed
autolens_assistant#59("first calibration campaign") was closed on/before 2026-07-15, to release the repo claim that was blocking the JOSS paper task. Nothing was open on GitHub — the campaign has been dead since July. The only issue the registry still referenced was#57, the parent (the benchmark package itself), which shipped in July and is untouched here.What survived was drift: the prompt sitting in
active/, aparked.mdentry, and a dashboard row.Changes
active/benchmark_calibration_runs.md→complete/archive/shelved/— the repo's home for deliberately-shelved prompts, skipped bylifecycle.py check/indexparked.md—benchmark-calibration-runsentry removeddashboard.md— regenerated (140 → 139 prompts); zero references remainResolves an open question in the JOSS record
complete/2026/07/pyautolens-assistant-joss-paper.mdhad been carrying an unresolved flag since July: benchmark prose was left future-tense becauseRESULTS.mdshowed zero runs despite#59being closed, noted as "either results exist un-ingested or the campaign closed without producing them; worth a look before submission."Verified against
autolens_assistantmain:benchmarks/runs/contains only.gitkeep, so #59 closed without ever producing runs — nothing was un-ingested. The resolution is recorded on that record; the paper's future-tense benchmark prose is correct as written.Rubric findings preserved, not filed as work
A read-only audit of the four cards and
autoassistant/benchmark.pysurfaced issues that would bite any future campaign. They are preserved inside the shelved record rather than kept as a live task, so a revival does not rediscover them. Two are real defects in the shipped package independent of whether benchmarks ever run:autoassistant/tests/test_benchmark.py:170-176asserts onlyprompt.strip() in readme, for three of four cards; no test binds a card's prompt text to itsversion. A coordinated README+card edit passes CI withversionunchanged, someta.yamlkeeps recording the sameprompt_versionacross a changed prompt.parse_scoresums whatever the operator typed; the "M rows need verifiable evidence" line is prose in the generatedscore.md, not a check.Plus gameable rows, evidence double-counted across bands on the hard card, and several undefined discriminators — all detailed in the shelved record.
Verification
python3 scripts/lifecycle.py check→ OKdashboard.mdregenerated viapyauto-brain intake --apply dashboard, not hand-edited🤖 Generated with Claude Code
https://claude.ai/code/session_01GFgC9kZLtHGcbbdyodgqCx
Generated by Claude Code