docs(state): record PR #413 planning-docs batch (5 Idea→Plan)

state.md

@@ -2,7 +2,7 @@
 
 > Read this first. A one-page snapshot: current focus, the last few merges, what's in flight, what's queued, and where the project sits in the MVP1 → MVP2 → MVP3 → GA roadmap. **Historical feature-merge narrative + chained execution context lives in [`state_history.md`](state_history.md)** — new merge entries land there, not here (per `chore_state_md_size_compression`, 2026-05-29). Keep this file loadable in a single `Read` call.
 
-**Last updated:** 2026-06-02 (`infra_solr_smoke_stability` merged, PR #383 `d32b9714` — smoke RED on merge per D-6 fast-lane posture; diagnostics fold-in + Lever 0 perms fix + Lever 1 heap-cap shipped; remaining bottleneck — Playwright demo-ubi reseed exceeds the smoke-job budget — captured as follow-up [`infra_smoke_reseed_runtime_budget`](docs/00_overview/planned_features/02_mvp2/infra_smoke_reseed_runtime_budget/idea.md) and linked from PR body before merge.)
+**Last updated:** 2026-06-02 (planning-docs batch PR #413 `69af77eb` merged — 5 idea-only MVP2 folders advanced Idea→Plan with cross-model-reviewed spec + impl-plan pairs: `chore_studies_post_arq_spy_fixture`, `bug_judgment_header_omits_click_bucket`, `bug_baseline_phase_test_isolation`, `chore_template_library_expansion`, `chore_ubi_reader_search_after_pagination`. No source/migration; all `/impl-execute`-ready follow-ups.)
 
 ## Where the roadmap sits
 
@@ -24,16 +24,17 @@ MVP1 (v0.1) **shipped** — all six differentiators live (Bayesian/TPE optimizer
 
 Detail + reasoning for each is in [`state_history.md`](state_history.md).
 
+- **2026-06-02** — Planning-docs batch (PR #413, squash-merged `69af77eb`). Took **5 idea-only MVP2 folders** from Idea → cross-model-reviewed **feature spec + implementation plan** (`idea-preflight → spec-gen → impl-plan-gen`, each spec & plan GPT-5.5-reviewed with full adjudication logs): `chore_studies_post_arq_spy_fixture` (arq_pool_spy no-enqueue assertions, 2 stories), `bug_judgment_header_omits_click_bucket` (render the UBI `click` source bucket, 2 stories, frontend-only), `bug_baseline_phase_test_isolation` (lazy `get_settings()` read + `os.environ` leak root cause, 2 stories + regression test), `chore_template_library_expansion` (6-template library + 3 per-engine cheatsheets, 8 stories), `chore_ubi_reader_search_after_pagination` (unified `scan_all`: ES/OS PIT+search_after, Solr cursorMark, 5 stories, SearchAdapter Protocol surface). **No source code, no migration** (head stays `0022`). Gemini: 4 findings — 3 accepted (edismax `pf`/`ps`, reader Settings-injection, byte-overhead chunking), 1 rejected as stale (`UBI_EVENTS_INDEX` is a real module-level constant). All 5 are `/impl-execute`-ready follow-ups.
 - **2026-06-02** — `infra_solr_smoke_stability` (PR #383, squash-merged `d32b9714`, smoke RED on merge per D-6 fast-lane posture). Smoke-job half of the Solr CI debt (sibling of `infra_solr_ci_readiness` Phase 1 / PR #367). Diagnostics fold-in (FR-1): smoke-test job's failure-diagnostics step now captures `solr` + `opensearch` Compose logs + per-container `docker inspect` exit-state (exit/oom/error/health/started/finished). Lever 1 heap-cap (FR-2): `SOLR_HEAP_SIZE: "256m"` step-env on `make up` + `COMPOSE_PROJECT_NAME: "relyloop"` at job-level env. Plus new runbook [`docs/03_runbooks/smoke-solr-stability.md`](docs/03_runbooks/smoke-solr-stability.md) with evidence-mapped lever cascade + CLAUDE.md Key Runbooks row. **The interesting story is the inline-fix iteration during CI watch:** the spec's locked lever cascade (heap/start_period/smoke-tolerance) all assumed JVM crash modes; the diagnostics fold-in immediately surfaced THREE failure modes the cascade never anticipated — (1) Solr container exited (1) in 542ms with filesystem permissions error (`Cannot write to /var/solr as 8983:8983`); (2) Playwright `beforeAll` 30s timeout (Solr unblocked → reseed seeds Solr scenario → longer wall-clock); (3) job-level `timeout-minutes: 15` cap. All three fixed inline per the new fix-inline-by-default rule (canonical "implement-over-defer" example). The fourth iteration timed out the new 25-min cap — Playwright demo-ubi reseed exceeds any reasonable per-PR smoke budget. Captured as the follow-up [`infra_smoke_reseed_runtime_budget`](docs/00_overview/planned_features/02_mvp2/infra_smoke_reseed_runtime_budget/idea.md) (linked from PR body before merge per D-6 forcing function — three candidate fixes documented with Option A as default). 6 commits + 1 follow-up commit. No migration (head stays `0022`). Cross-model: spec GPT-5.5 3 cycles (15 → 16 → 10 findings, all accepted); plan GPT-5.5 3 cycles (13 → 10 → 6, all accepted).
 - **2026-06-01** — `infra_solr_ci_readiness` Phase 1 (PR #367, squash-merged `214cdfcd`). Makes the demo reseed **engine-tolerant** so the `pr.yml` backend job is green again without a Solr service container: the orchestrator + CLI probe each engine before dispatch (`is_engine_reachable` / `snapshot_engine_reachability`, total never-raises probe, slug-keyed, covers the 5 `SCENARIOS` + the separately-seeded rich ESCI scenario) and **skip** unreachable engines instead of `ConnectError`-ing the whole reseed. Partial completion → `status="complete"` + additive `ReseedStatusResponse.scenarios_skipped` (the `ReseedStatusLiteral` enum is unchanged) + one summary WARN; all-engines-unreachable → typed `AllEnginesUnreachableError` → `status="failed"` + stable `failed_reason="all_engines_unreachable"` token + skip list (avoids the Arq `keep_result` retry wedge). The heavy-lane `test_demo_seeding_ubi_full` now computes counts dynamically (8/8 with Solr absent, 10/10 with Solr). Dashboard reseed dialog shows a "partial completion — N scenario(s) skipped" hint → new runbook. No migration (head stays `0022`). 6 stories / 1 epic; 2095 unit + 327 contract + 998 UI vitest. Cross-model: GPT-5.5 phase-gate (5 findings — incl. the all-unreachable path not being total until the pre-loop wipes were gated) + Gemini (3, probe-caching) + GPT-5.5 final (2 fixed, 1 rejected as a stdlib-vs-structlog mis-read) — all adjudicated, CI-verified. **Phase 2 (smoke Solr stability) deferred** to `phase2_idea.md`; the `smoke` job stays red. Captured `chore_demo_reseed_partial_completion_fast_test`.
 - **2026-06-01** — MVP2 backlog batch (PR #364). Fixed the **P1 caplog-isolation bug** (`bug_backend_suite_nondeterministic_caplog_isolation`): root cause was `configure_logging()` replacing the structlog processors *list instance* on every call (not `pytest-randomly`, which isn't installed) — loggers cached against the stale instance went blind to `capture_logs()`; fixed by mutating the list in place + fixing a second polluter fixture in `test_position_bias_prior.py`; regression test added. Also fixed 3 other pre-existing bugs to green the suite: `bug_contract_allowlists_outdated_after_mvp2_features` (3 stale `'solr'`/endpoint allowlists), the `test_judgment_generate` `click:0` source-breakdown drift, and the `test_migration_0021` downgrade target (was `-1` assuming 0021 was head; head is now 0022). Landed **6 cross-model-reviewed spec/plan pairs** (idea-preflight → spec-gen → impl-plan-gen, each GPT-5.5-reviewed) for `feat_apply_path_normalizer_declaration` (design-ahead), `feat_overnight_studies_summary_card`, `feat_query_normalizer_typed_pipeline` (design-ahead), `infra_generated_artifact_freshness_gate`, `chore_arq_pool_aclose_deprecation`, `chore_cluster_detail_rung_badge`. 3168 backend tests pass; merged over 2 pre-existing Solr-CI reds (`infra_solr_ci_readiness`). Gemini: 1 finding (defensive fixture guard) accepted.
 - **2026-06-01** — `feat_study_convergence_indicator` (PR #352, squash-merged `0eee17a9`). MVP2 ergonomics feature: every completed study carries a plain-language **convergence verdict** — `converged` / `still_improving` / `too_few_trials` — backed by a best-so-far metric curve, answering "did the optimizer finish learning, or did I stop it too early?" **No migration** (reads existing `trials` columns; Alembic head stays `0022`). 11 stories / 7 epics: (1) hoist the lift epsilon to `AUTO_FOLLOWUP_LIFT_EPSILON` + an **AST/grep guard** that fails CI on any bare `0.005` in a lift/epsilon-shaped context under `backend/app/` — surfaced + collapsed a pre-existing duplicate in `chain_summary.py`; (2) pure-domain `classify_convergence(...)` → `StudyConvergenceShape | None` (trailing-window-flat, direction-aware running max/min, window clamp `min(20, max(5, total//5))`, warmup floor 50); (3) repo helper `list_complete_optuna_trials_for_study` (`status='complete' AND is_baseline IS NOT TRUE AND primary_metric IS NOT NULL`); (4) service `fetch_study_convergence` with in-flight short-circuit + invalid-direction WARN + classifier-exception shielding (GET never 500s); (5) additive `StudyDetail.convergence` (renamed class `StudyConvergenceShape` to coexist with `confidence.py`'s `ConvergenceShape` — winner-trial timing vs metric plateau — without OpenAPI name collision); (6) `ConvergencePanel` (Recharts curve + `ReferenceArea` trailing-window shade + 3 null-state badges + 3 glossary entries deep-linking the runbook) mounted on `/studies/[id]` with the `CONVERGENCE_VERDICT_VALUES` enum-discipline pair value-locked both sides; (7) digest worker + Jinja `<convergence>` block + system-prompt rule that leads with "re-run with a larger trial budget" on `still_improving` / `too_few_trials`; (8) `ConvergenceVerdict` exported as the FR-7 soft contract for the autopilot chain panel (AC-16 lives in that PR's lane — no autopilot files touched here); (9) operator runbook + CLAUDE.md Key Runbooks row + `data-model.md` / `ui-architecture.md` patches. Tests: 40 domain unit + 11 prompt unit + 5 docs unit + 18 integration + 4 contract + 13 frontend vitest + 1 Playwright smoke; mypy clean. Cross-model: Gemini (1 Medium accepted — `is False`→`not getattr` None-safety, `644feeed`) + GPT-5.5 final (4 findings: SQL/domain filter-mirror `is_not(True)` accepted `ad72e297`; 3 rejected as review-window-truncation artifacts). Captured `bug_contract_allowlists_outdated_after_mvp2_features` (3 pre-existing contract failures from prior MVP2 PRs not updating hand-maintained allowlists).
-- **2026-05-31** — `feat_demo_reseed_solr_and_steplog` (PR #348, squash-merged `66323aba`). Completes the deferred `infra_adapter_solr` Story A13 — the async home-button demo reseed (`make seed-demo`) now seeds the **Solr** scenario (`acme-kb-docs-solr`) end-to-end, where before it crashed. Fixed layer by layer (each verified by the live reseed getting further): Solr host-URL→Compose-DNS mapping (`8983`); the dispatcher now creates the Solr collection from its configset + bulk-indexes (reusing `seed_solr_products`) instead of the ES `index_mapping` PUT; engine-aware **synthetic-UBI write** (`ubi_queries`/`ubi_events` Solr collections + Solr `/update`, per-engine ensure gate); study **search-space cardinality** (Solr scenario tunes 2 boosts, `estimate_cardinality` floats=100 → ≤3-float cap). **Surfaced + fixed a real product bug:** the UBI **read** path (`ubi_readiness`, `UbiReader`) built Elasticsearch query DSL → `UBI judgment generation was broken on Solr for every operator` (contradicting the "works everywhere from day one" claim); now builds Solr `q`/`fq`/`{!terms}`/`rows`/`fl` when `adapter.engine_type=="solr"`. Plus a **reseed step-history log** (worker accumulates `steps[]` → status endpoint → scrolling UI panel). **Verified by a live `make seed-demo` completing 6/6** (10 studies, 10 proposals). 8 commits; backend 2012 unit + frontend 983 vitest. Gemini: 3 findings accepted+fixed (`{!terms}` query_id vs maxBooleanClauses — verified live; collision-safe synthetic ids; scroll-on-reopen dep). Final GPT-5.5 review clean. Captured `bug_reseed_failure_blocks_retry_arq_singleton_dedup` (a failed reseed deduped retries for ~1h via the Arq singleton result).
-_(older entries — full narrative in [`state_history.md`](state_history.md): `feat_overnight_autopilot` PR #343, `infra_adapter_solr` PR #336, …)_
+_(older entries — full narrative in [`state_history.md`](state_history.md): `feat_demo_reseed_solr_and_steplog` PR #348, `feat_overnight_autopilot` PR #343, `infra_adapter_solr` PR #336, …)_
 
 ## In flight
 
-- _None._ `infra_solr_smoke_stability` (PR #383) merged 2026-06-02. The remaining smoke debt is the Playwright reseed runtime budget, captured as P1 Idea-stage folder `02_mvp2/infra_smoke_reseed_runtime_budget/` (three candidate fixes documented; Option A — skip demo-ubi on smoke — recommended as the inline-cheap default). The 6 spec/plan pairs from PR #364 remain at **Plan** stage in `02_mvp2/` (two are design-ahead: `feat_apply_path_normalizer_declaration` + `feat_query_normalizer_typed_pipeline`, both gated on `feat_query_normalization_tuning` Phase 1 merging — do not `/impl-execute` until then).
+- _None._ `infra_solr_smoke_stability` (PR #383) merged 2026-06-02. The remaining smoke debt is the Playwright reseed runtime budget, captured as P1 Idea-stage folder `02_mvp2/infra_smoke_reseed_runtime_budget/` (three candidate fixes documented; Option A — skip demo-ubi on smoke — recommended as the inline-cheap default).
+- **Plan-stage, `/impl-execute`-ready (no gates):** PR #413 (2026-06-02) added 5 spec/plan pairs in `02_mvp2/`: `chore_studies_post_arq_spy_fixture`, `bug_judgment_header_omits_click_bucket`, `bug_baseline_phase_test_isolation`, `chore_template_library_expansion`, `chore_ubi_reader_search_after_pagination`. Plus the 6 pairs from PR #364 — of which two are **design-ahead** (`feat_apply_path_normalizer_declaration` + `feat_query_normalizer_typed_pipeline`, both gated on `feat_query_normalization_tuning` Phase 1 merging — do not `/impl-execute` until then); the other four (`feat_overnight_studies_summary_card`, `infra_generated_artifact_freshness_gate`, `chore_arq_pool_aclose_deprecation`, `chore_cluster_detail_rung_badge`) are ungated.
 
 ## Queued (priority-ordered by dashboard / dep graph)
 

state_history.md

@@ -4,6 +4,77 @@
 
 ---
 
+### Planning-docs batch — 5 idea-only MVP2 folders advanced Idea→Plan (PR #413, 2026-06-02)
+
+Squash-merged via [PR #413](https://github.com/SoundMindsAI/relyloop/pull/413)
+(merge commit `69af77eb`). A pure planning-docs PR — **no source code, no
+migration** (Alembic head stays `0022`) — mirroring the PR #364 precedent of
+landing several cross-model-reviewed spec/plan pairs in one docs PR. Five
+previously idea-only folders under `02_mvp2/` were each taken through the full
+`idea-preflight → spec-gen → impl-plan-gen` pipeline, with the mandatory GPT-5.5
+cross-model review run on **both** the spec and the plan for each (full
+adjudication logs live in the artifacts). Generated in parallel via five
+subagents, each scoped to its own folder; bundled + reviewed + merged by the
+main loop.
+
+The five, with plan shape and review intensity:
+
+- **`chore_studies_post_arq_spy_fixture`** — an `arq_pool_spy` fixture so
+  studies-POST integration tests can positively assert "no Arq job enqueued" on
+  rejection paths. 2 stories / 1 epic, no migration. The load-bearing
+  preflight correction: the studies-POST tests do NOT run without a pool — the
+  `async_client` fixture builds the real Arq pool via `LifespanManager` against
+  CI Redis, so the spy must **replace** the pool *after* lifespan, not supply a
+  missing one. GPT-5.5: spec 3 cycles 4/4 accepted, plan 3 cycles 4/4 accepted.
+- **`bug_judgment_header_omits_click_bucket`** — render the `click` (UBI)
+  source bucket in the judgment-list header (it currently shows only
+  `llm`/`human`). Pure render gap — the `click` bucket is already on the wire
+  (`schemas.py` `_SourceBreakdown`) and in `types.ts`. 2 stories / 1 epic,
+  frontend-only (vitest + real-backend Playwright). GPT-5.5: spec 2 cycles 4/4
+  accepted, plan 1 clean cycle.
+- **`bug_baseline_phase_test_isolation`** — fix the suite-ordering dependency
+  in `test_orchestrator_baseline_phase.py::TestComputeBaselineWaitS`
+  (passes in the full run, fails in isolation). Confirmed two-layer root cause:
+  (1) `_compute_baseline_wait_s` calls `get_settings()` even when an explicit
+  `trial_timeout_s` is present; (2) `test_main_lifespan.py` leaks
+  `DATABASE_URL_FILE`/`POSTGRES_PASSWORD_FILE` into `os.environ` at collection
+  time. Durable fix = lazy settings read; the env leak is documented but left
+  out of scope. 2 stories / 1 epic + a regression test that fails pre-fix.
+  GPT-5.5: spec 2 cycles 2/2 accepted, plan 2 cycles (1 finding rejected with
+  counter-evidence — `Any` already imported).
+- **`chore_template_library_expansion`** — a curated 6-template library +
+  3 per-engine tunable-params cheatsheets. 8 stories / 3 epics, no migration.
+  The two High GPT-5.5 findings reshaped the feature: a single ES-native hybrid
+  template is invalid on OpenSearch, and the render context injects no query
+  vector — so kNN + hybrid-RRF were demoted from runnable templates to
+  engine-correct **reference snippets** in the cheatsheets. Locked runnable set:
+  ES/OS `multi_match_basic`/`function_score_decay`/`bool_boosted`/`rescore_phrase`
+  + Solr `edismax_basic`/`boost_decay`. GPT-5.5: spec 7 cycles 12/12 accepted,
+  plan 5 cycles 8/8 accepted.
+- **`chore_ubi_reader_search_after_pagination`** — exact full-traffic UBI
+  aggregation via a new engine-neutral `SearchAdapter.scan_all` + `close_scan`
+  returning an opaque round-tripped cursor (ES/OS: PIT + `search_after` over
+  `[timestamp,_shard_doc]`, one impl with internal PIT-endpoint branch; Solr:
+  `cursorMark` via `POST /select`) — replacing the 10k-sample cap. Keeps
+  Absolute Rule #4 intact (reader never branches on engine for pagination).
+  5 stories / 3 epics, no migration. GPT-5.5: spec 5 cycles 16/16 accepted,
+  plan 5 cycles 14/14 accepted.
+
+Gemini Code Assist left 4 line-level findings on the planning docs; adjudicated
+3 accepted + 1 rejected (in fix commit `56a888ab`): (1) bake `pf` into Solr
+`edismax_basic.j2` so the tunable `ps` is not a no-op [accept]; (2) the reader
+sketch's bare `settings.X` is out of scope — fixed via constructor injection
+rather than the suggested `get_settings()`, preserving `UbiReader`'s deliberate
+decoupling from `Settings` [accept, remedy adjusted]; (3) byte-length chunking
+measures the fully-serialized filter fragment, not summed raw id lengths
+[accept]; (4) `UBI_EVENTS_INDEX` "NameError" [reject — it's a module-level
+constant at `ubi_reader.py:75`, already used directly at L515]. Heavy `pr.yml`
+is path-filtered out for docs-only changes; merged on the docs fast lane with
+DCO + secrets-defense green. All five are now `/impl-execute`-ready follow-ups
+(none gated, unlike PR #364's two design-ahead pairs).
+
+---
+
 ### infra_solr_ci_readiness Phase 1 — engine-tolerant demo reseed unblocks the pr.yml backend job (PR #367, 2026-06-01)
 
 The `pr.yml` **backend** job had been red on every branch since Apache Solr