diff --git a/CLAUDE.md b/CLAUDE.md index e66ebb20..886684a7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -424,7 +424,7 @@ If you slip and a stub leaks into a committed file, capture it as a `bug_` | 3 | [`infra_optuna_eval`](docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/) | **Complete (PR #23, merged 2026-05-10)** | | 4 | [`feat_study_lifecycle`](docs/00_overview/implemented_features/2026_05_10_feat_study_lifecycle/) | **Complete — Phase 1 (Schema) PR #18 + Phase 2 (Orchestrator + API) PR #25, both merged 2026-05-10/11** | | 5 | [`feat_llm_judgments`](docs/00_overview/implemented_features/2026_05_11_feat_llm_judgments/) | **Complete (PR #35, merged 2026-05-11)** | -| 6 | [`feat_digest_proposal`](docs/02_product/planned_features/feat_digest_proposal/) | Implementation complete on `feature/feat-digest-proposal`; PR pending | +| 6 | [`feat_digest_proposal`](docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/) | **Complete (PR #41, merged 2026-05-11)** | | 7 | [`feat_github_pr_worker`](docs/02_product/planned_features/feat_github_pr_worker/) | Spec approved, plan pending | | 8 | [`feat_github_webhook`](docs/02_product/planned_features/feat_github_webhook/) | Spec approved, plan pending | | 9 | [`feat_studies_ui`](docs/02_product/planned_features/feat_studies_ui/) | Spec approved, plan pending | diff --git a/docs/00_overview/MVP1_DASHBOARD.md b/docs/00_overview/MVP1_DASHBOARD.md index db055cb1..9e9f27b4 100644 --- a/docs/00_overview/MVP1_DASHBOARD.md +++ b/docs/00_overview/MVP1_DASHBOARD.md @@ -6,8 +6,8 @@ _Reflects feature-folder state as of **2026-05-11** (latest mtime of any planned | Metric | Value | |---|---| -| Features done | **5 / 12** (42%) | -| Path to MVP1 | **16** items remaining (features + bugs + chores) | +| Features done | **6 / 12** (50%) | +| Path to MVP1 | **15** items remaining (features + bugs + chores) | | Open bugs | 3 | | Open chores | 6 (idea-stage debt) | | Backlog ideas | 4 idea-only feat/infra (not yet scoped into MVP1) | @@ -15,10 +15,11 @@ _Reflects feature-folder state as of **2026-05-11** (latest mtime of any planned ## Pipeline -### Done (5) +### Done (6) | Feature | Type | One-liner | Depends on | Status | |---|---|---|---|---| +| [feat_digest_proposal](implemented_features/2026_05_11_feat_digest_proposal/feature_spec.md) | Feature | When a study transitions to `completed`, the digest worker generates: a narrative summary (LLM-authored), a parameter-importance map (computed by `optuna.importance`), and a recommended config. | `feat_study_lifecycle` `feat_llm_judgments` | [PR #41](https://github.com/SoundMindsAI/relyloop/pull/41) merged 2026-05-11 | | [feat_llm_judgments](implemented_features/2026_05_11_feat_llm_judgments/feature_spec.md) | Feature | A relevance engineer selects a query set + cluster + target + rubric and the system runs the current template to fetch top-K hits per query, asks OpenAI to rate each (query, doc) on a 0–3 scale with r | `infra_foundation` `infra_adapter_elastic` `feat_study_lifecycle` | [PR #35](https://github.com/SoundMindsAI/relyloop/pull/35) merged 2026-05-11 | | [feat_study_lifecycle](implemented_features/2026_05_10_feat_study_lifecycle/feature_spec.md) | Feature | A relevance engineer creates a study via API or chat, the orchestrator enqueues N parallel `run_trial` jobs, trials accumulate in real time on the study detail page, the orchestrator detects stop-cond | — | [PR #18](https://github.com/SoundMindsAI/relyloop/pull/18) merged 2026-05-10 | | [infra_adapter_elastic](implemented_features/2026_05_10_infra_adapter_elastic/feature_spec.md) | Infra | A single `ElasticAdapter` implements the `SearchAdapter` Protocol and serves both Elasticsearch (8.11+ / 9.x) and OpenSearch (2.x / 3.x), distinguished by a `engine_type` column. | — | [PR #16](https://github.com/SoundMindsAI/relyloop/pull/16) merged 2026-05-10 | @@ -29,11 +30,9 @@ _Reflects feature-folder state as of **2026-05-11** (latest mtime of any planned _None._ -### Plan (1) +### Plan (0) -| Feature | Type | One-liner | Depends on | Status | -|---|---|---|---|---| -| [feat_digest_proposal](../02_product/planned_features/feat_digest_proposal/feature_spec.md) | Feature | When a study transitions to `completed`, the digest worker generates: a narrative summary (LLM-authored), a parameter-importance map (computed by `optuna.importance`), and a recommended config. | `feat_study_lifecycle` `feat_llm_judgments` | [PR #25](https://github.com/SoundMindsAI/relyloop/pull/25) merged 2026-05-11 | +_None._ ### Spec (6) @@ -79,8 +78,6 @@ graph LR class chore_tutorial_polish spec; feat_chat_agent["chat agent"] class feat_chat_agent spec; - feat_digest_proposal["digest proposal"] - class feat_digest_proposal plan; feat_github_pr_worker["github pr worker"] class feat_github_pr_worker spec; feat_github_webhook["github webhook"] @@ -97,10 +94,10 @@ graph LR class infra_adapter_elastic done; infra_optuna_eval["optuna eval"] class infra_optuna_eval done; + feat_digest_proposal["digest proposal"] + class feat_digest_proposal done; feat_llm_judgments["llm judgments"] class feat_llm_judgments done; - feat_study_lifecycle --> feat_digest_proposal - feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_github_pr_worker infra_adapter_elastic --> feat_github_pr_worker feat_digest_proposal --> feat_github_pr_worker @@ -115,6 +112,8 @@ graph LR feat_digest_proposal --> feat_studies_ui feat_llm_judgments --> feat_studies_ui infra_adapter_elastic --> feat_studies_ui + feat_study_lifecycle --> feat_digest_proposal + feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_llm_judgments infra_adapter_elastic --> feat_llm_judgments feat_study_lifecycle --> feat_llm_judgments diff --git a/docs/02_product/planned_features/feat_digest_proposal/feature_spec.md b/docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/feature_spec.md similarity index 100% rename from docs/02_product/planned_features/feat_digest_proposal/feature_spec.md rename to docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/feature_spec.md diff --git a/docs/02_product/planned_features/feat_digest_proposal/implementation_plan.md b/docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/implementation_plan.md similarity index 99% rename from docs/02_product/planned_features/feat_digest_proposal/implementation_plan.md rename to docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/implementation_plan.md index 23ab2ee1..d77f6a20 100644 --- a/docs/02_product/planned_features/feat_digest_proposal/implementation_plan.md +++ b/docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/implementation_plan.md @@ -1,7 +1,7 @@ # Implementation Plan — feat_digest_proposal **Date:** 2026-05-11 -**Status:** Approved (3 GPT-5.5 review cycles to the cap; 20 findings accepted + applied; ready for `/impl-execute`) +**Status:** Complete (PR #41, merged 2026-05-11 squash commit `3753894`) **Primary spec:** [feature_spec.md](feature_spec.md) **Policy source(s):** [CLAUDE.md](../../../../CLAUDE.md), [docs/01_architecture/data-model.md](../../../01_architecture/data-model.md), [docs/01_architecture/llm-orchestration.md](../../../01_architecture/llm-orchestration.md), [docs/01_architecture/api-conventions.md](../../../01_architecture/api-conventions.md) diff --git a/docs/02_product/planned_features/feat_digest_proposal/pipeline_status.md b/docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/pipeline_status.md similarity index 80% rename from docs/02_product/planned_features/feat_digest_proposal/pipeline_status.md rename to docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/pipeline_status.md index a6a46ba4..5851a1c4 100644 --- a/docs/02_product/planned_features/feat_digest_proposal/pipeline_status.md +++ b/docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/pipeline_status.md @@ -23,7 +23,18 @@ Study-end digest narrative + proposal-population pipeline. Replaces the `digest_ - Tests planned: 3 unit + 26 integration + 1 contract + 1 benchmark = 31 test files. ## Implementation -- Status: Not started. Next: `/impl-execute docs/02_product/planned_features/feat_digest_proposal/implementation_plan.md --all`. +- Status: **Complete** — merged 2026-05-11. +- PR: [#41](https://github.com/SoundMindsAI/relyloop/pull/41) (squash commit `3753894`). +- CI: green (4 cycles — initial fail on test seed/setting/migration order, AC-11 fix, final-review fix, all green). +- Stories completed: 12 of 12 across 4 epics. +- Test counts: 350 unit + 26 integration + 1 contract + 1 benchmark. +- Final cross-model review: GPT-5.5 raised 6 findings; 5 accepted + applied, 1 rejected with cited counter-evidence (full adjudication on PR #41). +- Gemini Code Assist: N/A on this repo. + +## Done +- Status: **Deployed** (no remote staging in MVP1; merge to main IS the deploy event). +- Date: 2026-05-11 +- Follow-up tracked: [`bug_digest_param_importance_seam`](../bug_digest_param_importance_seam/idea.md) — AC-7 test fixture seam (xfail in this PR). ## Dependencies (all satisfied) diff --git a/docs/00_overview/mvp1_dashboard.html b/docs/00_overview/mvp1_dashboard.html index 77ed6fab..6d37307c 100644 --- a/docs/00_overview/mvp1_dashboard.html +++ b/docs/00_overview/mvp1_dashboard.html @@ -267,13 +267,13 @@

MVP1 Progress

Features done
-
5 / 12
-
42% of scoped MVP1 features
-
+
6 / 12
+
50% of scoped MVP1 features
+
Path to MVP1
-
16
+
15
items left = features + bugs + chores
@@ -547,28 +547,29 @@

Spec 6

-

Plan 1

+

Plan 0

+ +
+ +
+

Implementing 0

+ +
+ +
+

Done 6

-
Digest Proposal
+
Digest Proposal
Feature - PR #25merged 2026-05-11 + PR #41merged 2026-05-11
When a study transitions to `completed`, the digest worker generates: a narrative summary (LLM-authored), a parameter-importance map (computed by `optuna.importance`), and a recommended config.
depends on: feat_study_lifecyclefeat_llm_judgments
-
- -
-

Implementing 0

- -
- -
-

Done 5

Llm Judgments
@@ -646,8 +647,6 @@

Dependency graph (feat_ + infra_)

class chore_tutorial_polish spec; feat_chat_agent["chat agent"] class feat_chat_agent spec; - feat_digest_proposal["digest proposal"] - class feat_digest_proposal plan; feat_github_pr_worker["github pr worker"] class feat_github_pr_worker spec; feat_github_webhook["github webhook"] @@ -664,10 +663,10 @@

Dependency graph (feat_ + infra_)

class infra_adapter_elastic done; infra_optuna_eval["optuna eval"] class infra_optuna_eval done; + feat_digest_proposal["digest proposal"] + class feat_digest_proposal done; feat_llm_judgments["llm judgments"] class feat_llm_judgments done; - feat_study_lifecycle --> feat_digest_proposal - feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_github_pr_worker infra_adapter_elastic --> feat_github_pr_worker feat_digest_proposal --> feat_github_pr_worker @@ -682,6 +681,8 @@

Dependency graph (feat_ + infra_)

feat_digest_proposal --> feat_studies_ui feat_llm_judgments --> feat_studies_ui infra_adapter_elastic --> feat_studies_ui + feat_study_lifecycle --> feat_digest_proposal + feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_llm_judgments infra_adapter_elastic --> feat_llm_judgments feat_study_lifecycle --> feat_llm_judgments
@@ -695,8 +696,6 @@

Dependency graph (feat_ + infra_)

class chore_tutorial_polish spec; feat_chat_agent["chat agent"] class feat_chat_agent spec; - feat_digest_proposal["digest proposal"] - class feat_digest_proposal plan; feat_github_pr_worker["github pr worker"] class feat_github_pr_worker spec; feat_github_webhook["github webhook"] @@ -713,10 +712,10 @@

Dependency graph (feat_ + infra_)

class infra_adapter_elastic done; infra_optuna_eval["optuna eval"] class infra_optuna_eval done; + feat_digest_proposal["digest proposal"] + class feat_digest_proposal done; feat_llm_judgments["llm judgments"] class feat_llm_judgments done; - feat_study_lifecycle --> feat_digest_proposal - feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_github_pr_worker infra_adapter_elastic --> feat_github_pr_worker feat_digest_proposal --> feat_github_pr_worker @@ -731,6 +730,8 @@

Dependency graph (feat_ + infra_)

feat_digest_proposal --> feat_studies_ui feat_llm_judgments --> feat_studies_ui infra_adapter_elastic --> feat_studies_ui + feat_study_lifecycle --> feat_digest_proposal + feat_llm_judgments --> feat_digest_proposal infra_foundation --> feat_llm_judgments infra_adapter_elastic --> feat_llm_judgments feat_study_lifecycle --> feat_llm_judgments diff --git a/state.md b/state.md index 56c2e0d9..e4167f02 100644 --- a/state.md +++ b/state.md @@ -2,20 +2,21 @@ > Read this first. Snapshots the active branch, what just shipped, what's in flight, what's queued, and where the project currently sits in the MVP1 → GA roadmap. Updated whenever a feature lands or a priority shifts. -**Last updated:** 2026-05-11 (after `feat_digest_proposal` implementation complete, PR pending) +**Last updated:** 2026-05-11 (after `feat_digest_proposal` merged via PR #41) --- ## Current branch / execution context -- **Branch:** `feature/feat-digest-proposal` — 12 stories complete + Epic 4 docs/tests; PR not yet opened. -- **Active feature:** `feat_digest_proposal` in flight (study-end digest narrative + populates the pending proposal row). +- **Branch:** `main` — `feat_digest_proposal` merged via PR #41 (squash commit `3753894`); finalization happening on `docs/finalize-digest-proposal`. +- **Active feature:** none in flight. Next-up: `feat_github_pr_worker` (now consumes the `proposals` shape that `feat_digest_proposal` populates). - **Alembic head:** `0005_digests` (bumped by Story 1.1 of `feat_digest_proposal`). - **Coverage:** above the 80% gate. 350 unit tests + 2 xpassed + 80+ integration (including 26 new for digest worker + API) + 6 contract (including 1 new for digest+proposals API). ## Most recent meaningful changes (newest first) -- **2026-05-11 — `feat_digest_proposal` implementation complete (PR pending)** on `feature/feat-digest-proposal`. 12 stories across 4 epics covering FR-1..FR-6 + FR-2b + AC-1..AC-11. **One migration (`0005_digests`).** Plan went through **3 GPT-5.5 review cycles** (20 findings, all accepted + applied) before execution. +- **2026-05-11 — `feat_digest_proposal` merged into `main`** as PR #41 (squash commit `3753894`). 12 stories across 4 epics covering FR-1..FR-6 + FR-2b + AC-1..AC-11. **One migration (`0005_digests`).** Plan went through **3 GPT-5.5 review cycles** (20 findings, all accepted + applied) before execution. Final GPT-5.5 review raised 6 findings on the merged PR diff; 5 accepted + applied (commit `6267cc8`), 1 rejected with cited counter-evidence (`git diff main..HEAD --stat` showed the docs WERE in the diff — false positive caused by code-only diff slice). + - **CI cycles:** 4 total — initial push (FK ordering in seed helper + settings pollution between tests + post-0005 downgrade-target wrong on `test_judgments_migration.py`) → fix-1 (commit `81d512e`) → still failing on AC-11 capability fallback persisting wrong recommended_config → fix-2 (commit `83fd8c6`) → final-review fixes (commit `6267cc8`) → green. - **Epic 1 (foundations, 3 stories):** `0005_digests` migration creating the `digests` table per data-model.md (`id`, `study_id UNIQUE`, `narrative`, `parameter_importance JSONB`, `recommended_config JSONB`, `suggested_followups TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[]` per cycle-1 F1, `generated_by`, `generated_at`); `Digest` ORM model exported via `db/models/__init__.py`; new `db/repo/digest.py` (`create_digest`, `get_digest_for_study`) + 5 proposal repo extensions per FR-6 (`update_proposal_for_digest` conditional on `WHERE status='pending'` per cycle-3 F4, `list_proposals_paginated`, `count_proposals`, `reject_proposal` raises `InvalidStateTransition`, `list_pending_proposals_for_boot_scan`); `prompts/digest_narrative.system.md` + `digest_narrative.user.jinja` with `include_recommendation` toggle for the degraded path (cycle-3 F3); `backend/app/llm/digest_prompt.py` with `lru_cache`-cached `DigestPromptBundle` + `SandboxedEnvironment(autoescape=True)` mirroring `prompt_loader.py`. - **Epic 2 (worker, 2 stories):** `backend/workers/digest.py:generate_digest` (15-step contract) REPLACES `backend/workers/digest_stub.py` (file deleted) under the same Arq job name so the orchestrator's enqueue at `orchestrator.py:370` keeps firing. Key cycle revisions: pre-LLM idempotency guard via `get_digest_for_study` (cycle-1 F6); `pg_try_advisory_xact_lock` keyed on `blake2b("digest:{sid}")` held across LLM call + persist tx (cycle-2 F6 — prefix disjoint from orchestrator's replenish lock); zero-trials short-circuit moved BEFORE OpenAI preflights (cycle-2 F5 — AC-2 fires regardless of config); capability fallback as a mode flag, NOT short-circuit (cycle-3 F2 — pricing + budget still gate); deterministic `recommended_config` (cycle-1 F5 / cycle-2 F1 — best-trial params filtered to currently-declared template params; all-dropped sub-case persists empty-recommendation digest + DELETEs the pending proposal per cycle-2 F7); persist FIRST then `_safe_record_cost` (cycle-2 C2-F3 from feat_llm_judgments); conditional UPDATE on the pending proposal handles operator-reject-mid-LLM race (cycle-3 F4 — benign no-op). `WorkerSettings.on_startup` extended with the FR-2b boot scan (list_pending_proposals_for_boot_scan + deterministic `_job_id="generate_digest:{sid}"`). - **Epic 3 (API, 4 stories, 5 endpoints):** `backend/app/api/v1/proposals.py` (new router; registered in `main.py` alongside the existing v1 routers): `GET /studies/{id}/digest` (FR-3 / AC-3 / AC-4 — 404 STUDY_NOT_FOUND vs DIGEST_NOT_READY); `POST /proposals` (FR-4 / AC-6 — manual creation, study_id NULL); `GET /proposals` (cursor + status Literal + cluster_id filter + X-Total-Count); `GET /proposals/{id}` (detail with inline `study_summary` + `digest` to spare UI a fan-out query); `POST /proposals/{id}/reject` (FR-4 / AC-5 — pending → rejected; 409 INVALID_STATE_TRANSITION via repo's `InvalidStateTransition`). `ProposalStatusWire` and `ProposalPrStateWire` Literals carry source-of-truth comments per cycle-2 F4 / cycle-3 F1. @@ -166,7 +167,7 @@ ## In flight -- **`feat_digest_proposal`** on `feature/feat-digest-proposal` — 12 stories complete; PR pending. After merge, the queued list collapses to start with `feat_github_pr_worker`. +- None. Next feature pulled from the Queued list below. ## Queued (priority-ordered by dependency)