Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -424,7 +424,7 @@ If you slip and a stub leaks into a committed file, capture it as a `bug_<slug>`
| 3 | [`infra_optuna_eval`](docs/00_overview/implemented_features/2026_05_10_infra_optuna_eval/) | **Complete (PR #23, merged 2026-05-10)** |
| 4 | [`feat_study_lifecycle`](docs/00_overview/implemented_features/2026_05_10_feat_study_lifecycle/) | **Complete — Phase 1 (Schema) PR #18 + Phase 2 (Orchestrator + API) PR #25, both merged 2026-05-10/11** |
| 5 | [`feat_llm_judgments`](docs/00_overview/implemented_features/2026_05_11_feat_llm_judgments/) | **Complete (PR #35, merged 2026-05-11)** |
| 6 | [`feat_digest_proposal`](docs/02_product/planned_features/feat_digest_proposal/) | Implementation complete on `feature/feat-digest-proposal`; PR pending |
| 6 | [`feat_digest_proposal`](docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/) | **Complete (PR #41, merged 2026-05-11)** |
| 7 | [`feat_github_pr_worker`](docs/02_product/planned_features/feat_github_pr_worker/) | Spec approved, plan pending |
| 8 | [`feat_github_webhook`](docs/02_product/planned_features/feat_github_webhook/) | Spec approved, plan pending |
| 9 | [`feat_studies_ui`](docs/02_product/planned_features/feat_studies_ui/) | Spec approved, plan pending |
Expand Down
21 changes: 10 additions & 11 deletions docs/00_overview/MVP1_DASHBOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,20 @@ _Reflects feature-folder state as of **2026-05-11** (latest mtime of any planned

| Metric | Value |
|---|---|
| Features done | **5 / 12** (42%) |
| Path to MVP1 | **16** items remaining (features + bugs + chores) |
| Features done | **6 / 12** (50%) |
| Path to MVP1 | **15** items remaining (features + bugs + chores) |
| Open bugs | 3 |
| Open chores | 6 (idea-stage debt) |
| Backlog ideas | 4 idea-only feat/infra (not yet scoped into MVP1) |
| In flight | 0 feature(s) actively shipping |

## Pipeline

### Done (5)
### Done (6)

| Feature | Type | One-liner | Depends on | Status |
|---|---|---|---|---|
| [feat_digest_proposal](implemented_features/2026_05_11_feat_digest_proposal/feature_spec.md) | Feature | When a study transitions to `completed`, the digest worker generates: a narrative summary (LLM-authored), a parameter-importance map (computed by `optuna.importance`), and a recommended config. | `feat_study_lifecycle` `feat_llm_judgments` | [PR #41](https://github.com/SoundMindsAI/relyloop/pull/41) merged 2026-05-11 |
| [feat_llm_judgments](implemented_features/2026_05_11_feat_llm_judgments/feature_spec.md) | Feature | A relevance engineer selects a query set + cluster + target + rubric and the system runs the current template to fetch top-K hits per query, asks OpenAI to rate each (query, doc) on a 0–3 scale with r | `infra_foundation` `infra_adapter_elastic` `feat_study_lifecycle` | [PR #35](https://github.com/SoundMindsAI/relyloop/pull/35) merged 2026-05-11 |
| [feat_study_lifecycle](implemented_features/2026_05_10_feat_study_lifecycle/feature_spec.md) | Feature | A relevance engineer creates a study via API or chat, the orchestrator enqueues N parallel `run_trial` jobs, trials accumulate in real time on the study detail page, the orchestrator detects stop-cond | — | [PR #18](https://github.com/SoundMindsAI/relyloop/pull/18) merged 2026-05-10 |
| [infra_adapter_elastic](implemented_features/2026_05_10_infra_adapter_elastic/feature_spec.md) | Infra | A single `ElasticAdapter` implements the `SearchAdapter` Protocol and serves both Elasticsearch (8.11+ / 9.x) and OpenSearch (2.x / 3.x), distinguished by a `engine_type` column. | — | [PR #16](https://github.com/SoundMindsAI/relyloop/pull/16) merged 2026-05-10 |
Expand All @@ -29,11 +30,9 @@ _Reflects feature-folder state as of **2026-05-11** (latest mtime of any planned

_None._

### Plan (1)
### Plan (0)

| Feature | Type | One-liner | Depends on | Status |
|---|---|---|---|---|
| [feat_digest_proposal](../02_product/planned_features/feat_digest_proposal/feature_spec.md) | Feature | When a study transitions to `completed`, the digest worker generates: a narrative summary (LLM-authored), a parameter-importance map (computed by `optuna.importance`), and a recommended config. | `feat_study_lifecycle` `feat_llm_judgments` | [PR #25](https://github.com/SoundMindsAI/relyloop/pull/25) merged 2026-05-11 |
_None._

### Spec (6)

Expand Down Expand Up @@ -79,8 +78,6 @@ graph LR
class chore_tutorial_polish spec;
feat_chat_agent["chat agent"]
class feat_chat_agent spec;
feat_digest_proposal["digest proposal"]
class feat_digest_proposal plan;
feat_github_pr_worker["github pr worker"]
class feat_github_pr_worker spec;
feat_github_webhook["github webhook"]
Expand All @@ -97,10 +94,10 @@ graph LR
class infra_adapter_elastic done;
infra_optuna_eval["optuna eval"]
class infra_optuna_eval done;
feat_digest_proposal["digest proposal"]
class feat_digest_proposal done;
feat_llm_judgments["llm judgments"]
class feat_llm_judgments done;
feat_study_lifecycle --> feat_digest_proposal
feat_llm_judgments --> feat_digest_proposal
infra_foundation --> feat_github_pr_worker
infra_adapter_elastic --> feat_github_pr_worker
feat_digest_proposal --> feat_github_pr_worker
Expand All @@ -115,6 +112,8 @@ graph LR
feat_digest_proposal --> feat_studies_ui
feat_llm_judgments --> feat_studies_ui
infra_adapter_elastic --> feat_studies_ui
feat_study_lifecycle --> feat_digest_proposal
feat_llm_judgments --> feat_digest_proposal
infra_foundation --> feat_llm_judgments
infra_adapter_elastic --> feat_llm_judgments
feat_study_lifecycle --> feat_llm_judgments
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Implementation Plan — feat_digest_proposal

**Date:** 2026-05-11
**Status:** Approved (3 GPT-5.5 review cycles to the cap; 20 findings accepted + applied; ready for `/impl-execute`)
**Status:** Complete (PR #41, merged 2026-05-11 squash commit `3753894`)
**Primary spec:** [feature_spec.md](feature_spec.md)
**Policy source(s):** [CLAUDE.md](../../../../CLAUDE.md), [docs/01_architecture/data-model.md](../../../01_architecture/data-model.md), [docs/01_architecture/llm-orchestration.md](../../../01_architecture/llm-orchestration.md), [docs/01_architecture/api-conventions.md](../../../01_architecture/api-conventions.md)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,18 @@ Study-end digest narrative + proposal-population pipeline. Replaces the `digest_
- Tests planned: 3 unit + 26 integration + 1 contract + 1 benchmark = 31 test files.

## Implementation
- Status: Not started. Next: `/impl-execute docs/02_product/planned_features/feat_digest_proposal/implementation_plan.md --all`.
- Status: **Complete** — merged 2026-05-11.
- PR: [#41](https://github.com/SoundMindsAI/relyloop/pull/41) (squash commit `3753894`).
- CI: green (4 cycles — initial fail on test seed/setting/migration order, AC-11 fix, final-review fix, all green).
- Stories completed: 12 of 12 across 4 epics.
- Test counts: 350 unit + 26 integration + 1 contract + 1 benchmark.
- Final cross-model review: GPT-5.5 raised 6 findings; 5 accepted + applied, 1 rejected with cited counter-evidence (full adjudication on PR #41).
- Gemini Code Assist: N/A on this repo.

## Done
- Status: **Deployed** (no remote staging in MVP1; merge to main IS the deploy event).
- Date: 2026-05-11
- Follow-up tracked: [`bug_digest_param_importance_seam`](../bug_digest_param_importance_seam/idea.md) — AC-7 test fixture seam (xfail in this PR).

## Dependencies (all satisfied)

Expand Down
49 changes: 25 additions & 24 deletions docs/00_overview/mvp1_dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -267,13 +267,13 @@ <h2>MVP1 Progress</h2>
<div class="kpi-row">
<div class="kpi ">
<div class="label">Features done</div>
<div class="value">5 / 12</div>
<div class="sub">42% of scoped MVP1 features</div>
<div class="bar"><span style="width:42%"></span></div>
<div class="value">6 / 12</div>
<div class="sub">50% of scoped MVP1 features</div>
<div class="bar"><span style="width:50%"></span></div>
</div>
<div class="kpi warn">
<div class="label">Path to MVP1</div>
<div class="value">16</div>
<div class="value">15</div>
<div class="sub">items left = features + bugs + chores</div>
</div>
<div class="kpi bug">
Expand Down Expand Up @@ -547,28 +547,29 @@ <h3>Spec <span class="count">6</span></h3>
</div>

<div class="col plan">
<h3>Plan <span class="count">1</span></h3>
<h3>Plan <span class="count">0</span></h3>

</div>

<div class="col implement">
<h3>Implementing <span class="count">0</span></h3>

</div>

<div class="col done">
<h3>Done <span class="count">6</span></h3>

<div class="card feat" data-prefix="feat">
<div class="name"><a href="../../docs/02_product/planned_features/feat_digest_proposal/feature_spec.md">Digest Proposal</a></div>
<div class="name"><a href="../../docs/00_overview/implemented_features/2026_05_11_feat_digest_proposal/feature_spec.md">Digest Proposal</a></div>
<div class="meta">
<span class="badge feat">Feature</span>
<a class="pr" href="https://github.com/SoundMindsAI/relyloop/pull/25">PR #25</a><span>merged 2026-05-11</span>
<a class="pr" href="https://github.com/SoundMindsAI/relyloop/pull/41">PR #41</a><span>merged 2026-05-11</span>
</div>
<div class="one-liner">When a study transitions to `completed`, the digest worker generates: a narrative summary (LLM-authored), a parameter-importance map (computed by `optuna.importance`), and a recommended config.</div>

<div class="deps">depends on: <span class="dep-chip">feat_study_lifecycle</span><span class="dep-chip">feat_llm_judgments</span></div>
</div>

</div>

<div class="col implement">
<h3>Implementing <span class="count">0</span></h3>

</div>

<div class="col done">
<h3>Done <span class="count">5</span></h3>

<div class="card feat" data-prefix="feat">
<div class="name"><a href="../../docs/00_overview/implemented_features/2026_05_11_feat_llm_judgments/feature_spec.md">Llm Judgments</a></div>
Expand Down Expand Up @@ -646,8 +647,6 @@ <h2>Dependency graph (feat_ + infra_)</h2>
class chore_tutorial_polish spec;
feat_chat_agent[&quot;chat agent&quot;]
class feat_chat_agent spec;
feat_digest_proposal[&quot;digest proposal&quot;]
class feat_digest_proposal plan;
feat_github_pr_worker[&quot;github pr worker&quot;]
class feat_github_pr_worker spec;
feat_github_webhook[&quot;github webhook&quot;]
Expand All @@ -664,10 +663,10 @@ <h2>Dependency graph (feat_ + infra_)</h2>
class infra_adapter_elastic done;
infra_optuna_eval[&quot;optuna eval&quot;]
class infra_optuna_eval done;
feat_digest_proposal[&quot;digest proposal&quot;]
class feat_digest_proposal done;
feat_llm_judgments[&quot;llm judgments&quot;]
class feat_llm_judgments done;
feat_study_lifecycle --&gt; feat_digest_proposal
feat_llm_judgments --&gt; feat_digest_proposal
infra_foundation --&gt; feat_github_pr_worker
infra_adapter_elastic --&gt; feat_github_pr_worker
feat_digest_proposal --&gt; feat_github_pr_worker
Expand All @@ -682,6 +681,8 @@ <h2>Dependency graph (feat_ + infra_)</h2>
feat_digest_proposal --&gt; feat_studies_ui
feat_llm_judgments --&gt; feat_studies_ui
infra_adapter_elastic --&gt; feat_studies_ui
feat_study_lifecycle --&gt; feat_digest_proposal
feat_llm_judgments --&gt; feat_digest_proposal
infra_foundation --&gt; feat_llm_judgments
infra_adapter_elastic --&gt; feat_llm_judgments
feat_study_lifecycle --&gt; feat_llm_judgments</div>
Expand All @@ -695,8 +696,6 @@ <h2>Dependency graph (feat_ + infra_)</h2>
class chore_tutorial_polish spec;
feat_chat_agent[&quot;chat agent&quot;]
class feat_chat_agent spec;
feat_digest_proposal[&quot;digest proposal&quot;]
class feat_digest_proposal plan;
feat_github_pr_worker[&quot;github pr worker&quot;]
class feat_github_pr_worker spec;
feat_github_webhook[&quot;github webhook&quot;]
Expand All @@ -713,10 +712,10 @@ <h2>Dependency graph (feat_ + infra_)</h2>
class infra_adapter_elastic done;
infra_optuna_eval[&quot;optuna eval&quot;]
class infra_optuna_eval done;
feat_digest_proposal[&quot;digest proposal&quot;]
class feat_digest_proposal done;
feat_llm_judgments[&quot;llm judgments&quot;]
class feat_llm_judgments done;
feat_study_lifecycle --&gt; feat_digest_proposal
feat_llm_judgments --&gt; feat_digest_proposal
infra_foundation --&gt; feat_github_pr_worker
infra_adapter_elastic --&gt; feat_github_pr_worker
feat_digest_proposal --&gt; feat_github_pr_worker
Expand All @@ -731,6 +730,8 @@ <h2>Dependency graph (feat_ + infra_)</h2>
feat_digest_proposal --&gt; feat_studies_ui
feat_llm_judgments --&gt; feat_studies_ui
infra_adapter_elastic --&gt; feat_studies_ui
feat_study_lifecycle --&gt; feat_digest_proposal
feat_llm_judgments --&gt; feat_digest_proposal
infra_foundation --&gt; feat_llm_judgments
infra_adapter_elastic --&gt; feat_llm_judgments
feat_study_lifecycle --&gt; feat_llm_judgments</pre></noscript>
Expand Down
11 changes: 6 additions & 5 deletions state.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,21 @@

> Read this first. Snapshots the active branch, what just shipped, what's in flight, what's queued, and where the project currently sits in the MVP1 → GA roadmap. Updated whenever a feature lands or a priority shifts.

**Last updated:** 2026-05-11 (after `feat_digest_proposal` implementation complete, PR pending)
**Last updated:** 2026-05-11 (after `feat_digest_proposal` merged via PR #41)

---

## Current branch / execution context

- **Branch:** `feature/feat-digest-proposal` — 12 stories complete + Epic 4 docs/tests; PR not yet opened.
- **Active feature:** `feat_digest_proposal` in flight (study-end digest narrative + populates the pending proposal row).
- **Branch:** `main` — `feat_digest_proposal` merged via PR #41 (squash commit `3753894`); finalization happening on `docs/finalize-digest-proposal`.
- **Active feature:** none in flight. Next-up: `feat_github_pr_worker` (now consumes the `proposals` shape that `feat_digest_proposal` populates).
- **Alembic head:** `0005_digests` (bumped by Story 1.1 of `feat_digest_proposal`).
- **Coverage:** above the 80% gate. 350 unit tests + 2 xpassed + 80+ integration (including 26 new for digest worker + API) + 6 contract (including 1 new for digest+proposals API).

## Most recent meaningful changes (newest first)

- **2026-05-11 — `feat_digest_proposal` implementation complete (PR pending)** on `feature/feat-digest-proposal`. 12 stories across 4 epics covering FR-1..FR-6 + FR-2b + AC-1..AC-11. **One migration (`0005_digests`).** Plan went through **3 GPT-5.5 review cycles** (20 findings, all accepted + applied) before execution.
- **2026-05-11 — `feat_digest_proposal` merged into `main`** as PR #41 (squash commit `3753894`). 12 stories across 4 epics covering FR-1..FR-6 + FR-2b + AC-1..AC-11. **One migration (`0005_digests`).** Plan went through **3 GPT-5.5 review cycles** (20 findings, all accepted + applied) before execution. Final GPT-5.5 review raised 6 findings on the merged PR diff; 5 accepted + applied (commit `6267cc8`), 1 rejected with cited counter-evidence (`git diff main..HEAD --stat` showed the docs WERE in the diff — false positive caused by code-only diff slice).
- **CI cycles:** 4 total — initial push (FK ordering in seed helper + settings pollution between tests + post-0005 downgrade-target wrong on `test_judgments_migration.py`) → fix-1 (commit `81d512e`) → still failing on AC-11 capability fallback persisting wrong recommended_config → fix-2 (commit `83fd8c6`) → final-review fixes (commit `6267cc8`) → green.
- **Epic 1 (foundations, 3 stories):** `0005_digests` migration creating the `digests` table per data-model.md (`id`, `study_id UNIQUE`, `narrative`, `parameter_importance JSONB`, `recommended_config JSONB`, `suggested_followups TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[]` per cycle-1 F1, `generated_by`, `generated_at`); `Digest` ORM model exported via `db/models/__init__.py`; new `db/repo/digest.py` (`create_digest`, `get_digest_for_study`) + 5 proposal repo extensions per FR-6 (`update_proposal_for_digest` conditional on `WHERE status='pending'` per cycle-3 F4, `list_proposals_paginated`, `count_proposals`, `reject_proposal` raises `InvalidStateTransition`, `list_pending_proposals_for_boot_scan`); `prompts/digest_narrative.system.md` + `digest_narrative.user.jinja` with `include_recommendation` toggle for the degraded path (cycle-3 F3); `backend/app/llm/digest_prompt.py` with `lru_cache`-cached `DigestPromptBundle` + `SandboxedEnvironment(autoescape=True)` mirroring `prompt_loader.py`.
- **Epic 2 (worker, 2 stories):** `backend/workers/digest.py:generate_digest` (15-step contract) REPLACES `backend/workers/digest_stub.py` (file deleted) under the same Arq job name so the orchestrator's enqueue at `orchestrator.py:370` keeps firing. Key cycle revisions: pre-LLM idempotency guard via `get_digest_for_study` (cycle-1 F6); `pg_try_advisory_xact_lock` keyed on `blake2b("digest:{sid}")` held across LLM call + persist tx (cycle-2 F6 — prefix disjoint from orchestrator's replenish lock); zero-trials short-circuit moved BEFORE OpenAI preflights (cycle-2 F5 — AC-2 fires regardless of config); capability fallback as a mode flag, NOT short-circuit (cycle-3 F2 — pricing + budget still gate); deterministic `recommended_config` (cycle-1 F5 / cycle-2 F1 — best-trial params filtered to currently-declared template params; all-dropped sub-case persists empty-recommendation digest + DELETEs the pending proposal per cycle-2 F7); persist FIRST then `_safe_record_cost` (cycle-2 C2-F3 from feat_llm_judgments); conditional UPDATE on the pending proposal handles operator-reject-mid-LLM race (cycle-3 F4 — benign no-op). `WorkerSettings.on_startup` extended with the FR-2b boot scan (list_pending_proposals_for_boot_scan + deterministic `_job_id="generate_digest:{sid}"`).
- **Epic 3 (API, 4 stories, 5 endpoints):** `backend/app/api/v1/proposals.py` (new router; registered in `main.py` alongside the existing v1 routers): `GET /studies/{id}/digest` (FR-3 / AC-3 / AC-4 — 404 STUDY_NOT_FOUND vs DIGEST_NOT_READY); `POST /proposals` (FR-4 / AC-6 — manual creation, study_id NULL); `GET /proposals` (cursor + status Literal + cluster_id filter + X-Total-Count); `GET /proposals/{id}` (detail with inline `study_summary` + `digest` to spare UI a fan-out query); `POST /proposals/{id}/reject` (FR-4 / AC-5 — pending → rejected; 409 INVALID_STATE_TRANSITION via repo's `InvalidStateTransition`). `ProposalStatusWire` and `ProposalPrStateWire` Literals carry source-of-truth comments per cycle-2 F4 / cycle-3 F1.
Expand Down Expand Up @@ -166,7 +167,7 @@

## In flight

- **`feat_digest_proposal`** on `feature/feat-digest-proposal` — 12 stories complete; PR pending. After merge, the queued list collapses to start with `feat_github_pr_worker`.
- None. Next feature pulled from the Queued list below.

## Queued (priority-ordered by dependency)

Expand Down