feat(memory): episode lifecycle — dedup-on-write + eviction (cap/TTL)

[jkyberneees]

Problem

EpisodeStore had no dedup and no eviction — episodes accumulated forever and near-duplicate sessions piled up unbounded (disk growth, noisier recall). This is the first slice of memory lifecycle (#6).

Change

Bounded, de-duplicated episode storage. Config-gated, safe defaults, no on-disk format change.

Dedup-on-write (newest-wins, provenance-gated)

A new episode whose cosine similarity to an existing one ≥ episode_dedup_threshold (default 0.92) replaces it. Uses an ephemeral RP embedder (the existing NewRPRanker primitive) over full on-disk summaries — deliberately not the shared dirty-rebuild vector index, avoiding a synchronous mid-write rebuild/re-entrancy. Provenance-safe: an untrusted near-dup can never evict a trusted/approved episode (trustRank mirrors the recall filter).

Eviction (prune-on-write)

Applies TTL (episode_ttl_days, default 0/off) then a count cap (max_episodes, default 500), deleting both the .md file and the index entry. Crash-safe order: files → writeIndex → markDirty (a crash leaves at most a dangling index entry, which rebuild/recall tolerate). Also exposes EpisodeStore.Prune() for session-end/CLI use.

Locking + a latent-bug fix

Dedup + .md write + index update + prune + markDirty now run under a single e.mu hold (writeLocked). This also fixes a latent bug where re-writing the same sessionID appended a duplicate index entry.

Config

EpisodeDedupThreshold / MaxEpisodes / EpisodeTTLDays wired through MemoryConfig, DefaultMemoryConfig, both overlay sites, and a new NewEpisodeStoreWithLifecycle. The bare NewEpisodeStore keeps lifecycle off, so existing callers/tests are unaffected. Documented in docs/CONFIG.md.

Deferred: fact supersession — facts have no per-entry metadata (true supersession needs a format change) and already get semantic dedup via merge-on-write + session-end LLM consolidation.

Tests

Dedup replace/threshold/disabled, provenance safety (both directions), eviction by count + TTL, TTL-disabled, self-overwrite regression, evicted-id absent from recall, -race concurrency (16 goroutines), config defaults + overlay-to-store wiring.

Verification

go build ./...                              # ok
go vet ./...                                # clean
gofmt -l internal/memory internal/config   # empty
go test ./internal/memory/... ./internal/config/... -race   # ok
go test ./... -short -race                  # PASS

🤖 Generated with Claude Code

[jkyberneees]

🔍 AI Verification Protocol v5.2.7 — Certificate

Classification: NovelBehavior (new dedup/eviction logic) · LOC_filtered: ~587 — under the 1,500 standard-pipeline ceiling.

Adversarial pass (Agent D) — findings & fixes

ID	Severity	Finding	Fix
D-01	🔴 Security (defense-in-depth)	Eviction/dedup called os.Remove on sessionID read straight from index.json without validation. A crafted/corrupted index entry (e.g. "../victim") deleted a .md file outside the episodes dir — verified with a probe that deleted a file in the parent dir. Every other file op (Read/Write/Promote) already validates.	New removeEpisodeFile(sessionID) calls session.ValidateSessionID before os.Remove; all three eviction/dedup deletions routed through it. Regression test TestEpisodeEviction_TraversalSafe. Fixed in 844da80.

Probes that passed (no fix needed):

• Provenance-skip leaves the trusted .md intact.
• No index/file orphans after dedup chains, dedup+cap, or 6 sequential near-dup replacements.
• Empty/whitespace summary doesn't crash (added as distinct; excluded from the vector index by the existing empty-text skip).

Axes

Axis	Result	Evidence
2.1 Semantic correctness	✅	Dedup replace/threshold/disabled, eviction count+TTL, self-overwrite all tested; behaviors match contract.
2.3 Security surface	✅ (after D-01)	Path-traversal in file deletion closed; os.Remove now validated like the rest of the package.
2.4 Structural integrity	✅	Single e.mu hold over dedup→write→index→prune→markDirty; crash-safe file→index→markDirty ordering; ephemeral RP avoids shared dirty-rebuild re-entrancy.
2.5 Behavioral exploration	✅	Adversarial probes (traversal, provenance-skip, dedup+cap, orphans, empty summary) + 16-goroutine -race.
2.9 Documentation	✅	New exported NewEpisodeStoreWithLifecycle/Prune carry godoc; 3 config keys documented in docs/CONFIG.md. No removed/changed public signatures (NewEpisodeStore unchanged).

Signals

go build ✅ · go vet ✅ · gofmt -l empty ✅ · go test ./internal/memory/... ./internal/config/... -race ✅ · go test ./... -short -race ✅.

Verdict: AutoApprove after remediation. The one finding (D-01) is fixed and regression-tested before merge.