Historian child cleanup races late OpenCode part writes

[coleleavitt]

Summary

Magic Context deletes a successful magic-context-compartment child immediately after the prompt API returns. OpenCode can still have detached session writers persisting step/summary parts, so the late insert targets a deleted session_id/message_id and fails its foreign key.

Observed failure

Failed query: insert into "part" (...) values (...) on conflict (...) do update ...
params: ..., {"type":"step-start"}, ...

Observed sequence:

1. Historian child created and streamed for about five minutes.
2. Magic Context accepted/published the compartment and deleted the child.
3. OpenCode persisted a late step-start part for that child.
4. The insert failed because the child session/message rows were already gone.

Root cause

runHistorianPrompt() deleted successful children in finally. Prompt completion and even session.idle are not global quiescence barriers: OpenCode can fork detached summary/session writes.

Expected behavior

Never delete historian children on the prompt-return path. Retire them only through an age-gated maintenance sweep whose stale budget covers the configured timeout, outer retries, model-suggestion retry, and fallback chain. Preserve children when keep_subagents is enabled.

Regression requirements

• Successful run does not call session.status or session.delete inline.
• Recent historian children are never swept.
• Stale historian children are swept after the full configured attempt budget.
• Dreamer-disabled installations still run the sweep.
• Slow reasoning models receive at least a ten-minute default timeout.

[alfonso-magic-context]

Fixed on master, ships next release. Historian children are no longer deleted inline at publish — they ride the age-gated orphan sweep, whose stale threshold now composes the configured timeout, retries, fallback chain, and a detached-writer grace window, and the sweep runs even with the dreamer disabled. Your sequence diagnosis was exact; see also PR #337 (filed independently by @coleleavitt for the same defect) for the extraction credits.

[Sugaroverdose]

Regression: the #336 fix (historian children) does not cover the dreamer
children — the same finally-delete → FK race still fires for them. v0.41.4,
2026-09-08.

#336's fix (per the comment above) removed the inline session.delete for
historian children (magic-context-compartment) — they now ride the
age-gated orphan sweep with a detached-writer grace window. But the dreamer
children (magic-context-dream-*) still call session.delete in finally
(after the prompt completes), racing opencode's detached part-writers → the
same FK failure on the part.session_id → session.id insert → lost output
(failure_class=empty_completion + the GUI error sound).

Repro (v0.41.4, local 27B model, non-TUI / OpenCode desktop 1.18.29):

• Dreamer task (e.g. map-memories, verify, curate) creates a child
  session → the LLM runs → the child's finally fires session.delete
  before opencode's detached writers finish persisting the final parts.
• The writers' part.session_id → session.id FK fails → the upsert is
  rejected (rolled back, hence 0 orphans in the sweep) → the output is
  lost (resultChars=0, failure_class=empty_completion).
• Historian children are immune (by design — the Historian child cleanup races late OpenCode part writes #336 fix keeps them for
  the age-gated sweep, so the writers always finish before any delete).
  Dreamer children self-delete → they're the only ones that race the
  writers. That's exactly why zero FK errors on historian sessions and all
  on dreamer sessions.

Fix (same as #336, applied to the dreamer children): remove the inline
session.delete from the dreamer child finally — let them ride the
existing age-gated orphan sweep (which already targets dreamer children
by title + age, with the detached-writer grace window). Bounded cleanup, no
FK race. No config flag needed (the sweep runs when keep_subagents is
false — the default).

Cross-ref: #337 (the PR that fixed the historian children for the same
defect).

[Sugaroverdose]

Follow-up: session.archive is a better fix for both the historian
children and the dreamer children than the age-gated sweep (the #336 fix).

The #336 fix (remove the inline session.delete, let the age-gated sweep
handle it) relies on a time-driven assumption (staleMs = max(60min, maxTimeout×3)) — the sweep only deletes children older
than staleMs, which is a heuristic tuned to "big enough" (the
detached writers might still be flushing if the child is younger than
staleMs but older than the writers' flush time).

session.archive is better — it gives you a deterministic state
(the session is "ended"), not a time-driven assumption:

• session.archive (OpenCode's first-class soft-delete: a patch that
  sets time_archived, not a row removal) retains the session row
  → the detached writers land on a live row → no FK race (the bug
  disappears).
• The session is archived (ended) → safe to delete (the writers are
  done, because the session is "ended"). This is a correct signal
  (the archive state), not a heuristic (the time-based staleMs).

So the fix (for both the historian children and the dreamer
children): session.archive instead of session.delete in the
teardown, and let the existing age-gated sweep (title + age) provide
bounded cleanup (retire the archived children). No racy delete, no
unbounded pile-up, no time-driven assumption.

This applies to both agents:

• Historian children (magic-context-compartment): the Historian child cleanup races late OpenCode part writes #336 fix
  (remove the inline session.delete) works, but the archive fix is
  better (deterministic state, no time-driven assumption).
• Dreamer children (magic-context-dream-*): the Historian child cleanup races late OpenCode part writes #336 fix didn't
  cover them (they still use the finally-delete → FK race). The
  archive fix is the right fix for them.

So the recommendation: session.archive for both agents (the
historian children and the dreamer children), and let the existing
age-gated sweep (title + age) provide bounded cleanup (retire the archived
children). No racy delete, no unbounded pile-up, no time-driven assumption.

[magic-alfonso]

Thanks @Sugaroverdose — your diagnosis is right: the #336 fix covered historian children only, and the dreamer children (retrospective, curate/maintain-docs, and smart-note confirmation runs) still called session.delete inline in finally. That is fixed on master and ships in the next patch release: those children now retire through the same age-gated orphan sweep, and the privacy-sensitive ones (retrospective, curate) are still swept when keep_subagents is on — eligibility is max(60 min, 3× the task timeout + 15 min writer grace), so with the default 20-minute task timeout a child is cleaned within ~90 minutes of creation.

One correction to the mechanism, verified against the OpenCode 1.18.29 source: every part of the assistant reply is committed to SQLite before session.prompt resolves, so a delete after a completed prompt cannot lose output. What produces both symptoms together is a client-side timeout: our dreamer gave up on the prompt (20-minute task timeout, which a local 27B model can exceed), the finally deleted the session, and the server's run loop — which a client disconnect does not stop — kept writing parts into a session row that no longer existed. The FK errors were the evidence of that, and resultChars=0 came from the timed-out prompt, not from the FK failure. Removing the inline delete closes the FK class regardless; if you keep seeing empty_completion on that model, it is the timeout — raise dreamer.task_timeout_minutes for local models, and the failure detail in the dashboard will tell you which it was.

[magic-alfonso]

On session.archive: it is a good addition but not the completion signal you describe. Archiving patches time_archived on the row; nothing in OpenCode's run loop consults it, so an archived session's server-side loop keeps writing exactly as an unarchived one does — "archived" means "hidden", not "writers done". The signal we actually have (verified at source) is the prompt itself: all parts are committed before session.prompt resolves, and on our own timeout we already call session.abort on the child so the loop stops; the age gate on the sweep is the belt for the abort failing, not the primary mechanism.

What archive does buy is real: the child leaves the user's session list immediately instead of lingering until the sweep. We'll add it — archive at teardown, sweep deletes later — as an additive step in the same release.

[Sugaroverdose]

Live-validated refinement: gate the child-session teardown delete on prompt resolution — for both historian and dreamer children. (Complements the confirmed sweep fix; does not replace it.)

I implemented this against the live 0.41.4 dist and validated it under real dreamer runs today (local 27B model, ~15 child sessions in a single /ctx-dream run):

• Settled runs (prompt resolved) — 6 observed: child session created, then deleted in place; all vanished well under the sweep's 60-min floor, so only the success-path inline delete explains their absence.
• Unsettled runs (timeout / empty_completion / restart-abort) — 7 observed: rows retained with parts fully committed (10–111 parts each), zero FK constraint errors post-patch, vs. 6 FOREIGN KEY constraint failed errors pre-patch the same day. The output that used to be lost is preserved and retired by the sweep.

The gate (~3 lines per executor, applied to 12 call sites):

let promptSettled = false;
try {
  ...
  const run = await promptSyncWithValidatedOutputRetry(client, {...}, {...});
  promptSettled = true;   // only path where the prompt call resolved
  ...
} finally {
  if (agentSessionId && promptSettled && !shouldKeepSubagents()) {
    await client.session.delete({ path: { id: agentSessionId } }).catch(log);
  } else if (agentSessionId && !promptSettled) {
    log(`[dreamer] ${task}: prompt unsettled — session ${agentSessionId} left to the age-gated sweep`);
  }
}

It keys on a fact MC already observes — the prompt call's own resolution (per your note, all parts are committed before session.prompt resolves, so a settled prompt ⇒ the row is safe to delete now; an unsettled one ⇒ writers may still be flushing ⇒ keep the row for the sweep). No DB polling, no new OpenCode-side flag, no time constant.

Why both agents:

• Historian children (compartment) today defer all cleanups — success included — to the sweep (staleMs = max(60min, timeout×3)). The gate lets the success path delete immediately; only the rare failed run waits for the sweep.
• Dreamer privacy-sensitive children (retrospective / curate / maintain-docs / user-memories / refresh-primers / smart-note) get the same success-path improvement.

Coverage gap on master (the part that needs the most attention): the four executors map-memories, verify, classify, and compress-cues still call session.delete unconditionally in finally on master (map-memories.ts:417, verify.ts:363, classify.ts:446, compress-cues.ts:448 — "Delete on success AND failure") — so the FK race from this issue is still live for those tasks. And their child titles (magic-context-dream-map-memories, -verify, -classify, -compress-cues) are not in PRIVACY_SENSITIVE_CHILD_TITLE_MATCHES (retrospective-orphan-sweep.ts), so if the inline delete is simply removed there (as done for the privacy set), those rows would never be reaped by the sweep. So the complete fix is: the settled-gate on all 12 executors + adding those four titles to the sweep's match set.

Happy to send the full per-site diff (applied, syntax-checked, and live-validated as described above) if useful.

[magic-alfonso]

Thank you @Sugaroverdose for the live validation on 0.41.4 across ~15 child sessions, and especially for catching the four multiline delete sites and their missing sweep titles — our earlier grep was blind to client.session\n.delete( chained across lines, so the coverage claim was wrong and you were right.

Merged to master: every OpenCode child teardown path (13 sites, enumerated by AST rather than grep — the inventory also turned up sidekick and the historian runner) now gates on prompt settlement. Resolved prompts delete inline, which is safe given OC's ordering (assistant parts are committed before the prompt response resolves); timeout/abort/client-error paths are archived best-effort so the child leaves the session list immediately, and are left for the unchanged age-gated sweep. Privacy-sensitive children still delete after settlement even with keep_subagents enabled. map-memories, verify, classify and compress-cues are now in the sweep's title set, with positive and negative title tests, and a source-fence test fails CI on any future ungated delete site.

On session.archive as a writers-done signal: it isn't one — archiving only sets time.archived; the run loop never consults it. The settlement gate is the signal; archive is the UX step on the unsettled branch. Rides the next patch release.