Skip to content

fix(client-runtime): stop resubscribing threads the server reports missing - #9822

Open
lnieuwenhuis wants to merge 7 commits into
pingdotgg:mainfrom
lnieuwenhuis:fix/thread-missing-tombstone
Open

fix(client-runtime): stop resubscribing threads the server reports missing#9822
lnieuwenhuis wants to merge 7 commits into
pingdotgg:mainfrom
lnieuwenhuis:fix/thread-missing-tombstone

Conversation

@lnieuwenhuis

@lnieuwenhuis lnieuwenhuis commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Every foreground wakeup re-issues subscribeThread for a thread the server already reported missing, and the debounced persistence writer can resurrect the thread right after cache deletion.

Return a dedicated missing-thread error and stop the durable subscription across reconnects and foreground wakeups. Serialize cache saves and removal so queued or in-flight writes cannot restore a deleted thread. Other snapshot failures still retry.

Reimplements the intent of #8192 (which only drained the inner stream) on the current client-runtime layout, with regressions for single-attempt termination, no foreground resubscribe, and no persistence resurrection.

Built with muse-spark-1.3-contributor via OpenCode in T3 Code. Review follow-ups by GPT-6 via Codex.


Note

Medium Risk
Changes subscribeThread wire errors and client thread subscription, cache deletion, and persistence ordering; incorrect classification could stop sync early or leave stale cache.

Overview
Introduces OrchestrationThreadNotFoundError on the subscribeThread RPC and has the server emit it when a thread snapshot is unavailable (no replay fallback), instead of overloading OrchestrationGetSnapshotError.

On the client, subscribeDynamic gains a terminalFailure path that halts session- and wakeup-driven resubscribes after a classified failure. Thread sync treats not-found as terminal: it sets a tombstone latch, marks the thread deleted, removes cache, and does not retry on foreground wakeups or session replacement. Persistence is tightened with a lock, skipping writes after deletion, polling the debounced queue before removeThread, so stale snapshots cannot resurrect a deleted thread.

Other snapshot failures still use the existing retry and resubscribe behavior. Tests cover RPC decoding, terminal vs retriable failures, and deletion parity with thread.deleted.

Reviewed by Cursor Bugbot for commit 74ce349. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Stop resubscribing threads the server reports as missing

  • Adds OrchestrationThreadNotFoundError to the contracts layer and the subscribeThread RPC error union; the server WebSocket layer now returns this typed error with the missing threadId instead of a generic snapshot error
  • Introduces a terminalFailure classifier in subscribeDynamic that halts all session-driven resubscription when every error in a cause matches the classifier, then runs a terminal handler once
  • makeEnvironmentThreadState classifies OrchestrationThreadNotFoundError as terminal, marks the thread deleted via setDeleted, and blocks foreground, probe, and session-replacement resubscriptions; generic snapshot errors remain retryable
  • setDeleted now drains the pending persistence queue and waits for an in-flight cache save before removing the cache entry; the persistence worker skips snapshots when the thread is already deleted
  • Risk: setDeleted in threads.ts acquires a persistence semaphore that serializes cache removal with saves — if persist blocks indefinitely on a cache write, deletion will also block; any caller that previously retried on a generic not-found message now needs to handle the typed error or it will fall into the ordinary retry path

Macroscope summarized 74ce349.

…ssing

Subscribe failures carrying threadDisposition not-found now end the
subscription terminally, tombstone the thread so foreground/probe
wakeups never resubscribe, and drain queued persistence before cache
removal so a debounced write cannot resurrect the deleted thread.
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Sep 4, 2026
Comment thread packages/client-runtime/src/state/threads.ts
@macroscopeapp

macroscopeapp Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This change modifies an existing production subscription path across the server protocol, client runtime, thread state, and cache persistence. It introduces terminal stream halting and concurrency-sensitive deletion ordering, so the runtime and wire-level effects are broader than a small isolated bug fix.

Not approved because:

  • Monthly spending limit reached (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings. You can add or adjust custom eligibility rules. Learn more.

…d miss

Session replacements re-issued subscribeThread after a not-found
tombstone because the terminal latch only filtered foreground wakeups
while the outer session stream in subscribeDynamic stayed alive.
Signal a halt Deferred from the terminalFailure handler and interrupt
the outer session stream so no new subscribe issues; non-matching
failures keep session-driven resubscription.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0895455. Configure here.

Comment thread packages/client-runtime/src/rpc/client.ts
A session replacement landing during the terminal handler's cache I/O
started a new inner subscribe before the post-handle halt landed. Signal
terminalHalt first so the outer session stream is already dead; the
handler still drains as the running inner.

Muse Spark (opencode)
@github-actions github-actions Bot added size:L 100-499 changed lines (additions + deletions). and removed size:M 30-99 changed lines (additions + deletions). labels Sep 4, 2026
Comment thread packages/contracts/src/orchestration.ts Outdated
Comment thread packages/client-runtime/src/state/threads.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant