Skip to content

fix(session): overflow trigger ignores cache reads; replay re-injects overflow payloads - #47832

Closed
padraig-myers-sh wants to merge 2 commits into
anomalyco:devfrom
padraig-myers-sh:fix/compaction-overflow-blindness-and-replay-duplication
Closed

fix(session): overflow trigger ignores cache reads; replay re-injects overflow payloads#47832
padraig-myers-sh wants to merge 2 commits into
anomalyco:devfrom
padraig-myers-sh:fix/compaction-overflow-blindness-and-replay-duplication

Conversation

@padraig-myers-sh

Copy link
Copy Markdown

Problem — two production wedge mechanisms (12+ sessions, Aug–Sep)

1. Proactive compaction trigger is blind on Anthropic-style usage

isOverflow trusted the provider-reported total_tokens:

const count = input.tokens.total || input + output + cache.read + cache.write

total_tokens semantics are provider-dependent. OpenAI-style providers include cached tokens; Anthropic-style paths report them separately (excluding cache reads). A session with a near-100% prompt-cache hit ratio therefore reports total_tokens in the single digits forever while its real context grows unbounded — the trigger never fires. Confirmed in production: a session grew to 3.17M true tokens (cache reads) across hundreds of turns while the reported non-cached input stayed at 3–15 tokens; it wedged for 7+ days. Live-confirmed on opus as a latent hazard: input=1 token/turn with 68k–80k riding on cache reads.

2. Overflow-replay re-injects the payload that caused the overflow

After an overflow-triggered compaction, the replay path copies the overflowed user turn's parts verbatim into a new user message. Its only substitution covers media-file parts — so multi-MB user document attachments (JSON files stored as file/text parts, truncated nowhere else in the pipeline) get re-injected exactly as-is. Production forensics: byte-identical 4.52MB attachment sets in two user messages 34 seconds apart; every retry cycle re-injected them (+1.19M tokens per cycle) until the session passed every model's context window and compaction itself could no longer run (ContextOverflowError: Conversation history too large to compact). User document text is untruncatable by design (the 2k cap covers tool outputs only), so the wedge is permanent.

Fix

  1. overflow.ts — compare the full context size against the usable window: count = Math.max(total ?? 0, input + output + cache.read + cache.write). Cache-inclusive providers are unaffected (max picks the larger, identical value); cache-excluding providers now fire correctly.
  2. compaction.ts — new replayPartFor() used by the replay loop: all file attachments become [Attached <mime>: <filename>] reference placeholders (previously only media), and text parts over REPLAY_PART_MAX_CHARS become [Document omitted: N characters]. Small typed text (the actual replayed prompt) passes through untouched.

Tests

  • isOverflow regression: { input: 15, cache.read: 2_900_000, total: 115 } against a 300k-context model → true (was false on v1.18.18 — this is the production wedge signature).
  • replayPartFor: json/image file parts → placeholders; 5,000-char text part → omitted; small text → verbatim.
  • bun test test/session/compaction.test.ts: 59 pass / 0 fail; tsgo --noEmit clean. (revert-compact/prompt suites have 14 pre-existing failures on the clean v1.18.18 tag — unchanged by this PR.)

opencode and others added 2 commits August 13, 2026 01:14
…e-injecting overflow payloads

Two production wedge mechanisms observed across 12+ sessions (Aug-Sep):

1. isOverflow trusted the provider-reported total_tokens, whose cache
   accounting convention varies by provider. Anthropic-style paths report
   total_tokens EXCLUDING cache reads, so a session with a near-100%
   prompt-cache hit ratio reported a tiny total forever while its real
   context grew to 3M+ tokens - the proactive compaction trigger never
   fired (overflow.ts). Fix: always compare max(total, input+output+cache)
   against the usable window.

2. The overflow-replay path copied the overflowed user turn's parts
   verbatim into a new user message, re-injecting the exact payload that
   caused the overflow. Multi-MB user document attachments (stored as
   file/text parts, truncated nowhere in the pipeline) were therefore
   guaranteed to re-overflow after compaction, wedging the session
   permanently (compaction.ts). Fix: replayPartFor() replaces all file
   attachments and oversized text parts with reference placeholders.
@github-actions github-actions Bot added the needs:compliance This means the issue will auto-close after 2 hours. label Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

  • PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs:compliance This means the issue will auto-close after 2 hours. needs:issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant