fix(session): overflow trigger ignores cache reads; replay re-injects overflow payloads - #47832
Closed
padraig-myers-sh wants to merge 2 commits into
Conversation
…e-injecting overflow payloads Two production wedge mechanisms observed across 12+ sessions (Aug-Sep): 1. isOverflow trusted the provider-reported total_tokens, whose cache accounting convention varies by provider. Anthropic-style paths report total_tokens EXCLUDING cache reads, so a session with a near-100% prompt-cache hit ratio reported a tiny total forever while its real context grew to 3M+ tokens - the proactive compaction trigger never fired (overflow.ts). Fix: always compare max(total, input+output+cache) against the usable window. 2. The overflow-replay path copied the overflowed user turn's parts verbatim into a new user message, re-injecting the exact payload that caused the overflow. Multi-MB user document attachments (stored as file/text parts, truncated nowhere in the pipeline) were therefore guaranteed to re-overflow after compaction, wedging the session permanently (compaction.ts). Fix: replayPartFor() replaces all file attachments and oversized text parts with reference placeholders.
Contributor
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
Contributor
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem — two production wedge mechanisms (12+ sessions, Aug–Sep)
1. Proactive compaction trigger is blind on Anthropic-style usage
isOverflowtrusted the provider-reportedtotal_tokens:total_tokenssemantics are provider-dependent. OpenAI-style providers include cached tokens; Anthropic-style paths report them separately (excluding cache reads). A session with a near-100% prompt-cache hit ratio therefore reportstotal_tokensin the single digits forever while its real context grows unbounded — the trigger never fires. Confirmed in production: a session grew to 3.17M true tokens (cache reads) across hundreds of turns while the reported non-cached input stayed at 3–15 tokens; it wedged for 7+ days. Live-confirmed on opus as a latent hazard:input=1token/turn with 68k–80k riding on cache reads.2. Overflow-replay re-injects the payload that caused the overflow
After an overflow-triggered compaction, the replay path copies the overflowed user turn's parts verbatim into a new user message. Its only substitution covers media-file parts — so multi-MB user document attachments (JSON files stored as
file/textparts, truncated nowhere else in the pipeline) get re-injected exactly as-is. Production forensics: byte-identical 4.52MB attachment sets in two user messages 34 seconds apart; every retry cycle re-injected them (+1.19M tokens per cycle) until the session passed every model's context window and compaction itself could no longer run (ContextOverflowError: Conversation history too large to compact). User document text is untruncatable by design (the 2k cap covers tool outputs only), so the wedge is permanent.Fix
overflow.ts— compare the full context size against the usable window:count = Math.max(total ?? 0, input + output + cache.read + cache.write). Cache-inclusive providers are unaffected (max picks the larger, identical value); cache-excluding providers now fire correctly.compaction.ts— newreplayPartFor()used by the replay loop: allfileattachments become[Attached <mime>: <filename>]reference placeholders (previously only media), andtextparts overREPLAY_PART_MAX_CHARSbecome[Document omitted: N characters]. Small typed text (the actual replayed prompt) passes through untouched.Tests
isOverflowregression:{ input: 15, cache.read: 2_900_000, total: 115 }against a 300k-context model →true(wasfalseon v1.18.18 — this is the production wedge signature).replayPartFor: json/image file parts → placeholders; 5,000-char text part → omitted; small text → verbatim.bun test test/session/compaction.test.ts: 59 pass / 0 fail;tsgo --noEmitclean. (revert-compact/promptsuites have 14 pre-existing failures on the clean v1.18.18 tag — unchanged by this PR.)