feat: spill oversized tool outputs to a spill tape - #274
Merged
Conversation
Large tool results (>4096 estimated tokens) are written once to a dedicated spill tape (the same store as session tapes) and replaced in the session tape with a compact ref: handle, shape, and bounded preview. The full payload is read back on demand through the bounded read_tool_result tool. A hard cap on the serialized request body clamps any remaining oversized tool message before it reaches the provider, so a single huge tool result can no longer fail a turn with 413. - ToolExecutor rewrites oversized string results after after_tool_call observers, so tracing still sees the original result. - Spill entries bypass the session-tape fork and land in the shared spill tape immediately. - No built-in cleanup: retention is the user's choice, exactly like session tapes. - Settings: BUB_TOOL_SPILL_THRESHOLD (0 disables), BUB_MAX_REQUEST_BYTES.
- agent.py normalizes a sync tape store through AsyncTapeStoreAdapter exactly once, so the spill store is always async. - maybe_spill takes a nullable store and folds the missing-store case into its single guard; ToolExecutor no longer branches on it. - SpillStore protocol is now append-only; read_spilled types against AsyncTapeStore instead of a bespoke query-capable protocol. - model_runner extracts one clamp_tool_messages helper for both the 2000-char pass and the per-message budget pass.
Collaborator
Author
|
Rebased onto main (now includes merged #273 fix/unknown-tool-recovery). Re-ran the full suite:\n\n- |
PsiACE
marked this pull request as ready for review
August 11, 2026 17:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #249
What's going on
A huge tool result — say
grep -Rovernode_moduleswith one very long line — was being re-sent in full on every later model request. Make it big enough and the provider answers413 Request Entity Too Large; the whole turn dies.The approach
Stop putting big outputs into the model request. Write them once, keep them around, and hand the model a pointer.
BUB_TOOL_SPILL_THRESHOLD(default 4096 estimated tokens, ~4 chars/token) are written once to aspilltape, in the same store as the session tapes. The session tape keeps a small ref instead: handle + shape + a head/tail preview.read_tool_resulttool reads slices of a spilled payload on demand (offset/limit/from_end/ literalpattern), capped at 1000 lines / 50k chars. Its own results never spill.BUB_MAX_REQUEST_BYTES(default 256 KB) clamps any oversized tool message before it reaches the provider, so even a plugin that bypasses spilling can't send an unbounded body.Errors are never spilled — the model needs the full error text to recover. If writing a spill fails, the original result is used; a spill problem never fails a turn.
How it fits
ToolExecutorrewrites oversized string results after theafter_tool_callobservers, so tracing still sees the original; only what lands in the tape and the model request gets smaller.Settings
BUB_TOOL_SPILL_THRESHOLD40960disablesBUB_MAX_REQUEST_BYTES2621440disablesVerification
uv run pytest -q— 282 passeduv run ruff check src tests— cleanuv run mypy src— cleantests/test_spill.py— 13 behavior/regression tests: spill + read-back roundtrip, small refs, small results and errors untouched, write-failure degrade, fork passthrough, read bounds and literal pattern, unknown handle, the 413 scenario (the next request never carries the payload), and the hard-cap clamp.