Skip to content

feat: spill oversized tool outputs to a spill tape - #274

Merged
PsiACE merged 3 commits into
bubbuild:mainfrom
PsiACE:lody/34026e74-20f
Aug 11, 2026
Merged

feat: spill oversized tool outputs to a spill tape#274
PsiACE merged 3 commits into
bubbuild:mainfrom
PsiACE:lody/34026e74-20f

Conversation

@PsiACE

@PsiACE PsiACE commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Closes #249

What's going on

A huge tool result — say grep -R over node_modules with one very long line — was being re-sent in full on every later model request. Make it big enough and the provider answers 413 Request Entity Too Large; the whole turn dies.

The approach

Stop putting big outputs into the model request. Write them once, keep them around, and hand the model a pointer.

  1. Spill — tool results over BUB_TOOL_SPILL_THRESHOLD (default 4096 estimated tokens, ~4 chars/token) are written once to a spill tape, in the same store as the session tapes. The session tape keeps a small ref instead: handle + shape + a head/tail preview.
  2. Read back — a read_tool_result tool reads slices of a spilled payload on demand (offset / limit / from_end / literal pattern), capped at 1000 lines / 50k chars. Its own results never spill.
  3. Last-resort capBUB_MAX_REQUEST_BYTES (default 256 KB) clamps any oversized tool message before it reaches the provider, so even a plugin that bypasses spilling can't send an unbounded body.
  4. No cleanup — the spill tape is shared across sessions and has no TTL or pruning. Retention is the user's call, same as the session tapes: delete it when done.

Errors are never spilled — the model needs the full error text to recover. If writing a spill fails, the original result is used; a spill problem never fails a turn.

How it fits

  • ToolExecutor rewrites oversized string results after the after_tool_call observers, so tracing still sees the original; only what lands in the tape and the model request gets smaller.
  • Spill entries bypass the session-tape fork and land in the shared spill tape immediately, so they survive merge-back and stay readable across turns and sessions.
  • It reuses the tape store end to end: no new storage, no sidecar files, no new hooks.

Settings

Env Default Meaning
BUB_TOOL_SPILL_THRESHOLD 4096 estimated tokens above which a tool result is spilled; 0 disables
BUB_MAX_REQUEST_BYTES 262144 hard cap on the serialized request body; 0 disables

Verification

  • uv run pytest -q — 282 passed
  • uv run ruff check src tests — clean
  • uv run mypy src — clean
  • tests/test_spill.py — 13 behavior/regression tests: spill + read-back roundtrip, small refs, small results and errors untouched, write-failure degrade, fork passthrough, read bounds and literal pattern, unknown handle, the 413 scenario (the next request never carries the payload), and the hard-cap clamp.

PsiACE added 3 commits August 11, 2026 03:01
Large tool results (>4096 estimated tokens) are written once to a
dedicated spill tape (the same store as session tapes) and replaced in
the session tape with a compact ref: handle, shape, and bounded preview.
The full payload is read back on demand through the bounded
read_tool_result tool. A hard cap on the serialized request body
clamps any remaining oversized tool message before it reaches the
provider, so a single huge tool result can no longer fail a turn with
413.

- ToolExecutor rewrites oversized string results after after_tool_call
  observers, so tracing still sees the original result.
- Spill entries bypass the session-tape fork and land in the shared
  spill tape immediately.
- No built-in cleanup: retention is the user's choice, exactly like
  session tapes.
- Settings: BUB_TOOL_SPILL_THRESHOLD (0 disables), BUB_MAX_REQUEST_BYTES.
- agent.py normalizes a sync tape store through AsyncTapeStoreAdapter
  exactly once, so the spill store is always async.
- maybe_spill takes a nullable store and folds the missing-store case
  into its single guard; ToolExecutor no longer branches on it.
- SpillStore protocol is now append-only; read_spilled types against
  AsyncTapeStore instead of a bespoke query-capable protocol.
- model_runner extracts one clamp_tool_messages helper for both the
  2000-char pass and the per-message budget pass.
@PsiACE

PsiACE commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (now includes merged #273 fix/unknown-tool-recovery). Re-ran the full suite:\n\n- uv run pytest -q — 287 passed\n- uv run ruff check src tests — clean\n- uv run mypy src — clean\n\nNo conflicts; the unknown-tool placeholder and the spill path compose cleanly (placeholder errors are never spilled, small refs stay intact).

@PsiACE
PsiACE marked this pull request as ready for review August 11, 2026 17:43
@PsiACE
PsiACE merged commit 86b055a into bubbuild:main Aug 11, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

工具输出过大时应截断,避免下一轮请求触发 413

1 participant