Skip to content

emrg: count request-level tool schemas in usage estimator (issue #1090) - #1091

Merged
argszero merged 1 commit into
masterfrom
feature/usage-estimator-tool-schemas
Aug 31, 2026
Merged

emrg: count request-level tool schemas in usage estimator (issue #1090)#1091
argszero merged 1 commit into
masterfrom
feature/usage-estimator-tool-schemas

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

Fixes issue #1090 (community finding by pm25coder via Dev.to 3dom2 / izgorodin).

Root cause: _estimate_tokens() counted message content and embedded tool_calls but ignored the request-level tools array that the API bills as part of prompt_tokens. Mid-session tool-set growth (skills/dynamic tools loaded between rounds) was therefore invisible to the auto-compact projection — it silently drifts low as more tools are added.

Changes

  • Add _estimate_tools(): JSON-char-counting each tool schema +3 per-tool overhead (mirroring how message metadata is counted).
  • Thread the tool-loop's tools_openai through _estimate_tokens and _refresh_usage_anchor so the anchor and auto-compact projection reflect exactly what was sent.
  • +5 tests: no-tools default, tool cost added, empty/None handling, multiple-tools scaling, and mid-session tool-growth visibility.
  • Agent.md Python count 1195 → 1200 (doc-count guard).

Verification

uv run pytest tests/: 1199 passed + 1 skipped (was 1194+1); import + CLI green.

The usage estimator (_estimate_tokens) counted message content and embedded
tool_calls but ignored the request-level `tools` array that the API bills as
part of prompt_tokens. Mid-session tool-set growth (skills/dynamic tools
loaded between rounds) was therefore invisible to the auto-compact projection,
which silently drifts low as more tools are added.

Add _estimate_tools(), JSON-char-counting each tool schema +3 overhead
(mirroring message metadata). Thread the tool-loop's tools_openai through
_estimate_tokens and _refresh_usage_anchor so the anchor and projection
reflect exactly what was sent. +5 tests (incl. mid-session tool-growth
visibility). Agent.md Python count 1195 -> 1200.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle

@pm25coder

Copy link
Copy Markdown
Collaborator

I tested this PR against the root cause of issue #1090 (which I filed from the Dev.to 3dom2 / izgorodin finding) and verified the fix end to end:

  1. Root cause confirmed: _estimate_tokens() counted message content + embedded tool_calls but ignored the request-level tools array. I grepped every _estimate_tokens / _refresh_usage_anchor call site in the current codebase: the auto-compact projection (daemon.py:2452) and the anchor refresh (daemon.py:2610 -> 3293) were the two places where the missing tools cost biased the projection — both are now threaded with tools_openai.
  2. Estimate matches the billed request: tools_openai (daemon.py:2427, with the empty-list fallback at 2444) is the exact array passed to chat_stream(messages, tools=tools_openai) at daemon.py:2542 — so the anchor now reflects precisely what the API bills for prompt_tokens. No drift between estimated and billed tool cost.
  3. Backward compatibility verified: tools defaults to None in both _estimate_tokens and _refresh_usage_anchor; _estimate_tools(None/[]) returns 0. The planted-fire drill's synthetic round (daemon.py:3198/3212) calls both without the arg — unchanged behavior, no new coupling.
  4. Tests cover the ± states: default no-tools (0 cost), tools add cost, empty/None -> 0, multiple-tools scaling (monotonic), and the key regression test — mid-session tool-set growth raises the projection so auto-compact sees the growth instead of a flat context.
  5. Consistency: the +3 per-tool overhead mirrors the +3 per-message metadata allowance, and JSON-char-counting matches the message/tool_calls convention — internally consistent estimation model. Agent.md 1195 -> 1200 matches the +5 tests.
  6. CI: test + test-windows both SUCCESS (run 33403933228).

Non-blocking observations (for the record):

  • The per-tool overhead is fixed (+3) regardless of schema size — reasonable, since the char-count dominates for real schemas.
  • The drill's synthetic round intentionally omits tools (its estimate is fabricated as the drift baseline) — no change needed there.

No issues found; the implementation closes the #1090 blind spot.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle

@argszero
argszero merged commit bae1bae into master Aug 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants