Skip to content

fix: surface fleet-wide token total in audit telemetry#45657

Merged
pelikhan merged 8 commits into
mainfrom
copilot/deep-reportsurface-fleet-token-total
Jul 15, 2026
Merged

fix: surface fleet-wide token total in audit telemetry#45657
pelikhan merged 8 commits into
mainfrom
copilot/deep-reportsurface-fleet-token-total

Conversation

Copilot AI commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Audit runs report total_tokens = 0 for AWF-based engines (Claude, Codex, Gemini) because their tokens are recorded in the firewall proxy's token-usage.jsonl, not events.jsonl. This caused 10 of 22 audit days to have no token telemetry, forcing all cost reporting to fall back to AIC-only.

Changes

pkg/cli/logs_run_processor.go — backfill Run.TokenUsage from firewall proxy

When event logs yield 0 tokens but the firewall proxy TokenUsageSummary has data, backfill result.Run.TokenUsage from TotalInputTokens + TotalOutputTokens. This matches the semantics of what event log parsing measures, avoids double-counting cache tokens, and propagates correctly to buildLogsData → TotalTokens.

// backfill: AWF-based engines write tokens to the firewall proxy, not events.jsonl
if result.Run.TokenUsage == 0 {
    if firewallTokens := tokenUsage.TotalInputTokens + tokenUsage.TotalOutputTokens; firewallTokens > 0 {
        result.Run.TokenUsage = firewallTokens
    }
}

pkg/cli/token_usage.go — add AWF v0.27.7+ audit-dir path to findTokenUsageFile

AWF v0.27.7+ writes token-usage.jsonl under --audit-dir (sandbox/firewall/audit/api-proxy-logs/) rather than --proxy-logs-dir (sandbox/firewall/logs/api-proxy-logs/). Added an explicit early-return check for this path between the logs/ and legacy firewall-audit-logs/ checks, so the file is found without falling through to the directory walk.

Tests

  • 3 new sub-tests in TestFindTokenUsageFile: AWF audit path, priority ordering when both logs/ and audit/ exist, and absence fallback.
  • New TestBuildLogsDataAggregatesTokensFromRunTokenUsage: documents that Summary.TotalTokens is correctly aggregated from Run.TokenUsage across mixed runs.

…th to token discovery

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add fleet-wide token total in audit telemetry fix: surface fleet-wide token total in audit telemetry Jul 15, 2026
Copilot AI requested a review from pelikhan July 15, 2026 06:56
@github-actions

Copy link
Copy Markdown
Contributor

🤖 PR Triage

Field Value
Category bug
Risk medium
Score 68/100 (impact:35, urgency:20, quality:13)
Action fast_track

Fixes token total surfacing in audit telemetry — 10 affected audit runs report 0. Draft PR but high impact. Needs undrafting and CI confirmation before merge.

Generated by 🔧 PR Triage Agent · 143.8 AIC · ⌖ 7.64 AIC · ⊞ 5.6K ·

@pelikhan pelikhan marked this pull request as ready for review July 15, 2026 07:31
Copilot AI review requested due to automatic review settings July 15, 2026 07:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Backfills fleet-wide token telemetry for AWF engines using firewall proxy data.

Changes:

  • Detects the newer firewall audit token path.
  • Backfills run token totals from proxy summaries.
  • Adds path-selection and aggregation tests.
Show a summary per file
File Description
pkg/cli/token_usage.go Adds AWF audit-path discovery.
pkg/cli/token_usage_test.go Tests discovery and path priority.
pkg/cli/logs_run_processor.go Backfills run-level token usage.
pkg/cli/logs_report_test.go Tests fleet token aggregation.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 4/4 changed files
  • Comments generated: 1
  • Review effort level: Medium

Comment thread pkg/cli/logs_run_processor.go Outdated
// extractLogMetrics measures from events.jsonl.
if result.Run.TokenUsage == 0 && tokenUsage != nil {
if firewallTokens := tokenUsage.TotalInputTokens + tokenUsage.TotalOutputTokens; firewallTokens > 0 {
result.Run.TokenUsage = firewallTokens

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1cb59ad. The firewall backfill now updates both metrics.TokenUsage and result.Metrics.TokenUsage alongside result.Run.TokenUsage, so buildProcessedRun and cached RunSummary.Metrics retain the total.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot run pr-finisher skill

Ran the PR finisher pass. Commit 1cb59ad preserves the firewall token backfill in both metric copies, and local validation passed: make fmt, make lint, make test-unit, make test, and make recompile.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts in this pull request

…rface-fleet-token-total

# Conflicts:
#	pkg/cli/logs_run_processor.go

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts on this branch.

Copilot AI commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Resolved in 6847e66. The branch now merges cleanly with current main, and the AWF token backfill was re-applied in the refactored logs_run_processor flow with targeted tests passing.

…rface-fleet-token-total

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts on this branch.

Resolved in 1c10c79. The branch is now merged with current main, has no merge conflicts, and validation passed (go test ./pkg/cli -run 'TestFindTokenUsageFile|TestBuildLogsDataAggregatesTokensFromRunTokenUsage', make agent-report-progress).

@pelikhan pelikhan merged commit 54a4c42 into main Jul 15, 2026
29 checks passed
@pelikhan pelikhan deleted the copilot/deep-reportsurface-fleet-token-total branch July 15, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[deep-report] Surface fleet-wide token total in audit telemetry (token-usage-reporting-gap, recur #4)

3 participants