fix: surface fleet-wide token total in audit telemetry#45657
Conversation
…th to token discovery Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
🤖 PR Triage
Fixes token total surfacing in audit telemetry — 10 affected audit runs report 0. Draft PR but high impact. Needs undrafting and CI confirmation before merge.
|
There was a problem hiding this comment.
Pull request overview
Backfills fleet-wide token telemetry for AWF engines using firewall proxy data.
Changes:
- Detects the newer firewall audit token path.
- Backfills run token totals from proxy summaries.
- Adds path-selection and aggregation tests.
Show a summary per file
| File | Description |
|---|---|
pkg/cli/token_usage.go |
Adds AWF audit-path discovery. |
pkg/cli/token_usage_test.go |
Tests discovery and path priority. |
pkg/cli/logs_run_processor.go |
Backfills run-level token usage. |
pkg/cli/logs_report_test.go |
Tests fleet token aggregation. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Medium
| // extractLogMetrics measures from events.jsonl. | ||
| if result.Run.TokenUsage == 0 && tokenUsage != nil { | ||
| if firewallTokens := tokenUsage.TotalInputTokens + tokenUsage.TotalOutputTokens; firewallTokens > 0 { | ||
| result.Run.TokenUsage = firewallTokens |
There was a problem hiding this comment.
Fixed in 1cb59ad. The firewall backfill now updates both metrics.TokenUsage and result.Metrics.TokenUsage alongside result.Run.TokenUsage, so buildProcessedRun and cached RunSummary.Metrics retain the total.
|
@copilot run pr-finisher skill |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot resolve the merge conflicts in this pull request |
…rface-fleet-token-total # Conflicts: # pkg/cli/logs_run_processor.go Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot resolve the merge conflicts on this branch. |
…rface-fleet-token-total Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Audit runs report
total_tokens = 0for AWF-based engines (Claude, Codex, Gemini) because their tokens are recorded in the firewall proxy'stoken-usage.jsonl, notevents.jsonl. This caused 10 of 22 audit days to have no token telemetry, forcing all cost reporting to fall back to AIC-only.Changes
pkg/cli/logs_run_processor.go— backfillRun.TokenUsagefrom firewall proxyWhen event logs yield 0 tokens but the firewall proxy
TokenUsageSummaryhas data, backfillresult.Run.TokenUsagefromTotalInputTokens + TotalOutputTokens. This matches the semantics of what event log parsing measures, avoids double-counting cache tokens, and propagates correctly tobuildLogsData → TotalTokens.pkg/cli/token_usage.go— add AWF v0.27.7+ audit-dir path tofindTokenUsageFileAWF v0.27.7+ writes
token-usage.jsonlunder--audit-dir(sandbox/firewall/audit/api-proxy-logs/) rather than--proxy-logs-dir(sandbox/firewall/logs/api-proxy-logs/). Added an explicit early-return check for this path between thelogs/and legacyfirewall-audit-logs/checks, so the file is found without falling through to the directory walk.Tests
TestFindTokenUsageFile: AWF audit path, priority ordering when bothlogs/andaudit/exist, and absence fallback.TestBuildLogsDataAggregatesTokensFromRunTokenUsage: documents thatSummary.TotalTokensis correctly aggregated fromRun.TokenUsageacross mixed runs.