feat(tracing): OTel span queue and export telemetry (SGPINF-1863)

[james-cardenas]

Summary

SGPINF-1863 - Validate SDK Connection Fix (PR #362) to Unblock 2x Export Capacity

• Add OpenTelemetry metrics for async span queue processing: depth, batch lag, drain duration, and shutdown flush timing.
• Record SGP export success/failure counters on async upsert_batch completion with bounded HTTP status labels.
• Introduce AGENTEX_TRACING_METRICS=0|false|no|off kill switch to disable SDK-side recording without code changes.

Stacked on PR362 (stas/tracing-perf-linger-keepalive) for load-test observability before M5 re-run.

Test plan

• uv run pytest src/agentex/lib/core/observability/tests/test_tracing_metrics.py src/agentex/lib/core/observability/tests/test_tracing_metrics_recording.py tests/lib/core/tracing/test_span_queue.py tests/lib/core/tracing/processors/test_sgp_tracing_processor.py -o addopts=
• Pin SDK in rocket mock agent and verify metrics appear in staging Mimir during a short load test
• Confirm AGENTEX_TRACING_METRICS=0 disables recording with no export overhead regression

Greptile Summary

This PR adds OpenTelemetry metrics instrumentation to the async span queue and SGP exporter: queue depth, batch lag, drain duration, export success/failure counters, and shutdown flush timing. An AGENTEX_TRACING_METRICS kill switch disables recording without code changes. All three issues flagged in the prior review (queue depth under-reporting, unbounded http_code label cardinality, and hard-coded "sgp" in record_export_success) have been addressed.

• tracing_metrics.py / tracing_metrics_recording.py: New modules define all OTel instruments and thin "best-effort" recording helpers that lazy-load the OTel SDK only on first use, keeping hot-path import overhead near zero.
• span_queue.py: Recording calls are woven into enqueue, drain-batch coalesce, per-phase dispatch timing, retry-exhausted failure, permanent failure, and shutdown-timeout paths; the record_batch_coalesced call correctly passes self._queue.qsize() + len(batch) to capture pre-drain depth.
• sgp_tracing_processor.py: record_export_success is called after each successful upsert_batch, taking an explicit processor label rather than a hard-coded string.

Confidence Score: 5/5

Safe to merge. All metrics calls are wrapped in best-effort try/except blocks and gated by the kill switch, so a misconfigured OTel provider cannot disrupt span export.

All three issues raised in the previous review have been correctly resolved. The failure-metric paths in _handle_failure are mutually exclusive — the retryable branch returns early, so the permanent-failure counter at the bottom is never double-fired. Recording helpers lazy-load OTel and swallow all exceptions, keeping the hot path safe. Test coverage is thorough across unit and integration layers.

No files require special attention.

Important Files Changed

Filename	Overview
src/agentex/lib/core/observability/tracing_metrics.py	New module: defines all OTel instruments for span queue + export telemetry. Cardinality is well bounded; classify_export_error now maps out-of-range codes to the sentinel 'other', preventing new timeseries per unique integer.
src/agentex/lib/core/observability/tracing_metrics_recording.py	New recording helpers: lazy OTel load, kill-switch guard, best-effort try/except on every call. record_export_success now accepts an explicit processor parameter. Logic is clean and test-covered.
src/agentex/lib/core/tracing/span_queue.py	Metrics integration across enqueue, drain-batch coalesce (depth fix with + len(batch)), per-phase timing, and failure paths. Failure paths are mutually exclusive (retryable returns early), so no double-counting.
src/agentex/lib/core/tracing/processors/sgp_tracing_processor.py	Adds record_export_success after each successful upsert_batch with explicit processor='sgp' label; called only after the await completes successfully so failures propagate normally.
tests/lib/core/tracing/test_span_queue.py	New TestAsyncSpanQueueMetrics class covers depth-inclusive batch coalescing, enqueued counter, dropped-on-shutdown counter, export failure on processor error, and kill-switch overhead check.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[enqueue] -->|put_nowait| B[_SpanQueueItem\nenqueued_at = monotonic]
    A -->|QueueFull| C[record_span_dropped\nreason=queue_full]
    A -->|_stopping| D[record_span_dropped\nreason=shutdown]
    A --> E[record_span_enqueued]
    B --> F[_drain_loop\ncollect batch]
    F --> G[record_batch_coalesced\nqueue_depth = qsize + len_batch]
    G --> H[_process_items START]
    H --> I[await processor.on_spans_start]
    I -->|success| J[record_export_success]
    I -->|exception| K{_is_retryable_exc?}
    K -->|yes, exhausted| L[record_export_failure\nspan_count=exhausted]
    K -->|yes, retriable| M[_reenqueue items]
    K -->|no| N[record_export_failure\nspan_count=all items]
    G --> O[await barrier]
    O --> P[_process_items END]
    P --> Q[await processor.on_spans_end]
    Q -->|success| R[record_export_success]
    Q -->|exception| S{_is_retryable_exc?}
    S -->|yes, exhausted| T[record_export_failure]
    S -->|yes, retriable| U[_reenqueue items]
    S -->|no| V[record_export_failure]
    W[shutdown timeout] --> X[record_shutdown_timeout]

Loading

_{Reviews (11): Last reviewed commit: "Standardize on len(spans) in tracing pro..." | Re-trigger Greptile}

[smoreinis]

left a few minor comments inline, but nothing to block getting this in - I did merge the underlying PR into next so you will need a rebase but hopefully it should be relatively painless. will be great to have more observability on the spans behavior!!

[socket-security]

No dependency changes detected. Learn more about Socket for GitHub.

👍 No dependency changes detected in pull request