Python: add feature-usage User-Agent telemetry - #7420
Conversation
There was a problem hiding this comment.
Automated Code Review
Reviewers: 5 | Confidence: 68%
✓ Correctness
No actionable issues found in this dimension.
✓ Security Reliability
No actionable issues found in this dimension.
✓ Test Coverage
Test coverage is uneven: orchestrations sequential/magentic builders and purview process_content have explicit feature-marking assertions, but several other instrumented call sites lack corresponding tests. The tools/shell package has two production files instrumented with no feature-marking tests in the diff. The comprehensive registry validation test provides structural coverage (every FeatureIndex is referenced) but does not verify runtime marking behavior at each call site.
✓ Failure Modes
No actionable issues found in this dimension.
✓ Design Approach
The new feature-mask plumbing is mostly consistent, but the Foundry OpenAI integration now unconditionally swaps in package-owned HTTP clients on both the chat-client and agent paths. That conflicts with the PR’s stated requirement that caller-owned transports remain unchanged, and it can silently discard custom retry/auth/event-hook behavior supplied through a caller-provided project/OpenAI client.
Suggestions
- The tools/shell package adds mark_feature_used(FeatureIndex.TOOLS_SHELL) in both _docker.py:447 and _tool.py:246, but no tests in the diff verify these calls. Consider adding a test (similar to the pattern in other packages) that patches mark_feature_used and asserts it is called with FeatureIndex.TOOLS_SHELL when run() is invoked.
Automated review by eavanvalkenburg's agents
There was a problem hiding this comment.
Pull request overview
This PR implements ADR-0033/SPEC-004 “feature-usage” telemetry for the Python SDK by maintaining a process-global monotonic 128-bit feature mask, stamping it into the Agent Framework User-Agent only on approved first-party request destinations, and adding per-package feature index ownership/activation plus registry parity validation.
Changes:
- Adds core feature-mask support (
mark_feature_used, token formatting, opt-out env var) and switches the base Agent Framework User-Agent version source to the installedagent-framework-coredistribution. - Introduces per-package
FeatureIndexdeclarations and marks “first meaningful activation” across many packages (clients/providers/orchestrations/tools/etc.), with expanded test coverage. - Adds destination-aware User-Agent stamping for Azure OpenAI (httpx hook) and Foundry (Azure pipeline policy + OpenAI client hook), plus documentation updates.
Reviewed changes
Copilot reviewed 151 out of 151 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| python/samples/README.md | Documents env vars for User-Agent/feature token controls. |
| python/packages/tools/agent_framework_tools/shell/_tool.py | Marks shell tool feature usage. |
| python/packages/tools/agent_framework_tools/shell/_docker.py | Marks shell tool feature usage (docker impl). |
| python/packages/tools/agent_framework_tools/_feature_usage.py | Declares tools package feature index. |
| python/packages/redis/tests/test_providers.py | Adds assertions for Redis feature marking. |
| python/packages/redis/agent_framework_redis/_history_provider.py | Marks Redis provider usage in history operations. |
| python/packages/redis/agent_framework_redis/_feature_usage.py | Declares Redis package feature index. |
| python/packages/redis/agent_framework_redis/_context_provider.py | Marks Redis provider usage in context operations. |
| python/packages/purview/tests/purview/test_purview_client.py | Adds assertions for Purview feature marking. |
| python/packages/purview/agent_framework_purview/_feature_usage.py | Declares Purview package feature index. |
| python/packages/purview/agent_framework_purview/_client.py | Marks Purview client feature usage. |
| python/packages/orchestrations/tests/test_sequential.py | Validates orchestration vs core workflow attribution. |
| python/packages/orchestrations/tests/test_magentic.py | Validates Magentic orchestration feature marking. |
| python/packages/orchestrations/agent_framework_orchestrations/_workflow_builder.py | Adds orchestration-specific WorkflowBuilder override to suppress core workflow bit. |
| python/packages/orchestrations/agent_framework_orchestrations/_sequential.py | Marks sequential orchestration feature usage. |
| python/packages/orchestrations/agent_framework_orchestrations/_orchestration_request_info.py | Uses orchestration builder override. |
| python/packages/orchestrations/agent_framework_orchestrations/_magentic.py | Marks Magentic orchestration feature usage. |
| python/packages/orchestrations/agent_framework_orchestrations/_handoff.py | Marks handoff orchestration feature usage. |
| python/packages/orchestrations/agent_framework_orchestrations/_group_chat.py | Marks group chat orchestration feature usage. |
| python/packages/orchestrations/agent_framework_orchestrations/_feature_usage.py | Declares orchestration feature indices. |
| python/packages/orchestrations/agent_framework_orchestrations/_concurrent.py | Marks concurrent orchestration feature usage. |
| python/packages/openai/tests/openai/test_openai_shared.py | Tests Azure OpenAI feature-stamping hook + client cleanup behavior. |
| python/packages/openai/tests/openai/test_openai_embedding_client.py | Tests OpenAI embedding feature marking. |
| python/packages/openai/tests/openai/test_openai_chat_completion_client.py | Tests OpenAI streaming feature marking timing. |
| python/packages/openai/agent_framework_openai/_shared.py | Injects feature-usage-aware http client for Azure OpenAI clients. |
| python/packages/openai/agent_framework_openai/_feature_usage.py | Adds OpenAI feature index + approved-origin stamping client. |
| python/packages/openai/agent_framework_openai/_embedding_client.py | Marks OpenAI embedding usage. |
| python/packages/openai/agent_framework_openai/_chat_completion_client.py | Marks OpenAI chat-completion usage (incl. streaming timing). |
| python/packages/openai/agent_framework_openai/_chat_client.py | Marks OpenAI Responses client usage, including continuation flows. |
| python/packages/ollama/tests/ollama/test_ollama_embedding_client.py | Adds assertions for Ollama feature marking. |
| python/packages/ollama/agent_framework_ollama/_feature_usage.py | Declares Ollama package feature index. |
| python/packages/ollama/agent_framework_ollama/_embedding_client.py | Marks Ollama embedding usage. |
| python/packages/ollama/agent_framework_ollama/_chat_client.py | Marks Ollama chat usage. |
| python/packages/monty/agent_framework_monty/_provider.py | Marks Monty provider usage. |
| python/packages/monty/agent_framework_monty/_feature_usage.py | Declares Monty package feature index. |
| python/packages/mistral/tests/mistral/test_mistral_embedding_client.py | Adds assertions for Mistral feature marking. |
| python/packages/mistral/agent_framework_mistral/_feature_usage.py | Declares Mistral package feature index. |
| python/packages/mistral/agent_framework_mistral/_embedding_client.py | Marks Mistral embedding usage. |
| python/packages/mem0/tests/test_mem0_context_provider.py | Adds assertions for Mem0 feature marking. |
| python/packages/mem0/agent_framework_mem0/_feature_usage.py | Declares Mem0 package feature index. |
| python/packages/mem0/agent_framework_mem0/_context_provider.py | Marks Mem0 provider usage. |
| python/packages/lab/tau2/agent_framework_lab_tau2/runner.py | Marks Lab usage in tau2 runner. |
| python/packages/lab/pyproject.toml | Adds shared lab-common package + pyright paths. |
| python/packages/lab/lightning/agent_framework_lab_lightning/init.py | Marks Lab usage in lightning tracer init. |
| python/packages/lab/gaia/agent_framework_lab_gaia/gaia.py | Marks Lab usage across GAIA entry points. |
| python/packages/lab/common/agent_framework_lab_common/_feature_usage.py | Declares Lab common feature index. |
| python/packages/lab/common/agent_framework_lab_common/init.py | Adds package initializer. |
| python/packages/hyperlight/agent_framework_hyperlight/_provider.py | Marks Hyperlight provider usage. |
| python/packages/hyperlight/agent_framework_hyperlight/_feature_usage.py | Declares Hyperlight package feature index. |
| python/packages/hosting/agent_framework_hosting/_state.py | Marks hosting state usage on initialization. |
| python/packages/hosting/agent_framework_hosting/_feature_usage.py | Declares hosting package feature index. |
| python/packages/hosting-telegram/agent_framework_hosting_telegram/_rendering.py | Marks Telegram hosting feature usage during rendering. |
| python/packages/hosting-telegram/agent_framework_hosting_telegram/_parsing.py | Marks Telegram hosting feature usage during parsing. |
| python/packages/hosting-telegram/agent_framework_hosting_telegram/_feature_usage.py | Declares Telegram hosting feature index. |
| python/packages/hosting-responses/agent_framework_hosting_responses/_parsing.py | Marks Responses hosting feature usage. |
| python/packages/hosting-responses/agent_framework_hosting_responses/_feature_usage.py | Declares Responses hosting feature index. |
| python/packages/hosting-mcp/agent_framework_hosting_mcp/_workflow_tool.py | Marks MCP hosting usage on tool operations. |
| python/packages/hosting-mcp/agent_framework_hosting_mcp/_feature_usage.py | Declares MCP hosting feature index. |
| python/packages/hosting-mcp/agent_framework_hosting_mcp/_agent_tool.py | Marks MCP hosting usage on tool operations. |
| python/packages/hosting-a2a/agent_framework_hosting_a2a/_feature_usage.py | Declares A2A hosting feature index. |
| python/packages/hosting-a2a/agent_framework_hosting_a2a/_conversion.py | Marks A2A hosting usage in conversions. |
| python/packages/github_copilot/tests/test_github_copilot_agent.py | Adds assertions for GitHub Copilot agent feature marking. |
| python/packages/github_copilot/agent_framework_github_copilot/_feature_usage.py | Declares GitHub Copilot package feature index. |
| python/packages/github_copilot/agent_framework_github_copilot/_agent.py | Marks GitHub Copilot agent usage. |
| python/packages/gemini/tests/test_gemini_client.py | Adds assertions for Gemini feature marking. |
| python/packages/gemini/agent_framework_gemini/_feature_usage.py | Declares Gemini package feature index. |
| python/packages/gemini/agent_framework_gemini/_chat_client.py | Marks Gemini chat usage. |
| python/packages/foundry/tests/foundry/test_foundry_memory_provider.py | Updates expectations around Foundry client policy wiring. |
| python/packages/foundry/tests/foundry/test_foundry_embedding_client.py | Updates expectations around Foundry client policy wiring. |
| python/packages/foundry/tests/foundry/test_foundry_chat_client.py | Adds tests for Foundry feature policy refresh + redirect stripping. |
| python/packages/foundry/tests/foundry/test_foundry_agent.py | Adds tests for Foundry/OpenAI feature-bit separation + policy wiring. |
| python/packages/foundry/agent_framework_foundry/_memory_provider.py | Adds Foundry policy/hook wiring and marks Foundry memory usage. |
| python/packages/foundry/agent_framework_foundry/_foundry_evals.py | Wires feature-usage-aware OpenAI client for evals and marks eval usage. |
| python/packages/foundry/agent_framework_foundry/_feature_usage.py | Declares Foundry feature indices + policy/http-client factories. |
| python/packages/foundry/agent_framework_foundry/_embedding_client.py | Wires Foundry policy and marks embedding usage. |
| python/packages/foundry/agent_framework_foundry/_chat_client.py | Wires Foundry policy/http-client and sets Foundry chat feature index. |
| python/packages/foundry/agent_framework_foundry/_agent.py | Wires Foundry policy/http-client and sets Foundry agent feature index. |
| python/packages/foundry_local/tests/test_foundry_local_client.py | Adds assertion for Foundry Local feature index ownership. |
| python/packages/foundry_local/agent_framework_foundry_local/_foundry_local_client.py | Sets Foundry Local feature index. |
| python/packages/foundry_local/agent_framework_foundry_local/_feature_usage.py | Declares Foundry Local feature index. |
| python/packages/foundry_hosting/tests/test_toolbox.py | Adds assertions for toolbox feature marking + index ownership. |
| python/packages/foundry_hosting/agent_framework_foundry_hosting/_toolbox.py | Marks toolbox usage on connect. |
| python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py | Marks Foundry hosting usage. |
| python/packages/foundry_hosting/agent_framework_foundry_hosting/_invocations.py | Marks Foundry hosting usage. |
| python/packages/foundry_hosting/agent_framework_foundry_hosting/_feature_usage.py | Declares Foundry hosting feature indices. |
| python/packages/durabletask/agent_framework_durabletask/_worker.py | Marks Durable Task usage when worker starts. |
| python/packages/durabletask/agent_framework_durabletask/_shim.py | Marks Durable Task usage on shim run. |
| python/packages/durabletask/agent_framework_durabletask/_feature_usage.py | Declares Durable Task feature index. |
| python/packages/devui/agent_framework_devui/_feature_usage.py | Declares DevUI feature index. |
| python/packages/devui/agent_framework_devui/init.py | Marks DevUI usage when starting server. |
| python/packages/declarative/tests/test_workflow_factory.py | Adds assertions for declarative workflow feature marking. |
| python/packages/declarative/tests/test_declarative_loader.py | Adds assertions for declarative agent feature marking. |
| python/packages/declarative/agent_framework_declarative/_workflows/_factory.py | Marks declarative workflow usage on successful creation. |
| python/packages/declarative/agent_framework_declarative/_loader.py | Marks declarative agent usage on successful creation. |
| python/packages/declarative/agent_framework_declarative/_feature_usage.py | Declares declarative feature indices. |
| python/packages/core/tests/core/test_telemetry.py | Adds comprehensive feature-mask + registry validator tests. |
| python/packages/core/README.md | Documents telemetry controls (mask-only vs full UA disable). |
| python/packages/core/AGENTS.md | Documents _telemetry.py module responsibility. |
| python/packages/core/agent_framework/_workflows/_workflow_builder.py | Adds core workflow feature marking via _FEATURE_USAGE_INDEX. |
| python/packages/core/agent_framework/_telemetry.py | Implements feature mask, token formatting, env var controls, and token (un)stamping helpers. |
| python/packages/core/agent_framework/_skills.py | Marks skills provider and skills sources feature usage. |
| python/packages/core/agent_framework/_sessions.py | Marks in-memory and file history provider usage. |
| python/packages/core/agent_framework/_mcp.py | Marks MCP usage on connect. |
| python/packages/core/agent_framework/_harness/_tool_approval.py | Marks tool-approval harness usage. |
| python/packages/core/agent_framework/_harness/_todo.py | Marks todo harness usage. |
| python/packages/core/agent_framework/_harness/_mode.py | Marks agent mode provider usage. |
| python/packages/core/agent_framework/_harness/_memory.py | Marks memory provider usage. |
| python/packages/core/agent_framework/_harness/_file_access.py | Marks file access provider usage. |
| python/packages/core/agent_framework/_harness/_background_agents.py | Marks background agents provider usage. |
| python/packages/core/agent_framework/_harness/_agent.py | Marks harness agent creation usage. |
| python/packages/core/agent_framework/_compaction.py | Marks compaction provider usage. |
| python/packages/core/agent_framework/_agents.py | Marks core agent usage on run. |
| python/packages/core/agent_framework/init.py | Resolves version from installed agent-framework-core distribution. |
| python/packages/copilotstudio/tests/test_copilot_agent.py | Adds assertions for Copilot Studio feature marking. |
| python/packages/copilotstudio/agent_framework_copilotstudio/_feature_usage.py | Declares Copilot Studio feature index. |
| python/packages/copilotstudio/agent_framework_copilotstudio/_agent.py | Marks Copilot Studio usage on run/stream. |
| python/packages/claude/tests/test_claude_agent.py | Adds assertions for Claude feature marking. |
| python/packages/claude/agent_framework_claude/_feature_usage.py | Declares Claude feature index. |
| python/packages/claude/agent_framework_claude/_agent.py | Marks Claude usage on query/receive path. |
| python/packages/chatkit/agent_framework_chatkit/_streaming.py | Marks ChatKit usage on streaming conversion. |
| python/packages/chatkit/agent_framework_chatkit/_feature_usage.py | Declares ChatKit feature index. |
| python/packages/chatkit/agent_framework_chatkit/_converter.py | Marks ChatKit usage on message conversion. |
| python/packages/bedrock/tests/test_bedrock_client.py | Adds assertions for Bedrock feature marking. |
| python/packages/bedrock/agent_framework_bedrock/_feature_usage.py | Declares Bedrock feature index. |
| python/packages/bedrock/agent_framework_bedrock/_embedding_client.py | Marks Bedrock embedding usage. |
| python/packages/bedrock/agent_framework_bedrock/_chat_client.py | Marks Bedrock chat usage. |
| python/packages/azurefunctions/agent_framework_azurefunctions/_feature_usage.py | Declares Azure Functions feature index. |
| python/packages/azurefunctions/agent_framework_azurefunctions/_app.py | Marks Azure Functions app usage on init. |
| python/packages/azure-cosmos/tests/test_cosmos_history_provider.py | Adds assertions for Cosmos feature marking. |
| python/packages/azure-cosmos/agent_framework_azure_cosmos/_history_provider.py | Marks Cosmos history provider usage. |
| python/packages/azure-cosmos/agent_framework_azure_cosmos/_feature_usage.py | Declares Cosmos feature index. |
| python/packages/azure-cosmos/agent_framework_azure_cosmos/_checkpoint_storage.py | Marks Cosmos checkpoint storage usage. |
| python/packages/azure-cosmos-memory/tests/test_context_provider.py | Adds assertions for Cosmos memory feature marking. |
| python/packages/azure-cosmos-memory/agent_framework_azure_cosmos_memory/_feature_usage.py | Declares Cosmos memory feature index. |
| python/packages/azure-cosmos-memory/agent_framework_azure_cosmos_memory/_context_provider.py | Marks Cosmos memory provider usage. |
| python/packages/azure-contentunderstanding/tests/cu/test_context_provider.py | Adds assertions for Content Understanding feature marking. |
| python/packages/azure-contentunderstanding/agent_framework_azure_contentunderstanding/_feature_usage.py | Declares Content Understanding feature index. |
| python/packages/azure-contentunderstanding/agent_framework_azure_contentunderstanding/_context_provider.py | Marks Content Understanding provider usage. |
| python/packages/azure-ai-search/tests/test_aisearch_context_provider.py | Adds assertions for Azure AI Search feature marking. |
| python/packages/azure-ai-search/agent_framework_azure_ai_search/_feature_usage.py | Declares Azure AI Search feature index. |
| python/packages/azure-ai-search/agent_framework_azure_ai_search/_context_provider.py | Marks Azure AI Search provider usage. |
| python/packages/anthropic/tests/test_anthropic_client.py | Adds assertions for Anthropic feature marking. |
| python/packages/anthropic/agent_framework_anthropic/_feature_usage.py | Declares Anthropic feature index. |
| python/packages/anthropic/agent_framework_anthropic/_chat_client.py | Marks Anthropic chat usage. |
| python/packages/ag-ui/agent_framework_ag_ui/_workflow.py | Marks AG-UI workflow usage. |
| python/packages/ag-ui/agent_framework_ag_ui/_feature_usage.py | Declares AG-UI feature index. |
| python/packages/ag-ui/agent_framework_ag_ui/_client.py | Marks AG-UI client streaming usage. |
| python/packages/ag-ui/agent_framework_ag_ui/_agent.py | Marks AG-UI agent usage. |
| python/packages/a2a/agent_framework_a2a/_feature_usage.py | Declares A2A feature index. |
| python/packages/a2a/agent_framework_a2a/_agent.py | Marks A2A usage in agent mapping path. |
| python/packages/a2a/agent_framework_a2a/_a2a_executor.py | Marks A2A usage in executor path. |
|
Addressed the automated review summary in 6187492: caller-provided Foundry project clients now retain their transport, conversation creation reuses the existing OpenAI client, and LocalShellTool plus DockerShellTool have explicit activation-time |
There was a problem hiding this comment.
Automated Code Review
Reviewers: 5 | Confidence: 62%
✓ Correctness
No actionable issues found in this dimension.
✓ Security Reliability
No actionable issues found in this dimension.
✓ Test Coverage
The PR adds comprehensive feature-usage telemetry marks across ~20 packages. Core telemetry logic is well-tested (mask accumulation, threading, disable controls, registry parity). Most leaf-package marks have dedicated tests. However, several packages show mark_feature_used calls without corresponding test assertions visible in the diff: AG-UI (3 call sites), A2A (2 call sites), ChatKit (2 call sites), DevUI, Azure Functions, DurableTask (2 call sites), Bedrock embedding client, and most core harness/session/skills providers. While 53+ additional test files are noted as omitted from the diff, the coverage pattern across the visible packages suggests some of these may genuinely lack unit-level mark verification. The PR adds feature-usage marking across ~25 packages. Most packages with visible test changes have appropriate test assertions verifying mark_feature_used is called with the correct FeatureIndex. The structural registry-parity test in test_telemetry.py provides broad declarative coverage but does not verify runtime call-site behavior. The new extra_headers pass-through in the ChatCompletion client also lacks direct test coverage.
✓ Failure Modes
No actionable issues found in this dimension.
✓ Design Approach
I found one non-blocking design concern: some hosting feature bits are marked during object construction rather than at first meaningful use, which can over-report capabilities that were instantiated during startup but never actually exercised. The rest of the chunk is consistent with the PR’s feature-usage approach and the approved-origin transport handling.
Suggestions
- The AG-UI package adds mark_feature_used(FeatureIndex.AG_UI) in three distinct entry points (_agent.py:144, _client.py:400, _workflow.py:300). If these lack dedicated tests (not visible in this diff chunk), consider adding at least one test per entry point to match the pattern used by other packages.
Automated review by eavanvalkenburg's agents
Add the 128-bit feature accumulator, package-local indexes, activation markers, and destination-scoped User-Agent emission for the initial Python implementation slice. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
6187492 to
fcb8f57
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 346bf168-b668-4c4a-a8db-a67282ee5e5f
Motivation & Context
Agent Framework's User-Agent identifies the Python SDK, but it does not show
which framework capabilities have been observed together in a process. ADR-0033
defines a transparent, first-party-only feature mask so product and support
analysis can answer coarse adoption and co-occurrence questions without adding a
new telemetry pipeline or turning the signal into per-call tracking.
Description & Review Guide
mechanism: a thread-safe, process-global 128-bit monotonic mask; package-local
FeatureIndexdeclarations and activation marks for all 63 assigned Pythonregistry rows; a strict repository validator for exact registry parity,
complete coverage, valid range, and zero overlap; and the dedicated
AGENT_FRAMEWORK_FEATURE_MASK_DISABLEDopt-out. Approved Azure OpenAI andFoundry request hooks append
(feat=v1.<hex>)only for reviewed HTTPS origins,refresh it at request time, and remove it on unapproved redirect hops. OpenAI
hooks retain SDK HTTP defaults and cleanup behavior, while Foundry policies
preserve Azure SDK custom hooks. The base User-Agent now resolves its version
from the installed
agent-framework-coredistribution.a bounded feature token representing capabilities observed at least once in
the current process. The bits are binary and non-countable: repeated requests
do not represent additional feature invocations or request-level attribution.
Third-party/custom origins and caller-owned transports remain unchanged.
and redirect boundaries, process-global privacy semantics, transport lifecycle
preservation, and the registry completeness test.
Related Issue
Fixes #7421. Contributes the Python implementation for #7240. No other open PR
implements this issue.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.