Add AI agent detection to CLI analytics

[amcaplan]

WHY are these changes introduced?

CLI analytics can't distinguish AI-agent runs from human or scripted ones. The AI toolkit has agents declare themselves through SHOPIFY_CLI_AGENT_INFO / SHOPIFY_CLI_AGENT_IDS; agents outside it stay invisible.

https://github.com/shop/issues-develop/issues/23864

WHAT is this pull request doing?

Detect the agent with @vercel/detect-agent and report it through the existing sensitive env_shopify_variables field, so there's no Monorail schema change. Detected names reuse the packed format producers already send — SHOPIFY_CLI_AGENT_INFO="n:<name>" — so the agent column resolves identically whether declared or detected. Only n: is set; detection can't resolve version, provider or model.

Decisions worth review:

• Detection is skipped when either declared variable is present: writing SHOPIFY_CLI_AGENT_INFO ourselves would clobber the producer's whole packed value, not just the name.
• SHOPIFY_CLI_AGENT_DETECTED=true marks a derived name. It's deliberately off the allowlist, which filters before the merge, so only the CLI can set it.
• | becomes _ in detected names — it separates tags with no escape sequence, so one could otherwise fabricate tags like v:.
• Not gated on analytics delivery: determineAgent() measures 15µs worst case, against an event that does a network POST.

How to manually test your changes?

bin/dev.js runs the bundle, so pnpm nx bundle --skip-nx-cache first. Detected, nothing declared:

env -u SHOPIFY_CLI_AGENT_INFO -u SHOPIFY_CLI_AGENT_IDS AI_AGENT=codex SHOPIFY_CLI_ALWAYS_LOG_ANALYTICS=1 node packages/cli/bin/dev.js version --verbose | grep env_shopify_variables

Declared, which detection must leave untouched and unmarked; declared (cursor) and detected (codex) differ so you can see which won:

env -u SHOPIFY_CLI_AGENT_IDS AI_AGENT=codex SHOPIFY_CLI_AGENT_INFO='n:cursor|v:2.1.0|p:openai|m:gpt-5' SHOPIFY_CLI_ALWAYS_LOG_ANALYTICS=1 node packages/cli/bin/dev.js version --verbose | grep env_shopify_variables

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate findings must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds fallback AI-agent detection to CLI analytics using @vercel/detect-agent, while preserving declared agent metadata.

Changes:

• Adds and locks the detection dependency.
• Detects, sanitizes, and reports agent names through sensitive analytics fields.
• Adds detection and analytics integration tests.

Review findings include two critical issues in analytics gating and test fixtures, four moderate issues involving send eligibility and name handling, and one documentation nit.

File summaries

File	Description
pnpm-lock.yaml	Locks the new detection dependency.
packages/cli-kit/src/public/node/analytics.ts	Provides analytics skip-check behavior.
packages/cli-kit/src/public/node/analytics.test.ts	Tests analytics integration and opt-out behavior.
packages/cli-kit/src/private/node/context/agent.ts	Implements agent detection, mapping, and sanitization.
packages/cli-kit/src/private/node/context/agent.test.ts	Tests detection behavior and normalization.
packages/cli-kit/src/private/node/analytics.ts	Integrates detected variables into analytics payloads.
packages/cli-kit/package.json	Adds the detection dependency.

Review details

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

Suppressed comments (2)

packages/cli-kit/src/private/node/analytics.ts:136

• env_shopify_variables is only delivered in the Monorail payload, but this gate treats alwaysLogMetrics as sufficient to run detection. With analytics disabled and only the metrics override enabled, monorailAnalyticsSkipped() remains true, so sendAnalyticsEvent drops the Monorail payload and the detected values are discarded after the detection work. Gate this enrichment on Monorail delivery instead; the related test should not require detection for a metrics-only send.
  packages/cli-kit/src/private/node/context/agent.ts:33
• The lookup happens after replacing |, so an arbitrary AI_AGENT value can collide with a canonical detector name. With the pinned detector names, AI_AGENT=claude|code becomes claude_code and is then remapped to claude-code, falsely attributing it to the Shopify toolkit instead of preserving the sanitized custom name. Apply the known-agent mapping to rawDetectedName before sanitizing the fallback.

    const detectedName = rawDetectedName.replaceAll('|', '_').trim()

    return {
      SHOPIFY_CLI_AGENT_INFO: `n:${toolkitAgentNamesByDetectedName[detectedName] ?? detectedName}`,

• Files reviewed: 6/7 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

• Some seemingly private modules might be re-exported through public modules.
• If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

packages/cli-kit/dist/private/node/context/agent.d.ts

export declare function detectedAgentEnvironmentVariables(env?: NodeJS.ProcessEnv): Promise<NodeJS.ProcessEnv>;

Existing type declarations

We found no diffs with existing type declarations