Add shopify validate command for offline validation#8142
Add shopify validate command for offline validation#8142craigmichaelmartin wants to merge 18 commits into
shopify validate command for offline validation#8142Conversation
Consolidate the 20 Shopify-AI-Toolkit skills into a single .agents/skills/shopify/SKILL.md. The description matches liberally across every Shopify surface (Admin/Storefront/Customer/Partner/Payments GraphQL, custom data, Functions, Hydrogen, Liquid, Polaris + UI extensions, onboarding, CLI ops, UCP, App Store review, docs search) within the 1024-char limit. The body makes two steps mandatory and non-skippable: search dev docs first via `shopify doc search`, then validate the final outcome via `shopify validate [topic]`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review feedback: be explicit that `shopify commands` lists the available commands and `--help` at any level describes commands/subcommands/flags, so the agent confirms the real surface (including `shopify validate`) instead of guessing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Distinguish two outcomes in the validate step: a chat-reply example must be a complete, fully valid artifact (no snippets or placeholder syntax) and validated via a temp file; a filesystem change must validate every touched file with the appropriate topic, not just the main one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude Code 2.1.86 caps skill descriptions at 250 chars in the /skills listing (to reduce context), so the previous 995-char description was truncated mid-list — ~12 of 20 surfaces (Hydrogen, themes, extensions, onboarding, CLI, UCP, App Store review, docs) fell past the cut and never reached the matcher. Replace it with a 248-char gateway that names every surface up front and survives the cap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review feedback: the helper-reuse note should point to @shopify/cli, not @shopify/cli-kit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the four validation scripts from ai-toolkit-source's
shopify-dev-tools package into first-class CLI subcommands:
- `shopify validate theme` Liquid/theme code via @shopify/theme-check-*
- `shopify validate graphql` GraphQL operations against bundled API schemas
(+ required offline access scopes)
- `shopify validate functions` Shopify Functions input queries
- `shopify validate components` UI-extension component code (TSX/JSX/HTML)
type-checked in a virtual TypeScript environment
All four share one dependency-free engine (result contract, formatter,
pure version resolution, and a lazy gunzip walk-up data-loader) under
packages/cli/src/cli/services/validate/engine. The barrel-decoupled
contract keeps the TypeScript compiler out of the non-component
validators. Reference schemas and type definitions are bundled for fully
offline, deterministic validation. Telemetry rides cli-kit's
addPublicMetadata via new cmd_validate_* fields; the source scripts'
agent-only plumbing (artifact IDs, network instrumentation) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
🤖 Code Review · Skipped — PR has too many files for review (limit: 1000) |
The doc-search --api-name flag is free-form: valid values are defined by shopify.dev (not the CLI), documented examples are admin/storefront/ hydrogen/functions, and unrecognized values are silently ignored (no error, unlike --api-version). Stop inventing values (customer/partner/ payments/liquid/polaris), tell the agent to omit it when unsure, and fix the illustrative polaris -> app-home. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Model omitting --api-name when the value isn't a documented one, consistent with the clarified guidance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add unified Shopify umbrella skill
|
/snapit |
|
🫰✨ Thanks @craigmichaelmartin! Your snapshot has been published to npm. Test the snapshot by installing your package globally: pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260721185120Caution After installing, validate the version by running |
List the full set of shopify.dev doc-search API names in the "search the docs first" step so agents pass only valid --api-name values instead of inventing them: admin, admin-extensions, checkout-ui-extensions, customer-account-ui-extensions, pos-ui-extensions, app-home, storefront, partner, customer, payments-apps, hydrogen, liquid, storefront-web-components, functions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Undo the changes from 80bb226 ("Clarify --api-name is an optional, unvalidated hint") and the enumeration layered on top of it, restoring the prior wording: the inline example list in the doc-search snippet, the simple "scope to a surface / omit to search all" bullet, the Themes --api-name liquid example, the polaris label, and the original gotcha. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
List the full set of shopify.dev doc-search API names in the "search the docs first" step, phrased as a plain enumeration without dismissive framing: admin, admin-extensions, checkout-ui-extensions, customer-account-ui-extensions, pos-ui-extensions, app-home, storefront, partner, customer, payments-apps, hydrogen, liquid, storefront-web-components, functions. Also correct the illustrative snippet's `payments` to `payments-apps`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flip the methodology from "search the docs first, validate last" to "attempt from knowledge, validate, and search only when validation fails." A passing validator proves recall was current, so a forced up-front search is wasted work; a failing validator is the signal that training data is stale and must be corrected from the docs, then re-validated in a loop. Validation is now the mandatory ground-truth gate; doc search is the failure-triggered remediation. The one exception is an outcome with no `shopify validate` surface (merchant onboarding guidance, App Store review analysis, UCP flows, bare doc questions), which is grounded in an up-front search since the safety net can't run. Preserves the CLI command-discovery pattern, the family table, the full --api-name valid set, the Outcome A/B validation rules, and the store-safety gotchas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
doc fetch pulls the entire document, which is token-expensive. Prefer the ranked chunks from `shopify doc search`; only fetch a full page when it is genuinely needed verbatim and doing so is cheaper than several follow-up searches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This stack of pull requests is managed by Graphite. Learn more about stacking. |
There are exactly four `shopify validate` topics — theme, graphql, functions, and components (graphql/functions/components take --api). State this in step 2 and the family table, and run `shopify validate --help` to confirm. Reconcile the table and examples, which referenced ~10 topics that don't exist (admin, customer, checkout-extension, config, ucp, app-store-review, ...). Each family now maps to a real topic (graphql --api, functions, theme, components --api) or is marked as having no validator. Families with no validator — onboarding guidance, CLI config (use `shopify app config validate`), UCP flows, Hydrogen recipe code, App Store review — are grounded with an up-front doc search per the no-validator exception. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…' into skill-validate-first-search-fallback # Conflicts: # .agents/skills/shopify/SKILL.md
…lback Invert shopify skill to validate-first, search-on-failure
Differences in type declarationsWe detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:
New type declarationsWe found no new type declarations in this PR Existing type declarationspackages/cli-kit/dist/public/node/metadata.d.ts@@ -34,7 +34,7 @@ export type SensitiveSchema<T> = T extends RuntimeMetadataManager<infer _TPublic
* @returns A container for the metadata.
*/
export declare function createRuntimeMetadataContainer<TPublic extends AnyJson, TSensitive extends AnyJson = Record<string, never>>(defaultPublicMetadata?: Partial<TPublic>): RuntimeMetadataManager<TPublic, TSensitive>;
-type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
+type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'cmd_validate_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
declare const coreData: RuntimeMetadataManager<CmdFieldsFromMonorail, {
commandStartOptions: {
startTime: number;
packages/cli-kit/dist/public/node/monorail.d.ts@@ -77,6 +77,11 @@ export interface Schemas {
cmd_app_validate_valid?: Optional<boolean>;
cmd_app_validate_issue_count?: Optional<number>;
cmd_app_validate_file_count?: Optional<number>;
+ cmd_validate_subcommand?: Optional<string>;
+ cmd_validate_result?: Optional<string>;
+ cmd_validate_api?: Optional<string>;
+ cmd_validate_api_version?: Optional<string>;
+ cmd_validate_json?: Optional<boolean>;
cmd_dev_tunnel_type?: Optional<string>;
cmd_dev_tunnel_custom_hash?: Optional<string>;
cmd_dev_urls_updated?: Optional<boolean>;
|
Assisted-By: devx/8a5f8cc7-d76e-49ff-a591-4193978ea1b9

What
Adds a new top-level
shopify validatecommand that ports the four validation scripts fromai-toolkit-source'sshopify-dev-toolspackage into first-class CLI subcommands for offline, deterministic validation:shopify validate theme@shopify/theme-check-*(already shipped by the CLI)shopify validate graphqlgraphql+ bundled introspection JSONshopify validate functionsshopify validate components@shopify/ui-extensionstype defsHow
All four subcommands sit on one shared engine under
packages/cli/src/cli/services/validate/engine/:ValidationResultconst-union,ValidationResponse) and a barrel-decoupled formatter — so the non-component validators never pull in the TypeScript compiler./assetsalike.Behavior preserved from the source tools (flags,
--jsonshape{success, responses, resolvedVersion?}, exit codes, stdin), while adapting to CLI conventions: cli-kit rendering,outputResult+AbortSilentErrorfor--json, telemetry viaaddPublicMetadata(newcmd_validate_*fields). Agent-only plumbing (random-UUID artifact IDs,shopify.devnetwork instrumentation) is dropped for deterministic output. Bad--file/empty stdin now return a structured FAILED payload instead of crashing.Reference data (~13.7 MB compressed: GraphQL schemas + UI-extension
.d.ts.gz) is bundled for fully offline validation and copied intodist/via a recursive esbuild rule.typescriptbecomes a lazily-loaded, esbuild-external runtime dependency (used only byvalidate components).Test plan
nx type-check cli✅ ·nx lint cli✅nx vitest clifor the validate suites: 81/81 pass (real temp dirs, no fs mocking)nx bundle cli✅ — bundled binary smoke-tested for all four subcommands:validate graphql --api admin --code 'query { shop { name } }'→success, exit 0validate graphqlwith an invalid field →failed, exit 1 (Cannot query field "nope" on type "Shop")validate functions --api functions_discount(valid input query) →successvalidate theme(codeblock) →successvalidate components --api polaris-app-home --code '<s-text>hello</s-text>' --language html→success,validatedComponents: ["s-text"]oclif.manifest.json,README.md, and dev-docs; changeset added.Notes / reviewer callouts
typescript(lazy, esbuild-external) forvalidate components.packages/cli/assets/validate/.gitignorere-includes the vendored type packages'dist/folders (otherwise the repo-widedistignore would drop ~320.d.ts.gzreference files).🤖 Generated with Claude Code