Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .fallowrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@
},
{
"comment": "Tool config default exports, loaded by the tool rather than imported.",
"file": "{tsdown.config.ts,vitest.mutation.config.ts,website/rspress.config.ts,test/skillgym/skillgym.config.ts,test/skillgym/suites/*.ts}",
"file": "{tsdown.config.ts,vitest.mutation.config.ts,website/rspress.config.ts}",
"exports": ["default"]
},
{
Expand All @@ -88,7 +88,7 @@
"exports": ["ownedTestFiles"]
},
{
"comment": "Help-benchmark conformance seams: helpTopicIds feeds the topic-coverage gate and PRIVATE_AX_RECOVERY_SAMPLE feeds the skillgym suite; both consumers are test-tree files outside --production analysis.",
"comment": "Help-benchmark conformance seams: helpTopicIds feeds the topic-coverage gate and PRIVATE_AX_RECOVERY_SAMPLE feeds the bench sample producers; both consumers are test-tree files outside --production analysis.",
"file": "{src/cli/parser/cli-help.ts,scripts/help-conformance-sample-outputs.mjs}",
"exports": ["helpTopicIds", "PRIVATE_AX_RECOVERY_SAMPLE"]
},
Expand Down
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ xcuserdata/
*.app
*.xctestrun
*.xcarchive
.skillgym-results/
android/snapshot-helper/build/
android/snapshot-helper/dist/
android/ime-helper/build/
Expand Down
1 change: 0 additions & 1 deletion .oxfmtrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
"ignorePatterns": [
"dist/**",
"node_modules/**",
"**/.skillgym-results/**",
"**/*.md",
"scripts/maestro-conformance/corpus/**",
"fallow-baselines/**",
Expand Down
6 changes: 2 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,10 +233,8 @@ connect errors, retry policy, or command typing, start in
layer after writing is how the same contract ends up duplicated across two of them.
`docs/agents/cli-flags.md` walks the layers for the flag case.
- Decide docs impact with the change, not after. For behavior/CLI-surface changes: update
help/metadata, README or `website/docs/**` when user-facing, and a SkillGym case in
`test/skillgym/suites/agent-device-smoke-suite.ts` when command-planning guidance changes.
- Keep SkillGym cases behavioral and command-planning oriented: assert the user-visible contract and
expected command family, forbid known bad patterns, avoid brittle exact output.
help/metadata, README or `website/docs/**` when user-facing, and a help-conformance bench case
(`scripts/help-conformance-*.mjs`) when command-planning guidance changes.
- State in the final summary whether docs/skills were updated, and why not if they weren't.

When guidance conflicts, Hard Rules win, then scope, then testing, then style.
2 changes: 1 addition & 1 deletion docs/agents/pull-requests.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
observes the artifact/state the feature claims to handle. If live verification is blocked, state
the blocker and the exact command/device needed, and downgrade the PR to residual risk rather than
calling it ready.
- Command-surface changes preserve CLI, Node.js, daemon, MCP, help, docs, and SkillGym coverage
- Command-surface changes preserve CLI, Node.js, daemon, MCP, help, and docs coverage
where that surface is affected, without duplicating command contracts across layers.
- Runtime output stays agent-friendly: compact defaults, top offenders first for diagnostics/perf,
bounded arrays in JSON, artifact paths for large raw data, progressive lookup for deeper detail.
Expand Down
10 changes: 3 additions & 7 deletions docs/agents/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,8 @@ The mapping it encodes, for when you need to run a gate directly or reason about
| iOS runner / Swift | `pnpm build:xcuitest` |
| CLI help/guidance (`src/cli/parser/cli-help.ts`, `src/cli-schema/`) | `pnpm exec vitest run src/cli/parser/__tests__ src/cli-schema/command-schema-guards.test.ts scripts/__tests__` — the `scripts/__tests__` gates enforce help-topic benchmark coverage and pin the bench's quoted CLI samples to the real renderers |
| Help benchmark cases (`scripts/help-conformance-*.mjs`) | `pnpm exec vitest run scripts/__tests__` (deterministic gates); model-backed: `pnpm bench:help-conformance` (paid LLM calls, local only) |
| SkillGym prompts/assertions | `pnpm test:skillgym:case <case-id>` (broad: `pnpm test:skillgym`, filter with `-- --tag fixture-smoke` or `-- --tag skill-guidance`) — agentic routing + local-help-consumption proof only; command-planning knowledge checks belong in the help bench |
| `.ad` grammar (`src/replay/script.ts`, gesture arity, replay vars) | `pnpm exec vitest run --project unit-core test/replay-compat` — the frozen replay-compat corpus asserts which released script surfaces still parse; a flipped verdict is edited in `test/replay-compat/manifest.ts`, never in the script. Adding or re-pinning a corpus entry also runs `pnpm check:replay-compat`, which re-derives each entry from its release tag in git history |
| Anything in `src/`, `test/`, `skills/` | `pnpm format` |
| Anything in `src/`, `test/` | `pnpm format` (`skills/` is Markdown-only guidance: oxfmt ignores `**/*.md`, and the affected-check selector classifies it docs-only) |
| A decision kernel or its tests (`src/kernel/errors.ts`, `src/daemon/ref-frame.ts`, `src/commands/interaction/runtime/settle.ts`, `src/utils/scroll-edge-state.ts`, `src/selectors/`) | `pnpm mutation:affected --base origin/main` (minutes; GitHub runs it per PR — see the mutation ratchet section) |

Two traps worth naming:
Expand Down Expand Up @@ -88,8 +87,8 @@ hand-maintained path map:
import ownership. Dynamic-import relationships remain outside Vitest's
analysis; GitHub's authoritative full suites still cover that boundary.
- **Non-Vitest suites** retain explicit ownership. Root
`test/integration/*.ts` files use the Node integration lane, SkillGym owns its
harness and skill guidance, and platform/build tools keep their native gates.
`test/integration/*.ts` files use the Node integration lane, and
platform/build tools keep their native gates.
- **Always-on gates** (`lint`, `typecheck`, `layering`, `fallow`, `format`) fire
for their input categories and are never silently skipped. Platform source
also selects the provider-integration and coverage gates required by the
Expand All @@ -99,9 +98,6 @@ hand-maintained path map:
- A **small explicit build-ownership layer** covers the paths whose owning build
cannot be derived: Swift runner, Android helpers, macOS helper, MCP metadata,
and the public package surface (itself derived from `package.json` `exports`).
- **SkillGym ownership** covers skill guidance (`skills/`) and the SkillGym
harness (`test/skillgym/`) — those changes select the (local-only) SkillGym
suite, and their Markdown is treated as skill/harness input, not inert docs.

Changed-file discovery folds working-tree state into the local plan: in the
default local mode (`--head HEAD`) it unions the committed `base..HEAD` diff with
Expand Down
5 changes: 2 additions & 3 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,5 @@ way a Node consumer or the agent-device-cloud bridge would. Each file:

## test-app/

[`test-app/`](./test-app) is the Expo dogfood fixture used for `agent-device` and `skillgym`
experiments (see its own [README](./test-app/README.md)) — it is a test fixture, not an SDK usage
example.
[`test-app/`](./test-app) is the Expo dogfood fixture used for `agent-device` experiments (see
its own [README](./test-app/README.md)) — it is a test fixture, not an SDK usage example.
3 changes: 1 addition & 2 deletions examples/test-app/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
# Agent Device Tester

`Agent Device Tester` is a minimal Expo Router fixture app for `agent-device` and `skillgym` experiments.
`Agent Device Tester` is a minimal Expo Router fixture app for `agent-device` experiments.

It is intentionally small, but each surface is dense with durable accessibility targets so a few screens cover a large share of the workflows we care about.

## Why this app exists

- It gives `agent-device` a stable React Native target that we control.
- It makes `skillgym` prompts concrete: the agent can inspect real app files instead of answering against an imagined UI.
- It keeps the number of screens low while still covering roughly 50 practical interaction and verification cases.

## Screens
Expand Down
2 changes: 1 addition & 1 deletion examples/test-app/src/screens/FormScreen.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ export function FormScreen(props: FormScreenProps) {
title="Android IME capture"
testID="android-ime-capture-fixture"
>
{/* SkillGym fixture: static diagnostic copy, not live state. */}
{/* Fixture: static diagnostic copy, not live state. */}
<TextField
accessibilityLabel="Android IME target field"
autoCapitalize="none"
Expand Down
5 changes: 0 additions & 5 deletions fallow-baselines/health.json
Original file line number Diff line number Diff line change
Expand Up @@ -570,11 +570,6 @@
"crap_moderate": {
"count": 1
}
},
"test/skillgym/suites/agent-device-smoke-suite.ts": {
"crap_moderate": {
"count": 1
}
}
},
"runtime_coverage_findings": [],
Expand Down
4 changes: 0 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,6 @@
"check:layering": "node --experimental-strip-types --test scripts/layering/model.test.ts scripts/layering/zone-policy.test.ts && node --experimental-strip-types scripts/layering/check.ts",
"depgraph": "node --experimental-strip-types scripts/depgraph/build.ts",
"depgraph:test": "node --experimental-strip-types --test scripts/depgraph/model.test.ts scripts/depgraph/affected.test.ts",
"repo-health": "node --experimental-strip-types scripts/repo-health/run.ts",
"check:production-exports": "fallow dead-code --config fallow-production-exports.json --production --unused-exports --fail-on-issues",
"check:bundle-owner-files": "node --experimental-strip-types scripts/check-bundle-owner-files.ts",
"check:command-docs": "vitest run --project unit-core src/__tests__/command-doc-coverage.test.ts",
Expand Down Expand Up @@ -166,8 +165,6 @@
"test:fixture-cache": "node --test test/ci/trusted-fixture-artifact.test.mjs",
"test:output-economy": "vitest run --project output-economy",
"test:smoke:web": "pnpm build && node --test test/integration/smoke-web-platform.test.ts",
"test:skillgym": "node test/skillgym/runner-environment.ts && pnpm build && skillgym run ./test/skillgym/suites/agent-device-smoke-suite.ts --config ./test/skillgym/skillgym.config.ts",
"test:skillgym:case": "node test/skillgym/runner-environment.ts && pnpm build && skillgym run ./test/skillgym/suites/agent-device-smoke-suite.ts --config ./test/skillgym/skillgym.config.ts --case",
"test:smoke": "node --test test/integration/smoke-*.test.ts",
"test:integration:node": "node --test test/integration/*.test.ts",
"test:integration": "pnpm test:integration:node && pnpm test:integration:provider",
Expand Down Expand Up @@ -254,7 +251,6 @@
"oxc-parser": "^0.138.0",
"oxfmt": "^0.42.0",
"oxlint": "^1.69.0",
"skillgym": "^0.9.1",
"tsdown": "^0.22.4",
"typescript": "^7.0.2",
"vite": "^8.0.16",
Expand Down
Loading
Loading