Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ All notable changes to `@testsprite/testsprite-cli` are documented here. The for

### Added

- **Declarative backend Suitefiles.** New `suite validate`, `suite graph`, `suite plan`, and `suite apply` commands validate versioned manifests and Python sources, compile dependency declarations into deterministic execution waves, detect remote metadata/code drift, and reconcile creates/updates with explicit confirmation. Adjacent atomic lockfiles preserve stable remote identities and interrupted-create idempotency; name collisions and changed pending creates fail as conflicts, while unmanaged remote tests are reported and never deleted.
- **`test cancel <run-id...>`** — user-initiated cancel of in-flight runs (the real stop button; Ctrl-C only detaches). A single id renders the run card with status `cancelled` (plus an advisory when it was already cancelled); multiple ids print a `{cancelled, alreadyCancelled, conflicts, notFound}` summary. Exit codes: 4 when any id is not found, else 6 on conflicts, else 0. `--dry-run` supported.
- **Graceful Ctrl-C during `--wait`** — SIGINT/SIGTERM now detaches cleanly instead of killing the process mid-poll: the in-flight request aborts immediately, stdout gets the same partial `{runId, status: "running"}` envelope as the request-timeout path, and stderr states the truth — the server-side run keeps executing (and billing) — with a re-attach hint and a `test cancel` pointer. Exit 130/143/129 per the documented signal contract; a second signal forces a hard exit. Interrupting never cancels the server-side run — that's what `test cancel` is for.
- **`project delete <project-id> --confirm`** — permanently delete a project and everything under it (its frontend/backend sub-projects, all their tests, and backend fixtures), mirroring the Portal's cascade delete. Requires `--confirm` (the CLI never prompts); `--dry-run` previews the response shape without a network call. Standard exit codes: 0 success, 3 auth, 4 not-found (or already-deleted), 5 validation (e.g. missing `--confirm`).
Expand Down
64 changes: 63 additions & 1 deletion DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The full reference for the TestSprite CLI: install verification, manual setup, e
- [Manual setup](#manual-setup)
- [The complete agent loop](#the-complete-agent-loop)
- [Agent onboarding (`agent install`)](#agent-onboarding-agent-install)
- [Declarative backend suites (`suite`)](#declarative-backend-suites-suite)
- [Command reference](#command-reference)
- [Read commands](#read-commands)
- [Write commands](#write-commands)
Expand Down Expand Up @@ -45,7 +46,7 @@ testsprite --version
testsprite project list --dry-run --output json
```

`--dry-run` is a global flag that skips the network, credentials, and the local filesystem and emits a canned sample matching the API contract. It's the right way to confirm an install or learn the surface before configuring auth — the response _shapes_ match the wire contract, but the data is fake.
`--dry-run` is a global flag that skips the network, credentials, and filesystem writes and emits a canned sample matching the API contract. File-backed plan inputs and Suitefile manifests/code are still read and validated locally. It's the right way to confirm an install or learn the surface before configuring auth — the response _shapes_ match the wire contract, but the remote data is fake.

## Manual setup

Expand Down Expand Up @@ -131,6 +132,67 @@ The `codex` target uses **managed-section mode** — it writes only a sentinel-d

Re-running with `--force` on **own-file targets** (claude, cursor, cline, antigravity, kiro, windsurf, copilot) backs up the existing file to `<path>.bak` first.

## Declarative backend suites (`suite`)

A Suitefile keeps a backend test suite in source control: test identity, metadata, Python source paths, and data dependencies. The CLI validates the complete local definition, compiles `produces`/`consumes` into deterministic execution waves, compares it with the remote project, then performs only the reviewed creates and updates.

```json
{
"schemaVersion": 1,
"projectId": "proj_xxxxxxxx",
"tests": [
{
"key": "auth/session",
"name": "Create API session",
"codeFile": "tests/auth_session.py",
"priority": "p0",
"produces": ["session_token"],
"category": "setup"
},
{
"key": "orders/create",
"name": "Create order",
"codeFile": "tests/create_order.py",
"consumes": ["session_token"]
},
{
"key": "orders/cleanup",
"name": "Clean up order data",
"codeFile": "tests/cleanup.py",
"category": "teardown"
}
]
}
```

Paths are relative to the Suitefile and must resolve to contained `.py` files. Unknown fields, duplicate keys/test ids, missing or ambiguous producers, self-dependencies, cycles, lexical or symlink path escapes, and oversized code files fail locally before credentials or network are touched. `teardown` and `cleanup` tests are forced behind non-teardown tests.

```bash
# Pure-local checks
testsprite suite validate ./testsprite.suite.json
testsprite suite graph ./testsprite.suite.json --output json

# Live comparison; makes no changes
testsprite suite plan ./testsprite.suite.json

# Fully validate locally without credentials/network
testsprite suite plan ./testsprite.suite.json --dry-run --output json

# Apply exactly the create/update plan
testsprite suite apply ./testsprite.suite.json --confirm
```

The adjacent `testsprite.suite.lock.json` records stable suite-key → remote-test identity, code version, and desired-state hash. Check it into source control with the manifest. You can override its location with `--lock-file <path>`.

Safety rules are deliberately strict:

- Existing tests are adopted only through an explicit manifest `testId` or the lockfile. A same-name remote test is a conflict, not an implicit match.
- Unmanaged remote tests are reported by `plan` and are **never deleted** by `apply`.
- Backend tests only are supported in schema version 1; an adopted frontend test is a conflict.
- Creates and updates use deterministic idempotency keys. The lockfile is written atomically, and interrupted creates retain their request identity for safe replay; changing the definition while a create is pending produces a conflict.
- `name`, Python code, `produces`, and `consumes` are authoritative. Omitted `produces`/`consumes` mean empty lists. `priority` and `category` are changed only when declared.
- `apply` requires `--confirm` whenever the plan contains a mutation. It refuses any plan containing conflicts.

## Command reference

Every command supports the [global flags](#global-flags), and every example below pairs a real call with a `--dry-run` companion that works on a fresh install with no auth.
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ Prefer to configure each step by hand (or learn the surface offline with `--dry-
| | `test failure summary` | One-screen triage card (no media download) |
| | `test diff` | Compare two runs — verdict, failure kind, per-step status flips, code-version drift |
| **Write** | `test scaffold` / `test lint` | Author plans locally: emit a schema-correct starter, validate plan files offline — no network, no credentials |
| | `suite validate` / `suite graph` / `suite plan` | Compile a declarative backend Suitefile into dependency waves and preview drift against the remote project |
| | `suite apply` | Reconcile backend test metadata and Python code with guarded, idempotent writes; unmanaged remote tests are never deleted |
| | `test create` / `test create-batch` | Create a test (or bulk-create from a plan file); `--produces` / `--needs` / `--category` wire BE dependency metadata |
| | `test update` / `test delete` / `test delete-batch` | Edit metadata and BE dependency declarations (`--produces` / `--needs` / `--category`) / permanently delete (no restore window; `--confirm` required) |
| | `test code put` | Replace generated code (etag-guarded) |
Expand Down
Loading
Loading