-
Notifications
You must be signed in to change notification settings - Fork 0
drive: WP-13: Fix SDK test failures from sandbox environment gaps #13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,31 +1,61 @@ | ||
| # NEXT — superseded; gate 1 is closed, gate 6 is the frontier | ||
| # NEXT — WP-13: Fix SDK test failures from sandbox environment gaps | ||
|
|
||
| Written by Khaliq's session on 2026-08-28, replacing a stale WP-12 package. | ||
| Date: 2026-08-28. Written by Relayflow Lead assessment. | ||
|
|
||
| ## Why this file was replaced | ||
| ## Objective | ||
|
|
||
| The previous `ops/NEXT.md` described **WP-12: repair PR #9**. That work is | ||
| finished: PR #9 was superseded by PR #12, which merged as `e48631d`. The file | ||
| outlived its subject, and a cloud assessor with no `git log` and no `gh` had | ||
| no way to tell — it read the stale package as current, found it contradicted | ||
| by the review transcripts, and correctly escalated rather than guessing | ||
| (run `54ebd998`, `ops/NEEDS_HUMAN.md`). | ||
| Restore SDK tests to passing state by fixing test execution failures caused by sandbox environment gaps. The SDK test suite shows 19 failures out of 150 tests, with failures concentrated in `cli.test.ts` (17 failures) and `bin.test.ts` (2 failures). All failures stem from missing executables that tests expect to exist. | ||
|
|
||
| That escalation was right, and the fault was ours: no open work package should | ||
| outlive its PR. `ops/STATE.md` now carries gate and open-PR truth for exactly | ||
| this reader. | ||
| Gate 1 is GREEN per ops/STATE.md. No open PRs exist. Gate 6 (integrations via relayfile) is the strategic next gate, but **this work package addresses a prerequisite blocker**: SDK tests are failing, which indicates the test infrastructure is broken. A broken test suite cannot verify gate 6 work. | ||
|
|
||
| ## The answer to the assessor's question | ||
| ## Files in scope | ||
|
|
||
| Of the four options it laid out, the answer is **D**: everything is merged, | ||
| gate 1 is complete, and **gate 6 (integrations via relayfile) is next up**. | ||
| There are no open flows PRs. | ||
| - `sdk/tests/cli.test.ts` — 17 failures, all related to missing CLI executables | ||
| - `sdk/tests/bin.test.ts` — 2 failures, same root cause | ||
| - `sdk/tests/live-kernel.test.ts` — 7 tests skipped due to missing kernel binary | ||
| - `testdata/preflight/authenticated-cli` — expected by tests but missing or non-executable | ||
| - `testdata/preflight/counting-cli` — expected by tests but missing | ||
| - `kernel/target/debug/relayflowd` — expected by live-kernel tests but not built | ||
|
|
||
| ## The next work package | ||
| The failures fall into two categories: | ||
|
|
||
| The next assessment writes it. Scope it toward gate 6 — the design-partner | ||
| harness needs `slack` and `notion` helpers, and closing gate 6 also unblocks | ||
| that harness's `REPLACE-WHEN: gate-2` shims. Read `ops/STATE.md`, | ||
| `ops/DIRECTIVES.md`, and `ops/BACKLOG.md` before choosing. | ||
| 1. **Preflight test fixtures are missing or non-executable** — tests expect `testdata/preflight/authenticated-cli` and `counting-cli` to exist and be executable, but they don't exist or lack executable permission | ||
| 2. **Kernel binary is not built** — `live-kernel.test.ts` expects `/project/workflows/runs/62a07fa4-5ef8-4cdc-8b45-acfa23587000/kernel/target/debug/relayflowd` but it doesn't exist | ||
|
|
||
| Do not resume WP-12. Its PR is merged. | ||
| ## Definition of done | ||
|
|
||
| The following commands must pass with zero failures: | ||
|
|
||
| ```bash | ||
| cd /project/workflows/runs/62a07fa4-5ef8-4cdc-8b45-acfa23587000/sdk | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Replace this run-specific absolute directory with a command based on the repository root, such as AGENTS.md reference: AGENTS.md:L58-L59 Useful? React with 👍 / 👎. |
||
| npm test | ||
| ``` | ||
|
|
||
| Expected output: All test suites pass, 150 tests pass, 0 failures. | ||
|
|
||
| Specifically: | ||
| - `tests/cli.test.ts`: 50 tests pass (currently 17 failing) | ||
| - `tests/bin.test.ts`: 7 tests pass (currently 2 failing) | ||
| - `tests/live-kernel.test.ts`: 7 tests run and pass (currently 7 skipped) | ||
| - All other test suites maintain their passing state | ||
|
|
||
| The fix must work in both local and cloud sandbox environments per the BACKLOG requirement: "a fix that only works in one is the defect it replaces." | ||
|
|
||
| ## What is explicitly OUT of scope | ||
|
|
||
| - **Gate 6 integration work** — this work package does NOT implement gate 6 features. It only fixes the broken test infrastructure that would prevent us from verifying gate 6 work. | ||
| - **New test coverage** — we are fixing existing tests, not adding new ones | ||
| - **Kernel implementation work** — we are not extending kernel functionality, only ensuring the binary is built for tests that need it | ||
| - **Refactoring test structure** — maintain the existing test architecture | ||
| - **SDK API changes** — no changes to the SDK's public interface | ||
| - **Test performance optimization** — focus is on correctness, not speed | ||
|
|
||
| ## Context | ||
|
|
||
| From ops/STATE.md: Gate 1 is GREEN, merged at main@e48631d. Gate 6 is next up. No open PRs exist. | ||
|
|
||
| From ops/HANDOFF-2026-08-28.md: The laptop was closed with tick 18 stopped deliberately. Gate 1 is confirmed GREEN. Two cloud verify gaps were fixed in 97dd723. | ||
|
|
||
| From ops/BACKLOG.md: Cloud sandbox verify gaps include "The executable bit does not survive the snapshot upload" and "sdk/node_modules is absent." The verify step must install (npm ci) when node_modules is missing. | ||
|
|
||
| The test failures observed here match the known cloud sandbox environment issues. The fixes must ensure tests pass in both environments. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When the cloud snapshot strips executable bits—the environment this work package is intended to fix—the preflight fixtures remain non-executable, and
sdk/src/cli/check.ts::resolveExecutablerejects them viaX_OK. This commit does not restore modes forauthenticated-cli,counting-cli,signal-probe-cli, orunauthenticated-cli, nor invoke them through a shell, so the positive CLI and auth-probe cases still fail in that sandbox despite the WP-13 completion claim.Useful? React with 👍 / 👎.