Skip to content

test: fix flaky watch + cwd + argv test-runner test#64372

Open
pimterry wants to merge 1 commit into
nodejs:mainfrom
pimterry:fix-flaky-test-cwd-test
Open

test: fix flaky watch + cwd + argv test-runner test#64372
pimterry wants to merge 1 commit into
nodejs:mainfrom
pimterry:fix-flaky-test-cwd-test

Conversation

@pimterry

@pimterry pimterry commented Jul 8, 2026

Copy link
Copy Markdown
Member

This fixes the flaky test recently introduced in #63690.

I have seen #64084 which tried to fix this as well but was closed as it didn't reproduce under stress test on main, so it couldn't be verified.

I think that that PR's diagnosis is correct (writing in the watched folder causes the flake). This PR takes a slightly different approach: removing the 'marker' file write entirely, to verify on executed test names instead (dropping any possibility of races via IO or watch behaviour, by doing zero writes during the test).

I'm pretty sure I know the exact failure scenario:

  • The test runner is watching cwd
  • The executed inner test writes to cwd (to the marker file)
  • This triggers a change event, which re-runs the test, but debounces by 200ms (debounce defined here)
  • test:pass fires from the first inner run, passing the mustCall(1) in the outer test.
  • There's now a 200ms window, for a race between test:pass + test:watch:drain firing (stopping all events and ending the test) vs the change event leading to a test re-run that fires test:pass a 2nd time.
  • The test:watch:drain event can easily be delayed unpredictably, since it's fired by the child process shutdown signal from the OS (plus a few other checks & async steps en route). >200ms delay between the write & the drain => flake.

As noted there, this is hard to verify as flaky in the CI job, but it is verifiable with a manual delay:

  • In the original test, add a >200ms delay inside the test:watch:drained if:
    }).on('data', function({ type }) {
      if (type === 'test:watch:drained') {
        setTimeout(() => {                          // <-- inject delay here
          stream.removeAllListeners('test:fail');
          stream.removeAllListeners('test:pass');
          controller.abort();
        }, 250);
      }
    });
    
  • The test now always fails, in exactly same way as in most failed CI runs (double test:pass).
  • I.e. if anything in CI ever delays the child proc exit (or notification of it) by 200ms like this, the test will fail.

Dropping the IO solves this, since it guarantees no watch events will fire, and removes the race entirely. With this fix in place, it now passes 100% for me with the same or larger setTimeout delays added manually.

As a more general "prove a flake fix" argument, where it's difficult to repro, personally I think "repro with a manual delay in an unpredictable-duration step" is a reasonable demo, even if the stress test passes.

Signed-off-by: Tim Perry <pimterry@gmail.com>
@pimterry pimterry requested a review from trivikr July 8, 2026 20:16
@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to the tests. labels Jul 8, 2026
@pimterry

pimterry commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.25%. Comparing base (799e910) to head (832835f).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #64372      +/-   ##
==========================================
- Coverage   90.25%   90.25%   -0.01%     
==========================================
  Files         741      741              
  Lines      241165   241192      +27     
  Branches    45428    45435       +7     
==========================================
+ Hits       217667   217678      +11     
- Misses      15069    15085      +16     
  Partials     8429     8429              

see 39 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@trivikr

trivikr commented Jul 8, 2026

Copy link
Copy Markdown
Member

@trivikr trivikr added request-ci Add this label to start a Jenkins CI on a PR. flaky-test Issues and PRs related to the tests with unstable failures on the CI. labels Jul 8, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 9, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

flaky-test Issues and PRs related to the tests with unstable failures on the CI. needs-ci PRs that need a full CI run. test Issues and PRs related to the tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants