Refactor the WPT test runner to be a python test runner configuration

[joyeecheung]

While working on the flakes I noticed that there are still some drawbacks with the current WPT test runner (common/wpt.js):

1. To enable running tests in parallel the WPT test runner uses workers, which can be harder to debug especially when the use of inspector is needed to look into the JS land. If we just use the python runner, we get the parallelism for free, and debugging individual tests would be easier. We can use a fixture to set up the main context/harness and let the python runner preload that file before running the tests. To debug the test, just out/Release/node --inspect-brk -r test/common/wpt-fixture.js test/fixtures/wpt/... and connect to the inspector server with a client.
2. The use of workers for parallelism also mean that we lack coverage of WPT running on the main thread (I've been bitten by this before when adding lazy loading of Web APIs the main context snapshot). If we use the python test runner to run the WPT directly, we can get coverage on both the main thread and the worker thread (via the custom test suite in the CI that uses the --worker switch)
3. The status file for the current WPT runner doesn't allow conditions which is important for marking flakes on a specific platform. To add this to the JS-based WPT runner we need to reinvent the wheel again, and as far as I can tell we might as well just integrate the test runner into the existing python one and get a unified handling for flakes.

[joyeecheung]

cc @panva @legendecas based on git log of test/common/wpt.js

[panva]

I'm just worried even less people will be able to make heads or tails of what it is doing. I for one would not be able to contribute anymore.

[joyeecheung]

I personally find the Python runner easier to extend than the current WPT runner (and I wrote the initial implementation of the JS WPT runner - in retrospect I would've just do the harness in a preloaded script, instead of inventing a different runner, had I know that the WPT can start flaking). It's actually surprisingly simple, ignoring language differences.

[joyeecheung]

Also, the harness part would still be in JS as a preloaded fixture, only the runner part would be in Python (I think most of the runner code in common/wpt.js is already reinventing what the python runner is capable of anyway...).

[legendecas]

I agree that the current WPT runner is trying to reinvent the capability of python runner but there are unique features like generating wpt.fyi reports. But I believe it would be straightforward to re-implement it in python runner as well.

[joyeecheung]

Happened to be talking about this during the WinterCG breakout session at Web engine hackfest and it seems there are interest from Deno to work on a common test runner for JS runtimes and upstream it to WPT. If that common test runner could handle the points mentioned here that would be great and we would not need to maintain (alone) yet another test runner / adaptor for WPT (it was proposed mostly because of point 3, trying not to re-invent the wheel of conditional test running based on a status file, and the python test runner looks like a good target to piggy-back. But if there's something else that includes the wheel and another party helps maintain it, that sounds like a better piggy-backing target)

[github-actions]

This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

[github-actions]

This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.