test-runner: run in random order

[ziimakc]

What is the problem this feature will solve?

This feature will improve user tests quality by allowing to run them in random order to ensure that tests are properly isolated from each other.

What is the feature you are proposing to solve the problem?

Allow to run tests in random order based on random seed. See jest randomize or vitest shuffle

Seed will allow to rerun test in the same order to reproduce failed case for specific order.

[fasenderos]

Can I take this?

[MoLow]

Sure

[MoLow]

This PR might be helpful to figure out where the internal queue is manged, and where it should be tweaked in case we want to randomize it:
#48428

[HinataKah0]

And I checked that there isn't a good random shuffle function in the codebase yet (CMIIW, I was searching with keyword: shuffle, fisher, yates, ...)
So I think maybe we want to implement the Fisher-Yates shuffle as well? Assuming the shuffling is done offline (after all tests have been enqueued)

[cjihrig]

Assuming the shuffling is done offline (after all tests have been enqueued)

Just FYI - the test runner tries to execute tests as quickly as possible, while respecting the defined concurrency limit. It's pretty unlikely that all of the tests in a file will ever be enqueued.

[MoLow]

Just FYI - the test runner tries to execute tests as quickly as possible, while respecting the defined concurrency limit. It's pretty unlikely that all of the tests in a file will ever be enqueued.

I think it's ok to change the behavior in case randomization is on.
Also changing the tree root to be a suite instead of a test would achieve that

[cjihrig]

I think it's ok to change the behavior in case randomization is on.

As long as it doesn't impact performance, sure.

Also changing the tree root to be a suite instead of a test would achieve that

Wouldn't that only apply to top level tests?

[MoLow]

Wouldn't that only apply to top level tests?

Yes, but isn't that what we desire to randomize?

[cjihrig]

Good question. If I was opting into a randomization mode, I would expect subtests to be randomized as well, but maybe that's a bad assumption on my part?