Perform test discovery per-file when a file is updated/added.

[ericsnowcurrently]

Currently we only perform test discovery on the whole project. This is unnecessary when discovery happens in response to a file being created, updated, or deleted. We should be able to perform discovery on a per-file basis in those cases. Doing so will be more efficient and allow us to be more granular in how we present test status.

This is effectively blocked by a uniform test discovery API for the different test frameworks (e.g. like we're doing in #3911 for pytest).

[Conchylicultor]

Luciana Abud (@luabud) thank you for looking into this !!

When implementing per-file discovery, it would be awesome if all discovered tests gets displayed, even if other files fails. Currently a single wrong file cause the entire test discovery to fail. (See #13301 for context)

[shanewazabbas]

hey Eleanor Boyd (@eleanorjboyd) , since I dont have permission to respond to: #22582. I'll reply here. We decided to go with a modular monolith approach in our repo with each module having a conftest to avoid unnecessary analysis by pytest. Basically solving it by working around pytest. And we have noticed much improved time in test discovery. So no need to focus on #22582 if you dont have bandwidth.

[eleanorjboyd]

Thanks for the heads up! Glad you got a solution which worked for you

[senhalil]

Do you have any updates on this issue?

We have a monorepo and test discovery takes ~3 minutes at the moment. This makes vscode quite clunky -- i.e., after modifying/adding a test, one needs to wait for it to be "discovered" by vscode-python.

If the discovery was based on file modifications, one new test file or new/modified tests in an existing file would be picked up instantaneously -- i.e., it would not parse all the files with unchanged dates/hashes.

[eleanorjboyd]

Still something we would love to do! We prioritize feature requests based on upvotes- so the more upvotes the higher the feature will sit on our priority list. Amplifying this issue to those at your company who are impacted and getting them to upvote would help. The other option is putting in a PR with the fix and then I can take a look and would not have to take development time for it. In the meantime another user did add a feature request that could help- which is the setting for which files to rediscover tests on. Maybe this could help in the time-being. Thanks!