Should we test `-X lazy_imports=all` more?

[JelleZijlstra]

Bug report

Bug description:

Inspired by #149321 I decided to see how well the test suite works with PYTHON_LAZY_IMPORTS=all / -X lazy_imports=all, which makes all imports lazy. It's nowhere near as bad as the none option that we're considering removing, but a few parts of the stdlib are broken, and it seems worth considering if we want to fix these.

Concrete issues I noticed:

• PEP 810 lazy from imports ignore module __getattr__ when reifying #144957 breaks test_typing
• test_unittest fails because unittest.main becomes a module instead of a function.
• test_struct is broken because of a test that does exec("import struct") in a function; lazy imports are disallowed within functions. Similar issues affect test_enum, test_traceback, and test_future.
• test_symtable is broken because it fills its _flags list by iterating over globals(), which means it starts containing lazy import marker objects. Similarly test_inspect breaks because it iterates over vars() of a module.

A few tests fail for legitimate reasons; test_compileall tests that an import triggers pyc compilation; test_builtin asserts that __import__ is used when in fact __lazy_import__ gets used.

Full list of failed modules on my machine:

test.test_future_stmt.test_future
test.test_inspect.test_inspect
test.test_pydoc.test_pydoc
test___all__
test__interpreters
test_builtin
test_capi
test_clinic
test_compileall
test_crossinterp
test_datetime
test_enum
test_generated_cases
test_idle
test_import
test_importlib
test_interpreters
test_json
test_lazy_import
test_struct
test_subprocess
test_symtable
test_tools
test_trace
test_traceback
test_tracemalloc
test_typing
test_unittest
test_xmlrpc
test_zipfile

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-150565: Add additional tests for Lazy Imports #149739
• gh-149640: Add new GitHub action to test lazy_imports=all against test suite #151090
• gh-149640: Add new GitHub action to test lazy_imports=all against test suite #151105

[JelleZijlstra]

And to be clear the outcome I'm looking for here is that we start testing lazy_imports=all in some kind of CI (a buildbot?) and fix the current issues, either by skipping the test conditionally or by fixing some aspect of the implementation.

I'm now looking into the exec issue which feels like a bug; the unittest issue may also be a bug but I need to look further.

[brittanyrey]

Attempting to add the CI in the linked PR.
Also adding an exclusion list for failing tests to steadily chip away at