pluginmanager validation kicks in too early when setuptools plugins use pytest_plugins

[RonnyPfannschmidt]

after updating to pytest 3.2 i had to move a hook implementation into a plugin object

will follow up with a reference to that code once i pushed it

the code in question is only triggered in worker processes of a own test parallelizing tool,
so it could be very well a underlying bug in pytest we just don't trigger usually/with xdist

[nicoddemus]

@RonnyPfannschmidt can we close this?

[RonnyPfannschmidt]

@RonnyPfannschmidt the issue was not resolved/fixed at all

[nicoddemus]

I remember we doing something related to pytest_plugins, thought that might have been it. I will fix the label then. 👍

[RonnyPfannschmidt]

Posted by me. The investigation below was done by an AI agent (Claude Opus 5 via Claude Code): I prompted it to dig into this issue, and the analysis, the reproducers and the prose are its work. I have read it and I am posting it.

Still reproducible on current main (9.2.0.dev) and on every release from 3.1.3 through 8.4.2.

Minimal reproducer

from _pytest.config import get_config


class NewHooks:
    def pytest_my_hook(self):
        """a hookspec contributed by a plugin"""


class Plugin:
    def pytest_addhooks(self, pluginmanager):
        pluginmanager.add_hookspecs(NewHooks)

    def pytest_my_hook(self):
        pass


config = get_config([])
config.pluginmanager.register(Plugin())
print("spec present:", config.hook.pytest_my_hook.has_spec())
config.pluginmanager.check_pending()

spec present: False
...
pluggy._manager.PluginValidationError: unknown hook 'pytest_my_hook' in plugin <__main__.Plugin object at 0x...>

The plugin adds the hookspec and implements it, so it should be self-consistent — but its own pytest_addhooks is never called.

What is going on

pytest_addhooks.call_historic(...) is issued from exactly one place, the top of Config.parse():

https://github.com/pytest-dev/pytest/blob/main/src/_pytest/config/__init__.py#L1637

Plugin registration does not drive it. Historic replay only works once the call has been recorded, so on a Config that never went through parse(), no registered plugin ever gets pytest_addhooks called, add_hookspecs() never runs, and check_pending() (at Session.perform_collect, or called directly as above) then rejects that plugin's own hookimpl as an unknown hook.

So hookspec contribution is gated on Config.parse() rather than on plugin registration. That is the "validation kicks in too early" in the title: it is really "pytest_addhooks happens too late, or not at all".

The original setuptools / pytest_plugins shape

Same defect, more moving parts. Reproducer package:

files

# setup.py
from setuptools import setup, find_packages
setup(name='miqrepro', version='0.1', packages=find_packages(),
      entry_points={'pytest11': ['miqrepro = miqrepro.main_plugin']})

# miqrepro/main_plugin.py
pytest_plugins = ('miqrepro.sprout',)

# miqrepro/sprout.py
def pytest_addhooks(pluginmanager):
    pluginmanager.add_hookspecs(NewHooks)

def pytest_miq_node_shutdown(config, nodeinfo):
    pass

class NewHooks:
    def pytest_miq_node_shutdown(self, config, nodeinfo):
        pass

# worker.py  --  what the reporting code did
from _pytest.config import get_config
config = get_config()
config.args = ['test_dummy.py']
config._preparse(config.args, addopts=False)      # loads pytest11 + their pytest_plugins
config.args = config._parser.parse_setoption(config.args, config.option,
                                             namespace=config.option)
raise SystemExit(config.hook.pytest_cmdline_main(config=config))

INTERNALERROR> PluginValidationError: unknown hook 'pytest_miq_node_shutdown'
               in plugin <module 'miqrepro.sprout' from '.../miqrepro/sprout.py'>

_preparse() loaded the entry points and the pytest_plugins they pull in, but parse() — the only caller of pytest_addhooks.call_historic — was skipped, so sprout's hookspec was never added. The code in question (fixtures/parallelizer/remote.py::_init_config in ManageIQ/integration_tests, described in its own comment as "a slightly modified form of Config.fromdictargs") had dropped the parse() call along the way, which is why only the parallelizer workers hit it.

Verified failing on 3.1.3, 3.2.5, 4.6.11, 5.4.3, 6.2.5, 7.4.4 and 8.4.2. On 9.0 this particular script stops working for an unrelated reason — _preparse was removed and inlined into parse() in 5a85b90 — which is not a fix; the minimal reproducer at the top still fails on main, as does get_config([...]) + pluginmanager.load_setuptools_entrypoints("pytest11") without parse().

When it started

3.1.0, commit 8bcf88e "try to consider all modules after registration as plugin", which made PytestPluginManager.register call consider_module so that pytest_plugins inside a setuptools entry-point plugin is honoured at all. On 3.0.7 miqrepro.sprout is simply never registered, so nothing complains. (Reported here after a 3.2 upgrade, but 3.1.0 is where the behaviour changed.)

Not affected

Ordinary pytest invocations, and Config.fromdictargs — which does call parse(), and passes the reproducer above on both 3.2.5 and main.