Skip to content

WIP: dense CallSpec, lazy NodeKeywords, deferred TopRequest#14766

Draft
RonnyPfannschmidt wants to merge 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:exp/dense-callspec-lazy-keywords
Draft

WIP: dense CallSpec, lazy NodeKeywords, deferred TopRequest#14766
RonnyPfannschmidt wants to merge 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:exp/dense-callspec-lazy-keywords

Conversation

@RonnyPfannschmidt

Copy link
Copy Markdown
Member

Summary

  • WIP / experimental exploration of collection memory for large parametrized suites (related to Very large test suites take *far* too much RAM #619).
  • Store CallSpec2 densely as parallel tuples (_argnames / _values / _indices / _scopes) with tuple marks, exposing dict-like views via _CallSpecMap.
  • Make NodeKeywords lazy: derive node.name, markers, function attributes, and callspec.id on read instead of eagerly copying into a per-item dict.
  • Defer TopRequest until first access (None → construct; False still means cleared after a run).
  • Add bench/collection_memory.py to measure RSS delta per collected item.

Local numbers (parametrized ints, --collect-only)

build items B/item
main 50k ~2860
this branch 50k ~2120 (−26%)
this branch 100k ~1910
python bench/collection_memory.py
python bench/collection_memory.py --counts 50000,100000 --json

Test plan

  • python bench/collection_memory.py --counts 1000,10000,50000,100000
  • pytest testing/test_mark.py testing/python/metafunc.py testing/python/fixtures.py::TestRequestBasic -q
  • Smoke: collect + run multi-arg parametrize and -k selection
  • Broader CI once ready to leave draft

Notes

  • No changelog yet (experiment).
  • Nodeid interning intentionally out of scope (follow-up after new nodeid classes).
  • Further density likely still needs compact single-arg CallSpec / lazy items.

Made with Cursor

Experimental changes aimed at large parametrized suites (pytest-dev#619): store
CallSpec2 as parallel tuples, derive NodeKeywords without eager marker
dicts, and defer TopRequest until first use. Adds
bench/collection_memory.py to measure RSS per collected item.

Co-authored-by: Cursor AI <ai@cursor.sh>
Co-authored-by: Cursor Grok 4.5 <grok@x.ai>
@RonnyPfannschmidt

Copy link
Copy Markdown
Member Author

this one doesn't bring in enough win yet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant