gh-153569: simplify tokenizer state - #156484

Draft

pablogsal wants to merge 11 commits into

python:mainfrom

pablogsal:gh-153569-tokenizer-validation-tools-clean

Commits on Aug 28, 2026

gh-153569: bound tokenizer input storage
pablogsal
committed
600a445
gh-153569: return tokenizer tokens as source spans
pablogsal
committed
c10fd82
gh-153569: simplify tokenizer buffer relocation
pablogsal
committed
8c4469b
gh-153569: streamline tokenizer buffer hot paths
pablogsal
committed
fced7ab
gh-153569: remove unused tokenizer cursor support
pablogsal
committed
64893b4
gh-153569: store f-string state as source spans
pablogsal
committed
d3290a7
gh-153569: track f-string comments as source spans
pablogsal
committed
f2bc3ec
gh-153569: simplify f-string span handling
pablogsal
committed
fa4714d
gh-153569: keep token span invariants explicit
pablogsal
committed
049f228
gh-153569: preserve comments after not-equal expressions
pablogsal
committed
eae3e33

Commits on Aug 29, 2026

gh-153569: remove duplicated tokenizer state
pablogsal
committed
3ce81ac