Skip to content

gh-148762: Speed up multiline regexes anchored by ^ #152339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
haampie wants to merge 4 commits into
base: main
Choose a base branch
from
+25 −0
Open

gh-148762: Speed up multiline regexes anchored by ^ #152339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ab4bf9f
gh-148762: speed up caret match in regexes
haampie Apr 23, 2026
134b179
Reduce control flow nesting
haampie Jun 26, 2026
d62cc5b
news
haampie Jun 26, 2026
5d1149a
drop cast cause not used consistently elsewhere
haampie Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2026-04-19-23-29-38.gh-issue-148762.HSCJka.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Multiline regexes starting with a caret, such as ``re.compile("^foo",
re.MULTILINE)``, now run significantly faster.
23 changes: 23 additions & 0 deletions Modules/_sre/sre_lib.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1848,6 +1848,29 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
ptr++;
RESET_CAPTURE_GROUP();
}
} else if (pattern[0] == SRE_OP_AT &&
pattern[1] == SRE_AT_BEGINNING_LINE) {
/* pattern is anchored at the start of a line (MULTILINE "^").
Only the start of the string and the character after a linebreak
can match, so jump from one line start to the next instead of
trying SRE(match) at every position. */
end = (SRE_CHAR *)state->end;
TRACE(("|%p|%p|SEARCH AT_BEGINNING_LINE\n", pattern, ptr));
state->start = state->ptr = ptr;
status = SRE(match)(state, pattern, 1);
state->must_advance = 0;
while (status == 0) {
/* skip to the next linebreak ... */
while (ptr < end && !SRE_IS_LINEBREAK(*ptr))
ptr++;
if (ptr >= end)
return 0;
Comment on lines +1864 to +1867

@eendebakpt eendebakpt Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be

Suggested change
while (ptr < end && !SRE_IS_LINEBREAK(*ptr))
ptr++;
if (ptr >= end)
return 0;
+#if SIZEOF_SRE_CHAR == 1
ptr = memchr(ptr, '\n', end - ptr);
if (ptr == NULL)
return 0;
#else
while (ptr < end && !SRE_IS_LINEBREAK(*ptr))
ptr++;
if (ptr >= end)
return 0;
#endif

(I did not benchmark, not sure it is worth the change)

@haampie haampie Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had another issue/PR where I tried something like this, but it's hard to optimize when you don't know the distribution of \n character in the "haystack". See #148729 (comment); on the macbook it was hard to beat a hand-written loop:

The regression on darwin is because the letter i has a density of 2.88% in the corpus; the cross-over density is apparently about 2%, below which memchr is faster.

Based on wc -cl $(find -type f -name '*.py') from cpython's own sources, there are 988799 lines and 36153429 bytes, or a newline character density of 2.7%, meaning on my macbook the memchr would likely be slightly worse. But it depends on the use case.

So, I would hold off on combining different types of optimizations. This PR is about reducing the number of expensive match function calls.

ptr++; /* ... and step past it, onto a line start */
RESET_CAPTURE_GROUP();
TRACE(("|%p|%p|SEARCH AT_BEGINNING_LINE\n", pattern, ptr));
state->start = state->ptr = ptr;
status = SRE(match)(state, pattern, 0);
}
} else {
/* general case */
assert(ptr <= end);
Expand Down
Loading