gh-155292: Skip updating unicodedata with mismatched interpreter - #157066
Merged
Conversation
mkstringprep uses things like str.lower(), so it generates the wrong result if run in an interpreter with a different Unicode data version than the target. This means that updating the Unicode version is a two-step process: run makeunicodedata.py, then compile, then run mkstringprep.py. The two steps can (and should) be combined when re-running regen-unicodedata to verify that the data is up to date. The pythonGH-155292 fix only considered that case. Change makeunicodedata.py to only run mkstringprep.py when the current interpreter is up to it. Otherwise, show a reminder. As an extra complication, download the input (RFC 3454) in the "first step", since an out-of-date stringprep.py's freshness assertion may prevent downloads.
encukou
commented
Sep 7, 2026
| data_file = open(FILENAME, encoding='utf-8') | ||
|
|
||
| with data_file: | ||
| with open("Tools/unicode/data/rfc3454.txt", encoding='utf-8') as data_file: |
Member
Author
There was a problem hiding this comment.
I figured we don't need the urllib call here: you shouldn't hit this in the usual case, and if you do, the FileNotFoundError makes it clear that you need rfc3454.txt and where to put it.
gpshead
approved these changes
Sep 8, 2026
sethmlarson
approved these changes
Sep 8, 2026
maurycy
reviewed
Sep 8, 2026
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
maurycy
added a commit
to maurycy/cpython
that referenced
this pull request
Sep 10, 2026
* main: (158 commits) pythongh-156924: Try reifying lazy imports in `ForwarRef.evaluate()` (python#156940) pythongh-156233: Fix typos and code snippet errors in the docs (python#156475) pythongh-156837: Refer to yield expressions in generator function definitions (pythonGH-156863) pythongh-155292: Skip updating unicodedata with mismatched interpreter (pythonGH-157066) pythongh-157170: Document the scope of global curses settings (pythonGH-157207) pythongh-155966: Correct handling of `math.tanpi` poles (python#155980) pythongh-157170: Restore use_env() after test_use_prescr_screen in test_curses (pythonGH-157171) pythongh-156910: fix deadlock in type_set_abstractmethods under free-threading (python#156948) pythongh-155648: Write the empty and placeholder IDLE tests (python#156260) pythongh-153569: centralize formatted-string state and source spans (python#156484) pythongh-157137: Mark the PEP 820 soft deprecations as 3.15, not `next` (python#157138) pythongh-152433: Use regular LoadLibrary in UWP for Windows system libs (pythonGH-156972) pythongh-121617: Fix Py_CLEAR() memcpy in C++: replace NULL with _Py_NULL (python#157188) pythongh-157135: Fix documentation errors in the `math.atan{2}pi` functions (python#157136) pythongh-121617: Fix Py_CLEAR() in C++: replace NULL with _Py_NULL (python#157067) pythongh-156774: Speed up pdb startup with asyncio guard (python#156775) pythongh-156109: Allow static, non-framework iOS builds (python#156110) pythongh-156780: Emscripten: add missing EM_JS_DEPS (python#156798) pythongh-152936: Make privileged functions available on Android (python#152977) pythongh-123018: Keep the libedit history file header when truncating (pythonGH-157165) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mkstringprep uses things like str.lower(), so it generates the wrong
result if run in an interpreter with a different Unicode data version
than the target.
This means that updating the Unicode version is a two-step process:
run makeunicodedata.py, then compile, then run mkstringprep.py.
The two steps can (and should) be combined when re-running
regen-unicodedata to verify that the data is up to date.
The #155292 fix only considered that case.
Change makeunicodedata.py to only run mkstringprep.py when the
current interpreter is up to it. Otherwise, show a reminder.
For an extra complication, download the input (rcf3454.txt) in
the "first step", since an out-of-date stringprep.py's freshness
assertion may break URL encoding in urllib.