From 1145afabb735e241f1c564fab381c0af6a7eee59 Mon Sep 17 00:00:00 2001 From: johnslavik Date: Sat, 18 Jul 2026 14:38:34 +0200 Subject: [PATCH 1/7] Document remaining PyCF compiler flag macros Document PyCF_DONT_IMPLY_DEDENT, PyCF_IGNORE_COOKIE, PyCF_SOURCE_IS_UTF8, PyCF_MASK and PyCF_COMPILE_MASK in the PyCompilerFlags section, with examples. --- Doc/c-api/veryhigh.rst | 81 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index 6256bf7a1454a9a..dc22fef8b017b01 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -343,11 +343,92 @@ the same library that the Python runtime is using. :py:mod:`!ast` Python module, which exports these constants under the same names. + .. c:macro:: PyCF_DONT_IMPLY_DEDENT + + By default, when compiling with the :c:var:`Py_single_input` start + symbol, reaching the end of the source text implicitly closes any + open indented blocks. With this flag set, open blocks are only + closed if the last line of the source ends with a newline; otherwise, + compilation fails with a :exc:`SyntaxError`: + + .. code-block:: c + + PyCompilerFlags flags = { + .cf_flags = 0, + .cf_feature_version = PY_MINOR_VERSION, + }; + const char *source = "if a:\n pass"; + + /* The "if" block is closed implicitly; + this returns a code object: */ + Py_CompileStringFlags(source, "", Py_single_input, &flags); + + /* With the flag, this fails with a SyntaxError, + because the last line does not end with a newline: */ + flags.cf_flags = PyCF_DONT_IMPLY_DEDENT; + Py_CompileStringFlags(source, "", Py_single_input, &flags); + + The :mod:`codeop` module uses this flag to detect incomplete + interactive input. While the user is still typing inside an + indented block, the source does not yet end with a newline, so it + fails to compile and the user is prompted for another line. + + .. c:macro:: PyCF_IGNORE_COOKIE + + Read the source text as UTF-8, ignoring its :pep:`263` encoding + declaration ("coding cookie"), if any: + + .. code-block:: c + + PyCompilerFlags flags = { + .cf_flags = 0, + .cf_feature_version = PY_MINOR_VERSION, + }; + const char *source = "# coding: latin-1\ns = '\xe9'\n"; + + /* The coding cookie is honored: byte 0xE9 is decoded as + Latin-1, and this returns a code object that sets s to "é": */ + Py_CompileStringFlags(source, "", Py_file_input, &flags); + + /* With the flag, the cookie is ignored and compilation fails + with a SyntaxError, because 0xE9 is not valid UTF-8: */ + flags.cf_flags = PyCF_IGNORE_COOKIE; + Py_CompileStringFlags(source, "", Py_file_input, &flags); + + The :func:`compile`, :func:`eval` and :func:`exec` built-in functions + set this flag when the source is a :class:`str` object, because they + pass the text to the parser encoded as UTF-8. + + .. c:macro:: PyCF_SOURCE_IS_UTF8 + + Mark the source text as known to be UTF-8 encoded. + The :func:`compile`, :func:`eval` and :func:`exec` built-in functions + set this flag, but it currently has no effect. + The "``PyCF``" flags above can be combined with "``CO_FUTURE``" flags such as :c:macro:`CO_FUTURE_ANNOTATIONS` to enable features normally selectable using :ref:`future statements `. See :ref:`c_codeobject_flags` for a complete list. + The following masks combine several flags: + + .. c:macro:: PyCF_MASK + + Bitmask of all ``CO_FUTURE`` flags (see :ref:`c_codeobject_flags`), + which select features normally enabled by + :ref:`future statements `. + When code compiled with a ``PyCompilerFlags *flags`` argument + contains a ``from __future__ import`` statement, the flag for the + imported feature is added to *flags*, so that code executed later + in the same context inherits it. + + .. c:macro:: PyCF_COMPILE_MASK + + Bitmask of all ``PyCF`` flags that change how the source is + compiled, such as :c:macro:`PyCF_ONLY_AST`. + The :func:`compile` built-in function uses this mask to validate + its *flags* argument. + .. _start-symbols: From be6279bb0327e98ef12bd35e94abe1d778b30f62 Mon Sep 17 00:00:00 2001 From: johnslavik Date: Sat, 18 Jul 2026 15:33:11 +0200 Subject: [PATCH 2/7] Remove newly documented flags from C API docs ignore list --- Tools/check-c-api-docs/ignored_c_api.txt | 5 ----- 1 file changed, 5 deletions(-) diff --git a/Tools/check-c-api-docs/ignored_c_api.txt b/Tools/check-c-api-docs/ignored_c_api.txt index fa53b205c4ff6af..448cdd203268eaf 100644 --- a/Tools/check-c-api-docs/ignored_c_api.txt +++ b/Tools/check-c-api-docs/ignored_c_api.txt @@ -31,12 +31,7 @@ PY_BIG_ENDIAN PyCFunction_GET_CLASS # cpython/compile.h PyCF_ALLOW_INCOMPLETE_INPUT -PyCF_COMPILE_MASK -PyCF_DONT_IMPLY_DEDENT -PyCF_IGNORE_COOKIE -PyCF_MASK PyCF_MASK_OBSOLETE -PyCF_SOURCE_IS_UTF8 # cpython/descrobject.h PyDescr_NAME PyDescr_TYPE From 240487c5707491ba99f27c791e4c00b8fe343adf Mon Sep 17 00:00:00 2001 From: johnslavik Date: Sat, 18 Jul 2026 15:41:50 +0200 Subject: [PATCH 3/7] Document PyCF_ALLOW_INCOMPLETE_INPUT and PyCF_MASK_OBSOLETE Both entries lead with a caution against using the flag. Remove the two names from the C API docs ignore list. --- Doc/c-api/veryhigh.rst | 30 ++++++++++++++++++++++++ Tools/check-c-api-docs/ignored_c_api.txt | 3 --- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index dc22fef8b017b01..ff1c1f2f8452b1e 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -343,6 +343,24 @@ the same library that the Python runtime is using. :py:mod:`!ast` Python module, which exports these constants under the same names. + .. c:macro:: PyCF_ALLOW_INCOMPLETE_INPUT + + This flag is an implementation detail of the :mod:`codeop` module. + Do not use it elsewhere; its behavior is unsupported and may + change without warning. + + With this flag set, when compilation fails because the source text + ends where more input is expected, for example in the middle of an + indented block or an unterminated string literal, the error raised + is the undocumented ``_IncompleteInputError``, a subclass of + :exc:`SyntaxError`. The :mod:`codeop` module sets this flag, + together with :c:macro:`PyCF_DONT_IMPLY_DEDENT`, to tell input + that is incomplete apart from input with a real syntax error, so + that interactive interpreters know when to prompt for another + line instead of reporting an error. + + .. versionadded:: 3.11 + .. c:macro:: PyCF_DONT_IMPLY_DEDENT By default, when compiling with the :c:var:`Py_single_input` start @@ -422,6 +440,18 @@ the same library that the Python runtime is using. imported feature is added to *flags*, so that code executed later in the same context inherits it. + .. c:macro:: PyCF_MASK_OBSOLETE + + Do not use this mask in new code. It is kept only so that old + code passing its flags to :func:`compile` keeps working. + + Bitmask of flags for future features that are long obsolete. + It contains only :c:macro:`CO_NESTED` + (:py:data:`__future__.nested_scopes`), which has had no effect + since nested scopes became mandatory in Python 2.2. + The :func:`compile` built-in function accepts flags in this mask, + but ignores them. + .. c:macro:: PyCF_COMPILE_MASK Bitmask of all ``PyCF`` flags that change how the source is diff --git a/Tools/check-c-api-docs/ignored_c_api.txt b/Tools/check-c-api-docs/ignored_c_api.txt index 448cdd203268eaf..e04deffc64f5104 100644 --- a/Tools/check-c-api-docs/ignored_c_api.txt +++ b/Tools/check-c-api-docs/ignored_c_api.txt @@ -29,9 +29,6 @@ PY_DWORD_MAX PY_BIG_ENDIAN # cpython/methodobject.h PyCFunction_GET_CLASS -# cpython/compile.h -PyCF_ALLOW_INCOMPLETE_INPUT -PyCF_MASK_OBSOLETE # cpython/descrobject.h PyDescr_NAME PyDescr_TYPE From 943ba4e134a037bf5e70b6e60dd4a32846964893 Mon Sep 17 00:00:00 2001 From: johnslavik Date: Sat, 18 Jul 2026 15:46:31 +0200 Subject: [PATCH 4/7] Group low-level compiler flags under a rubric Introduce them with a note that code outside the standard library rarely needs them, and title the masks group for symmetry. --- Doc/c-api/veryhigh.rst | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index ff1c1f2f8452b1e..28258b42ac5c993 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -343,11 +343,17 @@ the same library that the Python runtime is using. :py:mod:`!ast` Python module, which exports these constants under the same names. + .. rubric:: Low-level flags + + The following flags serve narrow needs of the standard library and + interactive interpreters. Code outside the standard library rarely + has a reason to use them. + .. c:macro:: PyCF_ALLOW_INCOMPLETE_INPUT - This flag is an implementation detail of the :mod:`codeop` module. - Do not use it elsewhere; its behavior is unsupported and may - change without warning. + This flag is a private interface between the compiler and the + :mod:`codeop` module. Do not use it; its behavior is unsupported + and may change without warning. With this flag set, when compilation fails because the source text ends where more input is expected, for example in the middle of an @@ -428,6 +434,8 @@ the same library that the Python runtime is using. selectable using :ref:`future statements `. See :ref:`c_codeobject_flags` for a complete list. + .. rubric:: Flag masks + The following masks combine several flags: .. c:macro:: PyCF_MASK From ef8fae2bb8ec9feb636d72f5180bea921e35f00a Mon Sep 17 00:00:00 2001 From: johnslavik Date: Sat, 18 Jul 2026 16:00:50 +0200 Subject: [PATCH 5/7] Trim PyCF_MASK_OBSOLETE description --- Doc/c-api/veryhigh.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index 28258b42ac5c993..940cde4ffc58279 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -453,12 +453,8 @@ the same library that the Python runtime is using. Do not use this mask in new code. It is kept only so that old code passing its flags to :func:`compile` keeps working. - Bitmask of flags for future features that are long obsolete. - It contains only :c:macro:`CO_NESTED` - (:py:data:`__future__.nested_scopes`), which has had no effect - since nested scopes became mandatory in Python 2.2. - The :func:`compile` built-in function accepts flags in this mask, - but ignores them. + Bitmask of flags for obsolete future features that no longer + have any effect. .. c:macro:: PyCF_COMPILE_MASK From 4020463ce494ece284da05a6c27ea2412cf012c3 Mon Sep 17 00:00:00 2001 From: johnslavik Date: Sat, 18 Jul 2026 16:09:13 +0200 Subject: [PATCH 6/7] Cover the flag masks by the low-level flags rubric The masks serve the same narrow needs as the flags, so drop their separate rubric and let the low-level introduction apply to both. --- Doc/c-api/veryhigh.rst | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index 940cde4ffc58279..7466e795f0b2740 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -345,9 +345,9 @@ the same library that the Python runtime is using. .. rubric:: Low-level flags - The following flags serve narrow needs of the standard library and - interactive interpreters. Code outside the standard library rarely - has a reason to use them. + The following flags and masks serve narrow needs of the standard + library and interactive interpreters. Code outside the standard + library rarely has a reason to use them. .. c:macro:: PyCF_ALLOW_INCOMPLETE_INPUT @@ -434,8 +434,6 @@ the same library that the Python runtime is using. selectable using :ref:`future statements `. See :ref:`c_codeobject_flags` for a complete list. - .. rubric:: Flag masks - The following masks combine several flags: .. c:macro:: PyCF_MASK From 455118998e74694fd03762e6f6a700120158349b Mon Sep 17 00:00:00 2001 From: johnslavik Date: Sat, 18 Jul 2026 16:34:14 +0200 Subject: [PATCH 7/7] Note that low-level flags and masks are implementation details --- Doc/c-api/veryhigh.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index 7466e795f0b2740..bba5c7f8ecf7751 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -347,7 +347,8 @@ the same library that the Python runtime is using. The following flags and masks serve narrow needs of the standard library and interactive interpreters. Code outside the standard - library rarely has a reason to use them. + library rarely has a reason to use them. They are considered + implementation details and may change at any time. .. c:macro:: PyCF_ALLOW_INCOMPLETE_INPUT