From d489c41f2168f46ddef28618e1ec564bccab75a7 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 2 Feb 2022 17:18:45 +0000 Subject: [PATCH 01/18] bpo-31582 Created a new section describing sys.path initialization Moved common details from the Windows specific finding modules section to a new platform neutral section. Linked to the new section from the relevant module documenation. --- Doc/library/importlib.rst | 3 ++ Doc/library/site.rst | 4 ++- Doc/library/sys.rst | 7 ++-- Doc/tutorial/modules.rst | 2 ++ Doc/using/cmdline.rst | 68 +++++++++++++++++++++++++++++++++++++++ Doc/using/windows.rst | 25 ++------------ Doc/whatsnew/3.6.rst | 4 +-- Doc/whatsnew/3.7.rst | 2 +- Misc/ACKS | 1 + 9 files changed, 87 insertions(+), 29 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 23d908196669f51..e90ecd25492c191 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -51,6 +51,9 @@ managing aspects of Python packages: The :func:`.__import__` function The :keyword:`import` statement is syntactic sugar for this function. + :ref:`using-on-finding-modules` + The initialization of :data:`sys.path`. + :pep:`235` Import on Case-Insensitive Platforms diff --git a/Doc/library/site.rst b/Doc/library/site.rst index e2ad3c48f9754ee..0f5f1880e2006a2 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -276,4 +276,6 @@ value greater than 2 if there is an error. .. seealso:: - :pep:`370` -- Per user site-packages directory + * :pep:`370` -- Per user site-packages directory + * :ref:`using-on-finding-modules` -- The initialization of :data:`sys.path`. + diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 5e47201f88eae13..47d5e98d19f89d6 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1132,9 +1132,10 @@ always available. .. seealso:: - Module :mod:`site` This describes how to use .pth files to extend - :data:`sys.path`. - + * Module :mod:`site` This describes how to use .pth files to + extend :data:`sys.path`. + * :ref:`using-on-finding-modules` for further details about the + initialization of :data:`sys.path`. .. data:: path_hooks diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index f1d4957e37eb11a..0e6e9616add3b63 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -194,6 +194,8 @@ named :file:`spam.py` in a list of directories given by the variable * The installation-dependent default (by convention including a ``site-packages`` directory, handled by the :mod:`site` module). +More details are at :ref:`using-on-finding-modules`. + .. note:: On file systems which support symlinks, the directory containing the input script is calculated after the symlink is followed. In other words the diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index d341ea8bb43c883..3d3a7cab7557740 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -994,3 +994,71 @@ Debug-mode variables Need Python configured with the :option:`--with-trace-refs` build option. .. versionadded:: 3.11 + +.. _using-on-finding-modules: + +Finding modules +--------------- + +A module search path is initialized when Python starts. This module search path +may be accessed at :data:`sys.path`. + +The first entry in the module search path is the directoy containing the input +script (or the the current directory when no file is specified). + +On Windows a default Python installation adds the :file:`/Lib` directory and +the :file:`/Lib/site-packages` directory to the module search path. These +locations are relative to the :file:`python.exe` location. Standard modules +are usually found in the Lib directory and third-party modules in the +Lib/site-packages directory. + +On Unix the lib and lib/site-packages directories are added by looking for +:file:`/exec_prefix/lib/pythonversion` , :file:`/exec_prefix/lib/pythonversion/site-packages` , +:file:`/prefix/lib/pythonversion` and :file:`/prefix/lib/pythonversion/site-packages` . +Often ``exec_prefix`` and ``prefix`` are the same and a common result is +adding :file:`/usr/lib/pythonversion` and :file:`/usr/lib/pythonversion/site-packages` +to the module search path. See :ref:`using-on-unix` for details. + +The :envvar:`PYTHONPATH` environment variable is often used to add directories +to the search path. + +Another common way to set the search path is to create :mod:`sitecustomize` +or :mod:`usercustomize` modules as described in the :mod:`site` module +documentation. + +Virtual environments +~~~~~~~~~~~~~~~~~~~~ + +If Python is run in a virtual environment (as described at :ref:`tut-venv`) +the Lib and Lib/site-packages directories are specific to the virtual +environment. + +If a ``pyvenv.cfg`` file is found alongside the main executable, or in the +directory one level above the executable, the following variations apply: + +* If ``home`` is an absolute path and :envvar:`PYTHONHOME` is not set, this + path is used instead of the path to the main executable when deducing the + Lib and Lib/site-packages directory locations. + +_pth files +~~~~~~~~~~ + +To completely override :data:`sys.path`, create a ``._pth`` file with the same +name as the DLL/dylib/so (``python311._pth``) or the executable (``python._pth``) +and specify one line for each path to add to :data:`sys.path`. The file based on +the library name overrides the one based on the executable, which allows paths +to be restricted for any program loading the runtime if desired. + +When the file exists, all registry and environment variables are ignored, +isolated mode is enabled, and :mod:`site` is not imported unless one line in the +file specifies ``import site``. Blank paths and lines starting with ``#`` are +ignored. Each path may be absolute or relative to the location of the file. +Import statements other than to ``site`` are not permitted, and arbitrary code +cannot be specified. + +Note that ``.pth`` files (without leading underscore) will be processed normally +by the :mod:`site` module when ``import site`` has been specified. + +.. seealso:: + + :ref:`windows_finding_modules` for detailed Windows notes. \ No newline at end of file diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 041166f4f8584f2..a98b050a6363f23 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -946,32 +946,13 @@ target Python. -.. _finding_modules: +.. _windows_finding_modules: Finding modules =============== -Python usually stores its library (and thereby your site-packages folder) in the -installation directory. So, if you had installed Python to -:file:`C:\\Python\\`, the default library would reside in -:file:`C:\\Python\\Lib\\` and third-party modules should be stored in -:file:`C:\\Python\\Lib\\site-packages\\`. - -To completely override :data:`sys.path`, create a ``._pth`` file with the same -name as the DLL (``python37._pth``) or the executable (``python._pth``) and -specify one line for each path to add to :data:`sys.path`. The file based on the -DLL name overrides the one based on the executable, which allows paths to be -restricted for any program loading the runtime if desired. - -When the file exists, all registry and environment variables are ignored, -isolated mode is enabled, and :mod:`site` is not imported unless one line in the -file specifies ``import site``. Blank paths and lines starting with ``#`` are -ignored. Each path may be absolute or relative to the location of the file. -Import statements other than to ``site`` are not permitted, and arbitrary code -cannot be specified. - -Note that ``.pth`` files (without leading underscore) will be processed normally -by the :mod:`site` module when ``import site`` has been specified. +These notes supplement the description at :ref:`using-on-finding-modules` with +detailed Windows notes. When no ``._pth`` file is found, this is how :data:`sys.path` is populated on Windows: diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst index a56d7a592305f04..d35a0fd9d9da38e 100644 --- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -162,10 +162,10 @@ Windows improvements: * A ``._pth`` file can be added to force isolated mode and fully specify all search paths to avoid registry and environment lookup. See - :ref:`the documentation ` for more information. + :ref:`the documentation ` for more information. * A ``python36.zip`` file now works as a landmark to infer - :envvar:`PYTHONHOME`. See :ref:`the documentation ` for + :envvar:`PYTHONHOME`. See :ref:`the documentation ` for more information. diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 1199535c84dcaab..5ffbf77b2b880cc 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -2474,7 +2474,7 @@ Windows-only Changes The file used to override :data:`sys.path` is now called ``._pth`` instead of ``'sys.path'``. -See :ref:`finding_modules` for more information. +See :ref:`windows_finding_modules` for more information. (Contributed by Steve Dower in :issue:`28137`.) diff --git a/Misc/ACKS b/Misc/ACKS index cf023c9af927ab5..d3d7bfd6e988c9e 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1890,6 +1890,7 @@ Colin Watson David Watson Aaron Watters Alex Waygood +Russel Webber Henrik Weber Leon Weber Steve Weber From 367de561fadbb073feb7b7710754602915f9a0cd Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 2 Feb 2022 17:32:10 +0000 Subject: [PATCH 02/18] Fix trailing white spaces --- Doc/library/site.rst | 3 +-- Doc/library/sys.rst | 4 ++-- Doc/using/cmdline.rst | 36 ++++++++++++++++++------------------ Doc/using/windows.rst | 2 +- 4 files changed, 22 insertions(+), 23 deletions(-) diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 0f5f1880e2006a2..866cb6937a702eb 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -277,5 +277,4 @@ value greater than 2 if there is an error. .. seealso:: * :pep:`370` -- Per user site-packages directory - * :ref:`using-on-finding-modules` -- The initialization of :data:`sys.path`. - + * :ref:`using-on-finding-modules` -- The initialization of :data:`sys.path`. \ No newline at end of file diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 47d5e98d19f89d6..e6cfa8df74731d6 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1132,9 +1132,9 @@ always available. .. seealso:: - * Module :mod:`site` This describes how to use .pth files to + * Module :mod:`site` This describes how to use .pth files to extend :data:`sys.path`. - * :ref:`using-on-finding-modules` for further details about the + * :ref:`using-on-finding-modules` for further details about the initialization of :data:`sys.path`. .. data:: path_hooks diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 3d3a7cab7557740..f86b4f362397926 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1000,37 +1000,37 @@ Debug-mode variables Finding modules --------------- -A module search path is initialized when Python starts. This module search path +A module search path is initialized when Python starts. This module search path may be accessed at :data:`sys.path`. -The first entry in the module search path is the directoy containing the input +The first entry in the module search path is the directoy containing the input script (or the the current directory when no file is specified). -On Windows a default Python installation adds the :file:`/Lib` directory and -the :file:`/Lib/site-packages` directory to the module search path. These -locations are relative to the :file:`python.exe` location. Standard modules -are usually found in the Lib directory and third-party modules in the +On Windows a default Python installation adds the :file:`/Lib` directory and +the :file:`/Lib/site-packages` directory to the module search path. These +locations are relative to the :file:`python.exe` location. Standard modules +are usually found in the Lib directory and third-party modules in the Lib/site-packages directory. -On Unix the lib and lib/site-packages directories are added by looking for +On Unix the lib and lib/site-packages directories are added by looking for :file:`/exec_prefix/lib/pythonversion` , :file:`/exec_prefix/lib/pythonversion/site-packages` , :file:`/prefix/lib/pythonversion` and :file:`/prefix/lib/pythonversion/site-packages` . -Often ``exec_prefix`` and ``prefix`` are the same and a common result is -adding :file:`/usr/lib/pythonversion` and :file:`/usr/lib/pythonversion/site-packages` +Often ``exec_prefix`` and ``prefix`` are the same and a common result is +adding :file:`/usr/lib/pythonversion` and :file:`/usr/lib/pythonversion/site-packages` to the module search path. See :ref:`using-on-unix` for details. -The :envvar:`PYTHONPATH` environment variable is often used to add directories +The :envvar:`PYTHONPATH` environment variable is often used to add directories to the search path. -Another common way to set the search path is to create :mod:`sitecustomize` -or :mod:`usercustomize` modules as described in the :mod:`site` module -documentation. +Another common way to set the search path is to create :mod:`sitecustomize` +or :mod:`usercustomize` modules as described in the :mod:`site` module +documentation. Virtual environments ~~~~~~~~~~~~~~~~~~~~ -If Python is run in a virtual environment (as described at :ref:`tut-venv`) -the Lib and Lib/site-packages directories are specific to the virtual +If Python is run in a virtual environment (as described at :ref:`tut-venv`) +the Lib and Lib/site-packages directories are specific to the virtual environment. If a ``pyvenv.cfg`` file is found alongside the main executable, or in the @@ -1044,9 +1044,9 @@ _pth files ~~~~~~~~~~ To completely override :data:`sys.path`, create a ``._pth`` file with the same -name as the DLL/dylib/so (``python311._pth``) or the executable (``python._pth``) -and specify one line for each path to add to :data:`sys.path`. The file based on -the library name overrides the one based on the executable, which allows paths +name as the DLL/dylib/so (``python311._pth``) or the executable (``python._pth``) +and specify one line for each path to add to :data:`sys.path`. The file based on +the library name overrides the one based on the executable, which allows paths to be restricted for any program loading the runtime if desired. When the file exists, all registry and environment variables are ignored, diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index a98b050a6363f23..641bd9396ac3394 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -951,7 +951,7 @@ target Python. Finding modules =============== -These notes supplement the description at :ref:`using-on-finding-modules` with +These notes supplement the description at :ref:`using-on-finding-modules` with detailed Windows notes. When no ``._pth`` file is found, this is how :data:`sys.path` is populated on From 4ad7d6e5cf4745dd697ef80497403e370b300f97 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 2 Feb 2022 17:48:36 +0000 Subject: [PATCH 03/18] Fix typo --- Doc/using/cmdline.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index f86b4f362397926..a54ffaf2b87e69c 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1003,7 +1003,7 @@ Finding modules A module search path is initialized when Python starts. This module search path may be accessed at :data:`sys.path`. -The first entry in the module search path is the directoy containing the input +The first entry in the module search path is the directory containing the input script (or the the current directory when no file is specified). On Windows a default Python installation adds the :file:`/Lib` directory and From 69dc3ea7c01c509598e7ee48f26b7364c0980adb Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Thu, 3 Feb 2022 08:09:19 +0000 Subject: [PATCH 04/18] Update Doc/using/cmdline.rst Suggested edit from @eryksun Co-authored-by: Eryk Sun --- Doc/using/cmdline.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index a54ffaf2b87e69c..26c405471a395b7 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1003,8 +1003,10 @@ Finding modules A module search path is initialized when Python starts. This module search path may be accessed at :data:`sys.path`. -The first entry in the module search path is the directory containing the input -script (or the the current directory when no file is specified). +The first entry in the module search path is the directory that contains the +input script, if there is one. Otherwise, the first entry is the current +directory, which is the case when executing the REPL shell, a ``-c`` command, +or ``-m`` module. On Windows a default Python installation adds the :file:`/Lib` directory and the :file:`/Lib/site-packages` directory to the module search path. These From 826183f492484551bebb8a4fa76f9ab9158f99e6 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Thu, 3 Feb 2022 08:57:36 +0000 Subject: [PATCH 05/18] Code review comments --- Doc/using/cmdline.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 26c405471a395b7..88799c915132fd4 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1005,8 +1005,8 @@ may be accessed at :data:`sys.path`. The first entry in the module search path is the directory that contains the input script, if there is one. Otherwise, the first entry is the current -directory, which is the case when executing the REPL shell, a ``-c`` command, -or ``-m`` module. +directory, which is the case when executing the interactive shell, a ``-c`` +command, or ``-m`` module. On Windows a default Python installation adds the :file:`/Lib` directory and the :file:`/Lib/site-packages` directory to the module search path. These @@ -1015,8 +1015,8 @@ are usually found in the Lib directory and third-party modules in the Lib/site-packages directory. On Unix the lib and lib/site-packages directories are added by looking for -:file:`/exec_prefix/lib/pythonversion` , :file:`/exec_prefix/lib/pythonversion/site-packages` , -:file:`/prefix/lib/pythonversion` and :file:`/prefix/lib/pythonversion/site-packages` . +:file:`/{exec_prefix}/lib/pythonversion/site-packages` , :file:`/{exec_prefix}/lib/pythonversion/lib-dynload` , +:file:`/{prefix}/lib/pythonversion` and :file:`/{prefix}/lib/pythonversion/site-packages` . Often ``exec_prefix`` and ``prefix`` are the same and a common result is adding :file:`/usr/lib/pythonversion` and :file:`/usr/lib/pythonversion/site-packages` to the module search path. See :ref:`using-on-unix` for details. @@ -1046,7 +1046,7 @@ _pth files ~~~~~~~~~~ To completely override :data:`sys.path`, create a ``._pth`` file with the same -name as the DLL/dylib/so (``python311._pth``) or the executable (``python._pth``) +name as the DLL/dylib (``python311._pth``) or the executable (``python._pth``) and specify one line for each path to add to :data:`sys.path`. The file based on the library name overrides the one based on the executable, which allows paths to be restricted for any program loading the runtime if desired. From afa47cc82cb83ef97e39e0b35fa424ec8912eb27 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Sun, 6 Feb 2022 16:23:41 +0000 Subject: [PATCH 06/18] Introduce home, prefix and exec_prefix in the doc --- Doc/using/cmdline.rst | 74 +++++++++++++++++++++++++++++-------------- 1 file changed, 51 insertions(+), 23 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 88799c915132fd4..9f491f40cd82b8c 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1008,39 +1008,60 @@ input script, if there is one. Otherwise, the first entry is the current directory, which is the case when executing the interactive shell, a ``-c`` command, or ``-m`` module. -On Windows a default Python installation adds the :file:`/Lib` directory and -the :file:`/Lib/site-packages` directory to the module search path. These -locations are relative to the :file:`python.exe` location. Standard modules -are usually found in the Lib directory and third-party modules in the -Lib/site-packages directory. - -On Unix the lib and lib/site-packages directories are added by looking for -:file:`/{exec_prefix}/lib/pythonversion/site-packages` , :file:`/{exec_prefix}/lib/pythonversion/lib-dynload` , -:file:`/{prefix}/lib/pythonversion` and :file:`/{prefix}/lib/pythonversion/site-packages` . -Often ``exec_prefix`` and ``prefix`` are the same and a common result is -adding :file:`/usr/lib/pythonversion` and :file:`/usr/lib/pythonversion/site-packages` -to the module search path. See :ref:`using-on-unix` for details. - The :envvar:`PYTHONPATH` environment variable is often used to add directories -to the search path. - -Another common way to set the search path is to create :mod:`sitecustomize` -or :mod:`usercustomize` modules as described in the :mod:`site` module -documentation. +to the search path. If this environment variable is found then the contents are +added to the module search path. + +The next items added are the directories containing standard Python modules as +well as any shared libraries (DLL/dylib/so) that these modules depend on. The +directory with the platform-independent Python modules is called ``prefix``. +The directory with the shared libraries is called ``exec_prefix``. + +The :envvar:`PYTHONHOME` environment variable may be used to set the ``prefix`` +and ``exec_prefix`` locations. Otherwise these directories are found by using +the Python executable as a starting point and then looking for various 'landmark' +files and directories. Note that any symbolic links are followed so the real +Python executable location is used as the search starting point. On MacOS, the +:envvar:`PYTHONEXECUTABLE` environment variable may be used to set the Python +executable location. The Python executable location is called ``home``. + +Once ``home`` is determined, the ``prefix`` directory is found by first looking +for :file:`python{majorversion}{minorversion}.zip` (``python311.zip``). On Windows +the zip archive is searched for in :file:`Lib` and on Unix the archive is expected +to be in :file:`lib`. Note that the expected zip archive location is added to the +module search path even if the archive does not exist. If no archive was found, +Python on Windows will continue the search for ``prefix`` by looking for :file:`Lib\\os.py` +or :file:`Lib\\os.pyc`. Python on Unix will look for :file:`lib/python{majorversion}.{minorversion}/os.py` +(``lib/python3.11/os.py``) or :file:`lib/python{majorversion}.{minorversion}/os.pyc` (``lib/python3.11/os.pyc``). +On Windows ``prefix`` and ``exec_prefix`` are the same, however on other platforms +:file:`lib/python{majorversion}.{minorversion}/lib-dynload` (``lib/python3.11/lib-dynload``) +is searched for and used as an anchor for ``exec_prefix``. + +Once found, ``prefix`` and ``exec_prefix`` may be accessed at :data:`sys.prefix` +and :data:`sys.exec_prefix`. + +Finally, the :mod:`site` module is processed and :file:`site-packages` directories +are added to the module search path. A common way to customize the search path is +to create :mod:`sitecustomize` or :mod:`usercustomize` modules as described in +the :mod:`site` module documentation. + +.. note:: + + The effect of :mod:`site` on the module search path may be seen by running + Python with ``-S`` which will start Python without importing site. Virtual environments ~~~~~~~~~~~~~~~~~~~~ If Python is run in a virtual environment (as described at :ref:`tut-venv`) -the Lib and Lib/site-packages directories are specific to the virtual -environment. +then ``prefix`` and ``exec_prefix`` are specific to the virtual environment. If a ``pyvenv.cfg`` file is found alongside the main executable, or in the directory one level above the executable, the following variations apply: * If ``home`` is an absolute path and :envvar:`PYTHONHOME` is not set, this - path is used instead of the path to the main executable when deducing the - Lib and Lib/site-packages directory locations. + path is used instead of the path to the main executable when deducing ``prefix`` + and ``exec_prefix``. _pth files ~~~~~~~~~~ @@ -1061,6 +1082,13 @@ cannot be specified. Note that ``.pth`` files (without leading underscore) will be processed normally by the :mod:`site` module when ``import site`` has been specified. +Embedded Python +~~~~~~~~~~~~~~~ + +If Python is embedded within another application :c:func:`Py_SetPath` can be used to +bypass the initialization of the module search path. + .. seealso:: - :ref:`windows_finding_modules` for detailed Windows notes. \ No newline at end of file + * :ref:`windows_finding_modules` for detailed Windows notes. + * :ref:`using-on-unix` for Unix details. \ No newline at end of file From afbabf5143b780a8838dfdd5f8b1a6c7c85c4a4c Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Sun, 6 Feb 2022 16:29:33 +0000 Subject: [PATCH 07/18] Darn trailing whitespaces. --- Doc/using/cmdline.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 9f491f40cd82b8c..b1d90647335f2d8 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1043,7 +1043,7 @@ and :data:`sys.exec_prefix`. Finally, the :mod:`site` module is processed and :file:`site-packages` directories are added to the module search path. A common way to customize the search path is to create :mod:`sitecustomize` or :mod:`usercustomize` modules as described in -the :mod:`site` module documentation. +the :mod:`site` module documentation. .. note:: From a1e330777dece4ab87c637aa8049b752450e4a71 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 9 Feb 2022 14:34:03 +0000 Subject: [PATCH 08/18] Fix the expected zip archive location on Windows --- Doc/using/cmdline.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index b1d90647335f2d8..16885aacf6d13e6 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1027,7 +1027,7 @@ executable location. The Python executable location is called ``home``. Once ``home`` is determined, the ``prefix`` directory is found by first looking for :file:`python{majorversion}{minorversion}.zip` (``python311.zip``). On Windows -the zip archive is searched for in :file:`Lib` and on Unix the archive is expected +the zip archive is searched for in ``home`` and on Unix the archive is expected to be in :file:`lib`. Note that the expected zip archive location is added to the module search path even if the archive does not exist. If no archive was found, Python on Windows will continue the search for ``prefix`` by looking for :file:`Lib\\os.py` From db925d45078c1042cd5e29c5d94fa8fbb4415de7 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Thu, 3 Mar 2022 10:56:21 +0000 Subject: [PATCH 09/18] Edits suggested by Ned Deily --- Doc/using/cmdline.rst | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 16885aacf6d13e6..81b508a91a80a8f 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1012,30 +1012,36 @@ The :envvar:`PYTHONPATH` environment variable is often used to add directories to the search path. If this environment variable is found then the contents are added to the module search path. +.. note:: + + :envvar:`PYTHONPATH` will affect all installed Python versions/environments. + Be wary of setting this in your shell profile or global environment variables. + The :mod:`site` module offers more nuanced techniques as mentioned below. + The next items added are the directories containing standard Python modules as -well as any shared libraries (DLL/dylib/so) that these modules depend on. The -directory with the platform-independent Python modules is called ``prefix``. -The directory with the shared libraries is called ``exec_prefix``. +well as any :term:`extension module`\s that these modules depend on. Extension +modules are .dll files on Windows, .dylib files on macOS and .so files on Linux. +The directory with the platform-independent Python modules is called ``prefix``. +The directory with the extension modules is called ``exec_prefix``. The :envvar:`PYTHONHOME` environment variable may be used to set the ``prefix`` and ``exec_prefix`` locations. Otherwise these directories are found by using the Python executable as a starting point and then looking for various 'landmark' files and directories. Note that any symbolic links are followed so the real -Python executable location is used as the search starting point. On MacOS, the -:envvar:`PYTHONEXECUTABLE` environment variable may be used to set the Python -executable location. The Python executable location is called ``home``. +Python executable location is used as the search starting point. The Python +executable location is called ``home``. Once ``home`` is determined, the ``prefix`` directory is found by first looking for :file:`python{majorversion}{minorversion}.zip` (``python311.zip``). On Windows the zip archive is searched for in ``home`` and on Unix the archive is expected to be in :file:`lib`. Note that the expected zip archive location is added to the module search path even if the archive does not exist. If no archive was found, -Python on Windows will continue the search for ``prefix`` by looking for :file:`Lib\\os.py` -or :file:`Lib\\os.pyc`. Python on Unix will look for :file:`lib/python{majorversion}.{minorversion}/os.py` -(``lib/python3.11/os.py``) or :file:`lib/python{majorversion}.{minorversion}/os.pyc` (``lib/python3.11/os.pyc``). -On Windows ``prefix`` and ``exec_prefix`` are the same, however on other platforms -:file:`lib/python{majorversion}.{minorversion}/lib-dynload` (``lib/python3.11/lib-dynload``) -is searched for and used as an anchor for ``exec_prefix``. +Python on Windows will continue the search for ``prefix`` by looking for :file:`Lib\\os.py`. +Python on Unix will look for :file:`lib/python{majorversion}.{minorversion}/os.py` +(``lib/python3.11/os.py``). On Windows ``prefix`` and ``exec_prefix`` are the same, +however on other platforms :file:`lib/python{majorversion}.{minorversion}/lib-dynload` +(``lib/python3.11/lib-dynload``) is searched for and used as an anchor for +``exec_prefix``. Once found, ``prefix`` and ``exec_prefix`` may be accessed at :data:`sys.prefix` and :data:`sys.exec_prefix`. @@ -1067,10 +1073,10 @@ _pth files ~~~~~~~~~~ To completely override :data:`sys.path`, create a ``._pth`` file with the same -name as the DLL/dylib (``python311._pth``) or the executable (``python._pth``) +name as the extension module (``python311._pth``) or the executable (``python._pth``) and specify one line for each path to add to :data:`sys.path`. The file based on -the library name overrides the one based on the executable, which allows paths -to be restricted for any program loading the runtime if desired. +the extension module name overrides the one based on the executable, which allows +paths to be restricted for any program loading the runtime if desired. When the file exists, all registry and environment variables are ignored, isolated mode is enabled, and :mod:`site` is not imported unless one line in the From 3fe47e99b0af005cfa5cf198572293ffdf15f2c2 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Thu, 3 Mar 2022 11:04:09 +0000 Subject: [PATCH 10/18] Add missing new lines. --- Doc/library/site.rst | 3 ++- Doc/using/cmdline.rst | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 866cb6937a702eb..1cd6dec92208717 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -277,4 +277,5 @@ value greater than 2 if there is an error. .. seealso:: * :pep:`370` -- Per user site-packages directory - * :ref:`using-on-finding-modules` -- The initialization of :data:`sys.path`. \ No newline at end of file + * :ref:`using-on-finding-modules` -- The initialization of :data:`sys.path`. + diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 81b508a91a80a8f..fda35fc9f70bd51 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1097,4 +1097,5 @@ bypass the initialization of the module search path. .. seealso:: * :ref:`windows_finding_modules` for detailed Windows notes. - * :ref:`using-on-unix` for Unix details. \ No newline at end of file + * :ref:`using-on-unix` for Unix details. + From 59cbc07c783610c8f9204b8a5fed4881f3995a82 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Fri, 4 Mar 2022 10:43:56 +0000 Subject: [PATCH 11/18] Changes suggested by Victor Stinner and Jeremy Paige --- Doc/using/cmdline.rst | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index fda35fc9f70bd51..566fd223fec6aff 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1054,7 +1054,14 @@ the :mod:`site` module documentation. .. note:: The effect of :mod:`site` on the module search path may be seen by running - Python with ``-S`` which will start Python without importing site. + Python with ``-S`` which will start Python without importing site. ``-s`` will + skip adding the :data:`user site-packages directory ` to :data:`sys.path`. + + The effect of :envvar:`PYTHON*` environment variables may be seen by running + Python with ``-E`` which will start Python ignoring these environment variables. + + Finally, ``-I`` will start Python in isolated mode, simlar to ``-E`` and ``-s`` + but with further restrictions. Virtual environments ~~~~~~~~~~~~~~~~~~~~ @@ -1092,7 +1099,9 @@ Embedded Python ~~~~~~~~~~~~~~~ If Python is embedded within another application :c:func:`Py_SetPath` can be used to -bypass the initialization of the module search path. +bypass the initialization of the module search path. Alternatively Python can be +initialized with :c:func:`Py_InitializeFromConfig` and the :c:type:`PyConfig` structure, +the path specific details are described at :ref:`init-path-config`. .. seealso:: From 51a89190dcc7b294d873968e9b550a4fb5bd5be6 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Fri, 11 Mar 2022 11:12:07 +0000 Subject: [PATCH 12/18] Changes suggested by Ned Deily and Steve Dower --- Doc/using/cmdline.rst | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 566fd223fec6aff..abafe20737efa6f 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -59,6 +59,7 @@ all consecutive arguments will end up in :data:`sys.argv` -- note that the first element, subscript zero (``sys.argv[0]``), is a string reflecting the program's source. +.. _using-on-interface-option-c: .. cmdoption:: -c Execute the Python code in *command*. *command* can be one or more @@ -72,6 +73,7 @@ source. .. audit-event:: cpython.run_command command cmdoption-c +.. _using-on-interface-option-m: .. cmdoption:: -m Search :data:`sys.path` for the named module and execute its contents as @@ -252,6 +254,7 @@ Miscellaneous options options). See also :envvar:`PYTHONDEBUG`. +.. _using-on-misc-option-uppercase-e: .. cmdoption:: -E Ignore all :envvar:`PYTHON*` environment variables, e.g. @@ -269,6 +272,7 @@ Miscellaneous options raises an exception. See also :envvar:`PYTHONINSPECT`. +.. _using-on-misc-option-uppercase-i: .. cmdoption:: -I Run Python in isolated mode. This also implies -E and -s. @@ -334,6 +338,7 @@ Miscellaneous options .. versionadded:: 3.2.3 +.. _using-on-misc-option-s: .. cmdoption:: -s Don't add the :data:`user site-packages directory ` to @@ -344,6 +349,7 @@ Miscellaneous options :pep:`370` -- Per user site-packages directory +.. _using-on-misc-option-uppercase-s: .. cmdoption:: -S Disable the import of the module :mod:`site` and the site-dependent @@ -1005,8 +1011,8 @@ may be accessed at :data:`sys.path`. The first entry in the module search path is the directory that contains the input script, if there is one. Otherwise, the first entry is the current -directory, which is the case when executing the interactive shell, a ``-c`` -command, or ``-m`` module. +directory, which is the case when executing the interactive shell, a :ref:`-c ` +command, or :ref:`-m ` module. The :envvar:`PYTHONPATH` environment variable is often used to add directories to the search path. If this environment variable is found then the contents are @@ -1020,8 +1026,8 @@ added to the module search path. The next items added are the directories containing standard Python modules as well as any :term:`extension module`\s that these modules depend on. Extension -modules are .dll files on Windows, .dylib files on macOS and .so files on Linux. -The directory with the platform-independent Python modules is called ``prefix``. +modules are .pyd files on Windows and .so files on other platforms. The +directory with the platform-independent Python modules is called ``prefix``. The directory with the extension modules is called ``exec_prefix``. The :envvar:`PYTHONHOME` environment variable may be used to set the ``prefix`` @@ -1053,15 +1059,8 @@ the :mod:`site` module documentation. .. note:: - The effect of :mod:`site` on the module search path may be seen by running - Python with ``-S`` which will start Python without importing site. ``-s`` will - skip adding the :data:`user site-packages directory ` to :data:`sys.path`. - - The effect of :envvar:`PYTHON*` environment variables may be seen by running - Python with ``-E`` which will start Python ignoring these environment variables. - - Finally, ``-I`` will start Python in isolated mode, simlar to ``-E`` and ``-s`` - but with further restrictions. + Certain command line options may further affect path calculations. + See :ref:`-E `, :ref:`-I `, :ref:`-s ` and :ref:`-S ` for further details. Virtual environments ~~~~~~~~~~~~~~~~~~~~ @@ -1080,9 +1079,9 @@ _pth files ~~~~~~~~~~ To completely override :data:`sys.path`, create a ``._pth`` file with the same -name as the extension module (``python311._pth``) or the executable (``python._pth``) +name as the shared library (``python311._pth``) or the executable (``python._pth``) and specify one line for each path to add to :data:`sys.path`. The file based on -the extension module name overrides the one based on the executable, which allows +the shared library name overrides the one based on the executable, which allows paths to be restricted for any program loading the runtime if desired. When the file exists, all registry and environment variables are ignored, From be21efad37e469d4a8764ac8809a17acb3798353 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Fri, 11 Mar 2022 12:11:35 +0000 Subject: [PATCH 13/18] Update Doc/using/cmdline.rst Co-authored-by: Victor Stinner --- Doc/using/cmdline.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index abafe20737efa6f..da95df9a637aa72 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1026,7 +1026,7 @@ added to the module search path. The next items added are the directories containing standard Python modules as well as any :term:`extension module`\s that these modules depend on. Extension -modules are .pyd files on Windows and .so files on other platforms. The +modules are ``.pyd`` files on Windows and ``.so`` files on other platforms. The directory with the platform-independent Python modules is called ``prefix``. The directory with the extension modules is called ``exec_prefix``. From b57d32ee3596fe35a80fb0e4b6f84e685fbc196d Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 16 Mar 2022 10:36:13 +0000 Subject: [PATCH 14/18] Changes following feedback from Victor Stinner and Eryk Sun --- Doc/library/importlib.rst | 2 +- Doc/library/site.rst | 2 +- Doc/library/sys.rst | 4 ++-- Doc/tutorial/modules.rst | 2 +- Doc/using/cmdline.rst | 40 +++++++++++++++++++-------------------- Doc/using/windows.rst | 2 +- 6 files changed, 25 insertions(+), 27 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index e90ecd25492c191..783653831b507d4 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -51,7 +51,7 @@ managing aspects of Python packages: The :func:`.__import__` function The :keyword:`import` statement is syntactic sugar for this function. - :ref:`using-on-finding-modules` + :ref:`sys-path-init` The initialization of :data:`sys.path`. :pep:`235` diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 1cd6dec92208717..5941739ee6942f7 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -277,5 +277,5 @@ value greater than 2 if there is an error. .. seealso:: * :pep:`370` -- Per user site-packages directory - * :ref:`using-on-finding-modules` -- The initialization of :data:`sys.path`. + * :ref:`sys-path-init` -- The initialization of :data:`sys.path`. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index e6cfa8df74731d6..2ff15d76f4abce5 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1126,6 +1126,8 @@ always available. current directory first. Notice that the script directory is inserted *before* the entries inserted as a result of :envvar:`PYTHONPATH`. + The initialization of :data:`sys.path`is documented at :ref:`sys-path-init`. + A program is free to modify this list for its own purposes. Only strings and bytes should be added to :data:`sys.path`; all other data types are ignored during import. @@ -1134,8 +1136,6 @@ always available. .. seealso:: * Module :mod:`site` This describes how to use .pth files to extend :data:`sys.path`. - * :ref:`using-on-finding-modules` for further details about the - initialization of :data:`sys.path`. .. data:: path_hooks diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index 0e6e9616add3b63..a4e1cefb26b71d3 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -194,7 +194,7 @@ named :file:`spam.py` in a list of directories given by the variable * The installation-dependent default (by convention including a ``site-packages`` directory, handled by the :mod:`site` module). -More details are at :ref:`using-on-finding-modules`. +More details are at :ref:`sys-path-init`. .. note:: On file systems which support symlinks, the directory containing the input diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index da95df9a637aa72..45c5d35d1d5a9df 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -59,7 +59,6 @@ all consecutive arguments will end up in :data:`sys.argv` -- note that the first element, subscript zero (``sys.argv[0]``), is a string reflecting the program's source. -.. _using-on-interface-option-c: .. cmdoption:: -c Execute the Python code in *command*. *command* can be one or more @@ -73,7 +72,6 @@ source. .. audit-event:: cpython.run_command command cmdoption-c -.. _using-on-interface-option-m: .. cmdoption:: -m Search :data:`sys.path` for the named module and execute its contents as @@ -254,7 +252,6 @@ Miscellaneous options options). See also :envvar:`PYTHONDEBUG`. -.. _using-on-misc-option-uppercase-e: .. cmdoption:: -E Ignore all :envvar:`PYTHON*` environment variables, e.g. @@ -338,7 +335,6 @@ Miscellaneous options .. versionadded:: 3.2.3 -.. _using-on-misc-option-s: .. cmdoption:: -s Don't add the :data:`user site-packages directory ` to @@ -349,7 +345,6 @@ Miscellaneous options :pep:`370` -- Per user site-packages directory -.. _using-on-misc-option-uppercase-s: .. cmdoption:: -S Disable the import of the module :mod:`site` and the site-dependent @@ -1001,7 +996,7 @@ Debug-mode variables .. versionadded:: 3.11 -.. _using-on-finding-modules: +.. _sys-path-init: Finding modules --------------- @@ -1011,8 +1006,8 @@ may be accessed at :data:`sys.path`. The first entry in the module search path is the directory that contains the input script, if there is one. Otherwise, the first entry is the current -directory, which is the case when executing the interactive shell, a :ref:`-c ` -command, or :ref:`-m ` module. +directory, which is the case when executing the interactive shell, a :option:`-c` +command, or :option:`-m` module. The :envvar:`PYTHONPATH` environment variable is often used to add directories to the search path. If this environment variable is found then the contents are @@ -1047,10 +1042,11 @@ Python on Unix will look for :file:`lib/python{majorversion}.{minorversion}/os.p (``lib/python3.11/os.py``). On Windows ``prefix`` and ``exec_prefix`` are the same, however on other platforms :file:`lib/python{majorversion}.{minorversion}/lib-dynload` (``lib/python3.11/lib-dynload``) is searched for and used as an anchor for -``exec_prefix``. +``exec_prefix``. On some platforms :file:`lib` may be :file:`lib64` or another value, +see :data:`sys.platlibdir` and :envvar:`PYTHONPLATLIBDIR`. -Once found, ``prefix`` and ``exec_prefix`` may be accessed at :data:`sys.prefix` -and :data:`sys.exec_prefix`. +Once found, ``prefix`` and ``exec_prefix`` are available at :data:`sys.prefix` and +:data:`sys.exec_prefix` respectively. Finally, the :mod:`site` module is processed and :file:`site-packages` directories are added to the module search path. A common way to customize the search path is @@ -1060,7 +1056,7 @@ the :mod:`site` module documentation. .. note:: Certain command line options may further affect path calculations. - See :ref:`-E `, :ref:`-I `, :ref:`-s ` and :ref:`-S ` for further details. + See :option:`-E`, :option:`-I`, :option:`-s` and :option:`-S` for further details. Virtual environments ~~~~~~~~~~~~~~~~~~~~ @@ -1078,11 +1074,13 @@ directory one level above the executable, the following variations apply: _pth files ~~~~~~~~~~ -To completely override :data:`sys.path`, create a ``._pth`` file with the same -name as the shared library (``python311._pth``) or the executable (``python._pth``) -and specify one line for each path to add to :data:`sys.path`. The file based on -the shared library name overrides the one based on the executable, which allows -paths to be restricted for any program loading the runtime if desired. +To completely override :data:`sys.path` create a ``._pth`` file with the same +name as the shared library or executable (``python._pth`` or ``python311._pth``). +The shared library path is always known on Windows, however it may not be +available on other platforms. In the ``._pth`` file specify one line for each path +to add to :data:`sys.path`. The file based on the shared library name overrides +the one based on the executable, which allows paths to be restricted for any +program loading the runtime if desired. When the file exists, all registry and environment variables are ignored, isolated mode is enabled, and :mod:`site` is not imported unless one line in the @@ -1097,10 +1095,10 @@ by the :mod:`site` module when ``import site`` has been specified. Embedded Python ~~~~~~~~~~~~~~~ -If Python is embedded within another application :c:func:`Py_SetPath` can be used to -bypass the initialization of the module search path. Alternatively Python can be -initialized with :c:func:`Py_InitializeFromConfig` and the :c:type:`PyConfig` structure, -the path specific details are described at :ref:`init-path-config`. +If Python is embedded within another application :c:func:`Py_InitializeFromConfig` and +the :c:type:`PyConfig` structure can be used to initialize Python. The path specific +details are described at :ref:`init-path-config`. Alternatively the older :c:func:`Py_SetPath` +can be used to bypass the initialization of the module search path. .. seealso:: diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 641bd9396ac3394..c92f361aaeea325 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -951,7 +951,7 @@ target Python. Finding modules =============== -These notes supplement the description at :ref:`using-on-finding-modules` with +These notes supplement the description at :ref:`sys-path-init` with detailed Windows notes. When no ``._pth`` file is found, this is how :data:`sys.path` is populated on From a5ee617d0dae2b95a587e9e4ffe92f0e6d57d8f5 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 16 Mar 2022 10:47:33 +0000 Subject: [PATCH 15/18] Fix missing space --- Doc/library/sys.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 2ff15d76f4abce5..b93e9c47531ced5 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1126,7 +1126,7 @@ always available. current directory first. Notice that the script directory is inserted *before* the entries inserted as a result of :envvar:`PYTHONPATH`. - The initialization of :data:`sys.path`is documented at :ref:`sys-path-init`. + The initialization of :data:`sys.path` is documented at :ref:`sys-path-init`. A program is free to modify this list for its own purposes. Only strings and bytes should be added to :data:`sys.path`; all other data types are From 2616cc7e3ee6c81e71a4a10cc467af8a9386e3be Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 16 Mar 2022 11:13:46 +0000 Subject: [PATCH 16/18] Moved to a new document --- Doc/library/modules.rst | 1 + Doc/library/sys_path_init.rst | 108 +++++++++++++++++++++++++++++++++ Doc/using/cmdline.rst | 110 ---------------------------------- 3 files changed, 109 insertions(+), 110 deletions(-) create mode 100644 Doc/library/sys_path_init.rst diff --git a/Doc/library/modules.rst b/Doc/library/modules.rst index 131dfca9576a384..6cf6eb28a1e058f 100644 --- a/Doc/library/modules.rst +++ b/Doc/library/modules.rst @@ -19,3 +19,4 @@ The full list of modules described in this chapter is: importlib.rst importlib.resources.rst importlib.metadata.rst + sys_path_init.rst diff --git a/Doc/library/sys_path_init.rst b/Doc/library/sys_path_init.rst new file mode 100644 index 000000000000000..b022a316cebfa85 --- /dev/null +++ b/Doc/library/sys_path_init.rst @@ -0,0 +1,108 @@ +.. _sys-path-init: + +The initialization of the :data:`sys.path` module search path +============================================================= + +A module search path is initialized when Python starts. This module search path +may be accessed at :data:`sys.path`. + +The first entry in the module search path is the directory that contains the +input script, if there is one. Otherwise, the first entry is the current +directory, which is the case when executing the interactive shell, a :option:`-c` +command, or :option:`-m` module. + +The :envvar:`PYTHONPATH` environment variable is often used to add directories +to the search path. If this environment variable is found then the contents are +added to the module search path. + +.. note:: + + :envvar:`PYTHONPATH` will affect all installed Python versions/environments. + Be wary of setting this in your shell profile or global environment variables. + The :mod:`site` module offers more nuanced techniques as mentioned below. + +The next items added are the directories containing standard Python modules as +well as any :term:`extension module`\s that these modules depend on. Extension +modules are ``.pyd`` files on Windows and ``.so`` files on other platforms. The +directory with the platform-independent Python modules is called ``prefix``. +The directory with the extension modules is called ``exec_prefix``. + +The :envvar:`PYTHONHOME` environment variable may be used to set the ``prefix`` +and ``exec_prefix`` locations. Otherwise these directories are found by using +the Python executable as a starting point and then looking for various 'landmark' +files and directories. Note that any symbolic links are followed so the real +Python executable location is used as the search starting point. The Python +executable location is called ``home``. + +Once ``home`` is determined, the ``prefix`` directory is found by first looking +for :file:`python{majorversion}{minorversion}.zip` (``python311.zip``). On Windows +the zip archive is searched for in ``home`` and on Unix the archive is expected +to be in :file:`lib`. Note that the expected zip archive location is added to the +module search path even if the archive does not exist. If no archive was found, +Python on Windows will continue the search for ``prefix`` by looking for :file:`Lib\\os.py`. +Python on Unix will look for :file:`lib/python{majorversion}.{minorversion}/os.py` +(``lib/python3.11/os.py``). On Windows ``prefix`` and ``exec_prefix`` are the same, +however on other platforms :file:`lib/python{majorversion}.{minorversion}/lib-dynload` +(``lib/python3.11/lib-dynload``) is searched for and used as an anchor for +``exec_prefix``. On some platforms :file:`lib` may be :file:`lib64` or another value, +see :data:`sys.platlibdir` and :envvar:`PYTHONPLATLIBDIR`. + +Once found, ``prefix`` and ``exec_prefix`` are available at :data:`sys.prefix` and +:data:`sys.exec_prefix` respectively. + +Finally, the :mod:`site` module is processed and :file:`site-packages` directories +are added to the module search path. A common way to customize the search path is +to create :mod:`sitecustomize` or :mod:`usercustomize` modules as described in +the :mod:`site` module documentation. + +.. note:: + + Certain command line options may further affect path calculations. + See :option:`-E`, :option:`-I`, :option:`-s` and :option:`-S` for further details. + +Virtual environments +-------------------- + +If Python is run in a virtual environment (as described at :ref:`tut-venv`) +then ``prefix`` and ``exec_prefix`` are specific to the virtual environment. + +If a ``pyvenv.cfg`` file is found alongside the main executable, or in the +directory one level above the executable, the following variations apply: + +* If ``home`` is an absolute path and :envvar:`PYTHONHOME` is not set, this + path is used instead of the path to the main executable when deducing ``prefix`` + and ``exec_prefix``. + +_pth files +---------- + +To completely override :data:`sys.path` create a ``._pth`` file with the same +name as the shared library or executable (``python._pth`` or ``python311._pth``). +The shared library path is always known on Windows, however it may not be +available on other platforms. In the ``._pth`` file specify one line for each path +to add to :data:`sys.path`. The file based on the shared library name overrides +the one based on the executable, which allows paths to be restricted for any +program loading the runtime if desired. + +When the file exists, all registry and environment variables are ignored, +isolated mode is enabled, and :mod:`site` is not imported unless one line in the +file specifies ``import site``. Blank paths and lines starting with ``#`` are +ignored. Each path may be absolute or relative to the location of the file. +Import statements other than to ``site`` are not permitted, and arbitrary code +cannot be specified. + +Note that ``.pth`` files (without leading underscore) will be processed normally +by the :mod:`site` module when ``import site`` has been specified. + +Embedded Python +--------------- + +If Python is embedded within another application :c:func:`Py_InitializeFromConfig` and +the :c:type:`PyConfig` structure can be used to initialize Python. The path specific +details are described at :ref:`init-path-config`. Alternatively the older :c:func:`Py_SetPath` +can be used to bypass the initialization of the module search path. + +.. seealso:: + + * :ref:`windows_finding_modules` for detailed Windows notes. + * :ref:`using-on-unix` for Unix details. \ No newline at end of file diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 45c5d35d1d5a9df..1ed3ab3a6241365 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -995,113 +995,3 @@ Debug-mode variables Need Python configured with the :option:`--with-trace-refs` build option. .. versionadded:: 3.11 - -.. _sys-path-init: - -Finding modules ---------------- - -A module search path is initialized when Python starts. This module search path -may be accessed at :data:`sys.path`. - -The first entry in the module search path is the directory that contains the -input script, if there is one. Otherwise, the first entry is the current -directory, which is the case when executing the interactive shell, a :option:`-c` -command, or :option:`-m` module. - -The :envvar:`PYTHONPATH` environment variable is often used to add directories -to the search path. If this environment variable is found then the contents are -added to the module search path. - -.. note:: - - :envvar:`PYTHONPATH` will affect all installed Python versions/environments. - Be wary of setting this in your shell profile or global environment variables. - The :mod:`site` module offers more nuanced techniques as mentioned below. - -The next items added are the directories containing standard Python modules as -well as any :term:`extension module`\s that these modules depend on. Extension -modules are ``.pyd`` files on Windows and ``.so`` files on other platforms. The -directory with the platform-independent Python modules is called ``prefix``. -The directory with the extension modules is called ``exec_prefix``. - -The :envvar:`PYTHONHOME` environment variable may be used to set the ``prefix`` -and ``exec_prefix`` locations. Otherwise these directories are found by using -the Python executable as a starting point and then looking for various 'landmark' -files and directories. Note that any symbolic links are followed so the real -Python executable location is used as the search starting point. The Python -executable location is called ``home``. - -Once ``home`` is determined, the ``prefix`` directory is found by first looking -for :file:`python{majorversion}{minorversion}.zip` (``python311.zip``). On Windows -the zip archive is searched for in ``home`` and on Unix the archive is expected -to be in :file:`lib`. Note that the expected zip archive location is added to the -module search path even if the archive does not exist. If no archive was found, -Python on Windows will continue the search for ``prefix`` by looking for :file:`Lib\\os.py`. -Python on Unix will look for :file:`lib/python{majorversion}.{minorversion}/os.py` -(``lib/python3.11/os.py``). On Windows ``prefix`` and ``exec_prefix`` are the same, -however on other platforms :file:`lib/python{majorversion}.{minorversion}/lib-dynload` -(``lib/python3.11/lib-dynload``) is searched for and used as an anchor for -``exec_prefix``. On some platforms :file:`lib` may be :file:`lib64` or another value, -see :data:`sys.platlibdir` and :envvar:`PYTHONPLATLIBDIR`. - -Once found, ``prefix`` and ``exec_prefix`` are available at :data:`sys.prefix` and -:data:`sys.exec_prefix` respectively. - -Finally, the :mod:`site` module is processed and :file:`site-packages` directories -are added to the module search path. A common way to customize the search path is -to create :mod:`sitecustomize` or :mod:`usercustomize` modules as described in -the :mod:`site` module documentation. - -.. note:: - - Certain command line options may further affect path calculations. - See :option:`-E`, :option:`-I`, :option:`-s` and :option:`-S` for further details. - -Virtual environments -~~~~~~~~~~~~~~~~~~~~ - -If Python is run in a virtual environment (as described at :ref:`tut-venv`) -then ``prefix`` and ``exec_prefix`` are specific to the virtual environment. - -If a ``pyvenv.cfg`` file is found alongside the main executable, or in the -directory one level above the executable, the following variations apply: - -* If ``home`` is an absolute path and :envvar:`PYTHONHOME` is not set, this - path is used instead of the path to the main executable when deducing ``prefix`` - and ``exec_prefix``. - -_pth files -~~~~~~~~~~ - -To completely override :data:`sys.path` create a ``._pth`` file with the same -name as the shared library or executable (``python._pth`` or ``python311._pth``). -The shared library path is always known on Windows, however it may not be -available on other platforms. In the ``._pth`` file specify one line for each path -to add to :data:`sys.path`. The file based on the shared library name overrides -the one based on the executable, which allows paths to be restricted for any -program loading the runtime if desired. - -When the file exists, all registry and environment variables are ignored, -isolated mode is enabled, and :mod:`site` is not imported unless one line in the -file specifies ``import site``. Blank paths and lines starting with ``#`` are -ignored. Each path may be absolute or relative to the location of the file. -Import statements other than to ``site`` are not permitted, and arbitrary code -cannot be specified. - -Note that ``.pth`` files (without leading underscore) will be processed normally -by the :mod:`site` module when ``import site`` has been specified. - -Embedded Python -~~~~~~~~~~~~~~~ - -If Python is embedded within another application :c:func:`Py_InitializeFromConfig` and -the :c:type:`PyConfig` structure can be used to initialize Python. The path specific -details are described at :ref:`init-path-config`. Alternatively the older :c:func:`Py_SetPath` -can be used to bypass the initialization of the module search path. - -.. seealso:: - - * :ref:`windows_finding_modules` for detailed Windows notes. - * :ref:`using-on-unix` for Unix details. - From d2da921fb4c52d4dabe26224a55bab78eb0f72a8 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 16 Mar 2022 11:18:53 +0000 Subject: [PATCH 17/18] Add missing newline --- Doc/library/sys_path_init.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/sys_path_init.rst b/Doc/library/sys_path_init.rst index b022a316cebfa85..72c1387344c2ad0 100644 --- a/Doc/library/sys_path_init.rst +++ b/Doc/library/sys_path_init.rst @@ -105,4 +105,4 @@ can be used to bypass the initialization of the module search path. .. seealso:: * :ref:`windows_finding_modules` for detailed Windows notes. - * :ref:`using-on-unix` for Unix details. \ No newline at end of file + * :ref:`using-on-unix` for Unix details. From 66b3e597154225c75e3f5daf23355409948d4450 Mon Sep 17 00:00:00 2001 From: Russel Webber <24542073+RusselWebber@users.noreply.github.com> Date: Wed, 16 Mar 2022 11:31:20 +0000 Subject: [PATCH 18/18] Remove a label that is no longer needed. --- Doc/using/cmdline.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 1ed3ab3a6241365..d341ea8bb43c883 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -269,7 +269,6 @@ Miscellaneous options raises an exception. See also :envvar:`PYTHONINSPECT`. -.. _using-on-misc-option-uppercase-i: .. cmdoption:: -I Run Python in isolated mode. This also implies -E and -s.