pythonGH-126985: move pyvenv.cfg detection from site to getpath (python#126987)

Author: FFY00

Doc/c-api/init_config.rst

@@ -1590,9 +1590,22 @@ If a ``._pth`` file is present:
 * Set :c:member:`~PyConfig.site_import` to ``0``.
 * Set :c:member:`~PyConfig.safe_path` to ``1``.
 
+If :c:member:`~PyConfig.home` is not set and a ``pyvenv.cfg`` file is present in
+the same directory as :c:member:`~PyConfig.executable`, or its parent,
+:c:member:`~PyConfig.prefix` and :c:member:`~PyConfig.exec_prefix` are set that
+location. When this happens, :c:member:`~PyConfig.base_prefix` and
+:c:member:`~PyConfig.base_exec_prefix` still keep their value, pointing to the
+base installation. See :ref:`sys-path-init-virtual-environments` for more
+information.
+
 The ``__PYVENV_LAUNCHER__`` environment variable is used to set
 :c:member:`PyConfig.base_executable`.
 
+.. versionchanged:: 3.14
+
+   :c:member:`~PyConfig.prefix`, and :c:member:`~PyConfig.exec_prefix`, are now
+   set to the ``pyvenv.cfg`` directory. This was previously done by :mod:`site`,
+   therefore affected by :option:`-S`.
 
 .. _pyinitconfig_api:
 

Doc/library/site.rst

@@ -49,14 +49,22 @@ added path for configuration files.
    identified by the "t" suffix in the version-specific directory name, such as
    :file:`lib/python3.13t/`.
 
-If a file named "pyvenv.cfg" exists one directory above sys.executable,
-sys.prefix and sys.exec_prefix are set to that directory and
-it is also checked for site-packages (sys.base_prefix and
-sys.base_exec_prefix will always be the "real" prefixes of the Python
-installation). If "pyvenv.cfg" (a bootstrap configuration file) contains
-the key "include-system-site-packages" set to anything other than "true"
-(case-insensitive), the system-level prefixes will not be
-searched for site-packages; otherwise they will.
+.. versionchanged:: 3.14
+
+   :mod:`site` is no longer responsible for updating :data:`sys.prefix` and
+   :data:`sys.exec_prefix` on :ref:`sys-path-init-virtual-environments`. This is
+   now done during the :ref:`path initialization <sys-path-init>`. As a result,
+   under :ref:`sys-path-init-virtual-environments`, :data:`sys.prefix` and
+   :data:`sys.exec_prefix` no longer depend on the :mod:`site` initialization,
+   and are therefore unaffected by :option:`-S`.
+
+.. _site-virtual-environments-configuration:
+
+When running under a :ref:`virtual environment <sys-path-init-virtual-environments>`,
+the ``pyvenv.cfg`` file in :data:`sys.prefix` is checked for site-specific
+configurations. If the ``include-system-site-packages`` key exists and is set to
+``true`` (case-insensitive), the system-level prefixes will be searched for
+site-packages, otherwise they won't.
 
 .. index::
    single: # (hash); comment

Doc/library/sys.rst

@@ -130,27 +130,26 @@ always available.
 
 .. data:: base_exec_prefix
 
-   Set during Python startup, before ``site.py`` is run, to the same value as
-   :data:`exec_prefix`. If not running in a
-   :ref:`virtual environment <venv-def>`, the values will stay the same; if
-   ``site.py`` finds that a virtual environment is in use, the values of
-   :data:`prefix` and :data:`exec_prefix` will be changed to point to the
-   virtual environment, whereas :data:`base_prefix` and
-   :data:`base_exec_prefix` will remain pointing to the base Python
-   installation (the one which the virtual environment was created from).
+   Equivalent to :data:`exec_prefix`, but refering to the base Python installation.
+
+   When running under :ref:`sys-path-init-virtual-environments`,
+   :data:`exec_prefix` gets overwritten to the virtual environment prefix.
+   :data:`base_exec_prefix`, conversely, does not change, and always points to
+   the base Python installation.
+   Refer to :ref:`sys-path-init-virtual-environments` for more information.
 
    .. versionadded:: 3.3
 
 
 .. data:: base_prefix
 
-   Set during Python startup, before ``site.py`` is run, to the same value as
-   :data:`prefix`. If not running in a :ref:`virtual environment <venv-def>`, the values
-   will stay the same; if ``site.py`` finds that a virtual environment is in
-   use, the values of :data:`prefix` and :data:`exec_prefix` will be changed to
-   point to the virtual environment, whereas :data:`base_prefix` and
-   :data:`base_exec_prefix` will remain pointing to the base Python
-   installation (the one which the virtual environment was created from).
+   Equivalent to :data:`prefix`, but refering to the base Python installation.
+
+   When running under :ref:`virtual environment <venv-def>`,
+   :data:`prefix` gets overwritten to the virtual environment prefix.
+   :data:`base_prefix`, conversely, does not change, and always points to
+   the base Python installation.
+   Refer to :ref:`sys-path-init-virtual-environments` for more information.
 
    .. versionadded:: 3.3
 
@@ -483,11 +482,19 @@ always available.
 
    .. note::
 
-      If a :ref:`virtual environment <venv-def>` is in effect, this
-      value will be changed in ``site.py`` to point to the virtual environment.
-      The value for the Python installation will still be available, via
-      :data:`base_exec_prefix`.
+      If a :ref:`virtual environment <venv-def>` is in effect, this :data:`exec_prefix`
+      will point to the virtual environment. The value for the Python installation
+      will still be available, via :data:`base_exec_prefix`.
+      Refer to :ref:`sys-path-init-virtual-environments` for more information.
 
+   .. versionchanged:: 3.14
+
+      When running under a :ref:`virtual environment <venv-def>`,
+      :data:`prefix` and :data:`exec_prefix` are now set to the virtual
+      environment prefix by the :ref:`path initialization <sys-path-init>`,
+      instead of :mod:`site`. This means that :data:`prefix` and
+      :data:`exec_prefix` always point to the virtual environment, even when
+      :mod:`site` is disabled (:option:`-S`).
 
 .. data:: executable
 
@@ -1483,10 +1490,21 @@ always available.
    argument to the :program:`configure` script.  See
    :ref:`installation_paths` for derived paths.
 
-   .. note:: If a :ref:`virtual environment <venv-def>` is in effect, this
-      value will be changed in ``site.py`` to point to the virtual
-      environment. The value for the Python installation will still be
-      available, via :data:`base_prefix`.
+   .. note::
+
+      If a :ref:`virtual environment <venv-def>` is in effect, this :data:`prefix`
+      will point to the virtual environment. The value for the Python installation
+      will still be available, via :data:`base_prefix`.
+      Refer to :ref:`sys-path-init-virtual-environments` for more information.
+
+   .. versionchanged:: 3.14
+
+      When running under a :ref:`virtual environment <venv-def>`,
+      :data:`prefix` and :data:`exec_prefix` are now set to the virtual
+      environment prefix by the :ref:`path initialization <sys-path-init>`,
+      instead of :mod:`site`. This means that :data:`prefix` and
+      :data:`exec_prefix` always point to the virtual environment, even when
+      :mod:`site` is disabled (:option:`-S`).
 
 
 .. data:: ps1

Doc/library/sys_path_init.rst

@@ -47,8 +47,15 @@ however on other platforms :file:`lib/python{majorversion}.{minorversion}/lib-dy
 ``exec_prefix``. On some platforms :file:`lib` may be :file:`lib64` or another value,
 see :data:`sys.platlibdir` and :envvar:`PYTHONPLATLIBDIR`.
 
-Once found, ``prefix`` and ``exec_prefix`` are available at :data:`sys.prefix` and
-:data:`sys.exec_prefix` respectively.
+Once found, ``prefix`` and ``exec_prefix`` are available at
+:data:`sys.base_prefix` and :data:`sys.base_exec_prefix` respectively.
+
+If :envvar:`PYTHONHOME` is not set, and a ``pyvenv.cfg`` file is found alongside
+the main executable, or in its parent directory, :data:`sys.prefix` and
+:data:`sys.exec_prefix` get set to the directory containing ``pyvenv.cfg``,
+otherwise they are set to the same value as :data:`sys.base_prefix` and
+:data:`sys.base_exec_prefix`, respectively.
+This is used by :ref:`sys-path-init-virtual-environments`.
 
 Finally, the :mod:`site` module is processed and :file:`site-packages` directories
 are added to the module search path. A common way to customize the search path is
@@ -60,18 +67,40 @@ the :mod:`site` module documentation.
    Certain command line options may further affect path calculations.
    See :option:`-E`, :option:`-I`, :option:`-s` and :option:`-S` for further details.
 
-Virtual environments
+.. versionchanged:: 3.14
+
+   :data:`sys.prefix` and :data:`sys.exec_prefix` are now set to the
+   ``pyvenv.cfg`` directory during the path initialization. This was previously
+   done by :mod:`site`, therefore affected by :option:`-S`.
+
+.. _sys-path-init-virtual-environments:
+
+Virtual Environments
 --------------------
 
-If Python is run in a virtual environment (as described at :ref:`tut-venv`)
-then ``prefix`` and ``exec_prefix`` are specific to the virtual environment.
+Virtual environments place a ``pyvenv.cfg`` file in their prefix, which causes
+:data:`sys.prefix` and :data:`sys.exec_prefix` to point to them, instead of the
+base installation.
+
+The ``prefix`` and ``exec_prefix`` values of the base installation are available
+at :data:`sys.base_prefix` and :data:`sys.base_exec_prefix`.
 
-If a ``pyvenv.cfg`` file is found alongside the main executable, or in the
-directory one level above the executable, the following variations apply:
+As well as being used as a marker to identify virtual environments,
+``pyvenv.cfg`` may also be used to configure the :mod:`site` initialization.
+Please refer to :mod:`site`'s
+:ref:`virtual environments documentation <site-virtual-environments-configuration>`.
+
+.. note::
+
+   :envvar:`PYTHONHOME` overrides the ``pyvenv.cfg`` detection.
+
+.. note::
 
-* If ``home`` is an absolute path and :envvar:`PYTHONHOME` is not set, this
-  path is used instead of the path to the main executable when deducing ``prefix``
-  and ``exec_prefix``.
+   There are other ways how "virtual environments" could be implemented, this
+   documentation referes implementations based on the ``pyvenv.cfg`` mechanism,
+   such as :mod:`venv`. Most virtual environment implementations follow the
+   model set by :mod:`venv`, but there may be exotic implementations that
+   diverge from it.
 
 _pth files
 ----------

Doc/library/venv.rst

@@ -25,6 +25,9 @@ A virtual environment is created on top of an existing
 Python installation, known as the virtual environment's "base" Python, and may
 optionally be isolated from the packages in the base environment,
 so only those explicitly installed in the virtual environment are available.
+See :ref:`sys-path-init-virtual-environments` and :mod:`site`'s
+:ref:`virtual environments documentation <site-virtual-environments-configuration>`
+for more information.
 
 When used from within a virtual environment, common installation tools such as
 :pypi:`pip` will install Python packages into a virtual environment

Lib/site.py

@@ -94,6 +94,12 @@ def _trace(message):
         print(message, file=sys.stderr)
 
 
+def _warn(*args, **kwargs):
+    import warnings
+
+    warnings.warn(*args, **kwargs)
+
+
 def makepath(*paths):
     dir = os.path.join(*paths)
     try:
@@ -619,7 +625,10 @@ def venv(known_paths):
                     elif key == 'home':
                         sys._home = value
 
-        sys.prefix = sys.exec_prefix = site_prefix
+        if sys.prefix != site_prefix:
+            _warn(f'Unexpected value in sys.prefix, expected {site_prefix}, got {sys.prefix}', RuntimeWarning)
+        if sys.exec_prefix != site_prefix:
+            _warn(f'Unexpected value in sys.exec_prefix, expected {site_prefix}, got {sys.exec_prefix}', RuntimeWarning)
 
         # Doing this here ensures venv takes precedence over user-site
         addsitepackages(known_paths, [sys.prefix])

Lib/sysconfig/__init__.py

@@ -173,7 +173,9 @@ def joinuser(*args):
 _PY_VERSION = sys.version.split()[0]
 _PY_VERSION_SHORT = f'{sys.version_info[0]}.{sys.version_info[1]}'
 _PY_VERSION_SHORT_NO_DOT = f'{sys.version_info[0]}{sys.version_info[1]}'
+_PREFIX = os.path.normpath(sys.prefix)
 _BASE_PREFIX = os.path.normpath(sys.base_prefix)
+_EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
 _BASE_EXEC_PREFIX = os.path.normpath(sys.base_exec_prefix)
 # Mutex guarding initialization of _CONFIG_VARS.
 _CONFIG_VARS_LOCK = threading.RLock()
@@ -465,10 +467,8 @@ def _init_config_vars():
     # Normalized versions of prefix and exec_prefix are handy to have;
     # in fact, these are the standard versions used most places in the
     # Distutils.
-    _PREFIX = os.path.normpath(sys.prefix)
-    _EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
-    _CONFIG_VARS['prefix'] = _PREFIX  # FIXME: This gets overwriten by _init_posix.
-    _CONFIG_VARS['exec_prefix'] = _EXEC_PREFIX  # FIXME: This gets overwriten by _init_posix.
+    _CONFIG_VARS['prefix'] = _PREFIX
+    _CONFIG_VARS['exec_prefix'] = _EXEC_PREFIX
     _CONFIG_VARS['py_version'] = _PY_VERSION
     _CONFIG_VARS['py_version_short'] = _PY_VERSION_SHORT
     _CONFIG_VARS['py_version_nodot'] = _PY_VERSION_SHORT_NO_DOT
@@ -541,7 +541,6 @@ def get_config_vars(*args):
     With arguments, return a list of values that result from looking up
     each argument in the configuration variable dictionary.
     """
-    global _CONFIG_VARS_INITIALIZED
 
     # Avoid claiming the lock once initialization is complete.
     if not _CONFIG_VARS_INITIALIZED:
@@ -552,15 +551,6 @@ def get_config_vars(*args):
             # don't re-enter init_config_vars().
             if _CONFIG_VARS is None:
                 _init_config_vars()
-    else:
-        # If the site module initialization happened after _CONFIG_VARS was
-        # initialized, a virtual environment might have been activated, resulting in
-        # variables like sys.prefix changing their value, so we need to re-init the
-        # config vars (see GH-126789).
-        if _CONFIG_VARS['base'] != os.path.normpath(sys.prefix):
-            with _CONFIG_VARS_LOCK:
-                _CONFIG_VARS_INITIALIZED = False
-                _init_config_vars()
 
     if args:
         vals = []

Lib/test/test_embed.py

@@ -1649,14 +1649,14 @@ def test_init_pyvenv_cfg(self):
             config = {
                 'base_prefix': sysconfig.get_config_var("prefix"),
                 'base_exec_prefix': exec_prefix,
-                'exec_prefix': exec_prefix,
+                'exec_prefix': tmpdir,
+                'prefix': tmpdir,
                 'base_executable': base_executable,
                 'executable': executable,
                 'module_search_paths': paths,
             }
             if MS_WINDOWS:
                 config['base_prefix'] = pyvenv_home
-                config['prefix'] = pyvenv_home
                 config['stdlib_dir'] = os.path.join(pyvenv_home, 'Lib')
                 config['use_frozen_modules'] = bool(not support.Py_DEBUG)
             else:

Lib/test/test_getpath.py

@@ -92,8 +92,8 @@ def test_venv_win32(self):
         ])
         expected = dict(
             executable=r"C:\venv\Scripts\python.exe",
-            prefix=r"C:\Python",
-            exec_prefix=r"C:\Python",
+            prefix=r"C:\venv",
+            exec_prefix=r"C:\venv",
             base_executable=r"C:\Python\python.exe",
             base_prefix=r"C:\Python",
             base_exec_prefix=r"C:\Python",
@@ -339,8 +339,8 @@ def test_venv_posix(self):
         ])
         expected = dict(
             executable="/venv/bin/python",
-            prefix="/usr",
-            exec_prefix="/usr",
+            prefix="/venv",
+            exec_prefix="/venv",
             base_executable="/usr/bin/python",
             base_prefix="/usr",
             base_exec_prefix="/usr",
@@ -371,8 +371,8 @@ def test_venv_changed_name_posix(self):
         ])
         expected = dict(
             executable="/venv/bin/python",
-            prefix="/usr",
-            exec_prefix="/usr",
+            prefix="/venv",
+            exec_prefix="/venv",
             base_executable="/usr/bin/python3",
             base_prefix="/usr",
             base_exec_prefix="/usr",
@@ -404,8 +404,8 @@ def test_venv_non_installed_zip_path_posix(self):
         ])
         expected = dict(
             executable="/venv/bin/python",
-            prefix="/path/to/non-installed",
-            exec_prefix="/path/to/non-installed",
+            prefix="/venv",
+            exec_prefix="/venv",
             base_executable="/path/to/non-installed/bin/python",
             base_prefix="/path/to/non-installed",
             base_exec_prefix="/path/to/non-installed",
@@ -435,8 +435,8 @@ def test_venv_changed_name_copy_posix(self):
         ])
         expected = dict(
             executable="/venv/bin/python",
-            prefix="/usr",
-            exec_prefix="/usr",
+            prefix="/venv",
+            exec_prefix="/venv",
             base_executable="/usr/bin/python9",
             base_prefix="/usr",
             base_exec_prefix="/usr",
@@ -652,8 +652,8 @@ def test_venv_framework_macos(self):
         ])
         expected = dict(
             executable=f"{venv_path}/bin/python",
-            prefix="/Library/Frameworks/Python.framework/Versions/9.8",
-            exec_prefix="/Library/Frameworks/Python.framework/Versions/9.8",
+            prefix=venv_path,
+            exec_prefix=venv_path,
             base_executable="/Library/Frameworks/Python.framework/Versions/9.8/bin/python9.8",
             base_prefix="/Library/Frameworks/Python.framework/Versions/9.8",
             base_exec_prefix="/Library/Frameworks/Python.framework/Versions/9.8",
@@ -697,8 +697,8 @@ def test_venv_alt_framework_macos(self):
         ])
         expected = dict(
             executable=f"{venv_path}/bin/python",
-            prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8",
-            exec_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8",
+            prefix=venv_path,
+            exec_prefix=venv_path,
             base_executable="/Library/Frameworks/DebugPython.framework/Versions/9.8/bin/python9.8",
             base_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8",
             base_exec_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8",
@@ -734,8 +734,8 @@ def test_venv_macos(self):
         ])
         expected = dict(
             executable="/framework/Python9.8/python",
-            prefix="/usr",
-            exec_prefix="/usr",
+            prefix="/framework/Python9.8",
+            exec_prefix="/framework/Python9.8",
             base_executable="/usr/bin/python",
             base_prefix="/usr",
             base_exec_prefix="/usr",