PEP 778: Supporting Symlinks in Wheels (#3786)

* Add PEP 778 and reserve 777

* Add Paul Moore as PEP delegate

Co-authored-by: Hugo van Kemenade <[email protected]>

* Add Paul Moore as PEP delegate

Co-authored-by: Hugo van Kemenade <[email protected]>

* Update peps/pep-0778.rst

Co-authored-by: Hugo van Kemenade <[email protected]>

* Remove extra underline

Co-authored-by: Hugo van Kemenade <[email protected]>

* Fix misspelling

Co-authored-by: Hugo van Kemenade <[email protected]>

* Remove PEP 777

The PEP has been split out into #4036

* Add codeowners

* Remove third p from suppport

Co-authored-by: Hugo van Kemenade <[email protected]>

* Mark PEP 778 deferred

* Apply suggestions from Barry to PEP text

Co-authored-by: Barry Warsaw <[email protected]>

* Fix lint error

---------

Co-authored-by: Hugo van Kemenade <[email protected]>
Co-authored-by: Barry Warsaw <[email protected]>

Author: 3 people

.github/CODEOWNERS

@@ -655,6 +655,7 @@ peps/pep-0774.rst  @savannahostrowski
 peps/pep-0775.rst  @encukou
 peps/pep-0776.rst  @hoodmane @ambv
 peps/pep-0777.rst  @warsaw
+peps/pep-0778.rst  @warsaw
 # ...
 peps/pep-0779.rst  @Yhg1s @colesbury @mpage
 peps/pep-0780.rst  @lysnikolaou

peps/pep-0778.rst

@@ -0,0 +1,277 @@
+PEP: 778
+Title: Supporting Symlinks in Wheels
+Author: Emma Harper Smith <[email protected]>
+Sponsor: Barry Warsaw <[email protected]>
+PEP-Delegate: Paul Moore <[email protected]>
+Discussions-To: https://discuss.python.org/t/pep-778-supporting-symlinks-in-wheels/53824
+Status: Deferred
+Type: Standards Track
+Topic: Packaging
+Requires: 777
+Created: 18-May-2024
+Post-History: `10-Oct-2024 <https://discuss.python.org/t/pep-778-supporting-symlinks-in-wheels/53824>`__
+
+Abstract
+========
+
+Wheels currently do not handle symlinks well, copying content instead of making symlinks when
+installed. To properly handle distributing libraries in wheels, we propose a new ``LINKS``
+metadata file to handle symlinks in a platform portable manner. This specification requires
+a new wheel major version, discussed in :pep:`777`.
+
+PEP Deferral
+============
+
+This PEP has been deferred until a better compatibility story for major changes to the wheel
+format is established. Once a compatibility story is established for wheels which allows backwards
+incompatible behavior in an unobtrusive way, the following points should be addressed in this PEP:
+
+- Re-focus this topic to just symlinks for shared libraries on POSIX platforms, perhaps tied to
+  platform tags?
+- Should the symlinks be materialized as file attributes in the archive or a ``LINKS`` file?
+  Could it be encoded in ``RECORD``?
+- Clarify that this PEP is insufficient to be useful for :pep:`660` editable installs since it will no
+  longer be cross platform.
+- Describe fallback behavior in instances where symlinks are unavailable on POSIX platforms.
+
+Motivation
+==========
+
+Today, symlinks in wheels get created as copies of files, as `the zipfile module
+<https://docs.python.org/3/library/zipfile.html>`_ in CPython `does not support handling symlinks
+in-place <https://github.com/python/cpython/issues/82102>`_ for security reasons.
+
+This `presents problems to projects that would like to ship large compiled libraries
+<https://pypackaging-native.github.io/other_issues/#lack-of-support-for-symlinks-in-wheels>`_ in
+wheels, as they must choose to either greatly increase the install size of the project on disk,
+or omit the symlink and potentially break some downstream use cases.
+
+To ship a library that can properly be loaded for runtime use or build time linking on POSIX, a
+library should follow the conventions of POSIX-style loader and linker search. The two main file names for
+the loader to use is the "soname" and the "real name". The "soname" is a file like
+``libfoo.so.3`` where ``3`` is a number that is incremented when the interface of the library
+changes. The "real name" is a file named like ``libfoo.so.3.1.4``, where the extra version
+information lets the loader find a specific version of a library. Finally, when compiling code to
+link against a library, the linker searches for a "linker name", named like ``libfoo.so``. A more
+detailed description is available in `this Linux documentation on shared libraries
+<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`_. To fully support all
+runtime and build time use cases, a project requires shipping all 3 files. Normally, this is
+handled on POSIX platforms by using symlinks, so that the library is not duplicated on disk 3 times.
+
+Returning to Python packaging, there are many popular projects which ship binary libraries, such as
+``numpy``, ``scipy``, and ``pyarrow``. Other site-packages ``dlopen`` libraries in other wheels, such as
+``pytorch`` and ``jax``. These projects currently rely on a single library in the wheel, but
+this can cause the linker to find the wrong library if there are system libraries that have a
+"real name" library version available.
+
+There is also the potential benefit that symlinks in wheels would allow for simpler editable
+installs by simply placing a symlink in the user's ``site-packages`` directory, but this PEP
+leaves that as an open question to be explored in a future PEP.
+
+Rationale
+=========
+
+To support the 3 main namings of a library used in loading and library linking on POSIX, we
+propose adding support for symlinks in Python wheels. To allow for tracking symlinks made, and to
+potentially support other platforms that may not support POSIX symlinks directly, we propose the
+use of a new wheel metadata file ``LINKS``, which will exist in the ``.dist-info`` directory alongside
+``METADATA``, ``RECORD``, and other metadata files.
+
+Using a ``LINKS`` file will allow for more cross-platform uses of symlink-like usage. On Windows,
+symlinks require either `a group policy allowing the user to make symlinks
+<https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/security-policy-settings/create-symbolic-links>`_
+(e.g. by enabling `Dev Mode
+<https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development>`_)
+or Administrative permissions. This means that it may be the case that symlinks are unsupported on
+some user systems. By using a ``LINKS`` file, installers will be able to potentially use other
+methods for handling symlinks, such as junctions on Windows, where otherwise the installer would
+have to fail.
+
+This PEP also describes checks that installers must make when installing an updated wheel. These
+checks exist to handle security risks from allowing wheels to install symlinks. For more
+information on why these checks are important, see `Security Implications`_.
+
+Specification
+=============
+
+Wheel Major Version Bump
+------------------------
+
+This PEP requires a wheel major version bump, so the ``Wheel-Version`` for wheels generated with
+``LINKS`` **MUST** be at least version ``2.0``, so that older installers do not silently fail to
+install symlinks and break user environments. For more see :pep:`777`.
+
+New ``LINKS`` Metadata File
+---------------------------
+
+To enable cross-platform symlinks, this PEP introduces a new wheel metadata file, ``LINKS``. An
+example of a ``LINKS`` file is below::
+
+    my_package/libfoo.so.3.1.4,my_package/libfoo.so.3
+    my_package/libfoo.so.3,my_package/libfoo.so
+
+The format of ``LINKS``, as can seen above, is ``source_path,target_path`` where ``source_path``
+is a path relative to the root of any namespace or package root in the wheel. ``target_path`` is a
+*non-dangling* path (i.e. a path that exists on the filesystem after the wheel's contents are
+extracted) in a package or namespace of any package in the wheel. This means that if a wheel
+contains multiple packages, all paths in packages in the wheel are acceptable.
+
+Installer Behavior Specification
+--------------------------------
+
+Installers **MUST** resolve the paths of any link contained in the ``LINKS`` file *before*
+deciding if any ``source_path`` or ``target_path`` are valid. Installers **MUST** verify that
+``source_path`` and ``target_path`` are located inside any namespace or package coming from the
+wheel.  Installers **MUST** reject cyclic symlinks in wheels. Installers **MAY** error if a long
+chain of symlinks (symlinks pointing to symlinks many times repeated) exceeds a limit set by the
+installer.
+
+Installers **MUST** follow the following steps when handling a wheel with symlinks:
+
+1. Check for the existence of a ``LINKS`` file in the ``.dist-info``. If it does not exist,
+   no further steps are required.
+2. Extract all files in the wheel packages and data directory as in wheel 1.x.
+3. Verify that for each ``source_path`` and ``target_path`` pairs, the ``target_path`` exists in
+   one of the package namespaces just extracted.
+4. Next, check that the installer can make some kind of link for each pair in the site directory.
+   If the installer cannot make a link for the file/folder ``target_path`` for the current
+   platform, an error **MUST** be raised. An example of a failure mode would be a POSIX symlink to
+   a file target, where the installer is running on Windows and the installer cannot make
+   symlinks but can make junctions. In this case the installer **MUST** error because it cannot
+   handle the link.
+5. Finally, the installer **MUST** add a platform-relevant link between ``source_path`` and
+   ``target_path``.
+
+Installers **MUST NOT** by default copy files instead of generating a symlink when handling
+symlinks. Installers **MAY** have such behavior available under an alternate configuration or
+command line flag.
+
+Build Backend Specification
+---------------------------
+
+When creating a wheel, build backends **MUST** treat symlinks in the same way as its target when
+deciding whether to include the symlink in a wheel. Build backends **MUST** verify that there are
+no dangling symlinks in the ``LINKS`` file. Build backends **SHOULD** recognize platform-relevant
+symlinks that would be included in builds. On POSIX systems this is typically symlinks, on Windows this
+includes symlinks and junctions.
+
+Backwards Compatibility
+=======================
+
+Introducing symlinks would require an increment to the wheel format major version. This would mean
+new wheels that use the new wheel format would raise an error on older installer tools, per the
+`wheel specification
+<https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-contents>`_.
+
+Please see :pep:`777` on "Wheel 2.0".
+
+Security Implications
+=====================
+
+Symlinks can be quite dangerous if not handled carefully. A simple example would be if a user were
+to run ``sudo pip install malicious``, and there were no protections, then the malicious package
+could overwrite ``/etc/shadow`` and replace the password hash on the system, allowing malicious
+logins.
+
+This PEP lists several requirements on checks to run by installers on symlinks in wheels to ensure
+attacks like the one described above cannot happen. This means it is **critical** that installers
+carefully implement these security safeguards and prevent malicious use on package installation.
+
+In particular, the following checks **MUST** be made by installers:
+
+1. That the symlinks do not point outside of any packages or namespaces coming from the wheel
+2. That the symlinks are not dangling (the target exists at install time)
+3. That the symlinks are not cyclical, stopping after a certain depth of checking to avoid denial
+   of service requests
+
+Do not follow symlinks on removal.
+
+How to Teach This
+=================
+
+End users should, once the changes have propagated through the ecosystem, transparently experience
+the benefits of symlinks in wheels. It is important for installers to give clear error messages if
+symlinks are unsupported on the platform, and explain why installation has failed.
+
+For people building libraries, documentation on ``packaging.python.org`` should describe the use
+cases and caveats (especially platform support) of symlinks in wheels. Otherwise it should be
+handled transparently by build backends in the same way any normal file would be handled.
+
+Reference Implementation
+========================
+
+TODO
+
+Rejected Ideas
+==============
+
+Just Use POSIX Symlinks Everywhere
+----------------------------------
+
+This PEP wants to allow for ``LINKS`` to be used for a potential future :pep:`660` editable
+installation. This future PEP should support Windows, so it may need to use junctions.
+
+Don't Use Junctions in ``LINKS``
+--------------------------------
+
+Junctions are a limited way to support symlinks between folders on Windows. They do not support
+files. This PEP allows for junctions as users may wish to only link folders to a different
+location, and future :pep:`660` implementations may need to rely on this feature.
+
+Put symlinks in the ``RECORD`` Metadata File
+--------------------------------------------
+
+While this could be done, it would clutter the ``RECORD`` file. Furthermore the most
+straightforward implementation would place the target at the end of the record. This would
+make it harder to scan across the line and visually see what symlinks exist in the wheel.
+
+Library Maintainers Should Use Python to Locate Libraries
+---------------------------------------------------------
+
+Using Python to locate libraries would be much easier. However, some libraries like ``libtorch``
+are used by extension modules and themselves require loading dependencies. Some compiled libraries
+cannot use Python to find their loader dependencies.
+
+Include Support for Hardlinks
+-----------------------------
+
+This PEP does not specify any behavior around hardlinks. This is intentional. This is left as an
+extension to a future PEP.
+
+Open Issues
+===========
+
+PEP 660 and Deferring Editable Installation Support
+---------------------------------------------------
+
+This PEP leaves the specification and implementation of a :pep:`660` editable installation
+mechanism as unresolved for a later PEP; should that be specified in this PEP?
+
+Security
+--------
+
+This PEP needs to be reviewed to make sure it would not allow for new security vulnerabilities.
+Are there other restrictions we should place on the source or target of symlinks to protect users?
+
+Allow inter-package symlinks
+----------------------------
+
+This could be useful for projects that want to shard dependencies such as large libraries between
+wheels but make them available in the main parent wheel.
+
+The Format of ``LINKS``
+-----------------------
+
+Currently the format is derived from ``RECORD``, but perhaps a better format exists.
+
+Previous Discussion
+===================
+
+https://discuss.python.org/t/symbolic-links-in-wheels/1945/25
+
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.