diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index c4cd5005378..e0c497f14a3 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -683,7 +683,7 @@ peps/pep-0804.rst @pradyunsg # ... peps/pep-0806.rst @JelleZijlstra peps/pep-0807.rst @dstufft -# ... +peps/pep-0808.rst @FFY00 peps/pep-0809.rst @zooba peps/pep-0810.rst @pablogsal @DinoV @Yhg1s # ... diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst new file mode 100644 index 00000000000..cddf3a0d2c6 --- /dev/null +++ b/peps/pep-0808.rst @@ -0,0 +1,390 @@ +PEP: 808 +Title: Partially dynamic project metadata +Author: Henry Schreiner , + Cristian Le +Sponsor: Filipe Laíns +PEP-Delegate: Paul Moore +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 19-Sep-2025 + + + +Abstract +======== + +This PEP relaxes the constraint on dynamic metadata listed in the ``[project]`` +section in ``pyproject.toml`` to allow the static portion of mixed metadata to +be defined in the normal location if the field is a table or array by having +the dynamic fields extend the static ones. + +This allows users to opt into allowing a backend to extend metadata while still +keeping the static portions of the metadata defined in the standard location in +``pyproject.toml``, and allows inspection tools to still be able to process the +static portions of the metadata. + + +Motivation +========== + +In the core metadata specification originally set out in :pep:`621`, metadata +can be specified in three ways. First, it can be listed in the ``[project]`` +table. This makes it statically inferable, meaning any tool (not just the +build backend) can reliably compute the value. Second, a field can be listed in +the ``project.dynamic`` list, which allows the build backend to compute the +value. Finally, a value could be missing from both the ``project`` table and +the ``project.dynamic`` list, in which case the matching metadata is guaranteed +to be empty. + +This system provided two important benefits to Python packaging. A standard +specification that all major backends have now adopted makes teaching much +easier; a single tutorial is now sufficient to cover the metadata portion of +configuring any backend. Users can now switch from a general purpose backend to +a specialized backend without changing their static metadata. Tooling like +schema validation tools can verify and catch configuration mistakes. + +The second benefit is improved support for static tools that read the source +files looking for metadata. This is useful for dependency chain analysis, such +as creating "used by" and "uses" graphs. It is used for code quality tooling to +detect the minimum supported version of Python. It is used by cibuildwheel_ to +automatically avoid building wheels that are not supported. It is not used, +however, to avoid wheel builds when the SDist is available; that was addressed +by METADATA 2.2, which a ``Dynamic`` field in the SDist metadata that lets a +tool know if the metadata can change when making a wheel - this is an easy +mistake to make due to the similarity of the names. + +Due to the rapidly increasing popularity of the project table, support from all +major backends, and a rise of backends supporting complex compiled extensions, +an issue with the restrictions applied in :pep:`621` is becoming more apparent. +In PEP 621, the metadata choice is all-or-nothing; metadata must be completely +static, or listed in the dynamic field and completely absent from the static +definition. For the most common use cases, this is fine; there is little +benefit to set the ``version`` statically if you are going to override it +dynamically. If you are using a custom README processor to filter or modify the +README for proper display, it's not a big deal to have to specify the +configuration in a custom ``tool.*`` section. But there is a specific class of +cases where the all-or-nothing approach is problematic: lists of items where +the backend needs to add items are currently forced to be fully dynamically +specified (that is, in a backend-specific configuration location). This causes +both of the original benefits (standard location and static tooling support) to +be lost. + +Rationale +========= + + +:pep:`621` includes the following statement: + + In an earlier version of this PEP, tools were allowed to extend data for + fields. For instance, build back-ends could take the version number and add + a local version for when they built the wheel. Tools could also add more + trove classifiers for things like the license or supported Python versions. + + In the end, though, it was thought better to start out stricter and + contemplate loosening how static the data could be considered based on + real-world usage. + +In this PEP, we are proposing a limited and explicit loosening of the +restrictions on the ``[project]`` table and ``project.dynamic`` list. + +Every list and every table with arbitrary keys will now be allowed to be +specified both statically, in the ``[project]`` table, and in the +``project.dynamic`` list. If it is present in both places, the build backend +can extend list items and add new keys, but not modify existing list items or +strings. + + +Use Cases +--------- + +There is an entire class of metadata fields where advanced use cases +would really benefit from a relaxation of this rule. Here are some use +cases that have come up: + +- Pinning dependency requirements when building the wheel. When building + PyTorch_ extensions, for example, the version you build with adds a constraint + to the wheel you create that is not present with the SDist. +- Generating extra scripts from a build system (this is a currently proposed in + scikit-build-core_). +- Adding entry points dynamically (validate-pyproject-schema-store_ could have + used this to generate an entry point for each schema present in the package.) +- Adding dependencies or optional dependencies based on configuration (such as + making an all dependency, or reading dependencies from dependency-groups, for + example). Adding constraints can also be useful; pybind11_ uses adds a ``global`` + extra that pins ``pybind11-global==``, as both packages are in the + same repository and released in sync. toga_ is a collection of packages that + currently is unable to set any static dependencies due to the same sort of + pinning problem. +- Adding classifiers; some backends can compute classifiers from other places + and inject them (Poetry_ being the best known example). +- Adding license files to the wheel based on what libraries are linked in (this + is an active discussion in followup to :pep:`639`). +- Adding SBom's when building - :pep:`770` had to remove the ``pyproject.toml`` + field specifically because you _want_ the build tool to add these, so the + ``[project]`` table setting would be useless, you'd almost never be able to + use it. + +All of these use cases have a similar feature: they are adding something +dynamically to a fixed list (possibly a narrower pin for the dependency case). + +You can implement these today, but it requires providing a completely separate +configuration location for the non-extended portion, and static analysis tools +lose the ability to detect anything. Since the current solution is to move all +the metadata out of the standard field, this proposal will increase the +availability of metadata for static tooling. + + +Example: pinning +---------------- + +For example, let's say you want to allow an imaginary build backend +(``my-build-backend``) to pin to the supported build of PyTorch_. Before this +PEP, you could do this: + +.. code-block:: toml + + [project] + dynamic = ["dependencies"] + + [tool.my-build-backend] + original-dependencies = ["torch", "packaging"] + pin-to-build-versions = ["torch=={exact}"] + +Which would effectively expand to the following SDist metadata: + +.. code-block:: text + + Dynamic: Requires-Dist + Requires-Dist: packaging + Requires-Dist: torch + +Which would then could make a wheel with this: + +.. code-block:: text + + Requires-Dist: packaging + Requires-Dist: torch + Requires-Dist: torch==2.8.0 + +Static tooling no longer can tell that ``torch`` and ``packaging`` are runtime +dependencies, and the build backend had to duplicate the dependency table, +making it harder for users to learn and read; the standardized place proposed +by :pep:`621` and adopted by all major build backends is lost. + +With this PEP, this could now be specified like this: + +.. code-block:: toml + + [project] + dependencies = ["torch", "packaging"] + dynamic = ["dependencies"] + + [tool.my-build-backend] + pin-to-build-versions = ["torch=={exact}"] + +Static tooling can now detect the static dependencies, and the build backend no +longer needs to create and document a new location for the standard +``project.dependencies`` field (the ``original-dependencies`` field above, for +example). + + + +Future Updates +-------------- + +Loosening this rule to allow purely additive metadata should address many of +the use cases that have been seen in practice. If further changes are needed, +this can be revisited in a future PEP; this PEP neither recommends or precludes +future updates like this. + +Terminology +=========== + +The keywords "MUST", "MUST NOT", "REQUIRED", +"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" +in this document are to be interpreted as described in :rfc:`2119`. + +Specification +============= + +Any field that is comprised of a list or a table with arbitrary entries will +now be allowed to be present in both the ``[project]`` table and the +``project.dynamic`` list. If a field is present in both places, then the build +backend is allowed to insert entries into the list or table, but not remove +entries, reorder entries, or modify the entries. Tables of arrays allow adding +a new table entry or extending an existing array according to the rules above. + +The fields that are arrays or tables with arbitrary entries are: + +* ``authors``, ``maintainers``: New author tables can be added to the list. + Existing authors cannot be modified (list of tables with pre-defined keys). +* ``classifiers``: Classifiers can be added to the list. +* ``dependencies``: New dependencies can be added, including more tightly + constrained existing dependencies. +* ``entry-points``: Entry points can be added, to either new or existing + groups. Existing entry points cannot be changed or removed. +* ``scripts``, ``gui-scripts``: New scripts can be added. Existing ones cannot + be changed or removed. +* ``keywords``: Keywords can be added to the list. +* ``license-files``: Files can be added to the list. +* ``optional-dependencies``: A new extra or new items can be added to a + existing extra. +* ``urls``: New urls can be added. Existing ones cannot be changed or removed. + +To add items, users must opt-in by listing the field in ``dynamic``; without +that, the metadata continues to be entirely static. + +A backend SHOULD error if a field is specified and it does not support +extending that field, to protect against possible user error. We recommend +being as strict as possible to avoid unnecessary entries in the ``dynamic`` +list. + +Static analysis tools, when detecting a field is both specified and in the +``project.dynamic`` array, SHOULD assume the field is incomplete, allowing for +new entries to be present when the package is built. + +The ``Dynamic`` field, as specified in :pep:`643`, is unaffected by this PEP, +and backends can continue to fill it as they chose. However, a backend MUST +ensure that both the SDist and the wheel metadata include the static metadata +portion of the project table. + +Reference Implementation +======================== + +The choice to support dynamic metadata for each field is already left up to +backends, and this PEP simply relaxes restrictions on what a backend is allowed +to do with dynamic metadata. + +The pyproject-metadata_ project, which is used by +several build backends, will need to modify the correctness check to account +for the possible extensions; this is in `a draft PR `__. + +The dynamic-metadata_ project, which provides a plugin +system that backends can use to share dynamic metadata plugins, was designed to +allow this possibility, and a similar PR to the one above will allow additive +metadata. + +Backwards Compatibility +======================= + +This does not affect any existing ``pyproject.toml``'s, since this was strictly +not allowed before this PEP. + +When users adopt this in a ``pyproject.toml``, the backend must support it; an +error will be correctly generated if it doesn't following the previous +standard. Frontends were never required to throw an error, though some +frontends may need to be updated to benefit from the partially static metadata. +Some frontends and other tooling may need updating, such as schema +validation, just like other ``pyproject.toml`` PEPs. + +Using metadata from SDists or wheels is unaffected. The METADATA version does +not need to be incremented. + +Security Implications +===================== + +There are no security concerns that are not already present, as this just adds +a static component to existing dynamic metadata support. + +How to Teach This +================= + +The current guides that state metadata must not be listed in both ``[project]`` +and ``project.dynamic`` can be updated to say that some fields can be extended +by ``project.dynamic``. Since dynamic metadata is already an advanced concept, +this will likely not affect most existing tutorial material aimed at +introductory packaging. + +The ``pyproject.toml`` `specification `__ will be updated to +include the behavior of fields when specified and also listed in the dynamic +field. + +It should also be noted that specifying something in dynamic will require any +tool that requires the full metadata to invoke the backend even if it is +partially statically specified, so it should not be used unless necessary. + + +Rejected Ideas +============== + +Special case some fields without adding dynamic +----------------------------------------------- + +This has come up specifically for the pinning build dependency use case, but +could also be applied to more of the use cases listed. This would not cover all +the use cases seen, though, and an explicit, opt-in approach is better for +static tooling. + + +Include string fields +--------------------- + +Some string fields could also be extended. Most notably, the ``license`` field +would benefit from being extendable, and due to the semantics of SPDX +expressions, extension could be defined through ``AND``. This was not added to +this PEP because that would require individual fields to have custom semantics. + +The other string fields, namely ``version`` and ``requires-python`` (``name`` +is not allowed to be specified dynamically), have less reason to be extended. +Fixed key tables, like the deprecated ``license.text``/``license.file`` or +``readme.text``/``readme.file`` also have no clear benefit being partially +dynamic. + + +Fully remove restrictions on backends +------------------------------------- + +Another option would be to simply allow backends to do whatever they wanted if +a field is statically defined and in the dynamic array. This would sacrifice +the ability for static tooling to infer anything about the field, and could +potentially confuse users by allowing the backend to ignore or change what they +entered. This is not worse than the status quo for static tooling and dynamic +metadata, but the current proposal improves the ability of static tooling to +infer some things about dynamic fields. Knowing some of the dependencies is +better for most applications than not knowing anything about the dependencies, +for example. + +Allow simplifications +--------------------- + +An earlier draft of this PEP had a clause allowing backends to simplify some +types of fields; most notably dependency specifiers would have allowed +"tightening", such as ``torch`` being replaced by ``torch>=1.2``, for example. +. This was removed due to it being impossible to ensure a variation will +resolve identically on all resolvers within the current specification, and to +simplify the contract with backends. Any other simplifications would be purely +cosmetic, and so were left out. The order in the current PEP is now required to +match the original static metadata, with the dynamic portion only allowing +insertions. + + +Add a general mechanism to specify dynamic-metadata +--------------------------------------------------- + +This PEP does not cover methods to specify dynamic metadata; that continues to +be entirely up to the backend. An earlier draft proposal did this, but it was +deemed better to develop that as a library (dynamic-metadata_, for the curious) +instead. This may be revisited in the future. + +References +========== + +.. _cibuildwheel: https://cibuildwheel.pypa.io +.. _pyprojectspec: https://packaging.python.org/en/latest/specifications/pyproject-toml +.. _pyproject-metadata: https://github.com/pypa/pyproject-metadata +.. _pyprojectmetadatapr: https://github.com/pypa/pyproject-metadata/pull/241 +.. _dynamic-metadata: https://github.com/scikit-build/dynamic-metadata +.. _PyTorch: https://pytorch.org/ +.. _scikit-build-core: https://github.com/scikit-build/scikit-build-core +.. _validate-pyproject-schema-store: https://pypi.org/project/validate-pyproject-schema-store/ +.. _pybind11: https://github.com/pybind/pybind11 +.. _Poetry: https://python-poetry.org/ +.. _setuptools: https://github.com/pypa/setuptools +.. _toga: https://github.com/beeware/toga + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.