PEP 735: apply community feedback, including reversion of [project] table changes (#3990)

Author: sirosen

peps/pep-0735.rst

@@ -51,6 +51,8 @@ which this PEP seeks to support:
 
 * non-package projects, such as data science projects
 
+Several motivating use cases are defined in detail in the :ref:`Use Cases Appendix <use_cases>`.
+
 Limitations of ``requirements.txt`` files
 -----------------------------------------
 
@@ -95,8 +97,11 @@ package specifiers which are published as part of a package's metadata, and
 which a user can request under that name, as in ``pip install 'foo[bar]'`` to
 install ``foo`` with the ``bar`` extra.
 
-Because ``extras`` are package metadata, they are not usable when a project
-does not build a distribution (i.e., is not a package).
+Because ``extras`` are package metadata, they are not guaranteed to be
+statically defined and may require a build system to resolve.
+Furthermore, definition of a ``[project.optional-dependencies]`` indicates to
+many tools that a project is a package, and may drive tool behaviors such as
+validation of the ``[project]`` table.
 
 For projects which are packages, ``extras`` are a common solution for defining
 development dependencies, but even under these circumstances they have
@@ -163,24 +168,6 @@ defined in greater detail in the :ref:`Use Cases Appendix <use_cases>`.
   be used as a location for locked dependency data)
 * Input data to an environment manager, such as tox, Nox, or Hatch
 * Configurable IDE discovery of test and linter requirements
-* Exposure of package dependencies for install, without the package itself
-
-Support for Inclusion from ``project.dependencies``
----------------------------------------------------
-
-The ``project.dependencies`` and ``project.optional-dependencies`` tables are
-allowed to include Dependency Groups, requiring an update to the specification
-of these tables.
-
-The drivers for such changes are that some usages are well solved by the
-addition of such support, and that failing to include support in the initial
-Dependency Group PEP, but adding such support later in a subsequent PEP,
-would make the support landscape significantly more difficult for tool
-maintainers.
-
-Inclusion of a Dependency Group in ``project.dependencies`` or
-``project.optional-dependencies`` takes the form of a Dependency Group Include,
-defined in the specification section below.
 
 Regarding Poetry and PDM Dependency Groups
 ------------------------------------------
@@ -255,12 +242,7 @@ Strings in requirement lists must be valid
 `Dependency Specifiers <https://packaging.python.org/en/latest/specifications/dependency-specifiers/>`__,
 as defined in :pep:`508`.
 
-Tables in requirement lists must be valid Dependency Object Specifiers,
-defined below.
-
-The ``project`` table in ``pyproject.toml`` is modified such that
-``project.dependencies`` and the values of ``project.optional-dependencies``
-may contain Dependency Object Specifiers.
+Tables in requirement lists must be valid Dependency Object Specifiers.
 
 Dependency Object Specifiers
 ----------------------------
@@ -310,21 +292,6 @@ Includes, in which case those includes should be expanded as well. Dependency
 Group Includes MUST NOT include cycles, and tools SHOULD report an error if
 they detect a cycle.
 
-``project`` Table Changes
--------------------------
-
-The ``[project]`` table, originally defined in :pep:`621` is extended in two ways.
-
-``dependencies``
-
-In addition to :pep:`508` strings, the array may contain Dependency Object
-Specifiers.
-
-``optional-dependencies``
-
-In addition to :pep:`508` strings, the array values in this table may contain
-Dependency Object Specifiers.
-
 Example Dependency Groups Table
 -------------------------------
 
@@ -404,11 +371,10 @@ Tools MAY choose to provide the same interfaces for installing Dependency
 Groups as they do for installing extras.
 
 Note that this specification does not forbid having an extra whose name matches
-a Dependency Group. In such cases, tools must define their own semantics for
-precedence order or disambiguation.
+a Dependency Group.
 
 Users are advised to avoid creating Dependency Groups whose names match extras.
-Tools SHOULD NOT treat such matching as an error.
+Tools MAY treat such matching as an error.
 
 Validation and Compatibility
 ----------------------------
@@ -482,7 +448,7 @@ The output is therefore valid ``requirements.txt`` data.
 
 
     def _resolve_dependency_group(
-        dependency_groups: dict, group: str, past_groups: tuple[str] = ()
+        dependency_groups: dict, group: str, past_groups: tuple[str, ...] = ()
     ) -> list[str]:
         if group in past_groups:
             raise ValueError(f"Cyclic dependency group include: {group} -> {past_groups}")
@@ -542,16 +508,15 @@ At time of writing, the ``dependency-groups`` namespace within a
 reserved for use only by standards specified at packaging.python.org,
 there are no direct backwards compatibility concerns.
 
-However, the introduction of the feature as a potential component of
-``project`` data has implications for a number of ecosystem tools.
+However, the introduction of the feature has implications for a
+number of ecosystem tools, especially those which attempt to support
+examination of data in ``setup.py`` and ``requirements.txt``.
 
 Audit and Update Tools
 ----------------------
 
 A wide range of tools understand Python dependency data as expressed in
-``project.dependencies`` and ``project.optional-dependencies`` and may
-additionally support ``setup.cfg``, ``requirements.txt``, and even
-``setup.py``. (e.g., Dependabot, Tidelift, etc)
+``requirements.txt`` files. (e.g., Dependabot, Tidelift, etc)
 
 Such tools inspect dependency data and, in some cases, offer tool-assisted or
 fully automated updates.
@@ -563,31 +528,6 @@ As a result, users of Dependency Groups would experience a degradation in their
 workflows and tool support at the time that they start using Dependency Groups.
 This is true of any new standard for where and how dependency data are encoded.
 
-Repackaging
------------
-
-Repackaging, and in particular tool-assisted repackaging such as Grayskull or
-PyInstaller, will need to contend with the change to package metadata
-definitions as well.
-
-Because repackagers for alternate ecosystems such as conda and linux distros
-are often distinct persons vs the package publishers, this compatibility
-concern is more difficult to address than cases in which the package
-maintainers experience the impact. Package maintainers may be unaware of the
-impact of beginning to use Dependency Groups, and may unknowingly make changes
-which harm downstream repackaging workflows.
-
-There are two primary ways in which this issue can be addressed:
-
-* via education and "How to Teach This" -- users whose packages are repackaged
-  should be made aware that using new standards may cause issues for downstream
-  package consumers
-
-* by driving more behavior through build-backends -- so long as dependency
-  metadata are gathered via :pep:`517` interfaces, downstream repackagers can
-  remain ignorant of which build system is being used and whether or not it
-  supports Dependency Groups
-
 Security Implications
 =====================
 
@@ -632,22 +572,6 @@ defined dynamically. Requirements loaded from ``requirements.txt`` files and
 definitions of static lists prior to ``setup()`` invocation readily analogize
 with Dependency Groups.
 
-Notes for Packages which are Repackaged
----------------------------------------
-
-A special note should be given to package maintainers whose packages are
-repackaged by linux distros, homebrew, conda, etc.
-
-Use of Dependency Group Includes in core project metadata,
-``project.dependencies`` and ``project.optional-dependencies``, may break these
-consumers' uses of your package. Because they may be consuming and directly
-interact with the source for your repository, their toolchains may not support
-Dependency Groups at the same time that the package maintainers' tools are
-updated.
-
-Ensure that repackaging consumers can contact you if there is an issue, and
-make sure to note transitions to use Dependency Groups in your changelogs.
-
 Interfaces for Use of Dependency Groups
 ---------------------------------------
 
@@ -779,18 +703,92 @@ has impacts on high-level tools like IDEs and Dependabot, which cannot support
 deep integration with these Dependency Groups. (For example, at time of writing
 Dependabot will not flag dependencies which are pinned in ``tox.ini`` files.)
 
+Deferred Ideas
+==============
+
+Why not support Dependency Group Includes in ``[project.dependencies]`` or ``[project.optional-dependencies]``?
+---------------------------------------------------------------------------------------------------------------
+
+Earlier drafts of this specification allowed Dependency Group Includes to be
+used in the ``[project]`` table.
+However, there were several issues raised during community feedback which led
+to its removal.
+
+Only a small number of additional use cases would be addressed by the inclusion
+of Dependency Groups, and it increased the scope of the specification
+significantly. In particular, this inclusion would increase the number of parties
+impacted by the addition. Many readers of the ``[project]`` table, including build
+backends, SBOM generators, and dependency analyzers are implicated by a change to
+``[project]`` but may continue to operate as-is in the presence of a new (but
+unconnected) ``[dependency-groups]`` table.
+
+Separately from the above concern, allowing inclusion of dependency groups from the
+``[project]`` table encourages package maintainers to move dependency metadata out
+of the current standard location.
+This complicates static ``pyproject.toml`` metadata and conflicts with the goal of
+:pep:`621` to store dependency metadata in a single location.
+
+Finally, exclusion of ``[project]`` support from this PEP is not final. The
+use of includes from that table, or an inclusion syntax from
+``[dependency-groups]`` into ``[project]``, could be introduced by a future
+PEP and considered on its own merits.
+
+Use Cases for Dependency Group Includes From ``[project]``
+''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+Although deferred in this PEP, allowing includes from the ``[project]``
+table would address several use cases.
+
+In particular, there are cases in which package developers would like to
+install only the dependencies of a package, without the package itself.
+
+For example:
+
+* Specify different environment variables or options when building dependencies
+  vs when building the package itself
+
+* Creating layered container images in which the dependencies are isolated from
+  the package being installed
+
+* Providing the dependencies to analysis environments (e.g., type checking)
+  without having to build and install the package itself
+
+For an example of the last case, consider the following sample
+``pyproject.toml``:
+
+.. code-block:: toml
+
+    [project]
+    dependencies = [{include = "runtime"}]
+    [optional-dependencies]
+    foo = [{include = "foo"}]
+    [dependency-groups]
+    runtime = ["a", "b"]
+    foo = ["c", "d"]
+    typing = ["mypy", {include = "runtime"}, {include = "foo"}]
+
+In this case, a ``typing`` group can be defined, with all of the package's
+runtime dependencies, but without the package itself. This allows uses of the
+``typing`` Dependency Group to skip installation of the package -- not only is
+this more efficient, but it may reduce the requirements for testing systems.
+
 Why not support Dependency Group Includes in ``[build-system.requires]``?
 -------------------------------------------------------------------------
 
-Although it may be interesting to allow this in the future, incorporating the
-proposal into the build-system table reduces the ability of users to rely on
-this to bootstrap support. :pep:`517` frontends would be required to support
-Dependency Groups in order to achieve this, and the versions of these frontends
-are not easily controlled by packages.
+Given that we will not allow for ``[project]`` usage of Dependency Groups,
+``[build-system.requires]`` can be considered in comparison with
+``[project.dependencies]``.
+
+There are fewer theoretical usages for build requirements specified in a group
+than package requirements. Additionally, the impact of such a change implicates
+:pep:`517` frontend, which would need to support Dependency Groups in order to
+prepare a build environment.
 
-By only defining build backend support, it is possible for packages to start
-leveraging the new syntax and capabilities without being concerned about
-controlling the environment in which the package is built and installed.
+Compared with changes to ``[project.dependencies]`` and
+``[project.optional-dependencies]``, changing the behaviors of
+``[build-system.requires]`` is higher impact and has fewer potential uses.
+Therefore, given that this PEP declines to make changes to the ``[project]``
+table, changing ``[build-system]`` is also deferred.
 
 .. _prior_art:
 
@@ -1411,42 +1409,6 @@ for various purposes.
 This declaration allows the project author's knowledge of the appropriate tools
 for the project to be shared with all editors of that project.
 
-Exposure of package dependencies without the package itself
------------------------------------------------------------
-
-There are a variety of use-cases in which package developers would like to
-install only the dependencies of a package, without the package itself.
-
-For example:
-
-* Specify different environment variables or options when building dependencies
-  vs when building the package itself
-
-* Creating layered container images in which the dependencies are isolated from
-  the package being installed
-
-* Providing the dependencies to analysis environments (e.g., type checking)
-  without having to build and install the package itself
-
-For an example of the last case, consider the following sample
-``pyproject.toml``:
-
-.. code-block:: toml
-
-    [project]
-    dependencies = [{include = "runtime"}]
-    [optional-dependencies]
-    foo = [{include = "foo"}]
-    [dependency-groups]
-    runtime = ["a", "b"]
-    foo = ["c", "d"]
-    typing = ["mypy", {include = "runtime"}, {include = "foo"}]
-
-In this case, a ``typing`` group can be defined, with all of the package's
-runtime dependencies, but without the package itself. This allows uses of the
-``typing`` Dependency Group to skip installation of the package -- not only is
-this more efficient, but it may reduce the requirements for testing systems.
-
 Copyright
 =========