From 99af225ab240808564fb9c9967aba346d4742199 Mon Sep 17 00:00:00 2001 From: Stephen Rosen Date: Mon, 16 Dec 2024 12:55:56 -0600 Subject: [PATCH 1/2] Add doc for dependency-groups, import from PEP This content is copied from the PEP, with the following characteristics: - the initial overview and examples are new - the "History" footer is new - all of the other sections are copied from the PEP verbatim --- source/specifications/dependency-groups.rst | 272 ++++++++++++++++++ .../section-distribution-metadata.rst | 1 + 2 files changed, 273 insertions(+) create mode 100644 source/specifications/dependency-groups.rst diff --git a/source/specifications/dependency-groups.rst b/source/specifications/dependency-groups.rst new file mode 100644 index 000000000..8206a7063 --- /dev/null +++ b/source/specifications/dependency-groups.rst @@ -0,0 +1,272 @@ +.. _dependency-groups: + +================= +Dependency Groups +================= + +This specification defines Dependency Groups, a mechanism for storing package +requirements in ``pyproject.toml`` files such that they are not included in +project metadata when it is built. + +Dependency Groups are suitable for internal development use-cases like linting +and testing, as well as for projects which are not built for distribution, like +collections of related scripts. + +Fundamentally, Dependency Groups should be thought of as being a standardized +subset of the capabilities of ``requirements.txt`` files (which are +``pip``-specific). + +Specification +============= + +Examples +-------- + +This is a simple table which shows a ``test`` group:: + + [dependency-groups] + test = ["pytest>7", "coverage"] + +and a similar table which defines ``test`` and ``coverage`` groups:: + + [dependency-groups] + coverage = ["coverage[toml]"] + test = ["pytest>7", {include-group = "coverage"}] + +The ``[dependency-groups]`` Table +--------------------------------- + +This PEP defines a new section (table) in ``pyproject.toml`` files named +``dependency-groups``. The ``dependency-groups`` table contains an arbitrary +number of user-defined keys, each of which has, as its value, a list of +requirements (defined below). These keys must be +`valid non-normalized names `__, +and must be +`normalized `__ +before comparisons. + +Tools SHOULD prefer to present the original, non-normalized name to users by +default. If duplicate names, after normalization, are encountered, tools SHOULD +emit an error. + +Requirement lists under ``dependency-groups`` may contain strings, tables +("dicts" in Python), or a mix of strings and tables. + +Strings in requirement lists must be valid +`Dependency Specifiers `__, +as defined in :pep:`508`. + +Tables in requirement lists must be valid Dependency Object Specifiers. + +Dependency Object Specifiers +---------------------------- + +Dependency Object Specifiers are tables which define zero or more dependencies. + +This PEP standardizes only one type of Dependency Object Specifier, a +"Dependency Group Include". Other types may be added in future standards. + +Dependency Group Include +'''''''''''''''''''''''' + +A Dependency Group Include includes the dependencies of another Dependency +Group in the current Dependency Group. + +An include is defined as a table with exactly one key, ``"include-group"``, +whose value is a string, the name of another Dependency Group. + +For example, ``{include-group = "test"}`` is an include which expands to the +contents of the ``test`` Dependency Group. + +Includes are defined to be exactly equivalent to the contents of the named +Dependency Group, inserted into the current group at the location of the include. +For example, if ``foo = ["a", "b"]`` is one group, and +``bar = ["c", {include-group = "foo"}, "d"]`` is another, then ``bar`` should +evaluate to ``["c", "a", "b", "d"]`` when Dependency Group Includes are expanded. + +Dependency Group Includes may specify the same package multiple times. Tools +SHOULD NOT deduplicate or otherwise alter the list contents produced by the +include. For example, given the following table: + +.. code:: toml + + [dependency-groups] + group-a = ["foo"] + group-b = ["foo>1.0"] + group-c = ["foo<1.0"] + all = ["foo", {include-group = "group-a"}, {include-group = "group-b"}, {include-group = "group-c"}] + +The resolved value of ``all`` SHOULD be ``["foo", "foo", "foo>1.0", "foo<1.0"]``. +Tools should handle such a list exactly as they would handle any other case in +which they are asked to process the same requirement multiple times with +different version constraints. + +Dependency Group Includes may include lists containing Dependency Group +Includes, in which case those includes should be expanded as well. Dependency +Group Includes MUST NOT include cycles, and tools SHOULD report an error if +they detect a cycle. + +Package Building +---------------- + +Build backends MUST NOT include Dependency Group data in built distributions as +package metadata. This means that PKG-INFO in sdists and METADATA in wheels +do not include any referencable fields containing Dependency Groups. + +It is valid to use Dependency Groups in the evaluation of dynamic metadata, and +``pyproject.toml`` files included in sdists will naturally still contain the +``[dependency-groups]`` table. However, the table contents are not part of a +published package's interfaces. + +Installing Dependency Groups +---------------------------- + +Tools which support Dependency Groups are expected to provide new options and +interfaces to allow users to install from Dependency Groups. + +No syntax is defined for expressing the Dependency Group of a package, for two +reasons: + +* it would not be valid to refer to the Dependency Groups of a third-party + package from PyPI (because the data is defined to be unpublished) + +* there is not guaranteed to be a current package for Dependency Groups -- part + of their purpose is to support non-package projects + +For example, a possible pip interface for installing Dependency Groups +would be: + +.. code:: shell + + pip install --dependency-groups=test,typing + +Note that this is only an example. This PEP does not declare any requirements +for how tools support the installation of Dependency Groups. + +Overlapping Install UX with Extras +'''''''''''''''''''''''''''''''''' + +Tools MAY choose to provide the same interfaces for installing Dependency +Groups as they do for installing extras. + +Note that this specification does not forbid having an extra whose name matches +a Dependency Group. + +Users are advised to avoid creating Dependency Groups whose names match extras. +Tools MAY treat such matching as an error. + +Validation and Compatibility +---------------------------- + +Tools supporting Dependency Groups may want to validate data before using it. +However, tools implementing such validation behavior should be careful to allow +for future expansions to this spec, so that they do not unnecessarily emit +errors or warnings in the presence of new syntax. + +Tools SHOULD error when evaluating or processing unrecognized data in +Dependency Groups. + +Tools SHOULD NOT eagerly validate the list contents of **all** Dependency +Groups. + +This means that in the presence of the following data, most tools will allow +the ``foo`` group to be used, and will only error when the ``bar`` group is +used: + +Reference Implementation +======================== + +The following Reference Implementation prints the contents of a Dependency +Group to stdout, newline delimited. +The output is therefore valid ``requirements.txt`` data. + +.. code-block:: python + + import re + import sys + import tomllib + from collections import defaultdict + + from packaging.requirements import Requirement + + + def _normalize_name(name: str) -> str: + return re.sub(r"[-_.]+", "-", name).lower() + + + def _normalize_group_names(dependency_groups: dict) -> dict: + original_names = defaultdict(list) + normalized_groups = {} + + for group_name, value in dependency_groups.items(): + normed_group_name = _normalize_name(group_name) + original_names[normed_group_name].append(group_name) + normalized_groups[normed_group_name] = value + + errors = [] + for normed_name, names in original_names.items(): + if len(names) > 1: + errors.append(f"{normed_name} ({', '.join(names)})") + if errors: + raise ValueError(f"Duplicate dependency group names: {', '.join(errors)}") + + return normalized_groups + + + def _resolve_dependency_group( + dependency_groups: dict, group: str, past_groups: tuple[str, ...] = () + ) -> list[str]: + if group in past_groups: + raise ValueError(f"Cyclic dependency group include: {group} -> {past_groups}") + + if group not in dependency_groups: + raise LookupError(f"Dependency group '{group}' not found") + + raw_group = dependency_groups[group] + if not isinstance(raw_group, list): + raise ValueError(f"Dependency group '{group}' is not a list") + + realized_group = [] + for item in raw_group: + if isinstance(item, str): + # packaging.requirements.Requirement parsing ensures that this is a valid + # PEP 508 Dependency Specifier + # raises InvalidRequirement on failure + Requirement(item) + realized_group.append(item) + elif isinstance(item, dict): + if tuple(item.keys()) != ("include-group",): + raise ValueError(f"Invalid dependency group item: {item}") + + include_group = _normalize_name(next(iter(item.values()))) + realized_group.extend( + _resolve_dependency_group( + dependency_groups, include_group, past_groups + (group,) + ) + ) + else: + raise ValueError(f"Invalid dependency group item: {item}") + + return realized_group + + + def resolve(dependency_groups: dict, group: str) -> list[str]: + if not isinstance(dependency_groups, dict): + raise TypeError("Dependency Groups table is not a dict") + if not isinstance(group, str): + raise TypeError("Dependency group name is not a str") + return _resolve_dependency_group(dependency_groups, group) + + + if __name__ == "__main__": + with open("pyproject.toml", "rb") as fp: + pyproject = tomllib.load(fp) + + dependency_groups_raw = pyproject["dependency-groups"] + dependency_groups = _normalize_group_names(dependency_groups_raw) + print("\n".join(resolve(pyproject["dependency-groups"], sys.argv[1]))) + +History +======= + +- October 2024: This specification was approved through :pep:`735`. diff --git a/source/specifications/section-distribution-metadata.rst b/source/specifications/section-distribution-metadata.rst index 0c0a8aaba..af7c1c3e6 100644 --- a/source/specifications/section-distribution-metadata.rst +++ b/source/specifications/section-distribution-metadata.rst @@ -10,6 +10,7 @@ Package Distribution Metadata version-specifiers dependency-specifiers pyproject-toml + dependency-groups inline-script-metadata platform-compatibility-tags well-known-project-urls From bb6f6cc042950bc052fe98bdba8768acd1b19563 Mon Sep 17 00:00:00 2001 From: Stephen Rosen Date: Mon, 16 Dec 2024 13:00:44 -0600 Subject: [PATCH 2/2] Refine dependency-groups documentation This is a significant revision from the PEP content to make the content appropriate for the packaging specs. In particular, the following changes are made: - references to "this PEP" are all removed - "Dependency Object Specifiers" are a layer of indirection around includes, and are removed from the documentation -- instead, the doc jumps straight to the (only) specific case now: the includes - the section on install UX and extras has been rewritten for brevity, removing the hypothetical example `pip` interface and folding together the install notes and the notes about extras - the section on validation/linting has been reorganized around an example, and the note on linters has been moved to a `note` admonition block --- source/specifications/dependency-groups.rst | 154 +++++++++----------- 1 file changed, 66 insertions(+), 88 deletions(-) diff --git a/source/specifications/dependency-groups.rst b/source/specifications/dependency-groups.rst index 8206a7063..b6a3e6fad 100644 --- a/source/specifications/dependency-groups.rst +++ b/source/specifications/dependency-groups.rst @@ -36,47 +36,33 @@ and a similar table which defines ``test`` and ``coverage`` groups:: The ``[dependency-groups]`` Table --------------------------------- -This PEP defines a new section (table) in ``pyproject.toml`` files named +Dependency Groups are defined as a table in ``pyproject.toml`` named ``dependency-groups``. The ``dependency-groups`` table contains an arbitrary number of user-defined keys, each of which has, as its value, a list of -requirements (defined below). These keys must be -`valid non-normalized names `__, -and must be -`normalized `__ -before comparisons. +requirements. -Tools SHOULD prefer to present the original, non-normalized name to users by -default. If duplicate names, after normalization, are encountered, tools SHOULD -emit an error. +``[dependency-groups]`` keys, sometimes also called "group names", must be +:ref:`valid non-normalized names `. Tools which handle Dependency +Groups MUST :ref:`normalize ` these names before +comparisons. -Requirement lists under ``dependency-groups`` may contain strings, tables -("dicts" in Python), or a mix of strings and tables. +Tools SHOULD prefer to present the original, non-normalized name to users, and +if duplicate names are detected after normalization, tools SHOULD emit an +error. -Strings in requirement lists must be valid -`Dependency Specifiers `__, -as defined in :pep:`508`. - -Tables in requirement lists must be valid Dependency Object Specifiers. - -Dependency Object Specifiers ----------------------------- - -Dependency Object Specifiers are tables which define zero or more dependencies. - -This PEP standardizes only one type of Dependency Object Specifier, a -"Dependency Group Include". Other types may be added in future standards. +Requirement lists, the values in ``[dependency-groups]``, may contain strings, +tables (``dict`` in Python), or a mix of strings and tables. Strings must be +valid :ref:`dependency specifiers `, and tables must be +valid Dependency Group Includes. Dependency Group Include -'''''''''''''''''''''''' - -A Dependency Group Include includes the dependencies of another Dependency -Group in the current Dependency Group. +------------------------ -An include is defined as a table with exactly one key, ``"include-group"``, -whose value is a string, the name of another Dependency Group. +A Dependency Group Include includes another Dependency Group in the current +group. -For example, ``{include-group = "test"}`` is an include which expands to the -contents of the ``test`` Dependency Group. +An include is a table with exactly one key, ``"include-group"``, whose value is +a string, the name of another Dependency Group. Includes are defined to be exactly equivalent to the contents of the named Dependency Group, inserted into the current group at the location of the include. @@ -84,94 +70,86 @@ For example, if ``foo = ["a", "b"]`` is one group, and ``bar = ["c", {include-group = "foo"}, "d"]`` is another, then ``bar`` should evaluate to ``["c", "a", "b", "d"]`` when Dependency Group Includes are expanded. -Dependency Group Includes may specify the same package multiple times. Tools -SHOULD NOT deduplicate or otherwise alter the list contents produced by the +Dependency Group Includes may specify the same package multiple times. +Tools SHOULD NOT deduplicate or otherwise alter the list contents produced by the include. For example, given the following table: -.. code:: toml +.. code-block:: toml [dependency-groups] group-a = ["foo"] group-b = ["foo>1.0"] group-c = ["foo<1.0"] - all = ["foo", {include-group = "group-a"}, {include-group = "group-b"}, {include-group = "group-c"}] + all = [ + "foo", + {include-group = "group-a"}, + {include-group = "group-b"}, + {include-group = "group-c"}, + ] The resolved value of ``all`` SHOULD be ``["foo", "foo", "foo>1.0", "foo<1.0"]``. Tools should handle such a list exactly as they would handle any other case in which they are asked to process the same requirement multiple times with different version constraints. -Dependency Group Includes may include lists containing Dependency Group -Includes, in which case those includes should be expanded as well. Dependency -Group Includes MUST NOT include cycles, and tools SHOULD report an error if -they detect a cycle. +Dependency Group Includes may include groups containing Dependency Group Includes, +in which case those includes should be expanded as well. Dependency Group Includes +MUST NOT include cycles, and tools SHOULD report an error if they detect a cycle. Package Building ---------------- Build backends MUST NOT include Dependency Group data in built distributions as -package metadata. This means that PKG-INFO in sdists and METADATA in wheels -do not include any referencable fields containing Dependency Groups. - -It is valid to use Dependency Groups in the evaluation of dynamic metadata, and -``pyproject.toml`` files included in sdists will naturally still contain the -``[dependency-groups]`` table. However, the table contents are not part of a -published package's interfaces. - -Installing Dependency Groups ----------------------------- - -Tools which support Dependency Groups are expected to provide new options and -interfaces to allow users to install from Dependency Groups. - -No syntax is defined for expressing the Dependency Group of a package, for two -reasons: +package metadata. This means that sdist ``PKG-INFO`` and wheel ``METADATA`` +files should not include referenceable fields containing Dependency Groups. -* it would not be valid to refer to the Dependency Groups of a third-party - package from PyPI (because the data is defined to be unpublished) +It is, however, valid to use Dependency Groups in the evaluation of dynamic +metadata, and ``pyproject.toml`` files included in sdists will still contain +``[dependency-groups]``. However, the table's contents are not part of a built +package's interfaces. -* there is not guaranteed to be a current package for Dependency Groups -- part - of their purpose is to support non-package projects +Installing Dependency Groups & Extras +------------------------------------- -For example, a possible pip interface for installing Dependency Groups -would be: +There is no syntax or specification-defined interface for installing or +referring to Dependency Groups. Tools are expected to provide dedicated +interfaces for this purpose. -.. code:: shell - - pip install --dependency-groups=test,typing - -Note that this is only an example. This PEP does not declare any requirements -for how tools support the installation of Dependency Groups. - -Overlapping Install UX with Extras -'''''''''''''''''''''''''''''''''' - -Tools MAY choose to provide the same interfaces for installing Dependency -Groups as they do for installing extras. - -Note that this specification does not forbid having an extra whose name matches -a Dependency Group. - -Users are advised to avoid creating Dependency Groups whose names match extras. -Tools MAY treat such matching as an error. +Tools MAY choose to provide the same or similar interfaces for interacting +with Dependency Groups as they do for managing extras. Tools authors are +advised that the specification does not forbid having an extra whose name +matches a Dependency Group. Separately, users are advised to avoid creating +Dependency Groups whose names match extras, and tools MAY treat such matching +as an error. Validation and Compatibility ---------------------------- Tools supporting Dependency Groups may want to validate data before using it. -However, tools implementing such validation behavior should be careful to allow -for future expansions to this spec, so that they do not unnecessarily emit -errors or warnings in the presence of new syntax. +When implementing such validation, authors should be aware of the possibility +of future extensions to the specification, so that they do not unnecessarily +emit errors or warnings. Tools SHOULD error when evaluating or processing unrecognized data in Dependency Groups. -Tools SHOULD NOT eagerly validate the list contents of **all** Dependency -Groups. +Tools SHOULD NOT eagerly validate the contents of *all* Dependency Groups +unless they have a need to do so. + +This means that in the presence of the following data, most tools should allow +the ``foo`` group to be used and only error if the ``bar`` group is used: + +.. code-block:: toml + + [dependency-groups] + foo = ["pyparsing"] + bar = [{set-phasers-to = "stun"}] + +.. note:: -This means that in the presence of the following data, most tools will allow -the ``foo`` group to be used, and will only error when the ``bar`` group is -used: + There are several known cases of tools which have good cause to be + stricter. Linters and validators are an example, as their purpose is to + validate the contents of all Dependency Groups. Reference Implementation ========================