PEP 751: drop packages.direct and add extras and dependency-groups (#4287)

Author: brettcannon

peps/pep-0751.rst

@@ -35,10 +35,11 @@ community (PDM_, ``pip freeze``, pip-tools_, Poetry_, and uv_), there seems to
 be an appetite for lock files in general.
 
 Those tools also vary in what locking scenarios they support. For instance,
-``pip freeze`` and pip-tools only generate lock files for the current
+``pip freeze`` and pip-tools only generate single-use lock files for the current
 environment while PDM, Poetry, and uv can/try to lock for multiple environments
-at once. There's also concerns around the lack of secure defaults in the face of
-supply chain attacks (e.g. including hashes for files).
+and use-cases at once. There's also concerns around the lack of secure defaults
+in the face of supply chain attacks by some tools (e.g. including hashes for
+files).
 
 The lack of a standard also has some drawbacks. For instance, any tooling that
 wants to work with lock files must choose which format to support, potentially
@@ -47,7 +48,7 @@ same for cloud providers who can do dependency installations on your behalf,
 etc.). It also impacts portability between tools, which causes vendor lock-in.
 By not having compatibility and interoperability it fractures tooling around
 lock files where both users and tools have to choose what lock file format to
-use upfront and making it costly to use/switch to other formats (e.g. tooling
+use upfront, making it costly to use/switch to other formats (e.g. tooling
 around auditing a lock file). Rallying around a single format removes this
 cost/barrier.
 
@@ -80,7 +81,9 @@ creating a lock file.
 
 The data in the file should be consumable by tools not written in Python. This
 allows for e.g. cloud hosting providers to write their own tool to perform
-installations in their preferred programming language.
+installations in their preferred programming language. This introduces the
+concept of *lockers* which write the lock file, and *installers* which install
+from a lock file (they can be the same tool).
 
 The file format should promote good security defaults. As the format is not
 meant to be human-writable, this means having tools provide security-related
@@ -92,6 +95,24 @@ pip-tools_ and ``pip freeze`` emit). This means the file format specified by
 this PEP can, at minimum, act as an export target for tools which have their own
 internal lock file format.
 
+Lock files can be *single-use* and *multi-use*. Single-use lock files are things
+like ``requirements.txt`` files, which serve a single use-case/purpose (hence
+why it isn't uncommon for a project to have multiple requirements files, each
+for a diffetent use-case). Multi-use lock files represent multiple use-cases
+within a single file, often expressed through
+:ref:`extras <packaging:core-metadata-provides-extra>` and
+:ref:`packaging:dependency-groups`. As such, this PEP supports additions to
+:ref:`packaging:dependency-specifiers-environment-markers` that allows for
+specifying extras and dependency groups as appropriate. This allows
+for a single lock file to support those cases. This not only cuts down on the
+number of potential lock files, but it also makes it easier when a single
+package is meant to be consistent across all use-cases (with multiple single-use
+lock files it requires coordinating across multiple lock files). This support
+also means that this PEP supports all package installation-related data that can
+be recorded in a ``pyproject.toml`` file. The hope is that this support will
+allow some tools to drop their internal lock file entirely and solely rely on
+what this PEP specifies.
+
 
 =============
 Specification
@@ -139,6 +160,7 @@ recorded in a consistent order (if inspiration is desired, this PEP has tried to
 write down keys in a logical order). As well, tools SHOULD sort arrays in
 consistent order. Usage of inline tables SHOULD also be kept consistent.
 
+.. File details
 
 ``lock-version``
 ================
@@ -177,6 +199,41 @@ consistent order. Usage of inline tables SHOULD also be kept consistent.
   (i.e. the minimum viable Python version for the lock file).
 
 
+``extras``
+==========
+
+- **Type**: Array of strings
+- **Required?**: no; defaults to ``[]``
+- **Inspiration**: :ref:`packaging:pyproject-tool-table`
+- The list of :ref:`extras <packaging:core-metadata-provides-extra>` supported
+  by this lock file.
+- Lockers MAY choose to not support writing lock files that support extras and
+  dependency groups (i.e. tools may only support exporting a single-use lock
+  file).
+- Tools supporting extras MUST also support dependency groups.
+- Tools should explicitly set this key to an empty array to signal that the
+  inputs used to generate the lock file had no extras (e.g. a ``pyproject.toml``
+  file had no ``[project.optional-dependencies]`` table), signalling that the
+  lock file is, in effect, multi-use even if it only looks to be single-use.
+
+
+``dependency-groups``
+=====================
+
+- **Type**: Array of strings
+- **Required?**: no; defaults to ``[]``
+- **Inspiration**: :ref:`packaging:pyproject-tool-table`
+- The list of :ref:`packaging:dependency-groups` supported by this lock file.
+- Lockers MAY choose to not support writing lock files that support extras and
+  dependency groups (i.e. tools may only support exporting a single-use lock
+  file).
+- Tools supporting dependency groups MUST also support extras.
+- Tools SHOULD explicitly set this key to an empty array to signal that the
+  inputs used to generate the lock file had no extras (e.g. a ``pyproject.toml``
+  file had no ``[dependency-groups]`` table), signalling that the lock file
+  is, in effect, multi-use even if it only looks to be single-use.
+
+
 ``created-by``
 ==============
 
@@ -269,18 +326,6 @@ consistent order. Usage of inline tables SHOULD also be kept consistent.
   informational for auditing purposes.
 
 
-.. Installation
-
-``packages.direct``
--------------------
-
-- **Type**: boolean
-- **Required?**: no; defaults to ``false``
-- **Inspiration**: :ref:`packaging:direct-url`
-- Represents whether the installation is via a
-  :ref:`direct URL reference <packaging:direct-url>`.
-
-
 .. Source
 
 ``[packages.vcs]``
@@ -297,6 +342,8 @@ consistent order. Usage of inline tables SHOULD also be kept consistent.
   and/or installation perspective.
 - Tools SHOULD provide a way for users to opt in/out of using version control
   systems.
+- Installation from a version control system is considered originating from a
+  :ref:`direct URL reference <packaging:direct-url>`.
 
 
 ``packages.vcs.type``
@@ -380,6 +427,8 @@ consistent order. Usage of inline tables SHOULD also be kept consistent.
 - Tools MAY choose to not support local directories, both from a locking
   and/or installation perspective.
 - Tools SHOULD provide a way for users to opt in/out of using local directories.
+- Installation from a directory is considered originating from a
+  :ref:`direct URL reference <packaging:direct-url>`.
 
 
 ``packages.directory.path``
@@ -415,11 +464,14 @@ See ``packages.vcs.subdirectory``.
 - **Type**: table
 - **Required?**: no
 - **Inspiration**: :ref:`packaging:direct-url-data-structure-archive`
-- An archive file containing a
-  :ref:`packaging:source-distribution-format-source-tree`.
+- A direct reference to an archive file to install from
+  (this can include wheels and sdists, as well as other archive formats
+  containing a source tree).
 - Tools MAY choose to not support archive files, both from a locking
   and/or installation perspective.
 - Tools SHOULD provide a way for users to opt in/out of using archive files.
+- Installation from an archive file is considered originating from a
+  :ref:`direct URL reference <packaging:direct-url>`.
 
 
 ``packages.archive.url``
@@ -445,6 +497,16 @@ See ``packages.vcs.path``.
   size is available via the Content-Length_ header from a HEAD_ HTTP request).
 
 
+``packages.archive.upload-time``
+''''''''''''''''''''''''''''''''
+
+- **Type**: datetime
+- **Required?**: no
+- **Inspiration**: :ref:`packaging:simple-repository-api`
+- The time the file was uploaded.
+- The date and time MUST be recorded in UTC.
+
+
 ``[packages.archive.hashes]``
 ''''''''''''''''''''''''''''''
 
@@ -507,11 +569,7 @@ See ``packages.vcs.subdirectory``.
 ``packages.sdist.upload-time``
 ''''''''''''''''''''''''''''''
 
-- **Type**: datetime
-- **Required?**: no
-- **Inspiration**: :ref:`packaging:simple-repository-api`
-- The time the file was uploaded.
-- The date and time MUST be recorded in UTC.
+See ``packages.archive.upload-time``.
 
 
 ``packages.sdist.url``
@@ -564,7 +622,7 @@ See ``packages.archive.hashes``.
 ``packages.wheels.upload-time``
 '''''''''''''''''''''''''''''''
 
-See ``packages.sdist.upload-time``.
+See ``packages.archive.upload-time``.
 
 
 ``packages.wheels.url``
@@ -638,6 +696,63 @@ See ``packages.archive.hashes``.
 - See ``packages.tool``.
 
 
+-------------------------------------
+Additions to marker expression syntax
+-------------------------------------
+
+This PEP proposes adding to the
+:ref:`packaging:dependency-specifiers-environment-markers` specification such
+that extras and dependency group reliationships for an entry in ``[[packages]]``
+can be expressed in ``packages.marker``. The additions outlined in this PEP
+ONLY apply in the context of lock files as defined by this PEP and not in other
+contexts where marker syntax is used (e.g. ``METADATA``, ``pyproject.toml``).
+
+First, two new markers will be introduced: ``extras`` and
+``dependency_groups``. They represent the extras and dependency groups that have
+been requested to be installed, respectively.
+
+Second, the marker specification will be changed to allow for containers and not
+just strings for values. ONLY the new markers introduced in this PEP MAY and
+MUST must be used with a :py:class:`collections.abc.Container` type for their
+value (which default to empty containers). An assumption of the type of
+container used MUST NOT be made (e.g. could be a set, list, or tuple).
+
+Third, the "<marker_op> operators that are not in <version_cmp>" will be changed
+from operating "the same as they do for *strings*" to "the same as they
+do for *containers*". There are no backwards-compatibility concerns as strings
+are containers themselves.
+
+Fourth, a tool MUST raise an error if an extra or dependency group is specified
+in a marker expression that does not exist in ``packages.extras`` and
+``packages.dependency-groups``, respectively.
+
+These changes, along with ``packages.extras``/ ``packages.dependency-groups``
+and marker expressions' Boolean logic support, allow for expressing arbitrary,
+exhaustive requirements for when a package should be installed based on the
+extras and dependency groups (not) requested. For example, if a lock file has
+``extras = ["extra-1", "extra-2"]``, you can specify if a package should be
+installed when:
+
+- Any of the extras are specified
+  (``'extra-1' in extras or 'extra-2' in extras``)
+- Only "extra-1" is specified
+  (``'extra-1' in extras and 'extra-2' not in extras``)
+- None of the extras are specified
+  (``'extra-1' not in extras and 'extra-2' not in extras``)
+
+(This list is not exhaustive of all possible Boolean logic expressions.)
+
+The same flexibility applies to dependency groups.
+
+How users tell a tool what extras and/or dependency groups they want installed
+is up to the tool. Tools MUST default to no extras or dependency groups being
+requested by the user if no extras or dependency groups are requested.
+Installers MUST support the marker expression syntax additions as
+proposed by this PEP. Lockers MAY support writing lock files that utilize the
+proposed marker expression syntax additions (i.e. lockers can choose to only
+support writing single-use lock files).
+
+
 -------
 Example
 -------
@@ -707,10 +822,10 @@ a suggestion):
       skip to the next package.
    #. If ``requires-python`` is specified, check if it is satisfied; an error
       MUST be raised if it isn't.
-   #. Check that no other instance of the package has been slated to be
-      installed; an error about the ambiguity MUST be raised otherwise.
+   #. Check that no other conflicting instance of the package has been slated to
+      be installed; an error about the ambiguity MUST be raised otherwise.
    #. Check that the source of the package is specified appropriately (i.e.
-      there are not conflicting sources in the package entry);
+      there are no conflicting sources in the package entry);
       an error MUST be raised if any issues are found.
    #. Add the package to the set of packages to install.
 
@@ -759,6 +874,7 @@ a suggestion):
         - If ``url`` is set, try to use it; tools MAY use
           ``packages.index`` or some tool-specific mechanism to download the
           file.
+
      #. Validate the file size and hash.
      #. Build the package.
      #. Install.
@@ -795,6 +911,8 @@ from the URL or path is also to help make reading the list of wheel files easier
 as it encodes information that can be useful when understanding and auditing a
 file. Recording the sdist file name is for the same reason.
 
+This PEP supports multi-use lock files while requirements files are single-use.
+
 
 =======================
 Backwards Compatibility
@@ -855,6 +973,20 @@ installed and not a malicious one that someone may have slipped in. It also
 lets one be more deliberate in upgrading their dependencies and thus making sure
 the change is on purpose and not one slipped in by a bad actor.
 
+Lock files can support only certain *environments*. What must be installed for
+the environment they are installing for may be different compared to another,
+different environment. Some lock files do try to be *universal*, though, and
+work for any possible environment (sdist and source tree installation success
+permitting).
+
+Lock files can be *single-use* or *multi-use*. Single-use lock files are for
+single use-cases. Multi-use lock files can be used for multiple
+use-cases based on extras and dependency groups. It is up to the tool(s) you use
+that decide whether multi-use lock files are possible. All tools dealing with
+lock files at least support single-use lock files. Neither type of lock file
+is better or worse than the other, it just changes how much can be written down
+in a single file (which can influence how manageable).
+
 
 ========================
 Reference Implementation
@@ -1045,30 +1177,21 @@ decide it wasn't worth trying to do in this PEP upfront. Instead, a future PEP
 could propose a solution.
 
 
-Recording possible extras and dependency groups
-===============================================
+A dedicated ``direct`` key
+==========================
+
+Earlier versions had a dedicated ``packages.direct`` key to flag when something
+should be considered as originating from a
+:ref:`direct URL reference <packaging:direct-url>`. But there are explicitly
+only three cases where a direct URL reference can occur (VCS, directory, and
+archive). Since all three cases are explicit in ``[[packages]]``, the setting of
+the key was technically superfluous.
 
-To expand the feature set of this PEP such that tools could use this PEP for
-their main lock file format, recording possible extras and dependency groups
-that a user may request was considered. The idea was that if you were locking
-from a ``pyproject.toml`` file then you would want to lock for all
-possibilities; that includes extras and dependency groups.
-
-To make this work one would need a way to declare when a package applied to an
-extra and/or dependency group. Due to the possibility that a combination of
-extras and/or dependency groups could change version requirements of a package
-(e.g. extra A needed the spam package at any version, but extra B needed the
-spam package older than version 2, thus making whether extra B was requested
-override what extra A requested), it would require either full Boolean logic
-support or only locking to a specific version of a package no matter what extras
-and/or dependency groups were requested. The former would at least require
-coming up with semantics around a marker for dependency groups, and the latter
-would require a separate lock file any time one didn't want a single version
-restriction.
-
-In the end, there wasn't enough interest from tools for using this PEP as
-their sole lock file format to warrant doing the work to implement this feature
-at this time.
+The single drawback of not having this key is for wheels and sdists that now
+fall under ``packages.archive``. With the separate key (or having it specified
+as a part of ``packages.sdist`` or ``packages.wheels``), it would allow for
+identifying in the lock file itself that an archive file is an sdist of wheel.
+As it currently stand, an installer must infer that detail itself.
 
 
 --------------