PEP 694: Several updates based on Michał Górny's feedback on DPO (#4323)

warsaw · web-flow · commit d4603557e713 · 2025-04-14T22:15:34.000Z
* Several updates based on Michal Gorny's feedback on DPO https://discuss.python.org/t/pep-694-pypi-upload-api-2-0/76316/2 Summary: * Add some error return code descriptions where they were underspecified * Fix some misspellings and poorly worded text * Combine the canceling and deleting of upload files sections * Simplify the rules for replacing a staged but not yet published file * Add open question about deferring stage previews * Fix a typo * The last chunk can be accepted asynchronously * Be explicit that the server MAY accepted parallel file uploads * Post-History * Fix reference link typo The server is allowed to respond with a 202 for the last chunk, to indicate it's processing that last chunk asynchronously. This fleshes out the process for checking for completed process of the file in that case, and improves the file status key definitions.
diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst
@@ -6,7 +6,9 @@ Status: Draft
 Type: Standards Track
 Topic: Packaging
 Created: 11-Jun-2022
-Post-History: `27-Jun-2022 <https://discuss.python.org/t/pep-694-upload-2-0-api-for-python-package-repositories/16879>`__
+Post-History: `27-Jun-2022 <https://discuss.python.org/t/pep-694-upload-2-0-api-for-python-package-repositories/16879>`__,
+              `06-Jan-2025 <https://discuss.python.org/t/pep-694-pypi-upload-api-2-0/76316>`__
+              `14-Apr-2025 <https://discuss.python.org/t/pep-694-pypi-upload-api-2-0/76316/9>`__
 
 
 Abstract
@@ -297,6 +299,7 @@ the following keys:
     wishes to communicate to the end user.  These notices are specific to the overall session, not
     to any particular file in the session.
 
+
 .. _session-links:
 
 Session Links
@@ -329,8 +332,15 @@ The ``files`` key contains a mapping from the names of the files uploaded in thi
 sub-mapping with the following keys:
 
 ``status``
-    A string with the same values and semantics as the :ref:`session status key <session-status>`,
-    except that it indicates the status of the specific referenced file.
+    A string with valid values ``partial``, ``pending``, ``complete``, and ``error``.  If a file
+    upload has not seen an ``Upload-Complete: ?1`` header, then ``partial`` will be returned.  If
+    ``Upload-Complete: ?1`` resulted in a ``202 Accepted``, then ``pending`` will be returned until
+    asynchronous processing of the last chunk and the full file has been completed.  If a ``201
+    Created`` was returned, or the last chunk processing is finished, ``complete`` will be returned.
+    If there was an error during upload, then clients should not assume the file is in any usable
+    state, ``error`` will be returned and it's best to :ref:`cancel or delete <cancel-an-upload>`
+    the file and start over.  This action would remove the file name from the ``files`` key of the
+    :ref:`session status response body <session-response>`.
 
 ``link``
     The *absolute* URL that the client should use to reference this specific file.  This URL is used
@@ -409,6 +419,9 @@ If the server determines that upload should proceed, it will return a ``201 Crea
 an empty body, and a ``Location`` header pointing to the URL that the file content should be
 uploaded to.  The :ref:`status <session-status>` of the session will also include the filename in
 the ``files`` mapping, with the above ``Location`` URL included in under the ``link`` sub-key.
+If the server determines the upload cannot proceed, it **MUST** return a ``409 Conflict``.  The
+server **MAY** allow parallel uploads of files, but is not required to.
+
 
 .. IMPORTANT::
 
@@ -521,15 +534,18 @@ For the second chunk representing bytes 1000 through 1999, include the following
 These requests would continue sequentially until the last chunk is ready to be uploaded.
 
 For each successful chunk, the server **MUST** respond with a ``202 Accepted`` header, except for
-the final chunk, which **MUST** be a ``201 Created``, and as with non-chunked uploads, the body of
-these responses has no content.
+the final chunk, which **MUST** be either:
 
-.. _complete-the-upload:
+* ``201 Created`` if the server accepts and processes the last chunk synchronously, completing the
+  file upload.
+* ``202 Accepted`` if the server accepts the last chunk, but must process it asynchronously.  In
+  this case, the client should query the :ref:`session status <session-response>` periodically until
+  the uploaded :ref:`file status <session-files>` transitions to ``complete``.
 
 The final chunk of data **MUST** include the ``Upload-Complete: ?1`` header, since at that point the
 entire file has been uploaded.
 
-With both chunked and non-chunked uploads, once completed successfully, the file **MUST** not be
+With both chunked and non-chunked uploads, once completed successfully, the file **MUST NOT** be
 publicly visible in the repository, but merely staged until the upload session is :ref:`completed
 <publish-session>`. If the server supports :ref:`previews <staged-preview>`, the file **MUST** be
 visible at the ``stage`` :ref:`URL <session-links>`.  Partially uploaded chunked files **SHOULD
@@ -541,9 +557,9 @@ multiple chunks:
 - A client **MUST NOT** perform multiple ``POST`` requests in parallel for the same file to avoid
   race conditions and data loss or corruption.
 
-- If the offset provided in ``Upload-Offset`` is not ``0`` or correctly specifies the byte offset of
-  the next chunk in an incomplete upload, then the server **MUST** respond with a ``409 Conflict``.
-  This means that a client **MAY NOT** upload chunks out of order.
+- If the offset provided in ``Upload-Offset`` is not ``0`` and does not correctly specify the byte
+  offset of the next chunk in an incomplete upload, then the server **MUST** respond with a ``409
+  Conflict``.  This means that a client **MUST NOT** upload chunks out of order.
 
 - Once a file upload has completed successfully, you may initiate another upload for that file,
   which **once completed**, will replace that file.  This is possible until the entire session is
@@ -578,47 +594,30 @@ multiple chunks as per the above protocol.
 
 .. _cancel-an-upload:
 
-Canceling an In-Progress Upload
-+++++++++++++++++++++++++++++++
-
-If a client wishes to cancel an upload of a specific file, for instance because they need to upload
-a different file, they may do so by issuing a ``DELETE`` request to the upload resource URL of the
-file they want to delete.
-
-A successful cancellation request **MUST** respond with a ``204 No Content``.
-
-Once deleting, a client **MUST NOT** assume that the previous upload resource URL can be reused.
-
-
-Delete a Partial or Fully Uploaded File
-+++++++++++++++++++++++++++++++++++++++
+Canceling and Deleting File Uploads
++++++++++++++++++++++++++++++++++++
 
-Similarly, for files which have already been completely uploaded, clients can delete the file by
-issuing a ``DELETE`` request to the upload resource  URL.
+A client can cancel an in-progress upload for a file, or delete a file that has been completely
+uploaded.  In both cases, the client performs this by issuing a ``DELETE`` request to the upload
+resource URL of the file they want to delete.
 
 A successful deletion request **MUST** response with a ``204 No Content``.
 
-Once deleting, a client **MUST NOT** assume that the previous upload resource URL can be reused.
+Once canceled or deleted, a client **MUST NOT** assume that the previous upload resource URL can be reused.
 
 
 Replacing a Partially or Fully Uploaded File
 ++++++++++++++++++++++++++++++++++++++++++++
 
-To replace a session file, the file upload **MUST** have been previously completed or deleted.  It
-is not possible to replace a file if the upload for that file is incomplete.  Clients have two
-options to replace an incomplete upload:
+To replace a session file, the file upload **MUST** have been previously completed, canceled, or
+deleted.  It is not possible to replace a file if the upload for that file is in-progress.
 
-- :ref:`Cancel the in-progress upload <cancel-an-upload>` by issuing a ``DELETE`` to the upload
-  resource URL for the file they want to replace.  After this, the new file upload can be initiated
-  by beginning the entire :ref:`file upload <file-uploads>` sequence over again.  This means
-  providing the metadata request again to retrieve a new upload resource URL.  Client **MUST NOT**
-  assume that the previous upload resource URL can be reused after deletion.
-
-- :ref:`Complete the in-progress upload <complete-the-upload>` by uploading a zero-length chunk
-  providing the ``Upload-Complete: ?1`` header.  This effectively truncates and completes the
-  in-progress upload, after which point the new upload can commence.  In this case, clients
-  **SHOULD** reuse the previous upload resource URL and do not need to begin the entire :ref:`file
-  upload <file-uploads>` sequence over again.
+To replace a session file, clients should :ref:`cancel and delete the in-progress upload
+<cancel-an-upload>` by issuing a ``DELETE`` to the upload resource URL for the file they want to
+replace.  After this, the new file upload can be initiated by beginning the entire :ref:`file upload
+<file-uploads>` sequence over again.  This means providing the metadata request again to retrieve a
+new upload resource URL.  Client **MUST NOT** assume that the previous upload resource URL can be
+reused after deletion.
 
 
 .. _session-status:
@@ -720,6 +719,9 @@ If a session is published that has no staged files, the operation is effectively
 where a new project name is being reserved.  In this case, the new project is created, reserved, and
 owned by the user that created the session.
 
+If an error occurs, the appropriate ``4xx`` code should be returned, as described in the
+:ref:`session-errors` section.
+
 
 .. _session-token:
 
@@ -932,6 +934,18 @@ The user that created the session will become the owner of the new project.
 Open Questions
 ==============
 
+Defer Stage Previews
+--------------------
+
+:ref:`Stage previews <staged-preview>` are an important and useful feature for testing new version
+wheel uploads before they are published.  They'd allow us to effectively decommission
+``test.pypi.org``, which has well-known deficiencies.
+
+However, the ability to preview stages before they're published does complicate the protocol and
+this proposal.  We could defer this feature for later, although if we do, we should still keep the
+optional ``nonce`` for token generation, in order to be easily future proof.
+
+
 Multipart Uploads vs tus
 ------------------------
 
