PEP 694: Abstract file upload mechanisms

peps/pep-0694.rst

@@ -1,6 +1,6 @@
 PEP: 694
 Title: Upload 2.0 API for Python Package Indexes
-Author: Barry Warsaw <[email protected]>, Donald Stufft <[email protected]>
+Author: Barry Warsaw <[email protected]>, Donald Stufft <[email protected]>, Ee Durbin <[email protected]>
 Discussions-To: https://discuss.python.org/t/pep-694-upload-2-0-api-for-python-package-repositories/16879
 Status: Draft
 Type: Standards Track
@@ -77,9 +77,11 @@ In addition, there are a number of major issues with the legacy API:
 * Creation of new projects requires the uploading of at least one file, leading to "stub" uploads
   to claim a project namespace.
 
-The new upload API proposed in this PEP solves all of these problems, providing for a much more
-flexible approach, with better error reporting, a better release testing
-experience, and atomic and simultaneous publishing of all release artifacts.
+The new upload API proposed in this PEP provides a solution to all of these problems,
+providing for a much more flexible approach, with support for servers to
+implement resumable and parallel uploads via mechanisms,
+better error reporting, better release testing experience,
+and atomic and simultaneous publishing of all release artifacts.
 
 
 Legacy API
@@ -157,6 +159,7 @@ these steps:
 #. Complete the upload session, publishing or discarding the stage.
 #. Optionally check the status of an upload session.
 
+.. _versioning:
 
 Versioning
 ----------
@@ -175,20 +178,17 @@ All URLs described here are relative to the "root endpoint", which may be locate
 the url structure of a domain. For example, the root endpoint could be
 ``https://upload.example.com/``, or ``https://example.com/upload/``.
 
-Specifically for PyPI, this PEP proposes to implement the root endpoint at
-``https://upload.pypi.org/2.0``.  This root URL will be considered provisional while the feature is
-being tested, and will be blessed as permanent after sufficient testing with live projects.
-
-
 .. _session-create:
 
 Create an Upload Session
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-A release starts by creating a new upload session.  To create the session, a client submits a ``POST`` request
-to the root URL, with a payload that looks like:
+A release starts by creating a new upload session.  To create the session, a client submits a
+``POST`` request to the root URL like:
 
-.. code-block:: json
+.. code-block:: text
+
+    Content-Type: application/vnd.pypi.upload.v2+json
 
     {
       "meta": {
@@ -237,9 +237,11 @@ requests.
 Response Body
 +++++++++++++
 
-The successful response includes the following JSON content:
+The successful response includes the following content:
+
+.. code-block:: text
 
-.. code-block:: json
+    Content-Type: application/vnd.pypi.upload.v2+json
 
     {
       "meta": {
@@ -365,9 +367,11 @@ After creating the session, the ``upload`` endpoint from the response's :ref:`se
 to those URLs from one session to the next.
 
 To initiate a file upload, a client first sends a ``POST`` request to the ``upload`` URL.  The
-request body has the following JSON format:
+request looks like:
+
+.. code-block:: text
 
-.. code-block:: json
+    Content-Type: application/vnd.pypi.upload.v2+json
 
     {
       "meta": {
@@ -431,9 +435,11 @@ it **MUST** return a ``422 Unprocessable Entity``.
 File Upload Session Response Body
 +++++++++++++++++++++++++++++++++
 
-The successful response includes the following JSON content:
+The successful response includes the following:
+
+.. code-block:: text
 
-.. code-block:: json
+    Content-Type: application/vnd.pypi.upload.v2+json
 
     {
       "meta": {
@@ -447,8 +453,10 @@ The successful response includes the following JSON content:
       "valid-for": 3600,
       "mechanism": {
         "http-post-application-octet-stream": {
-          "url": "..."
-        }
+          "file_url": "..."
+          "attestations_url": "..."
+        },
+        ...
       }
     }
 
@@ -490,9 +498,11 @@ To complete a file upload session, which indicates that the file upload mechanis
 and did not produce an error, a client issues a ``POST`` to the ``file-upload-session`` link in the
 file upload session creation response body.
 
-The JSON body of this requests looks like:
+The requests looks like:
+
+.. code-block:: text
 
-.. code-block:: json
+    Content-Type: application/vnd.pypi.upload.v2+json
 
     {
       "meta": {
@@ -566,9 +576,11 @@ allowed is left to the server.  To extend a session, a client issues a ``POST``
 given in the :ref:`session creation response body <session-response>`
 or :ref:`file upload session creation response body <file-upload-session-response>`, respectively.
 
-The JSON body of this request looks like:
+The request looks like:
+
+.. code-block:: text
 
-.. code-block:: json
+    Content-Type: application/vnd.pypi.upload.v2+json
 
     {
       "meta": {
@@ -616,9 +628,11 @@ To complete a session and publish the files that have been included in it, a cli
 ``POST`` request to the ``session`` :ref:`link <session-links>` given in the :ref:`session creation
 response body <session-response>`.
 
-The JSON body of this request looks like:
+The request looks like:
 
-.. code-block:: json
+.. code-block:: text
+
+    Content-Type: application/vnd.pypi.upload.v2+json
 
     {
       "meta": {
@@ -711,9 +725,11 @@ experience, but the details of this are left to installer tool maintainers.
 Errors
 ------
 
-All error responses that contain content will have a body that looks like:
+All error responses that contain content look like:
+
+.. code-block:: text
 
-.. code-block:: json
+    Content-Type: application/vnd.pypi.upload.v2+json
 
     {
       "meta": {
@@ -744,20 +760,68 @@ interpretation to aid in diagnosing underlying issue.
 File Upload Mechanisms
 ----------------------
 
-File Upload Mechanisms, with the exception of ``http-post-application-octet-stream`` are left as an
-implementation detail specific to each server. Servers **MUST** implement a
-``http-post-application-octet-stream`` mechanism. This serves as a fallback if no server specific implementations
-exist.
+Servers **MUST** implement :ref:`required file upload mechansisms <required-file-upload-mechanisms>`.
+Such mechanisms serve as a fallback if no server specific implementations exist.
 
-A given server may implement an arbitrary number of mechanisms and is responsible for documenting
-their usage. Implemenatations **SHOULD** be prefixed with a string that clearly identifies the
-server and is unique from other well known servers or implementations.
+Each major version of the Upload API **MUST** specify at least one required File Upload Mechanism.
 
-If a server intendes to match the behavior of another server's implementation, it **MAY** respond
+New required mechanisms **MUST NOT** be added and existing required mechanisms **MUST NOT** be removed
+without an update to the :ref:`major version <versioning>`.
+
+A given server **MAY** implement an arbitrary number of server specific mechanisms
+and is responsible for documenting their usage.
+Server specific implementations **MUST** be prefixed with ``vnd-``
+and **SHOULD** further contain a string that clearly identifies the server operator
+and is unique from other well known servers or implementations.
+
+For example:
+
+====================================== ================  ========================================================
+File Upload Mechanism string           Server Operator   Mechanism description
+====================================== ================  ========================================================
+``vnd-pypi-s3multipart-presigned``     PyPI              S3 multipart upload via pre-signed URL
+``vnd-pypi-fetch``                     PyPI              File delivered by instructing server to fetch from a URL
+``vnd-acmecorp-fetch``                 Acme Corp         File delivered by instructing server to fetch from a URL
+``vnd-acmecorp-postal``                Acme Corp         File delivered via postal mail
+``vnd-madscience-quantumentanglement`` Mad Science Labs  Upload via quantum entanglement
+====================================== ================  ========================================================
+
+If a server intends to match the behavior of another server's implementation, it **MAY** respond
 with that implementation's file upload mechanism name.
 
-All implementations of this PEP **MUST** implement the ``http-post-application-octet-stream`` file
-upload mechanism.
+.. _required-file-upload-mechanisms:
+
+Required File Upload Mechanisms
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``http-post-application-octet-stream``
+++++++++++++++++++++++++++++++++++++++
+
+A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the
+``http-post-application-octet-stream`` map of the ``mechanism`` map of the
+:ref:`file upload session creation response body <file-upload-session-response>` like:
+
+.. code-block:: text
+
+    Content-Type: application/octet-stream
+
+    <binary contents of the file to upload>
+
+Servers **MAY** support uploading of digital attestations for files (see :pep:`740`).
+This support will be indicated by inclusion of an ``attestations_url`` key in the
+``http-post-application-octet-stream`` map of the ``mechanism`` map of the
+:ref:`file upload session creation response body <file-upload-session-response>`.
+Attestations **MUST** be uploaded to the ``attestations_url`` before
+:ref:`file upload session completion <file-upload-session-completion>`.
+
+To upload an attestation, a client submits a ``POST`` request to the ``attestations_url``
+containing a JSON array of :pep:`attestation objects <740#attestation-objects>` like:
+
+.. code-block:: text
+
+    Content-Type: application/json
+
+    [{"version": 1, "verification_material": {...}, "envelope": {...}},...]
 
 
 Content Types
@@ -779,6 +843,9 @@ The structure of the ``Content-Type`` header for all other requests is:
 Since minor API version differences should never be disruptive, only the major version is included
 in the content type; the version number is prefixed with a ``v``.
 
+The minor API version specified in the ``.meta.api-version`` JSON key of client requests
+**MUST** match the ``Content-Type`` header for major version.
+
 Unlike :pep:`691`, this PEP does not change the existing *legacy* ``1.0`` upload API in any way, so
 servers are required to host the new API described in this PEP at a different endpoint than the
 existing upload API.
@@ -788,11 +855,6 @@ defined in this PEP **MUST** include a ``Content-Type`` header value of:
 
 - ``application/vnd.pypi.upload.v2+json``.
 
-As with :pep:`691`, a special "meta" version is supported named ``latest``, the purpose of which is
-to allow clients to request the latest version implemented by the server, without having to know
-ahead of time what that version is.  It is recommended however, that clients be explicit about what
-versions they support.
-
 Similar to :pep:`691`, this PEP also standardizes on using server-driven content negotiation to
 allow clients to request different versions or serialization formats, which includes the ``format``
 part of the content type.  However, since this PEP expects the existing legacy ``1.0`` upload API to