ipip-328: final editorial pass

Author: lidel

IPIP/0328-gateway-json-cbor-response-format.md

@@ -19,16 +19,16 @@ Currently, the gateway supports requesting data in the [DAG-PB], RAW, [CAR] and
 TAR formats. In addition, it allows for traversing of links encoded through CBOR
 Tag 42, as long as they are intermediate links, and not the final document.
 It works on both DAG-CBOR, and its JSON representation, DAG-JSON. However, it
-should be possible to download deserialized versions of data other than UnixFS,
-such as JSON and CBOR.
+should be possible to download deserialized versions of the final JSON/CBOR document
+in raw format (not wrapped in UnixFS).
 
 The main functional gap in the IPFS ecosystem is the lack of support for
 non-UnixFS DAGs on HTTP gateways. Users are able to create custom DAGs based on
 traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs][cbor-42]
 and DAG-JSON documents, but they are unable to load deserialized documents from
 a local gateway, which is severely decreasing the utility of non-UnixFS DAGs.
 
-Adding new responses types will also benefit UnixFS. DAG-PB has a
+Adding JSON and CBOR response types will also benefit UnixFS. DAG-PB has a
 [logical format][dag-pb-format] which makes it possible to represent a DAG-PB
 directory as a [DAG-JSON] document. This means that, if we support DAG-JSON in
 the gateway, then we would support
@@ -59,22 +59,22 @@ failing with `node type unknown`.
 
 The current gateway already supports different response formats via the
 `Accept` HTTP header and the `format` URL query. This IPIP proposes adding
-more supported formats to that list.
+JSON and CBOR formats to that list.
 
 In addition, the current gateway already supports traversing through DAG-CBOR
 and DAG-JSON links if they are intermediary documents. With this IPIP, we aim
 to be able to download the DAG-CBOR, DAG-JSON, JSON and CBOR documents
-themselves.
+themselves, with correct `Content-Type` headers.
 
 ### User benefit
 
 The user benefits from this change as they will now be able to retrieve
 content encoded in the traversable DAG-JSON and DAG-CBOR formats. This is
 something that has been [requested before][ipfs/go-ipfs/issues/7552].
 
-In addition, both UX and DX are significantly improved, since every block can
+In addition, both UX and DX are significantly improved, since every UnixFS directory can
 now be inspected in a regular web browser via `?format=json`. This can remove the
-need for additional tools, decoders, and/or libraries.
+need for parsing HTML with directory listing.
 
 ### Compatibility
 
@@ -99,6 +99,8 @@ strict enough:
 
 ### Alternatives
 
+#### Why four content types?
+
 If we do not introduce  DAG-JSON, DAG-CBOR, JSON and CBOR  response formats in
 the gateway, the usage of IPFS is constricted to files and directories represented
 by UnixFS (DAG-PB) codec. Therefore, if a user wants to store JSON and/or CBOR
@@ -110,7 +112,10 @@ supporting the generic variants, JSON and CBOR, would lead to poor UX. The
 ability to retrieve DAG-JSON as `application/json` is an important step
 for the interoperability of the HTTP Gateway with web browsers and other tools
 that expect specific Content Types. Namely, `Content-Type: application/json` with
-`Content-Disposition: inline` allows for JSON preview to be rendered in a web browser.
+`Content-Disposition: inline` allows for JSON preview to be rendered in a web browser
+and webdev tools.
+
+#### Why JSON/CBOR pathing is limited to full blocks?
 
 Finally, we considered supporting pathing within both DAG and non-DAG variants
 of the JSON and CBOR codecs. Pathing within these documents could lead to responses

http-gateways/PATH_GATEWAY.md

@@ -186,8 +186,8 @@ For example:
 - [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a root item which name is the same as the requested CID. Produces 400 Bad Request for content that is not UnixFS.
 - [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/). If the requested CID already has `dag-json` (0x0129) codec, data is validated as DAG-JSON before being returned as-is. Invalid DAG-JSON produces HTTP Error 500.
 - [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/). If the requested CID already has `dag-cbor` (0x71) codec,  data is validated as DAG-CBOR before being returned as-is. Invalid DAG-CBON produces HTTP Error 500.
-- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec already is `json` (0x0200). Then, the raw JSON block can be returned.
-- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec already is `cbor` (0x51). Then, the raw CBOR block can be returned.
+- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec already is `json` (0x0200). Then, the raw JSON block can be returned as-is without any conversion.
+- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec already is `cbor` (0x51). Then, the raw CBOR block can be returned as-is without any conversion.
 
 ### `Range` (request header)
 
@@ -590,32 +590,36 @@ A good practice is to always return it with HTTP error [status codes](#response-
 ## Response Payload
 
 Data sent with HTTP response depends on the type of the requested IPFS resource, and the requested response type.
-The following response formats are selected according to the codec of the resolved CID:
+
+By default, implicit deserialized response type is based on `Accept` header and the codec of the resolved CID:
 
 - UnixFS, either `dag-pb` (0x70) or `raw` (0x55)
-  - File
-    - Bytes representing file contents
+  - File or `raw` block
+    - Bytes representing file/block contents
+    - When `Range` is present, only the requested byte range is returned.
   - Directory
     - Generated HTML with directory index (see [additional notes here](#generated-html-with-directory-index))
-    - When `index.html` is present, gateway can skip generating directory index and return it instead
+    - When `index.html` is present, gateway MUST skip generating directory index and return content from `index.html` instead.
 - JSON (0x0200)
   - Bytes representing a JSON file, see [application/json](https://www.iana.org/assignments/media-types/application/json).
+  - Works exactly the same as `raw`, but returned `Content-Type` is `application/json`
 - CBOR (0x51)
   - Bytes representing a CBOR file, see [application/cbor](https://www.iana.org/assignments/media-types/application/cbor)
+  - Works exactly the same as `raw`, but returned `Content-Type` is `application/cbor`
 - DAG-JSON (0x0129)
-  - If the `Accept` header includes `text/html`: generated HTML with options to download DAG-JSON as-is, or converted to DAG-CBOR.
-  - Bytes representing a DAG-JSON file, see [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json)
+  - If the `Accept` header includes `text/html`, implementation should return a generated HTML with options to download DAG-JSON as-is, or converted to DAG-CBOR.
+  - Otherwise, response works exactly the same as `raw` block, but returned `Content-Type` is [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json)
 - DAG-CBOR (0x71)
-  - If the `Accept` header includes `text/html`: generated HTML with options to download DAG-CBOR as-is, or converted to DAG-JSON.
-  - Bytes representing a DAG-CBOR file, see [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor)
+  - If the `Accept` header includes `text/html`: implementation should return a generated HTML with options to download DAG-CBOR as-is, or converted to DAG-JSON.
+  - Otherwise, response works exactly the same as `raw` block, but returned `Content-Type` is [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor)
 
-The following response types are opt-in, can only be requested with [`format`](#format-request-query-parameter) query parameter or [`Accept`](#accept-request-header) header:
+The following response types require an explicit opt-in, can only be requested with [`format`](#format-request-query-parameter) query parameter or [`Accept`](#accept-request-header) header:
 
-- Raw Block
+- Raw Block (`?format=raw`)
   - Opaque bytes, see [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw).
-- CAR
+- CAR (`?format=car`)
   - Arbitrary DAG as a verifiable CAR file or a stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car).
-- TAR
+- TAR (`?format=tar`)
   - Deserialized UnixFS files and directories as a TAR file or a stream, see [IPIP-288](https://github.com/ipfs/specs/pull/288)
 
 # Appendix: notes for implementers
@@ -642,7 +646,7 @@ and [DNSLINK_GATEWAY.md](./DNSLINK_GATEWAY.md)).
 
 ### Traversing remaining path
 
-After the content route is found, the remaining of the path should be traversed
+After the content root CID is found, the remaining of the path should be traversed
 and resolved. Depending on the data type, that may occur through UnixFS pathing,
 or DAG-JSON, and DAG-CBOR pathing.
 
@@ -666,7 +670,7 @@ namespace, see [specification](https://ipld.io/specs/codecs/dag-json/spec/#links
 - DAG-CBOR: links are tagged with CBOR tag 42, indicating that they encode a CID,
 see [specification](https://ipld.io/specs/codecs/dag-cbor/spec/#links).
 
-Note: pathing into [IPLD Kind](https://ipld.io/docs/data-model/kinds/) other than Link (CID) is not supported at the moment. Implementations should return HTTP 501 Not Implemented when fully resolved content path has any remainder left.  This feature may be specified in a future IPIP that introduces data onboarding  and [IPLD Patch](https://ipld.io/specs/patch/) semantics.
+Note: pathing into [IPLD Kind](https://ipld.io/docs/data-model/kinds/) other than Link (CID) is not supported at the moment. Implementations should return HTTP 501 Not Implemented when fully resolved content path has any remainder left.  This feature may be specified in a future [IPIP that introduces data onboarding](https://github.com/ipfs/in-web-browsers/issues/189)  and [IPLD Patch](https://ipld.io/specs/patch/) semantics.
 
 ### Handling traversal errors
 
@@ -729,8 +733,14 @@ The usual optimizations involve:
 - Skipping size and type resolution for child UnixFS items, and using `Tsize`
   from [logical format][dag-pb-format] instead, allows gateway to respond much
   faster, as it no longer need to fetch root nodes of child items.
-  - Additional information about child nodes can be fetched lazily
-    with JS, but only for items in the browser's viewport.
+  - Instead of showing "file size" GUIs should show "IPFS DAG size". This
+    remains useful for quick inspection, but does not require fetching child
+    blocks, making directory listing fast, even with tens of thousands of
+    blocks.  Example with 10k items:
+    `bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm`.
+  - Additional information about child nodes, such as exact file size without
+    DAG overhead, can be fetched lazily with JS, but only for items in the
+    browser's viewport.
 
 - Alternative approach is resolving child items, but providing pagination UI.
   - Opening a big directory can return HTTP 302 to the current URL with