ipip: Streaming Routing V1 HTTP API

Author: hacdias

src/ipips/ipip-0410.md

@@ -0,0 +1,83 @@
+---
+title: "IPIP-0410: Streaming Routing V1 HTTP API"
+date: 2023-05-12
+ipip: proposal
+editors:
+  - name: Henrique Dias
+    github: hacdias
+    url: https://hacdias.com/
+relatedIssues:
+  - https://github.com/ipfs/specs/issues/344
+  - https://github.com/ipfs/boxo/pull/18
+  - https://github.com/ipfs/kubo/pull/9868
+  - https://github.com/ipfs/kubo/pull/9874
+order: 410
+tags: ['ipips']
+---
+
+Introduce backwards-compatible streaming support to the Routing V1 HTTP API.
+For this, we use the `Accept` HTTP header (:cite[rfc9110]) for content type negotiation, as well
+as the Newline Delimited JSON ([NDJSON]) format.
+
+## Motivation
+
+The main motivation for this change is to allow servers to respond faster to the
+client with provider records, as soon as they are available. In the current state,
+the client requests a list of providers for a CID from the server. Then, the client
+has to wait for the server to collect their final list of providers. After that,
+the server can respond with the full list of providers.
+
+With streaming support, the server is able to respond with provider records as soon
+as they are available. This reduces latency and allows for faster content discovery.
+
+In addition, streaming responses may produce an unlimited amount of results, which
+is not the case for non-streamed responses.
+
+## Detailed Design
+
+In summary, streaming is supported by using the `Accept` HTTP header, which is used
+for content type negotiation as described in :cite[rfc9110]. The client sends an
+`Accept` HTTP header starting with `application/x-ndjson`, which is the content
+type for [NDJSON]. The following happens:
+
+- The client adds the `Accept` HTTP header in the request starting with `application/x-ndjson`.
+- The server checks the `Accept` HTTP header from the request and, if it contains
+`application/x-ndjson`, they reply with NDJSON. If they don't support NDJSON, they
+can reply with JSON.
+- The server response MUST contain a `Content-Type` HTTP header indicating the
+response type, which may be either `application/json` for non-streaming responses,
+and `application/x-ndjson` for streamed responses.
+
+For more details regarding the design, check :cite[http-routing-v1].
+
+## Design Rationale
+
+This feature is designed such that it does not break compatibility with existing
+clients and servers. The `Accept` HTTP header is OPTIONAL. By default, the server
+MUST respond with `application/json` unless the client explicitly asked for
+`application/x-ndjson`. If the server does not support NDJSON, it is allowed
+to still respond with non-streamed JSON.
+
+### User Benefit
+
+Users (clients) will benefit from this change as the servers will now be able
+to respond more promptly to provider record requests. Instead of waiting for the whole
+list to be constructed, servers can now return each provider record one by one,
+in a streaming fashion.
+
+### Compatibility
+
+The introduced changes are backwards-compatible. The introduced header is completely
+optional, and a server that does not support streaming is able to respond with a non-streaming
+response to the client. Equally, non-streaming responses are the default. Therefore, a
+client that does not support streaming will not receive a streamed response.
+
+### Security
+
+Security considerations are equivalent as the ones in :cite[ipip-0337].
+
+### Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
+
+[NDJSON]: http://ndjson.org/

src/routing/http-routing-v1.md

@@ -11,6 +11,9 @@ editors:
     github: guseggert
   - name: Masih H. Derkani
     github: masih
+  - name: Henrique Dias
+    url: https://hacdias.com/
+    github: hacdias
 xref:
   - ipns-record
 order: 0
@@ -139,7 +142,23 @@ This API does not support pagination, but optional pagination can be added in a
 
 ## Streaming
 
-This API does not currently support streaming, however it can be added in the future through a backwards-compatible update by using a content type other than `application/json`.
+JSON-based endpoints support streaming requests made by sending an `Accept` HTTP Header containing
+`application/x-ndjson`. The response will be formatted as [Newline Delimited JSON (ndjson)](https://github.com/ndjson/ndjson-spec),
+with one *read provider record* per line:
+
+
+```json
+{"Protocol": "<protocol_name>", "Schema": "<schema>", ...}
+{"Protocol": "<protocol_name>", "Schema": "<schema>", ...}
+{"Protocol": "<protocol_name>", "Schema": "<schema>", ...}
+...
+```
+
+Streaming is backwards-compatible with clients that do not support streaming. Please note the following:
+
+- Requests without an `Accept` header MUST default to regular, non-streaming, responses.
+- The server MAY respond with non-streaming response even if the client requests streaming.
+- The server MUST NOT respond with streaming response if the client did not request so.
 
 ## Error Codes