Skip to content

Commit 383fd01

Browse files
lidellidel
authored
Merge pull request #410 from /issues/344
IPIP-410: Streaming Routing V1 HTTP API
2 parents b870ad5 + e55e319 commit 383fd01

File tree

2 files changed

+127
-2
lines changed

2 files changed

+127
-2
lines changed

‎src/ipips/ipip-0410.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
---
2+
title: "IPIP-0410: Streaming NDJSON in Routing HTTP API"
3+
date: 2023-05-12
4+
ipip: proposal
5+
editors:
6+
- name: Henrique Dias
7+
github: hacdias
8+
url: https://hacdias.com/
9+
relatedIssues:
10+
- https://github.com/ipfs/specs/issues/344
11+
- https://github.com/ipfs/boxo/pull/18
12+
- https://github.com/ipfs/kubo/pull/9868
13+
- https://github.com/ipfs/kubo/pull/9874
14+
order: 410
15+
tags: ['ipips']
16+
---
17+
18+
## Summary
19+
20+
Introduce backwards-compatible streaming support to the Routing V1 HTTP API.
21+
For this, we use the `Accept` HTTP header (:cite[rfc9110]) for content type negotiation, as well
22+
as the Newline Delimited JSON ([NDJSON]) format.
23+
24+
## Motivation
25+
26+
The main motivation for this change is to allow servers to respond faster to the
27+
client with provider records, as soon as they are available. In the current state,
28+
the client requests a list of providers for a CID from the server. Then, the client
29+
has to wait for the server to collect their final list of providers. After that,
30+
the server can respond with the full list of providers.
31+
32+
This is a big source of latency when `/routing/v1` is used for delegating DHT lookups,
33+
where the client is forced to wait for the server to finish DHT walk.
34+
35+
With streaming support, the server is able to respond with provider records as soon
36+
as they are available. This reduces latency and allows for faster content discovery.
37+
38+
In addition, streaming responses may produce an unlimited amount of results, which
39+
is not the case for non-streamed responses.
40+
41+
## Detailed Design
42+
43+
In summary, streaming is supported by using the `Accept` HTTP header, which is used
44+
for content type negotiation as described in :cite[rfc9110]. The client sends an
45+
`Accept` HTTP header starting with `application/x-ndjson`, which is the content
46+
type for [NDJSON]. The following happens:
47+
48+
- The client adds the `Accept` HTTP header in the request starting with `application/x-ndjson`.
49+
- The server checks the `Accept` HTTP header from the request and, if it contains
50+
`application/x-ndjson`, they reply with NDJSON. If they don't support NDJSON, they
51+
can reply with JSON.
52+
- The server response MUST contain a `Content-Type` HTTP header indicating the
53+
response type, which may be either `application/json` for non-streaming responses,
54+
and `application/x-ndjson` for streamed responses.
55+
56+
For more details regarding the design, check :cite[http-routing-v1].
57+
58+
## Design Rationale
59+
60+
This feature is designed such that it does not break compatibility with existing
61+
clients and servers. The `Accept` HTTP header is OPTIONAL. By default, the server
62+
MUST respond with `application/json` unless the client explicitly asked for
63+
`application/x-ndjson`. If the server does not support NDJSON, it is allowed
64+
to still respond with non-streamed JSON.
65+
66+
### User Benefit
67+
68+
Users (clients) will benefit from this change as the servers will now be able
69+
to respond more promptly to provider record requests. Instead of waiting for the whole
70+
list to be constructed, servers can now return each provider record one by one,
71+
in a streaming fashion.
72+
73+
The client will be able to close connection at any time, reducing load on both ends.
74+
75+
The main use cases for this IPIP are light clients and services which want to
76+
delegate DHT lookups to external service. With streaming, clients will be able
77+
to receive results as soon the delegated service learns about new record, which
78+
directly impacts the content load speeds perceived by the end user.
79+
80+
### Compatibility
81+
82+
The introduced changes are backwards-compatible. The introduced header is completely
83+
optional, and a server that does not support streaming is able to respond with a non-streaming
84+
response to the client. Equally, non-streaming responses are the default. Therefore, a
85+
client that does not support streaming will not receive a streamed response.
86+
87+
### Security
88+
89+
Security considerations are equivalent as the ones in :cite[ipip-0337].
90+
91+
### Copyright
92+
93+
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
94+
95+
[NDJSON]: http://ndjson.org/

‎src/routing/http-routing-v1.md‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,16 @@ editors:
1111
github: guseggert
1212
- name: Masih H. Derkani
1313
github: masih
14+
- name: Henrique Dias
15+
url: https://hacdias.com/
16+
github: hacdias
1417
xref:
1518
- ipns-record
1619
order: 0
1720
tags: ['routing']
1821
---
1922

20-
Delegated routing is a mechanism for IPFS implementations to use for offloading content routing and naming to another process/server. This specification describes an HTTP API for delegated content routing.
23+
Delegated routing is a mechanism for IPFS implementations to use for offloading content routing and naming to another process/server. This specification describes a vendor-agnostic HTTP API for delegated content routing.
2124

2225
## API Specification
2326

@@ -144,7 +147,34 @@ This API does not support pagination, but optional pagination can be added in a
144147

145148
## Streaming
146149

147-
This API does not currently support streaming, however it can be added in the future through a backwards-compatible update by using a content type other than `application/json`.
150+
JSON-based endpoints support streaming requests made
151+
with `Accept: application/x-ndjson` HTTP Header.
152+
153+
Steaming responses are formatted as
154+
[Newline Delimited JSON (ndjson)](https://github.com/ndjson/ndjson-spec),
155+
with one result per line:
156+
157+
```json
158+
{"Schema": "<schema>", ...}
159+
{"Schema": "<schema>", ...}
160+
{"Schema": "<schema>", ...}
161+
...
162+
```
163+
164+
:::note
165+
166+
Streaming is opt-in and backwards-compatibile with clients and servers that do
167+
not support streaming:
168+
169+
- Requests without the `Accept: application/x-ndjson` header MUST default to
170+
regular, non-streaming, JSON responses.
171+
- Legacy server MAY respond with non-streaming `application/json` response even
172+
if the client requested streaming. It is up to the client to inspect
173+
the `Content-Type` header before parsing the response.
174+
- The server MUST NOT respond with streaming response if the client did not
175+
explicitly request so.
176+
177+
:::
148178

149179
## Error Codes
150180

0 commit comments

Comments
 (0)