Skip to content

Commit 1b555b1

Browse files
vgonkivsvgonkivs
authored
Shrex client server specification (#4665)
1 parent 533c69c commit 1b555b1

File tree

1 file changed

+268
-0
lines changed

1 file changed

+268
-0
lines changed

‎specs/src/shrex/client-server.md‎

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
# SHREX Client/Server Specification
2+
3+
## Abstract
4+
5+
This specification defines a request-response protocol that enables nodes to retrieve data from peers. The protocol provides a simple interface for requesting specific data and receiving responses.
6+
7+
## Overview
8+
9+
It implements a client-server architecture for data exchange:
10+
11+
- **SHREX Client**: Makes requests to retrieve data from a specific peer
12+
- **SHREX Server**: Responds to requests by serving data from local storage
13+
14+
## Transport Protocol
15+
16+
It uses libp2p streaming as its transport layer.
17+
18+
### Protocol ID
19+
20+
The protocol ID follows the format:
21+
22+
```text
23+
/<network-id>/shrex/v0.1.0/<request-name>
24+
```
25+
26+
Where:
27+
28+
- `<network-id>`: Network identifier
29+
- `/shrex/v0.1.0/`: Protocol name and version
30+
- `<request-name>`: Type of request being made (e.g., "row", "sample", "eds")
31+
32+
### Stream Lifecycle
33+
34+
1. **Stream Opening**: Client opens a new stream to the server using the protocol ID
35+
2. **Request Phase**: Client writes request and closes the write side
36+
3. **Response Phase**: Server sends status message
37+
- If status is `OK`: Server sends data payload, then closes stream
38+
- If status is not `OK`: Server closes stream immediately
39+
4. **Stream Closing**: Stream is closed after response is complete
40+
41+
### Timeouts
42+
43+
Both client and server implement timeouts to protect resources:
44+
45+
- **Read Timeout**: Maximum time to read data from stream
46+
- **Write Timeout**: Maximum time to write data to stream
47+
- **Handle Request Timeout**: Maximum time for server to process request
48+
49+
## Message Format
50+
51+
### Encoding
52+
53+
Request and response messages use different encoding schemes:
54+
55+
- **Request Messages**: Binary encoded using `MarshalBinary()` method
56+
- **Response Messages**: Protocol Buffers (protobuf) encoded
57+
58+
### Request Messages
59+
60+
Requests use shwap id and associated encoding. Each request type includes:
61+
- Height: Block height for the requested data
62+
- Type-specific parameters
63+
64+
### Response Messages
65+
66+
#### Status Message
67+
68+
After handling requests server MUST send Status message indicating result of handling
69+
70+
```protobuf
71+
enum Status {
72+
INVALID = 0; // Invalid/unknown status
73+
OK = 1; // Data found and will be sent
74+
NOT_FOUND = 2; // Requested data not found
75+
INTERNAL = 3; // Internal server error
76+
}
77+
78+
message Response {
79+
Status status = 1;
80+
}
81+
```
82+
83+
#### Data Payload
84+
85+
Data is only sent when status is `OK`. After sending the OK status, the server streams the requested data and then closes the stream.
86+
87+
For all other status codes (`NOT_FOUND`, `INTERNAL`), the server closes the stream immediately after sending the status message without sending any data.
88+
89+
The format and encoding of the data payload is defined in the [SHWAP](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-019.md) specification.
90+
91+
For certain request types (e.g., GetNamespaceData), a non-inclusion proof may be sent instead of data when the namespace is not present.
92+
93+
## Supported Endpoints
94+
95+
SHREX supports multiple endpoint types for retrieving different kinds of data. All endpoints follow the same request-response pattern but return different data structures.
96+
97+
All ProtocolIDs MUST be prefixed with the `networkID`.
98+
99+
| ProtocolID | Request | Response |
100+
|--------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
101+
| `<networkID>/shrex/v0.1.0/row_v0` | [RowID](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-019.md#rowid) | [Row](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-019.md#row-container) |
102+
| `<networkID>/shrex/v0.1.0/sample_v0` | [SampleID](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-019.md#sampleid) | [Sample](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-019.md#sample-container) |
103+
| `<networkID>/shrex/v0.1.0/nd_v0` | [NamespaceDataID](https://github.com/celestiaorg/celestia-node/blob/2c991a8cbb3be9afd413472e127b0e2b6e770100/share/shwap/namespace_data_id.go#L21C1-L26C2) | [NamespaceData](https://github.com/celestiaorg/celestia-node/blob/2c991a8cbb3be9afd413472e127b0e2b6e770100/share/shwap/namespace_data.go#L20) |
104+
| `<networkID>/shrex/v0.1.0/eds_v0` | [EdsID](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-019.md#edsid) | [Eds](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-019.md#eds-container) |
105+
| `<networkID>/shrex/v0.1.0/rangeNamespaceData_v0` | [RangeNamespaceDataID](https://github.com/celestiaorg/celestia-node/blob/2c991a8cbb3be9afd413472e127b0e2b6e770100/share/shwap/range_namespace_data_id.go#L31-L37) | [RangeNamespaceData](https://github.com/celestiaorg/celestia-node/blob/2c991a8cbb3be9afd413472e127b0e2b6e770100/share/shwap/range_namespace_data.go#L38-L42) |
106+
107+
## Client
108+
109+
The SHREX client is responsible for:
110+
111+
- Connecting to a specified peer
112+
- Sending a data request
113+
- Receiving and validating the status response
114+
- Reading the data payload (if status is OK)
115+
- Returning the result to the caller
116+
117+
### Input and Output
118+
119+
**Input**:
120+
121+
- Target peer ID
122+
- Request parameters (height, type-specific data)
123+
124+
**Output**:
125+
126+
- Retrieved data (on success)
127+
- Error information (on failure)
128+
129+
### Request Flow
130+
131+
```text
132+
┌──────────┐ ┌────────────┐ ┌────────┐
133+
│ Getter │ │ Client │ │ Server │
134+
└────┬─────┘ └─────┬──────┘ └───┬────┘
135+
│ │ │
136+
│ Request(peer, data) │ │
137+
├────────────────────>│ │
138+
│ │ │
139+
│ │ Open Stream │
140+
│ ├───────────────────>│
141+
│ │ │
142+
│ │ Send Request │
143+
│ ├───────────────────>│
144+
│ │ │
145+
│ │ Close Write │
146+
│ ├───────────────────>│
147+
│ │ │
148+
│ │ Receive Status │
149+
│ │<───────────────────┤
150+
│ │ │
151+
│ │ Receive Data │
152+
│ │ (if status OK) │
153+
│ │<───────────────────┤
154+
│ │ │
155+
│ Return Data/Error │ │
156+
│<────────────────────┤ │
157+
│ │ │
158+
```
159+
160+
### Error Handling
161+
162+
The client maps status codes to errors:
163+
164+
| Status Code | Error | Description |
165+
|------------|-------|-------------|
166+
| `OK` | None | Success, data received |
167+
| `NOT_FOUND` | `ErrNotFound` | Requested data not available |
168+
| `INTERNAL` | `ErrInternalServer` | Server encountered an error |
169+
| `INVALID` or unknown | `ErrInvalidResponse` | Invalid or unexpected status |
170+
171+
Additional client-side errors:
172+
173+
- **Connection Errors**: Stream opening failures
174+
- **Timeout Errors**: Request exceeded time limits
175+
- **Rate Limit Errors**: Indicated by EOF when reading status (server closed stream without response)
176+
177+
## Server
178+
179+
The SHREX server is responsible for:
180+
181+
- Accepting incoming connections
182+
- Reading and validating requests
183+
- Retrieving data from local storage
184+
- Sending status responses
185+
- Sending data payloads (when applicable)
186+
- Sending non-inclusion proofs (for certain request types when data not found)
187+
188+
### Request Handling
189+
190+
```text
191+
Incoming Stream
192+
│
193+
▼
194+
┌──────────────┐
195+
│ Read Request │
196+
└──────┬───────┘
197+
│
198+
┌───────┴────────┐
199+
│ │
200+
Error Success
201+
│ │
202+
▼ ▼
203+
┌───────────┐ ┌─────────────┐
204+
│ Reset │ │ Close Read │
205+
│ Stream │ │ │
206+
└───────────┘ └──────┬──────┘
207+
│
208+
▼
209+
┌──────────────┐
210+
│ Validate │
211+
│ Request │
212+
└──────┬───────┘
213+
│
214+
┌───────┴────────┐
215+
│ │
216+
Invalid Valid
217+
│ │
218+
▼ ▼
219+
┌──────────┐ ┌─────────────┐
220+
│ Reset │ │ Get Data │
221+
│ Stream │ │ from Store │
222+
└──────────┘ └──────┬──────┘
223+
│
224+
▼
225+
┌──────────────┐
226+
│ Set Write │
227+
│ Deadline │
228+
└──────┬───────┘
229+
│
230+
┌───────┴────────┐
231+
│ │
232+
Found Not Found
233+
│ │
234+
▼ ▼
235+
┌──────────┐ ┌──────────┐
236+
│ Send OK │ │ Send NOT │
237+
│ Status │ │ FOUND │
238+
└────┬─────┘ │ Status │
239+
│ └──────────┘
240+
▼
241+
┌──────────┐
242+
│ Send │
243+
│ Data │
244+
└──────────┘
245+
```
246+
247+
### Status Response Behavior
248+
249+
The server sends status codes based on the following conditions:
250+
251+
| Status Code | Condition | Stream Behavior |
252+
|------------|-----------|-----------------|
253+
| `OK` | Data found and ready | Send status, then send data |
254+
| `NOT_FOUND` | Data not in store | Send status, close stream |
255+
| `INTERNAL` | Error retrieving data | Send status, close stream |
256+
| N/A (no status sent) | Invalid request | Reset stream without response |
257+
| N/A (no status sent) | Error reading request | Reset stream without response |
258+
259+
## Resource Protection
260+
261+
The protocol includes mechanisms to protect node resources:
262+
263+
- **Timeouts**: Requests have time limits to prevent resource exhaustion
264+
- **Validation**: Invalid requests are rejected early to save resources
265+
266+
## Requirements Language
267+
268+
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

0 commit comments

Comments
 (0)