RPC Raw Streams for Binary Data

[tegefaulkes]

Specification

RPC methods that need to send binary data such as secrets data for the client RPC or GIT pack data for the agent RPC. This needs to make use of the raw streams in the RPC system.

Raw streams take a header message and then act as a normal binary stream. The header can include any metadata you need. The binary data can then be streamed. To ensure there are no problems with Endianness we can chunk the data into protobuf messages.

Some things to note

1. Raw streams don't have a return header message/metadata currently. It could be implemented.
2. Raw streams don't support middleware.
3. If we use protobuf messages then we need a standard way of delimiting messages in the stream.

See #2 for clarification on the usage of binary data formats like CBOR or Protobuf. It's only needed in limited usecases, and can be done as part of the RPC caller and handler.

When dropping to raw streams there should still be an exchange protocol between client and server.

The client sends a "header" message, and then should expect the server to send back a "header" message. Both should be in JSON. Only then the client can drop down to binary. However this can be just one of the ways of using it. This primarily applies to streaming: client streaming, server streaming and duplex streaming.

For example:

• In client streaming, the client sends 1 header message, then starts sending binary data. The server then sends back a JSON message (possibly at the end or at any time?)
• In server streaming, the client sends 1 header message, the server send backs 1 header message, then drops down to binary.
• In duplex streaming. The client sends a header message. The server responds with a header message. Both client and server can start sending binary data (at any time?).

It seems we should enforce that there be a header message sent, even if it is empty like a {}.

Additional context

• Related Migrate from GRPC to JSON RPC (with binary variant for higher performance) Polykey#495 (comment)
• Related Migrate from GRPC to JSON RPC (with binary variant for higher performance) Polykey#495
• Related Client RPC migration Polykey#509
• Related Agent RPC migration Polykey#512

Tasks

1. Spec out a standard way of sending raw file/binary data.
2. Update all relevant handlers to use raw streaming for sending this data.

[tegefaulkes]

As just discussed. we need to add some changes to the RPC system here.

1. for raw duplex streams we need to support the server responding with leading metadata with a JSONRPC message.
2. Nice but not immediately needed, we can support middleware for the leading metadata messages for raw duplex streams.
3. Trailing metadata is optional, but we do need a standard way of serialising binary data for the raw stream. We may need a standard way of delimiting messages as well.

Either of these points might be a new issues. TBD.

[CMCDragonkai]

Also this one before extracting out to Polykey CLI?

[tegefaulkes]

I'm only going to get to this at the trailing end of the agent migration. In fact, I may have to complete the agent migration PR before the CLI change just so we have a clean history to start from. But maybe not, CLI is reasonably separate from agent migration..

[CMCDragonkai]

Some notes for binary data formats for encapsulation:

• If we use Protobuf - https://github.com/bufbuild/protobuf-es
• If we use CBOR - https://github.com/kriszyp/cbor-x

The protobuf-es appears to support dynamic messages that don't require creating a proto file.

The creation of a proto file is sort of unnecessary in most of our API due to our usage of JSON RPC.

It would be rare for us to need protobuf. As per the comment in MatrixAI/Polykey#495 (comment) the protobuf is only needed for mixed messages (and in most cases we will use JSON with base64url) and/or chunked processing (that isn't already governed by underlying protocols like git and filesystem).

Furthermore CBOR is pure JS, it runs in the browser too. So CBOR is likely the better solution since it would be used in the DB with MatrixAI/js-db#58 and then the same dependency can be shared. CBOR though is a less popular format... so third party clients may find it more difficult.

[CMCDragonkai]

We can't be sure that endianness actually matters here. Because we would just be sending bytes.

See the discussion here: https://chat.openai.com/share/8786aa4f-53c2-4b25-95a0-c29e7223da12

This basically means for file transfers, we should not need to care about endianness. This should apply to when we are doing vault cloning and pulling, and uploading/downloading files to the vault.

It would only matter if we are sending "structured" binary data that we expect to be interpreted by a program. But even then, this structured binary data would have to be as if we were to just serialise a C struct (literally raw memory) and send it over. Most binary formats now specify a specific endianness.

[CMCDragonkai]

So yes, if we were to send 1 as a ascii or utf-8 text, it would be fine. If we were to send 1 as a uint64_t, then endianness would be a problem. So to avoid the problem we would only either send 1 as utf-8 text OR within a protobuf/cbor message.

[tegefaulkes]

Maybe the terminology is a bit weird? We don't technically downgrade the stream. What we do is take the raw binary stream and pipe it into a JSON message parser and read out a single message. When then cancel the parser stream allowing the raw stream to be used again. That raw stream is then passed to the raw handler along with the initial message that was read.

Note that we only have 1 kind of raw stream which is a duplex stream. Client, server or unary behaviour has to be re-implemented within the raw handler.

All the normal style handlers are built on-top of the raw handler. It takes the raw stream and pipes it through the middle-ware and parsing streams before providing it to the handlers re-inserting the initial message into the stream while doing so.

We need to add the feature where a raw stream can respond with leading metadata. Ideally the middle-ware is used for leading metadata. In all cases the raw handler should respond immediately with leading metadata before using the stream. This may have to be done callback style.

[CMCDragonkai]

Oh I must have forgotten then. Only the duplex stream is raw.

Well in that case, duplex stream should allow the client and server to both send a header message.

However I think the leading header message should be required... before you continue using it in binary style.

One question is that I assume that ordering constraints don't need to exist. That is the client can immediately start sending binary data after sending the header without waiting for the response. Or the client can wait for the response. It's up to the client.

The question is whether we should force the existence of the leading JSON message first from both ends?

[CMCDragonkai]

Ok we can force the server stream to send a JSON message as the leading message before it can send anything else. If it has no header to send, it can send {}. But if it has nothing at all, then it can just close the stream.