data streaming / binary files transfer

[xmonader]

Requirements

There is a new requirement to support uploading files to remote destinations. This might be problematic now with the current protocol duo to the following limitations

• The protocol is designed for short independent messages. A message usually curry a request, where it expect a single answer.
• Each messages, is signed, validated and routed separately from the previous or next messages.
• Because of the above, there is an overhead over the actual message size (metadata including destination ids, signatures, timestamps, etc...) plus the overhead of the routing, http request to remote destination, etc...

This all makes it very inefficient to uploading binary files with the current protocol specs. Since for each file, the following will have to be done

• File has to be chunked into smaller bites that each can fit into a single rmb message.
• Each message, will has to be signed, routed, and sent over http to remote twin.
• The remote twin can receive the messages out of order, duo to the async nature of the rmb protocol, multiple messages to the same destination from the same source, are not granted any order. Hence it's up to the destination to re-assemble the parts in the right order. Which can be accomplished by providing an offset in each message (another piece of metadata that need to be included in the message)
• Now because many messages are related, a new form of 're-transmission' and a way to 'ask' for missing or corrupt chunk will need to be implemented.

So it's obvious that the current protocol specifications out of the box is not good enough for this kind of transfer. Hence we discussing here all the possible "solutions" that can be implemented:

Solution 1 - The naive

This solution does not require any changes. Only requires implementing a service that runs on the remote rmb where the files need to be updated. The service will just implement 3 methods to handle

• open request, takes only file as a parameter, this file can either be path, name or id based on the requirements. This can also be empty (and the server decide what to name the file or where to store it) this is all according to the service operation and requirements.
  • the open calls then return an upload-id
• write call, takes (upload-id, "data") as arguments, the method then (using the upload-id) append the data to the current open file.
• closecall, takes (upload-id), terminates the upload. Then takes whatever action is needed on the file.
• The client, basically to do the upload need to do the following
  • call open,
  • call write repeatedly with the upload-id returned by open for each chunk of the file
    • each call to write needs to wait until it receive the call reply, this to make sure no more write calls are made until we sure the previous chunk is received and processed by the remote server
  • call close after all write call are complete.

This solution while can be implemented now with no protocol changes, is obvious very inefficient, since for each call, an entire rmb messages is created, signed, encoded, and wired over http to remote destination, where it gets verified, unwrapped, and pushed to the service for processing.

[muhamadazmy]

Solution 2 - The negotiator

This solution might have better performance, but has a worst design. Since this requires rmb to intercept calls from client/server instead of just working as a postman for messages. Rmb now need to understand the messages passing through it to know when it needs to receive an upload. (at least for the first iteration of this solution, I will try to lay down the problems)

The idea is in this design RMB will has more active rule in upload process. Both local and remote RMB need to negotiate the upload before it's established:

Requirements

• Make sure that uploading is done ONLY if there is a service to handle this upload, otherwise I can flood remote RMBs with file uploads that are not expected, or will never be handled.
• Uploads must be secured, since RMB supports HTTP only (so far) file encryption is necessary.

Implementation

• Client create an upload message for file upload. This message must has a special know command name (let's say system.upload). The content of the message need to have a specific body

{
   "service": "<remote side specific service unique for the remote service>",
   "path": "/path/to/local/file/to/upload",
}

• Once the local rmb receive this command, it needs to do 2 things
  • Validate that this local file exists
  • Call a new entrypoint on remote RMB let's say /rmb-open with message service in the request. This is need to make sure that the remote server actually has a service that is expecting a file upload. TBD
• Once upload possibility is verified, the HTTP response should have an upload-id, plus a public key for encryption, and a signature
• Rmb then follow up with a request to /rmb-upload/<upload-id> that is a normal multi-part http file upload, the file is uploaded encrypted with the key received
• Remote RMB will then accept the upload (because the upload-id is valid) and store the file locally in a known upload location configured on rmb start
• Once upload is complete. Remote RMB create a message to the local service (the one bind with that upload id) that include the local file path.

Cons of this design

• It's a mess
• It's not generic solution for streams. It will only work for file uploads.
• It's not an improvement to the current protocol to work with this kind of "messages" it's instead a parallel protocol, that has some meetings points (for example the client create an upload message, but only after it puts the file somewhere on disk)
• It wastes a lot of storage because clients (and servers) needs to fist write the files on disk before they are uploaded, same on remote side on receive. There is no way we can open an actual realtime stream (think a video broadcast)

Pros

• It's more performant than Solution 1 since file upload is done in a single http multip-part request.

[ashraffouda]

what if we have something like this
1- there is a long running server on remote rmb side
2- the client sends a message saying I'm uploading this file like normal messages
3- the remote rmb receives this message and generate in local redis
4- it returns back this id to the client
5- the client does an http upload for example and do the transfer
6- the server method that handles the upload make sure the generated id above is in the request and hence handles the request otherwise returns bad request

[muhamadazmy]

While this is a simplified version of solution 2 but it ignores the following points

• File encryption is ignored.
• RMB need to know about the encryption key, because it's the one receiving the file, not the remote service, hence imho
• the Client shouldn't be the one doing the upload, rather local RMB itself (the one on the uploader side) the main reason is that we can't allow clients to talk to remote RMB directly because this way it's gonna become spagetti design and there is not gonna be anything to prevent clients from directly contacting remote RMBs.
• I would avoid making the client/ or server take extra steps more than the normal message handling, so setting keys in redis directly special for upload does not sound like a good idea. This makes it much harder to move to a different underlying protocol since will cause client/server breaking (imagine dropping redis all together) hence I prefer that all communication between RMBs and the local services to be over the messages only.

[muhamadazmy]

Solution 3 - The New Protocol

Okay, may be it's better to take a step back and rethink the entire protocol of RMB.
We already have issues with current protocol regarding

• encryption of data since there is no way to communication encryption keys
• it's not possible atm for RMB to verify if a command or local servier are available. Hence a malicious client can flood remote rmb redis queues with commands that will never get handled. (this is not directly a protocol change but rather a local service to rmb registration which is now not existent) but would require client change that would be better to include in the next major protocol change.
• each message need to be signed, and routed separately, that is a heavy process (to sign and verify) would be better to open a trusted channel between 2 peers, than have messages traverse this trusted secure channel instead.
• Protocol doesn't support generic streams (and specific file uploads)
• There is no back pressure support. If a service is under heavy load there is no way u can tell the client to slow down (during an upload, or other kind of messages)
  • what if server redis queues are filling up, or the service went down and is not available anymore ?

The problem is during implementation of current rmb we had to stick to the original protocol specs which did not try to solve those issues, so we inherited the same issues as is. The only possible way to solve this properly is to break out from this version of the protocol specs and move to a new version. backward compatibility will not be possible anymore.

Requirement

• Support both Request/Response model AND Stream model
• Encryption to be build in the protocol itself, the client/server don't need to encrypt their own data. it's RMB that takes care of the "reliable" connection.
• Faster handling of messages (avoid un-necessary signing)

Solution.

• Peer to Peer communication can be maintained with a web-socket
  • Why web-socket and not TCP (the idea is to possibly support running RMB inside the browser with WASM)
• On first time an RMB need to talk to another peer. It initialize a web-socket connection then go through a handshake procedure where both can communicate to setup a shared encryption key. This can go as follows
  • The local rmb sends a handshake message with it's own public encryption key + signature of the key (with twin sr/ed25519 key)
  • The remote rmb then replies with a random shared key generated encrypted with the local twin public key
  • Both rmbs now share a key and all following messages (in both directions) need to use this encryption key
• Once connection is stablished, any forwarded messges (request/reply) can traverse this channel.
• Twin can maintain a configurable number of active peer connections. Stall connections, can be dropped (LRU cache can be used)

This solves the encryption issue mainly, but doesn't improve much on the other issues. but there is more

• Payload of the message doesn't have to be json anymore, msgpack can be used which is more compact, so no more over-head, or need to actually encode the payload as base64.
• On remote service start they need to register with their local RMB so RMB need to know at all time if a certain command has a handler this can be implemented by setting a shared state in redis which can timeout if the server didn't update its TTL to mark it as a live, hence if the service died, RMB will eventually know that this service is not available anymore, hence not fill up redis queues for no reason. I can reply with service not available error as soon as possible.
• For the back pressure feature, this is more tricky because on the client side all requests are put on a single redis queue (the system local queue) hence any back pressure request from a twin will slow down for all clients, not only the one that is pushing a lot of data. This need to be thought of more thoroughly, hence we will leave it out of this discussion, only if we decided to take this approach.

Now, how does this solve the issue? well it solve it by being more efficient.

• There is no short lived connections anymore, as long as the connection between the peer is used, the connection will remain. Wasted processing time is eliminated because
  • There is no signing per message
  • There is no message overhead because you don't need to specify source, desitnation, or other attributes in the message because this is maintained by the connection state (since the connection is long lived we know that this connection is associated with this source twin) hence a single message can only hold an id, command, and body (this can be combat because of use of msgpack, plus that the body data is just a byte array that is not encoded, unlike the current version
• it's then easy to implement the file upload same way as we did with solution 1, there is still some over head and is probably a little bit slower than solution 2 (need a poc to test this) but it's much more cleaner and more generic and optimal solution.

Cons

• That's not a backward compatible protocol, it involves a partial rewrite of RMB (the peer to peer layer). Client and Server compatibility can be maintained (as far as i can tell) although won't be as efficient.
• RMBs need to maintain a list of peer active connections, this can be limited by the the system and memory. Usually easy with an LRU cache to vacuum stall connections. Can be a problem for entities like the grid proxy which probably has to establish connection with many nodes at the same time (hence this need to be configurable)

Pros

• This solves, encryption issue since this is part of handshake
• The same connection is used by both RMB to forward requests, replies. Hence a reply from remote rmb doesn't need to establish another connection to the local one, instead can push it's reply over the open multiplexed connection.
• Each message is really minimized since only id, and payload now need to sent to remote server with each request.
• If RMB now start using websockets, we can implement a light version of rmb that can run in browser hence they can make direct calls to remote RMBs knowing only their twin ids. (given that the host is running yggdrasil) Hence need for the proxy will be obsolete

[maxux]

I think we can simplify stuff as well: since we use HTTP for everything, let's use HTTP headers for metadata and HTTP body for payload ? So payload can be anything (even binary) and with some offset header, we can stream.

[muhamadazmy]

Yeah, what I suggest we can do with WS can definitely be done with HTTP. While we still can agree on a shared encryption key, it would be more efficient to do that over WS and also means we can know how long the key is valid (basically as long as the connection is alive).

WS also has binary/text messages which means we can have control messages over text and binary for actual user requests, or payloads.

Still few questions to answer regarding this idea anyway:

• Do we still support multiple destinations with streams, or streams has to be single destinations (i think this will complicate stuff with no actual use case)
• How to implement back pressure in a way that doesn't slow down the entire system, but only slow twins streams (this will be even harder to design correctly with multiple destinations)
• Problem of using redis is that it provide "virtual unlimited" buffer. So a back pressure implementation will require clients to play nicely. if their sending queues are filling up, they need to know that they have to slow down. I don't like to depend on clients to have a working system because malicious clients can still cause issues.

Suggestions

• On stream open request. both RMBs knows that they now need to create a queue for that stream on both sides (so a local queue, and a remote queue)
• The local rmb will push messages from the local queue to the remote RMB wich which will fill up its counterpart queue on the remote host.
  • If remote queue is filling up, the remote rmb need to push a control "slow down" message to the local rmb.
  • the local rmb slow down the pushing of local messages (increase delays between pushes)
  • the local client sees that it's upstream queue is filling up, so it need to slow down as well.
• client need to push EOF message as well to terminate the stream.
• Using separate queues for each stream will solve the "out of order" delivery of parts. since only one worker will be responsible for each stream.

This can also be implemented with HTTP as i mentioned, there is no "need" to do this via WS except having a bidirectional, and multiplexed connections (multiple requests, streams can basically go over the same connection)

Issues

• What if we lost a message, or connection was interrupted. How this going to affect a stream

[muhamadazmy]

A good sdk can abstract stream access to a Reader interface or trait.

[muhamadazmy]

The reasons I prefer moving to WS:

• One connection per peer which is multiplexed and bidirectional. regardless which peer initialized the connection, there will be no "client/server" (unlike http)
  • because it's multiplexed, multiple messages are delivered on one connection, same for file uploads and streams.
  • RMBs can have "negotiation" to communicate other things than user messages. Like encryption keys and transfer speed for streams. Which is not possible with normal HTTP.
• It's long lived, so avoid all normal connection initialization with normal HTTP (I know you can use keep-alive) with http but it's up to the http library implementation to decide if to reuse the connection or create another one.
• It's probably more challenging to implement :)

[despiegk]

dear friends, we did not ask for a redesign at this stage, we don't have the time for it
we need a quick solution for getting files from one side to the other

performance at this stage is not the main concern

main requirements are

• how fast can we develop it
• can we make sure that the interface at client side will be unchanged even if we change protocol later
• it needs to be just good enough for our requirements
• keep it all incredibly simple so its easy to fix
• its designed to be very peer2peer, its not client server, which means our communication is going to many many destinations and is chatty, its not for long running streams
• its designed to fire & forget, you send, the rmb keeps on resending, this to work around limits in bad network environments, reliability and simplicity where much more important in the current design compared to performance

some remarks

• the original suggested spec is very opinionated don't use words like obviously, ... not its not obviously because we are not taking all requirements as mentioned above in consideration
• overhead is not a good argument: 200 bytes on 1 MB is nothing
• even encryption overhead is nothing a 1 MB block, encryption overhead is +- nothing in relation to network speed over WAN
• actually fire & forget approach on file transfer might be faster because more parallel sending with little bit of smartness

if we make it a permanent connection we get other issues (websockets or tcp)

• websockets are known to be blocked or behave bad in many parts of the world, firewalls drop them
• long running tcp sockets same, lots of work needs to be done to keep them alive and put all kinds of mechanisms in to have fall back scenario's and often also blocked (firewalls again)
• tcp is known to become really slow for big file uploads on heavy latency lines, fire & forget chunks can be much faster at the end because you can have many more in parallel
• to do a long running network process and then use symmetric encryption we are basically re-implementing SSL and there are quite some known issues with it (key exchange is an issue), I see your suggestion which might be ok
• we are saying there is no way to exchange encryption keys, exactly that is the full point
• a design with keeping channels on websockets will be a heavy job to get it to work reliably because we are way more peer2peer as normal client server, so there will be many websockets open per twin, this will require lots of management and will be difficult to get it right especially in line to though network connections

[maxux]

The only way which come in my mind right now, to get it quickly and easily (a bit like naive solution): chunk the file and send parts as base64. But this introduce a minimum 40% overhead (1 MB file will be 1.4 MB). That's a hard limitation by json with current implementation :/ Not really efficient on slow network or bad network connectivity, but acceptable ?

[despiegk]

supporting json & binary is probably a good add on and should not have to change much, because indeed base64 we don't like
should be easy to internally use e.g. protobuf for the base message, part of it is base64 and can even use some light compression on the binary part = payload

schema e.g. protobuf to have even less overhead

• the metadata as we require, please point me to the specs (one is a bool to specify if binary or json)
• binary or json payload

[muhamadazmy]

Okay, I agree with both of you. I think we can easily switch to binary protocol between RMBs, to avoid the base64 overhead. Then we can work on the improvement later.

Chunking the file and sending each chunk separately is already one of the suggested solution.

[despiegk]

if we wanna re-design we need to use different requirements

• difference between delivery & processing (how can we know as sender it has been processed & committed rather than received)
• how can we deal with super bad connectivity (lots of packet loss, super high latency, ...)
• how can we deal with bad firewalls
• how can we deal with re-transmits better, need more reliability
• how can we do 1 to multi (so I wanna send a message to 100 destinations)
• how can we work around limitations of tcp (the retransmit issue)

ofcourse there are things we could do more efficiently like symmetric encryption but I don't think the cost right now is worth the hastle

the current ask can be done very easily, if we don't overengineer

[muhamadazmy]

While WS sounds better in theory it's indeed not as good in practice specially in a P2P envirotment. I was trying to work on a poc to see how good can we use it and it indeed wasn't as clean to implement/use in the system as I hoped.

I agree with u to stick to the current design except improving the message format to support binary natively.

[muhamadazmy]

This can be closed now since it's not relevant anymore to current system since rmb has been rewritten to phase out relying on yggdrasil. The new protocol is binary in nature (between peers) which can support efficient file uploads.