Cloudflare websocket error "Message is too large"

[jakubriedl]

Describe the bug

When synchronizing data over websockets to Cloudflare durable objects and with larger store it hits a limit of message size causing silent fail of sync. There may be many situations when the message gets over 1mb, we're experiencing it with messages 6 = GetCellDiff.

from CF docs:

Note
WebSocket messages received by a Worker have a size limit of 1 MiB (1048576). If a larger message is sent, the WebSocket will be automatically closed with a 1009 "Message is too large" response.

I've validated that by logging ws.onclose(console.log)

Probable solution to that is to slice the messages in smaller pieces which not only can resolve this issue but probably also speedup the sync on initial init.

I can look into that in few days, just listing here for visibility.

Your Example Website or App

No response

Steps to Reproduce the Bug or Issue

No response

Expected behavior

No response

Screenshots or Videos

No response

Platform

• OS: [e.g. macOS, Windows, Linux]
• Browser: [e.g. Chrome, Safari, Firefox]
• Version: [e.g. 91.1]

Additional context

No response

[jakubriedl]

I've rolled out a patched version of createWsSynchronizer and durable object (we already had our own) to fix this issue. The relevant code snippet can be found here.

What it does is

• adjust message format to include partition number and total count ${actorId}\n${requestId}\n${index + 1}/${chunks.length}\n${messageType}\n${chunk}
• when creating/serializing message it actually creates array of messages with individual chunks (the header is always the same)
• on the other end it buffers messages based on message id and once it has all of them JSON.parse(chunks.join(""))

I'll try to find time to port the changes to this repo and it's unique style. In the meantime if anyone has questions about details I'm happy to answer them here

[jamesgpearce]

Ok very interesting. You probably realize that TinyBase is trying to pack as many changes into a single response as it can to limit chatter and latency but I wonder whether that can be configurable per synchronizer...

[jakubriedl]

Yeah I think there's a sweet spot how large messages are the most efficient. Small messages have extra chatter and overhead, large messages don't fit through infras. I've seen somewhere the AWS API GW has limit 32kb which is way less than CF we've hit issues with. With the solution we got there isn't extra overhead with the split because if the message is under limit it's still just one message. I think it would be useful if the ws synchroniser supported similar chunking out of the box and allowed the consumer to set the chunk size based on infra they have.

[jamesgpearce]

I was thinking I would just set it on a 'how deep, how wide' sort of thing for shipping the tree. But what happens if a cell or value on its own is over that size? Maybe we need to have a packeting strategy that is ambivalent to the underlying structure (much as I am loathe to send fragments of unformed JSON around) and which puts them all back together. Wait are we reinventing TCP/IP?

[jakubriedl]

yeah I'm not fan of the unformed fragments as well, but that was the easiest & least invasive thing I could do to resolve our issue. It does feel like reinventing wheel, but I'm not aware of any solution that would work smooth with the various limitations that websocket IaaS have.