Implement "permessage-deflate" extension for gateway

[gudvinr]

There is RFC 7692 which describes concept "Compression Extensions" for WebSocket.

RFC 7692 supported by major browsers since at least 2014 and isn't a draft anymore since 2016.
It doesn't mean anything from technical standpoint because discord bots aren't browsers but it shows that it is quite mature set of extensions.
Event Discord client itself tries to handshake with Sec-WebSocket-Extensions: permessage-deflate in Upgrade request.

PMCE supported by variety of libraries with WebSocket client implementations: Python, Node, .NET and .NET 6 (not yet), C++ (see protocol support), etc.

At the same time Discord uses custom "zlib-stream" compression (apparently for both regular clients and bots) which isn't defined by widespread standards and requires to be manually implemented in order to work.

However, "zlib-stream" is essentially representing the exact same stream compression that you will get by using permessage-deflate without context takeover. Context takeover there means that LZ77 sliding window will be reused during for following stream (which is what "zlib-stream" does).
Only difference between "zlib-stream" and permessage-deflate stream is first 2 bytes of the very first message.

These 2 bytes contain zlib header which consists of 0111 1000 10 0 11100:

• 0111 - 32KiB window size
• 1000 - DEFLATE
• 10 - default compression level
• 0 - no dict
• 11100 - checksum

That's what used by standard DEFLATE implementations. No difference there either so it seems that server implementation uses most standard parameters for zlib and could either trim header or just use deflate as is.

So, in order to simplify compression support for clients, it is suggested to use widely-available standards instead of custom-made solution.

It should remove needs to support custom compression handlers in client applications with little to zero intervention. Basically, in most cases bot developer could just remove their own code for transport compression and toggle an option in their WebSocket client.

It also removes needs to include "compression" in get parameters to gateway since PMCE controlled by Sec-WebSocket-Extensions header.

[devsnek]

Thanks for posting this. We don't have any current plans to support this extension. The two major points are: 1) we don't want to deal with inflate bombs from client->server compressed messages and 2) having our own non-standard compression scheme means we can update it to our requirements in the future, which we can't with permessage-deflate.

One upside here is that thanks to how ubiquitous permessage-deflate is, there is lots of code you can use as a starting point to implement zlib-stream.

[gudvinr]

Thanks for the answer.

we don't want to deal with inflate bombs from client->server compressed messages

As per your documentation, current limit on payload is 4096 bytes. Assuming that it is a meaningful limitations based on payload content we can say that after decompression payload must be at most 4096 bytes too.
I don't know about what you use on your side, but at least 2 of deflate implementations that I know of (zlib and compress/flate from Go) will allow you to allocate buffer to decompress data to. There are non-zero chances that your implementation uses some zlib or zlib-compatible library under the hood.
So once you fill 4KiB buffer with meaningless data it'll be safe to assume that this payload could be rejected. Most likely, it'll happen within first few bytes of deflate stream because you don't even need to extract whole chunk for that.

having our own non-standard compression scheme means we can update it to our requirements in the future, which we can't with permessage-deflate

You still can use non-standard extensions by utilizing Sec-WebSocket-Extensions header. Updates to compression scheme will require support from developers anyway so it isn't very different from changing argument for compress parameter.
You can't easily add extension from JS in browser because they are negotiated on a protocol level but there are two arguments against doing so:

• you still can do that in Electron
• using custom compression from user-facing code means you must implement it in JS which is definitely slower for the client.
  Even if it is some kind of efficient algorithm compiled to wasm. Transport compression inside browser code (or native electron module) will always be faster because it's a native code that doesn't need to pass buffers between JS VM and transport layer.

thanks to how ubiquitous permessage-deflate is, there is lots of code you can use as a starting point to implement zlib-stream

It isn't really useful though. Compression bits are mostly hidden inside websocket implementations and tightly coupled with internals so it's easier to just get whatever library is available

[fumoboy007]

@devsnek Any chance y’all could reconsider? I think @gudvinr addressed your concerns?