Decompression attack in `MinecraftCompressDecoder#decode`

[WouterGritter]

Proxies of a large network I'm working for were constantly being restarted due to OOM kills. I highly suspect this is due to a malicious decompression attack.

I had added the following logs in MinecraftCompressDecoder#decode:

if (claimedUncompressedSize > UNCOMPRESSED_CAP * 0.95) {
  LOGGER.warn("Received a packet with a large uncompressed size: {} bytes ({}% of the {} byte cap) from {}",
      claimedUncompressedSize,
      (int) (claimedUncompressedSize * 100.0 / UNCOMPRESSED_CAP),
      UNCOMPRESSED_CAP,
      ctx.channel().remoteAddress());
}

When we were under attack we got spammed by the following logs:

[17:36:45 WARN]: Received a packet with a large uncompressed size: 8380416 bytes (99% of the 8388608 byte cap) from /x.x.x.x:16643

All from the same IP address, all with the same packet size. Moments later the proxy got OOM killed.

When we blacklisted this IP at the firewall level the proxies stayed alive.

I've implemented the following fix on a fork to try to block these kinds of attacks, at least on the short term: GemstoneGG@3fd2f11
However the limits (0.95 * UNCOMPRESSED_CAP and MAX_COMPRESSION_RATIO = 1024) are not really based on anything. Since we're running the fork with this commit we haven't been attacked again, so I'm not sure if the commit prevents the attack. Luckily, legitimate connections aren't being affected.

-Dvelocity.increased-compression-cap=true is NOT set, so the UNCOMPRESSED_CAP is 8MB.
Decompressing a really small packet to 8MB over and over again ideally shouldn't OOM kill a proxy. I wonder if the proper fix for this is a "rate-limiter" approach like in my temporary-ish fix commit, or to take a closer look at why allocating "just" 8MB constantly gets an OOM kill. Something might be going wrong with (de)allocating this buffer.

[Leymooo]

probably some plugin message packets + plugin what process PluginMessageEvent async or abusing loginPluginMessages queue.

or you have compression disabled between proxy <--> backend, so these buffers are stored inside netty's internal write queue. Looks like configuration state handlers do not have back pressure handling (writabilityChanged()), what propably should help to mitigate that type of attack.

[WouterGritter]

I was able to recreate the attack we we're experiencing by sending large unknown packets and large discardedpayload packets during the ClientConfigSessionHandler phase. #1743 fixes this attack, now it immediately disconnects the client.

I think it's still good to implement writabilityChanged() for the two handlers.