Gzip Middleware Tuning

[bencap]

Gzip middleware is currently active on the project with a minimum size of 1000 and a compression level of 5. We should evaluate the performance of the middleware with these settings and potentially tune their values to optimize them for our use case.

As part of this tuning, we should consider adding a new middleware that can inject a pre-Gzipped content length into a request/response cycle's logging context. This middleware would run immediately prior to the Gzip middleware and calculate the response content's size prior to being Gzipped. It should also calculate the time it takes to compress the content.

After gzipping, we can use the calculated values to calculate a few new logging outputs for our canonical log lines:

• uncompressed-bytes: The size of the pre-gzipped uncompressed response.
• compressed-bytes: The size of the post-gzipped compressed response.
• compression-time: The time it took to compress the response content.
• bytes_saved_per_ms: (uncompressed-bytes - compressed-bytes) / compression-time. The amount of bytes we save in the response per ms of compression time.

After releasing the new logging output, we should monitor the values to see:

• How many requests are gzipped at the current threshold level
• The average compression bytes/ms ratio
• The average compression amount

We should consider sampling representative payloads at different compression levels and observing the ratio. We should pick the level with the best ratio while evaluating the following factor:

• How sensitive we are to response latency (higher levels = more latency, but maybe we get that time back in data transfer)
• CPU budget (higher compression levels use more CPU)
• Content redundancy (compressing text with more repetition--csv/json-- is more effective at higher levels)
• Concurrency (Many simultaneous requests = lower levels)

Originally posted by @bencap in #548 (review)