Add optional request and response compression

[jer-k]

What changed?

ClickHouse supports request and response compression which is explained in the HTTP Interface - Compression docs.

• Added README information on how to set up the compression attributes in database.yml
• Added ActiveRecord::ConnectionAdapters::Clickhouse::Compression class to handle the compression and compression use cases
• Added unit tests (see the Real World Testing section below for additional tests)
  • ActiveRecord::ConnectionAdapters::Clickhouse::Statement::ResponseProcessor is covered
  • ActiveRecord::ConnectionAdapters::Clickhouse::Compression is covered
  • ActiveRecord::ConnectionAdapters::Clickhouse::SchemaStatements#execute tests added for a baseline and compression related.

Motivation

We have monkey patched in this functionality to our application and have been using brotli compression for a few years now.

Real World Testing

All these tests will be executed the same way, querying for a record by opening a Rails console and executing UserEvent.first. I used mitmproxy to view the request and their headers.

[1] pry(main)> UserEvent.first
  Clickhouse (system) (24.2ms)  SHOW TABLES WHERE name NOT LIKE '.inner_id.%'
  Clickhouse (6.1ms)  SELECT version()
  Clickhouse (system) (9.3ms)  SHOW COLUMNS FROM `user_events`
  Clickhouse UserEvent Load (10.0ms)  SELECT user_events.* FROM user_events ORDER BY user_events.event_name ASC, user_events.event_time ASC, user_events.user_id ASC LIMIT 1
  Clickhouse user_events (system) (4.1ms)  DESCRIBE TABLE `user_events`

This is ensuring the base line and then all the different gems / algorithms are working.

Baseline - No compression

# database.yml
# request_compression: gzip
# response_compression: gzip

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&password=&user=default
    • 200 OK text/plain 600b 84ms
  • Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
• Response - No Content-Encoding header is present

These are the Net::HTTP defaults

Baseline - Requesting compression without installing the required gems

No additional gem is installed here, but I'm requesting br compression. In this case, no compression will actually be added, requests function as normal.

# database.yml
request_compression: br
response_compression: br

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&password=&user=default
    • 200 OK text/plain 600b 17ms
  • Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
• Response - No Content-Encoding header is present

gzip

Note - no additional gem is required for gzip, Net::HTTP can do this natively

# database.yml
request_compression: gzip
response_compression: gzip

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&enable_http_compression=1&password=&user=default
    • 200 OK text/plain 326b 13ms
  • Accept-Encoding: gzip
  • Content-Encoding: gzip
• Response
  • Content-Encoding: gzip

deflate

Note - no additional gem is required for deflate, Net::HTTP can do this natively

# database.yml
request_compression: deflate
response_compression: deflate

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&enable_http_compression=1&password=&user=default
    • 200 OK text/plain 314b 9ms
  • Accept-Encoding: deflate
  • Content-Encoding: deflate
• Response
  • Content-Encoding: deflate

br

# database.yml
request_compression: br
response_compression: br

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&enable_http_compression=1&password=&user=default
    • 200 OK text/plain 315b 8ms
  • Accept-Encoding: br
  • Content-Encoding: br
• Response
  • Content-Encoding: br

zstd

# database.yml
request_compression: zstd
response_compression: zstd

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&enable_http_compression=1&password=&user=default
    • 200 OK text/plain 362b 8ms
  • Accept-Encoding: zstd
  • Content-Encoding: zstd
• Response
  • Content-Encoding: zstd

lz4

# database.yml
request_compression: lz4
response_compression: lz4

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&enable_http_compression=1&password=&user=default
    • 200 OK text/plain 415b 14ms
  • Accept-Encoding: lz4
  • Content-Encoding: lz4
• Response
  • Content-Encoding: lz4

bz2

# database.yml
request_compression: bz2
response_compression: bz2

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&enable_http_compression=1&password=&user=default
    • 200 OK text/plain 355b 9ms
  • Accept-Encoding: bz2
  • Content-Encoding: bz2
• Response
  • Content-Encoding: bz2

snappy

I tried to implement snappy but ran into a couple issues.

First, ClickHouse currently doesn't support the response compression for snappy. The issue is ClickHouse/ClickHouse#44885

Secondly, when trying to compress the request data I got

ActiveRecord::ActiveRecordError: Response code: 500: (ActiveRecord::ActiveRecordError)
Code: 638. DB::Exception: hadoop snappy decode error: TOO_LARGE_COMPRESSED_BLOCK. (SNAPPY_UNCOMPRESS_FAILED) (version 25.8.8.26 (official build))

from /clickhouse-activerecord/lib/active_record/connection_adapters/clickhouse/statement/response_processor.rb:51:in 'ActiveRecord::ConnectionAdapters::Clickhouse::Statement::ResponseProcessor#raise_generic!'

I didn't want to block this whole implementation on snappy so I decided to remove it as an option for now.

xz

One caveat with xz is that it does not work on Ruby 4.0.0 currently. This PR needs to be merged first win93/ruby-xz#13. When this is run on 4.0.0, the require will fail and no compression will be enabled. The request still works.

The test below is on Ruby 3.4.7.

# database.yml
request_compression:n xz
response_compression: xz

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&enable_http_compression=1&password=&user=default
  • 200 OK text/plain 384b 25ms
  • Accept-Encoding: xz
  • Content-Encoding: xz
• Response
  • Content-Encoding: xz

Mixed br + lz4

# database.yml
request_compression:n br
response_compression: lz4

• Request
  • POST http://localhost:8123/?database=clickhouse_testing_ch&enable_http_compression=1&password=&user=default
    • 200 OK text/plain 415b 22ms
  • Accept-Encoding: lz4
  • Content-Encoding: br
• Response
  • Content-Encoding: lz4

[jer-k]

@PNixx I rebased in the streaming work. It appears it will be a decent undertaking to make the response compression work with streaming responses, although ClickHouse does support it. I updated the code and README to have response compression do anything if the streaming is happening. It's a bit clunky, but would appreciate any feedback on the approach.

[PNixx]

rspec will run only single and cluster directories