Introduce more compression libraries and implementations

[mulugetam]

Lucene, and by extension OpenSearch, currently support only zlib and lz4 compression algorithms offering best compression ratio and best speed, respectively. There are a couple of limitations with this:

• Use cases that prefer a middle ground between best compression ratio and best speed, such as that offered by zstd, are not supported.
• The lz4 implementation is written entirely in Java and, based on our benchmark results, is less performant than a native lz4 implementation.

To address the limitations, we have implemented a small library that adds support for both zstd and lz4 (native) compressions to OpenSearch. The library requires a couple of source code changes to OpenSearch, EngineConfig.java and CodecService.java, but does not require any code changes to Lucene.

The charts below summarize the relative performance gain we measured for the different compression algorithms and implementations using the StoredFieldsBenchmark.

Indexing Time (relative, less is better):

Retrieval Time (relative, less is better):

Stored size (relative, less is better):

We have also used the OpenSearch-Benchmark with the nyc_taxis workload to compare the indexing performance of the different compression algorithms and implementations above and measured decent gains as well.

We'll be upstreaming our source code and patches in a couple of weeks and discuss on how best we can integrate this into OpenSearch and make them available for users. This post is intended to start the discussion.

[reta]

@mulugetam there is an issue on Apache Lucene to support ZSTD [1], [2], plus somewhat unrelated to index storage but ZSTD [3] for snapshots. I believe the main concern is related to usage of the JNI bindings, I suspect your implementation is also using that? Thank you.

[1] https://issues.apache.org/jira/browse/LUCENE-8739
[2] apache/lucene#439
[3] #2192

[dblock]

This is awesome @mulugetam, looking forward!

[dblock]

@reta on JNI, I think the issue is primarily native code, not the bindings themselves, as Lucene is strongly opposed to including native code? This isn't a limitation of OpenSearch, we have plugins, such as k-nn, that use native libraries.

I don't love JNI btw as it requires one to write JNI bindings in C. I would prefer to replace all native boilerplate code written in OpenSearch with https://github.com/java-native-access/jna (see opensearch-project/k-NN#35 for an example).

[reta]

thanks @dblock, yeah I think the native code + JNI bindings (as the consequence) is the concern for Apache Lucene (as per discussion thread), I am just curious what the compression implementation was used, as you said, OpenSearch has no such limitations or / and constraints.

[willyborankin]

Compression for snapshotting I added here: #2996. @mulugetam Do you implement Lucene codes with ZSTD compression support? Regarding lz4 Lucense has its own implementation the missing part is a support of lz4 InputStream.

[mulugetam]

@willyborankin Yes. From Lucene, you'd set your codec, e.g. zstd, using:

indexWriterConfig.setCodec(new Lucene92CustomCodec(Lucene91CustomCodec.Mode.ZSTD));

and lz4-jni (native) like:

indexWriterConfig.setCodec(new Lucene92CustomCodec(Lucene92CustomCodec.Mode.LZ4_NATIVE));

From OpenSearch, only the EngineConfig.java and CodecService.java source files need to be updated. You can then use it by setting the index.codec property.

Below is how I used it when running OpenSearch-Benchmark:

echo '{"index_settings":{"index.codec": "zstd"}}' > params.json
opensearch-benchmark execute_test \
--workload=nyc_taxis \
--workload-params=params.json \
--target-hosts=127.0.0.1:9200 \
--pipeline=benchmark-only

The current lz4 implementation in Lucene, which is written in pure Java, has a very poor performance and that is why we went for the native implementation.

Compression for snapshotting and lz4 InputStream is something we'd need to look into. I'll be upstreaming the patches as soon as I'm done with our internal process here.

[willyborankin]

@mulugetam This is perfect. I misread your comment about LZ4 looks really good. Regarding lz4 and InputStream, I can implement it like it was done for Kafka here: https://github.com/apache/kafka/tree/99b9b3e84f4e98c3f07714e1de6a139a004cbc5b/clients/src/main/java/org/apache/kafka/common/compress.

[willyborankin]

Maybe it makes sense to add a global parameter in opensearch.yml such this:
cluster.compression_type: which defines compression type both for indexes and snapshotting. WDYT?

[reta]

@willyborankin what would be the benefit to compress LZ4 indices with LZ4 again while snapshotting?

[willyborankin]

@reta Better compression but slow read even compare to ZSTD ... I think it is up to the end-user to decide. Wdyt?

[reta]

Ok, let me go back to your original question:

Maybe it makes sense to add a global parameter in opensearch.yml such this:
cluster.compression_type: which defines compression type both for indexes and snapshotting.

Let say the indices are compressed with LZ4 / ZSTD, what would be the benefit to compress them again with LZ4 / ZSTD while snapshotting? I am trying to understand this part.