[Demote] ZStd compression from GA / LTS to experimental and release a 2.9.1 patch

[nknize]

Is your feature request related to a problem? Please describe.

ZStd was introduced as an experimental compression option for Lucene indexes in #3577. This brought the implementation as a module (installed by default) under the sandbox directory. However, the feature would remain "expert" optional as it would only be installed if users built the distribution themselves and passed sandbox.enabled=true at JVM startup.

Unfortunately this code was prematurely released GA when it was migrated as a top level module a short time later in #7908 without any feature flags. This has now lead to users reporting memory leaks along with several other bugs in the zstd jni library. Additionally, Lucene has a long running discussion on the pros/cons of Zstd as an index compression option including the reasons it's not provided as a core capability. One of those reasons is the hard dependency on native compiled code, which often leads to portability issues due to glibc differences. These issues have been realized several times both in the OpenSearch bundle (e.g., see the KNN issue where users can't compile on M1 Mac) and legacy codebase .

For these reasons, we need to address the premature promotion of the zstd library as a GA / LTS feature in core, and (at minimum) release a patched bundle that fixes the critical performance issues and bugs.

Describe the solution you'd like

Quickly build and release a 2.9.1 bundle distribution with five patches.

1. Zstd-jni memory leak fix in: Close Zstd Dictionary after execution to avoid any memory leak. #9403
2. Add a new ZSTD_COMPRESSION_EXPERIMENTAL = "opensearch.experimental.feature.compression.zstd.enabled feature flag that is set to false by default (forcing users to opt in).
3. Apply the ZSTD_COMPRESSION_EXPERIMENTAL feature flag both to the CodecService constructor and the CompressionProvider.getCompressors() (used for BlobStoreRepository compression).
4. Add a DeprecationLogger message that the zstd feature will be moved to a plugin in the next release
5. Bump the zstd library dependency from 1.5.5-3 to 1.5.5-5 Bump zstd version to 1.5.5-5 #9431 (NEEDS TO BE BACKPORTED)

In 2.10.0 (or later even) we should decide the following:

1. Move the ZstdCodec and ZstdNoDictCodec out from being a default module into an optional location (e.g., either an optional plugin or library - note that the BlobStoreRepository compression is already in as an optional library, but its packaged and included by default so we still need to figure out how to make that optional).
2. Whether to switch to direct memory or introduce an expert setting that gives users the option to use direct or heap memory when using ZStd compression

Describe alternatives you've considered

1. Move ZStd codec and BlobStore compression to a module in a patch release.
2. Revert Moving zstd out of sandbox #7908 in a 2.9.1 patch release

[bbarani]

@nknize we cannot add new features in patch release (i.e 2.9.1) rather only bug fix and security patches as per SemVar guidelines. We can definitely get the bug fix in but I am not sure about adding experimental flag since that might be considered a feature, so we can target those changes only for next minor release.

[nknize]

...we cannot add new features in patch release (i.e 2.9.1) ....per SemVar guidelines. ...I am not sure if adding experimental flag since that would be considered a feature,...

@bbarani Unfortunately we can't allow ZStd to remain available by default because rolling this back requires users to reindex if they chose these codecs. We have to revert this feature ASAP before users start indexing. This was mistakenly released GA, hence the patch.

[andrross]

Regarding the option to revert #7908, I just want to note that will leave any users who indexed data with zstd using 2.9.0 in a bad spot. Their only option would be to reindex that data with 2.9.0 before upgrading to 2.9.1, since the 2.9.1 release will not contain the sandbox plugin functionality.

[nknize]

Great point @andrross! I'm adding context from follow on discussion from slack just so we don't lose it.

Barani Bikshandi (Barani): Yeah but can this be remediated with "revert" rather than adding experimental flag for 2.9.1? Basically remove support for zstd in 2.9.1? (edited)

I added this as an alternative to discuss.

[backslasht]

One of those reasons is the hard dependency on native compiled code, which often leads to portability issues due to glibc differences

@nknize - What is the general guideline for any native dependency? Are you saying OpenSearch will never support native dependencies going forward?

Move the ZstdCodec and ZstdNoDictCodec out from being a default module into an optional location (e.g., either an optional plugin or library - note that the BlobStoreRepository compression is already in as an optional library, but its packaged and included by default so we still need to figure out how to make that optional).

What is the intent of this? By moving to optional are you suggesting users to use Zstd at their own risk? Zstd is providing 30% reduction in storage size and latencies as good as best_speed. What is the alternative to this?

[nknize]

Are you saying OpenSearch will never support native dependencies going forward?

As plugins, sure. But likely not as default modules. Plugins are just like modules except users have to opt in by installing them.

By moving to optional are you suggesting users to use Zstd at their own risk?

Depends on how you define "risk". ldd errors due to libc issues is certainly a "risk". An alternative to the jni implementation would be to use the pure java solution. Then we could include it as a module without portability risks.

Zstd is providing 30% reduction in storage size and latencies as good as best_speed.

That's only part of the picture. Retrieval time per 10k docs is nearly five times slower than BEST_SPEED. So it's not without its tradeoffs.

[sarthakaggarwal97]

@nknize @backslasht sharing some search numbers from my runs with NYC Taxis

Cluster Configuration:

1. Server Setup
   a. One data node, r5.2xlarge backed by EBS GP3 volume of size 100gb.
   b. Three master nodes, r5.xlarge instance type

2. Benchmarking Setup
   Benchmark was run on a single node of type c5.4xlarge backed by EBS GP3 volume of size 500gb.
   a. Number of clients: 16 and bulk size: 1024.
   b. Workload: https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/nyc_taxis

3. Index Setup
   a. Number of shards: 1
   b. Number of replicas: 0

4. Benchmarks

[backslasht]

@sarthakaggarwal97 - Can you add details about the cluster configuration for the run?