Pluggable Module for custom codecs

[sarthakaggarwal97]

Is your feature request related to a problem? Please describe.
The idea is to extend the newly introduced codecs in the modules as a plugin. This would be prudent as it will provide us with the generic design to add more codecs to OpenSearch, and would be extensible.

Describe the solution you'd like
We can have a new module for custom codecs, where we would be able to introduce any number of codecs to OpenSearch. We can have a CodecPlugin interface, which will be generic enough to accept a number of arguments required to initialize a custom codec. Further, upon startup, we would be filtering the codecs, and populating the CodecService with the custom and default codecs. This would required us to wire up additional components like like IndicesServices, IndexService, IndexModule, IndexShard so that we are able to fetch the filtered codecs from the newly added module using PluginsService.

Tasks

• Design the generic Codec Plugin to filter codecs
• Introduce a Custom Codec Module
• • Implement a generic framework for new codecs
• • Move zstd and zstd_no_dict codec implementation to the new module
• • Wire up new components to pass the filtered codecs
• • Unit Tests
• • Plugin Compatibility (like KNN fixed over at Fix: avoid ZSTD codec from overriding service codec factory. #7037)
• • Stress Tests (100gb shard)
• Enable the index setting to accept new codec values via plugins

cc: @mgodwan @backslasht

[andrross]

@sarthakaggarwal97 Does this need to be updated after the creation of the custom-codecs repository? Namely the "Move zstd and zstd_no_dict codec implementation to the new module" item seems obsolete now.

[sarthakaggarwal97]

@andrross I think we can close this now.