Bulk doc value loading at codec level #128445
Description
Implement bulk doc values implementations that push down value loading to doc values codec level. This should be possible for our custom doc value codec that we use for logsdb and tsdb and with the compute engine would allow for bulk loading.
- Make use of low level implementations details of the codec when loading blocks. The tsdb doc value codec is organized values / ordinals in blocks of 128 values each. Today this gets decoded into an array which gets used in intermediate calls to public doc value apis. We could chose to append all 128 values in one operation to block loader builder when possible.
- Take another look at explicit vectorization and see whether this can be applied to value loading. Today we rely on auto vectorization, which maybe doesn't get applied all the time.
A first step is to push down doc value loading to the es819 doc values codec and add bulk loading support to es819 doc value codec. This is a relative large tasks but can be divided into smaller tasks:
- Add initial bulk loading to es819 doc value codec. (Push compute engine value loading for longs down to tsdb codec. #132622)
- Push down loading of singleton dense long based field types to the es819 doc value codec. (Push compute engine value loading for longs down to tsdb codec. #132622)
- Push down loading of singleton dense integer based field types to the es819 doc value codec.
- Push down loading of singleton dense float/doble based field types to the es819 doc value codec.
- Push down loading of singleton dense keyword field types to the es819 doc value codec. (Push compute engine value loading for dense singleton ordinals down to tsdb codec. #132715)
- After upgrading to Lucene 10.3, make use of
IndexedDISI#docIDRunEnd()implementation to in some cases support bulk loading in the es819 doc value codec. - Push down loading of singleton sparse number based field types to the es819 doc value codec.
- Push down loading of singleton sparse keyword field types to the es819 doc value codec.
- Push down loading of multi-valued number based field types to to the es819 doc value codec
- Push down loading of multi-valued keyword field types to to the es819 doc value codec