Switch over current scalar quantization formats to use `OptimizedScalarQuantizer`

[benwtrent]

Description

@vigyasharma found that single bit is actually really nice and sometimes better than the current half-byte mikemccand/luceneutil#435

As a user, this doesn't make sense. more bits means more information. Well, the current scalar quantization formats use the old ScalarQuantizer.

We should consider switching the formats to use OptimizedScalarQuantizer.

This will not be a simple change. But, I don't think it should be TOO complicated? (at least just as complicated as any other format change).

• On disk layout of the vectors needs to change (more corrective values)
• More metadata needs to be retained (e.g. centroid)
• Different vector scorers (the scoring function is different)

But, I would expect half-byte (4bit) quantization to achieve 90%+ recall without any oversampling & reranking for larger vector dimensions

7bit (yeah, we likely still need to worry about signed int8 being busted....) will also be even better.

Again, I expect this to be a big 'ole chunk of work :/.

[mccullocht]

Is anyone working on this? I was thinking about extending OSQ to work with LVQ two-level encoding and expose it as a reranker, but it would probably conflict with any work on this.

[mccullocht]

I merged the change to add OSQ based codecs and I'm working on backwards codec changes but I've run into a bit of a dependency cycle: I can't move Lucene99ScalarQuantizedVectorScorer to backwards_codecs because it's linked directly by DefaultVectorizationProvider, but I can no longer test Lucene99ScalarQuantizedVectorScorer if I move all the remaining codec files to backward_codecs because they directly use the format (and presumably the writer). I'm not sure if this can be fixed without removing the off-heap scalar quantized scoring code which was just merged :(. I think this layering issue would likely repeat if we decided to use something besides OSQ or make really significant format changes there.

cc @benwtrent @kaivalnp

[benwtrent]

Tricky indeed. I need to think about this.

I THINK we will end up adding a new getLucene104SQVectorScorer.

Then the goal we want is that Lucene99ScalarQuantizedVectorsFormat doesn't lose its read time performance and can still score off-heap and use panama vector optimized comparators.

The purpose behind the versioning is that it makes assumptions on how the vectors are laid out (byte sizes, etc.).

I need to think about how to best achieve this...

[mccullocht]

The Scorer seems like it wants to be package-private with both the codec implementation and the panama/vector util bits but that isn't really possible when the codec gets moved to a different package. Even on 11 when FFM is stabilized and you could place MemorySegment overrides in VectorUtil you'd still need to know whether or not you should be trying to use them.

I will work through a change that moves Lucene99HnswScalarQuantizedFormat to BWC since that would probably capture most uses anyway.