Adds new formats that use the new scalar formats from lucene #141601

benwtrent · 2026-01-30T17:12:05Z

Adds new ES formats that build on the Lucene formats.

This adds scorers & scorer suppliers.

elasticsearchmachine · 2026-01-30T17:12:30Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

benwtrent · 2026-01-30T18:26:34Z

Here is the difference in recall/etc.

this PR (1, 2, 4, 7) bits, all using the new format

ndex_name                      index_type  visit_percentage(%)  latency(ms)  net_cpu_time(ms)  avg_cpu_count      QPS  recall  visited  filter_selectivity  filter_cached  oversampling_factor  num_candidates  early_termination
------------------------------  ----------  -------------------  -----------  ----------------  -------------  -------  ------  -------  ------------------  -------------  -------------------  --------------  -----------------
cohere-wikipedia-docs-768d.vec        hnsw                0.000         0.43              0.00           0.00  2325.58    0.67  4103.22                1.00           true                 0.00             250              false
cohere-wikipedia-docs-768d.vec        hnsw                0.000         0.59              0.00           0.00  1694.92    0.79  3889.48                1.00           true                 0.00             250              false
cohere-wikipedia-docs-768d.vec        hnsw                0.000         0.68              0.00           0.00  1470.59    0.90  3796.08                1.00           true                 0.00             250              false
cohere-wikipedia-docs-768d.vec        hnsw                0.000         1.08              0.00           0.00   925.93    0.94  3812.40                1.00           true                 0.00             250              false

baseline: (1, 4, 7)

index_name                      index_type  visit_percentage(%)  latency(ms)  net_cpu_time(ms)  avg_cpu_count      QPS  recall  visited  filter_selectivity  filter_cached  oversampling_factor  num_candidates  early_termination
------------------------------  ----------  -------------------  -----------  ----------------  -------------  -------  ------  -------  ------------------  -------------  -------------------  --------------  -----------------
cohere-wikipedia-docs-768d.vec        hnsw                0.000         0.41              0.00           0.00  2439.02    0.67  4085.27                1.00           true                 0.00             250              false
cohere-wikipedia-docs-768d.vec        hnsw                0.000         1.69              0.00           0.00   591.72    0.55  4418.59                1.00           true                 0.00             250              false
cohere-wikipedia-docs-768d.vec        hnsw                0.000         0.42              0.00           0.00  2380.95    0.92  3787.26                1.00           true                 0.00             250              false

Obviously, 2, 4 are way better. Single bit might be a little slower. But int7 is significantly slower due to lack of native code support.

Recall is better across the board.

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java

...simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7uOSQVectorScorerSupplier.java

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java

libs/simdvec/src/main22/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorer.java

thecoop · 2026-02-04T09:53:25Z

We need some tests on the scorer, that the native and lucene implementations produce the same result - see Int7SQVectorScorerFactoryTests

gradle/verification-metadata.xml

benwtrent · 2026-02-04T21:56:34Z

@thecoop I am gonna close this and rebase & reopen against the new lucene_10_4 branch

adding tests [CI] Auto commit changes from spotless adding exposure via module iter fixing things [CI] Auto commit changes from spotless iter iter iter [CI] Auto commit changes from spotless Adding random vector scorer code iter iter fixing scorer supplier iter adding more tests

benwtrent · 2026-02-05T21:32:35Z

@thecoop sorry for the force push, but rebased on 10_4 and now merging there.

ldematte · 2026-02-06T07:42:35Z

libs/simdvec/src/main22/java/org/elasticsearch/simdvec/internal/Int7uOSQVectorScorer.java

+        }
+
+        @Override
+        float applyCorrections(float rawScore, int ord) throws IOException {


I like the new name, way better than some variant of score.
Can we have a follow up PR that renames all the others (e.g. in BBQ/DiskBBQ)?
I'm also leaning towards using the same names on native functions. Wdyt? CC @thecoop

I also like how you separated corrections into the different distance scorers, like we did in native code.

ldematte

Just gave a quick look over; looks good, just a couple of minor comments/questions

ldematte · 2026-02-06T07:46:11Z

...c/main/java/org/elasticsearch/index/codec/vectors/es94/ES94ScalarQuantizedVectorsFormat.java

+        public RandomVectorScorerSupplier getRandomVectorScorerSupplier(VectorSimilarityFunction sim, KnnVectorValues values)
+            throws IOException {
+            if (values instanceof QuantizedByteVectorValues quantizedValues && quantizedValues.getSlice() != null) {
+                // TODO: optimize int4, 2, and single bit quantization


I'm getting confused with all the formats :)
Maybe we can sync a bit on these?
Are these "striped" (like BBQ/DiskBBQ) or packed? (e.g. 2 Int4 in a byte?)

These are scalar quantized, so packed

2 bits is striped, int4 are packed.

I am not convinced that "striped" is the best option for int4 * int4 operations.

Lucene just packs with int4 & int4. Stripes int4 * int1 and double stripes int4 * int2 :D

I am not convinced that "striped" is the best option for int4 * int4 operations.

++
I have (SIMD) implementations for both, I just need some time to test and benchmark them.
My gut feeling is that for int4 packed/normal mul (or madd) is going to be faster.
Give me some time and I'll come back with numbers :)

libs/simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorer.java

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java

libs/simdvec/src/main22/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorer.java

libs/simdvec/src/test/java/org/elasticsearch/simdvec/AbstractVectorTestCase.java

...t/java/org/elasticsearch/index/codec/vectors/es94/ES94ScalarQuantizedVectorsFormatTests.java

tteofili

LGTM. but I think we need a few more benchmarks

ldematte · 2026-02-10T14:03:12Z

libs/simdvec/src/main22/java/org/elasticsearch/simdvec/internal/Int7uOSQVectorScorer.java

+            float y1 = quantizedComponentSum;
+            float score = ax * ay * values.dimension() + ay * lx * x1 + ax * ly * y1 + lx * ly * rawScore;
+            score += additionalCorrection + correctiveTerms.additionalCorrection() - values.getCentroidDP();
+            score = Math.clamp(score, -1, 1);


Do you think we can reuse the native code implementations here? Or we can expose a new one, but share the same "kernel"? Besides this clamp, I do not see other differences.

We likely could reuse native here.

benwtrent requested a review from thecoop January 30, 2026 17:12

benwtrent added >non-issue :Search Relevance/Vectors Vector search v9.4.0 labels Jan 30, 2026

elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Jan 30, 2026

benwtrent changed the title ~~Adds initial pass of the new scalar formats from lucene~~ Adds new formats that use the new scalar formats from lucene Feb 2, 2026

thecoop reviewed Feb 3, 2026

View reviewed changes

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 3, 2026

View reviewed changes

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 3, 2026

View reviewed changes

...simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7uOSQVectorScorerSupplier.java Show resolved Hide resolved

benwtrent requested a review from thecoop February 3, 2026 19:34

thecoop reviewed Feb 4, 2026

View reviewed changes

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 4, 2026

View reviewed changes

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 4, 2026

View reviewed changes

libs/simdvec/src/main22/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorer.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 4, 2026

View reviewed changes

libs/simdvec/src/main22/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorer.java Outdated Show resolved Hide resolved

benwtrent requested a review from a team as a code owner February 4, 2026 19:32

benwtrent commented Feb 4, 2026

View reviewed changes

gradle/verification-metadata.xml Outdated Show resolved Hide resolved

benwtrent force-pushed the add-new-scalar-formats branch from 53d421b to 0d5b790 Compare February 5, 2026 21:31

benwtrent requested review from a team as code owners February 5, 2026 21:31

benwtrent changed the base branch from lucene_snapshot to lucene_snapshot_10_4 February 5, 2026 21:31

benwtrent requested a review from thecoop February 5, 2026 21:32

tvernum removed the request for review from a team February 6, 2026 07:21

ldematte reviewed Feb 6, 2026

View reviewed changes

thecoop reviewed Feb 6, 2026

View reviewed changes

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 6, 2026

View reviewed changes

.../simdvec/src/main21/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorerSupplier.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 6, 2026

View reviewed changes

libs/simdvec/src/main22/java/org/elasticsearch/simdvec/internal/Int7OSQVectorScorer.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 6, 2026

View reviewed changes

libs/simdvec/src/test/java/org/elasticsearch/simdvec/AbstractVectorTestCase.java Outdated Show resolved Hide resolved

thecoop reviewed Feb 6, 2026

View reviewed changes

...t/java/org/elasticsearch/index/codec/vectors/es94/ES94ScalarQuantizedVectorsFormatTests.java Show resolved Hide resolved

addressing pr comments

3b98c8f

benwtrent requested review from ldematte and thecoop February 6, 2026 13:03

benwtrent and others added 2 commits February 6, 2026 10:46

fmt

2e0a64a

Merge branch 'lucene_snapshot_10_4' into add-new-scalar-formats

ecd6e91

tteofili approved these changes Feb 10, 2026

View reviewed changes

benwtrent merged commit 632a640 into elastic:lucene_snapshot_10_4 Feb 10, 2026
32 of 36 checks passed

benwtrent deleted the add-new-scalar-formats branch February 10, 2026 12:41

ldematte reviewed Feb 10, 2026

View reviewed changes

Adds new formats that use the new scalar formats from lucene #141601

Adds new formats that use the new scalar formats from lucene #141601

Uh oh!

Conversation

benwtrent commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 30, 2026

Uh oh!

benwtrent commented Jan 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thecoop commented Feb 4, 2026

Uh oh!

Uh oh!

benwtrent commented Feb 4, 2026

Uh oh!

benwtrent commented Feb 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldematte left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tteofili left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

benwtrent commented Jan 30, 2026 •

edited

Loading