MergedByteVectorValues is missing increasing `lastOrd` when advancing.

[0ctopus13prime]

Description

MergedByteVectorValues represents a unified view of byte vector values from multiple underlying Lucene segments. It provides an iterator interface that allows advancing to a specific document ID. In this contract, if a user advances five times using the KnnVectorValues iterator and then loads the next byte[], it is expected to return the corresponding next vector.

However, this expected behavior does not hold for MergedByteVectorValues. After advancing N times and attempting to load the next byte[], an error consistently occurs. This inconsistency does not exist in MergedFloat32VectorValues, where the behavior is correct and reliable.

The root cause is that MergedByteVectorValues does not update the internal lastOrd field when advancing, unlike MergedFloat32VectorValues. As a result, when attempting to load the next vector, the code checks the current ord against lastOrd, which remains zero, causing an exception to be thrown.

lucene/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java

Lines 450 to 461 in 7fc9fd3

	@Override
	public int nextDoc() throws IOException {
	current = docIdMerger.next();
	if (current == null) {
	docId = NO_MORE_DOCS;
	index = NO_MORE_DOCS;
	} else {
	docId = current.mappedDocID;
	++index;
	}
	return docId;
	}

To fix this, MergedByteVectorValues should be updated to increment lastOrd during advancement, mirroring the behavior of MergedFloat32VectorValues.

lucene/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java

Lines 340 to 352 in 7fc9fd3

	@Override
	public int nextDoc() throws IOException {
	current = docIdMerger.next();
	if (current == null) {
	docId = NO_MORE_DOCS;
	index = NO_MORE_DOCS;
	} else {
	docId = current.mappedDocID;
	++lastOrd;
	++index;
	}
	return docId;
	}

In OpenSearch, for fast vector index construction, we upload KNN vectors to a remote builder component and trigger the index build. To speed up uploads, vectors are logically partitioned, and multipart uploading is performed — a process that relies on the advance-then-load pattern. Due to lastOrd not being updated correctly in MergedByteVectorValues, this multipart upload mechanism currently fails, blocking fast uploads with byte vectors.

For more details use case in Opensearch, please refer to this : opensearch-project/k-NN#2803

Solution

The solution is straightforward: increment lastOrd during advancement, and ensure the ord check is made against lastOrd, not lastOrd + 1.

@Override
public byte[] vectorValue(int ord) throws IOException {
//  if (ord != lastOrd + 1) {
  if (ord != lastOrd) { <----------------- It now compares against `lastOrd` not `lastOrd + 1`.
    throw new IllegalStateException(
        "only supports forward iteration: ord=" + ord + ", lastOrd=" + lastOrd);
  } else {
    lastOrd = ord;
  }
  return current.values.vectorValue(current.index());
}


@Override
public int nextDoc() throws IOException {
  current = docIdMerger.next();
  if (current == null) {
    docId = NO_MORE_DOCS;
    index = NO_MORE_DOCS;
  } else {
    docId = current.mappedDocID;
    ++lastOrd; <------------------------ This line should be added
    ++index;
  }
  return docId;
}

Version and environment details

releases/lucene/10.2.2

[0ctopus13prime]

With change I can see that testing with the command ../../gradlew test in core is passing.

:lucene:core:test (SUCCESS): 7573 test(s), 236 skipped                                                                                                                                                                                                                                    
The slowest tests (exceeding 500 ms) during this run:                                                                                                                                                                                                                                     
  11.37s TestIndexOrDocValuesQuery.testUseIndexForSelectiveMultiValueQueries (:lucene:core)                                                                                                                                                                                               
   8.40s TestHnswByteVectorGraph.testRamUsageEstimate (:lucene:core)                                                                                                                                                                                                                      
   7.77s TestHnswFloatVectorGraph.testRamUsageEstimate (:lucene:core)                                                                                                                                                                                                                     
   6.61s TestLucene101PostingsFormat.testDocIDRunEnd (:lucene:core)                                                                                                                                                                                                                       
   6.51s TestLucene101PostingsFormatV0.testDocIDRunEnd (:lucene:core)                                                                                                                                                                                                                     
   6.17s TestPerFieldPostingsFormat.testDocIDRunEnd (:lucene:core)                                                                                                                                                                                                                        
   4.93s TestExitableDirectoryReader.testExitableTermsEnumSampleTimeoutCheck (:lucene:core)                                                                                                                                                                                               
   3.73s TestIndexWriterExceptions.testRandomExceptions (:lucene:core)                                                                                                                                                                                                                    
   3.71s TestNeverDelete.testIndexing (:lucene:core)                                                                                                                                                                                                                                      
   3.64s TestBagOfPostings.test (:lucene:core)                                                                                                                                                                                                                                            
The slowest suites (exceeding 1s) during this run:                                                                                                                                                                                                                                        
  12.41s TestLucene90DocValuesFormatMergeInstance (:lucene:core)                                                                                                                                                                                                                          
  11.94s TestIndexOrDocValuesQuery (:lucene:core)                                                                                                                                                                                                                                         
  11.12s TestLucene90DocValuesFormat (:lucene:core)                                                                                                                                                                                                                                       
  10.31s TestLucene101PostingsFormat (:lucene:core)                                                                                                                                                                                                                                       
  10.00s TestLucene101PostingsFormatV0 (:lucene:core)
   9.96s TestLucene90DocValuesFormatVariableSkipInterval (:lucene:core)
   9.92s TestBoolean2 (:lucene:core)
   9.22s TestSimpleExplanationsWithFillerDocs (:lucene:core)
   9.18s TestHnswByteVectorGraph (:lucene:core)
   8.62s TestHnswFloatVectorGraph (:lucene:core)

BUILD SUCCESSFUL in 2m 10s
21 actionable tasks: 13 executed, 8 up-to-date

Could anyone give a guidance whether I can raise a fix PR?

[vigyasharma]

Thanks for uncovering this, and the detailed explanation, @0ctopus13prime . The API promise should be the same in byte and float merged vector values, so this looks like an oversight.

A PR with your fix sounds good. Let's also add a test which fails without your fix in place?

[navneet1v]

Thanks @vigyasharma . @0ctopus13prime lets raise a PR when you have time.

[navneet1v]

@0ctopus13prime when we are planning to raise a PR for this?

[finnroblin]

@0ctopus13prime @navneet1v @vigyasharma Hi folks, I've raised a PR with the bugfix described by Dooyong and a unit test that fails without the bugfix. Could you please take a look if it resolves this issue? Thanks!