IVF format fails search due to serialization failure

[benwtrent]

Elasticsearch Version

main

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

ivf format fails to search due to invalid bytes in DocIdReader.

Steps to Reproduce

This occurred during a large vector indexing test, indexing about 100M+ vectors with aggressive merging.

Obviously, we have a serialization bug where we are miscounting bytes in the format.

Logs (if relevant)

[2025-06-17T02:58:50,200][WARN ][o.e.s.SearchService      ] [elasticsearch-0] [yt4RfBSmRcKGMzZS3IDE-w][msmarco-v2][0]: failed to execute search request for task [2783699]
org.elasticsearch.search.dfs.DfsPhaseExecutionException: Dfs Failed [Exception during dfs phase]
	at org.elasticsearch.search.dfs.DfsPhase.execute(DfsPhase.java:67) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:654) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.SearchService.lambda$executeDfsPhase$5(SearchService.java:643) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) ~[?:?]
	at java.lang.Thread.run(Thread.java:1447) ~[?:?]
Caused by: java.io.IOException: Unsupported number of bits per value: -47
	at org.elasticsearch.index.codec.vectors.DocIdsWriter.readInts(DocIdsWriter.java:242) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.codec.vectors.DefaultIVFVectorsReader$MemorySegmentPostingsVisitor.resetPostingsScorer(DefaultIVFVectorsReader.java:234) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.index.codec.vectors.IVFVectorsReader.search(IVFVectorsReader.java:279) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader.search(PerFieldKnnVectorsFormat.java:311) ~[lucene-core-10.2.1.jar:?]
	at org.apache.lucene.index.CodecReader.searchNearestVectors(CodecReader.java:272) ~[lucene-core-10.2.1.jar:?]
	at org.apache.lucene.index.FilterLeafReader.searchNearestVectors(FilterLeafReader.java:369) ~[lucene-core-10.2.1.jar:?]
	at org.apache.lucene.index.FilterLeafReader.searchNearestVectors(FilterLeafReader.java:369) ~[lucene-core-10.2.1.jar:?]
	at org.elasticsearch.search.internal.FieldUsageTrackingDirectoryReader$FieldUsageTrackingLeafReader.searchNearestVectors(FieldUsageTrackingDirectoryReader.java:233) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableLeafReader.searchNearestVectors(ExitableDirectoryReader.java:166) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.vectors.IVFKnnFloatVectorQuery.approximateSearch(IVFKnnFloatVectorQuery.java:95) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.getLeafResults(AbstractIVFKnnVectorQuery.java:151) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.searchLeaf(AbstractIVFKnnVectorQuery.java:137) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at org.elasticsearch.search.vectors.AbstractIVFKnnVectorQuery.lambda$rewrite$0(AbstractIVFKnnVectorQuery.java:123) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:328) ~[?:?]
	at org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) ~[lucene-core-10.2.1.jar:?]
	at org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) ~[lucene-core-10.2.1.jar:?]
	... 6 more

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[iverase]

This was most probably the issue here: #129690