initial changes to support dense vectors off-heap stats

Author: ChrisHegarty

distribution/tools/server-cli/src/main/java/org/elasticsearch/server/cli/SystemJvmOptions.java

@@ -62,6 +62,8 @@ static List<String> systemJvmOptions(Settings nodeSettings, final Map<String, St
                 "-Dio.netty.noUnsafe=true",
                 "-Dio.netty.noKeySetOptimization=true",
                 "-Dio.netty.recycler.maxCapacityPerThread=0",
+                // temporary until we get offheap vector stats in Lucene 10.3
+                "--add-opens=org.apache.lucene.core/org.apache.lucene.codecs.lucene99=org.elasticsearch.server",
                 // log4j 2
                 "-Dlog4j.shutdownHookEnabled=false",
                 "-Dlog4j2.disable.jmx=true",

docs/reference/cluster/nodes-stats.asciidoc

@@ -834,6 +834,27 @@ This is not shown for the `shards` level, since mappings may be shared across th
 `value_count`::
 (integer) Total number of dense vector indexed across all shards assigned to the node.
 
+`off_heap`::
+(object) Statistics about the desired size of off-heap memory for all vectors indexed across all shards assigned to the node.
++
+.Properties of `off_heap`
+[%collapsible%open]
+========
+`total_size_in_bytes`::
+(long) The total desired size in bytes of off-heap memory.
+
+`total_veb_size_in_bytes`::
+(long) The total desired size in bytes of off-heap memory for `bbq` quantized vectors` (<<dense-vector-file-extensions,veb>>).
+
+`total_vec_size_in_bytes`::
+(long) The total desired size in bytes of off-heap memory for all non-quantized vectors (<<dense-vector-file-extensions,vec>>).
+
+`total_veq_size_in_bytes`::
+(long) The total desired size in bytes of off-heap memory for vectors indexed with quantization: int4 or int8 (<<dense-vector-file-extensions,veq>>).
+
+`total_vex_size_in_bytes`::
+(long) The total desired size in bytes of off-heap memory for the HNSW graph (<<dense-vector-file-extensions,vex>>).
+
 =======
 
 `sparse_vector`::

docs/reference/how-to/knn-search.asciidoc

@@ -98,6 +98,7 @@ of datasets and configurations that we use for our nightly benchmarks.
 [[dense-vector-preloading]]
 include::search-speed.asciidoc[tag=warm-fs-cache]
 
+[[dense-vector-file-extensions]]
 The following file extensions are used for the approximate kNN search:
 Each extension is broken down by the quantization types.
 

docs/reference/rest-api/common-parms.asciidoc

@@ -508,7 +508,7 @@ Return all statistics.
 <<completion-suggester,Completion suggester>> statistics.
 
 `dense_vector`::
-Total number of dense vectors indexed.
+Statistics about dense vectors. Includes the total number of dense vectors indexed, and the desired size of off-heap memory.
 <<indices-refresh,Index refreshes>> can affect this statistic.
 
 `docs`::

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/cluster.stats/10_basic.yml

@@ -351,6 +351,7 @@
   - match: { indices.docs.count: 5 }
   - match: { indices.docs.deleted: 0 }
   - match: { indices.dense_vector.value_count: 8 }
+  - is_false: indices.dense_vector.off_heap
 
 ---
 "Sparse vector stats":