Add more dense_vector details for cluster stats field stats #113607
Conversation
benwtrent
added
auto-backport-and-merge
v8.16.0
:Search Relevance/Search
|
Documentation preview: |
elasticsearchmachine
added
the
Team:Search Relevance
|
Pinging @elastic/es-search-relevance (Team:Search Relevance) |
|
Hi @benwtrent, I've created a changelog YAML for you. |
| indexedVectorDimMax = UNSET; | ||
| } | ||
|
|
||
| DenseVectorFieldStats(StreamInput in) throws IOException { |
There was a problem hiding this comment.
I noticed that this isn't actually used anywhere. I am going to double check that this is ok. But the underlying fieldstats serialization isn't via named writables & consequently the DenseVectorFieldStats is never actually written or read.
…ub.com:benwtrent/elasticsearch into feature/add-dense-vector-details-cluster-stats
|
@elasticmachine update branch |
server/src/main/java/org/elasticsearch/action/admin/cluster/stats/DenseVectorFieldStats.java
Outdated
Show resolved
Hide resolved
| stats = fieldTypes.computeIfAbsent(type, DenseVectorFieldStats::new); | ||
| boolean indexed = fieldMapping.containsKey("index") ? (boolean) fieldMapping.get("index") : false; | ||
| DenseVectorFieldStats vStats = (DenseVectorFieldStats) stats; | ||
| if (fieldMapping.containsKey("similarity")) { |
There was a problem hiding this comment.
I thought similarity can only be provided when indexed=true? Should we put this under indexed (line 101)?
| DenseVectorFieldStats vStats = (DenseVectorFieldStats) stats; | ||
| Object indexOptions = fieldMapping.get("index_options"); | ||
| // NOTE, while the default for `float` is now `int8_hnsw`, that is actually added to the mapping | ||
| // if the value is truly missing & we are indexed, we default to hnsw. |
There was a problem hiding this comment.
Under what condition this could happen that the value for index_options.type is missing?
There was a problem hiding this comment.
@mayya-sharipova when we default to index: true, but the element type is byte. in that scenario we won't provide a value to the mapping and its instead hnsw.
There was a problem hiding this comment.
@benwtrent Thanks, nice addition to vector stats. I left a couple of comments.
...c/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/70_dense_vector_telemetry.yml
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/action/admin/cluster/stats/DenseVectorFieldStats.java
Outdated
Show resolved
Hide resolved
…ector-details-cluster-stats
benwtrent
added
the
auto-merge-without-approval
|
@elasticmachine update branch |
| Map<String, Integer> vectorIndexTypeCount; // count of mappings by index type | ||
| Map<String, Integer> vectorSimilarityTypeCount; // count of mappings by similarity | ||
| Map<String, Integer> vectorElementTypeCount; // count of mappings by element type |
There was a problem hiding this comment.
should we be concerned about these becoming huge (and hence OOMs & co)?
There was a problem hiding this comment.
@tteofili I wouldn't think so. We have a static number of element types, index types, and similarity types. Since we are counting each separately, we won't have combinatoric explosion. These each will be hashmaps of < 10 elements each or so.
…113607) This adds some more counts for dense_vector field mapping stats. This allows for seeing the number of mappings with a given element type, similarity, or index type.
💚 Backport successful
|
…#113884) This adds some more counts for dense_vector field mapping stats. This allows for seeing the number of mappings with a given element type, similarity, or index type. Co-authored-by: Elastic Machine <[email protected]>
…113607) This adds some more counts for dense_vector field mapping stats. This allows for seeing the number of mappings with a given element type, similarity, or index type.
6 participants
This adds some more counts for dense_vector field mapping stats. This allows for seeing the number of mappings with a given element type, similarity, or index type.