score_mode not working as expected in nested knn queries

[nit0906]

Elasticsearch Version

8.17.2

Installed Plugins

No response

Java Version

21.0.5

OS Version

docker-image = docker.elastic.co/elasticsearch/elasticsearch:8.17.2

Problem Description

I am trying to do a nested knn query on a vector field. I am trying to use the score_mode option to have the top level document score be decided based on the inner hits scores.

However it looks like only the mode max is working not matter which mode I select. For instance, I have 2 document hits, each with 2 inner hits.

doc1 :
innerHit1: 0.9055939
innerHit2: 0.8818512

doc2:
innerHit1: 0.90008545
innerHit2: 0.89920044

Now, even if I select score_mode=avg/min/sum - the doc1 score is always 0.9063721 and doc2 score is always 0.8989563, which tends to indicate that only max score_mode is working.

However if I select score_mode = none, the doc scores are 0 - so that seems to be working.

This is the query that I am testing out -

{ "query": { "nested": { "path": "foo.bar", "score_mode": "sum", "inner_hits": {}, "query": { "knn": { "boost": 1, "field": "foo.bar.vector", "query_vector": [ -0.0109421825, -0.023024175, 0.0008527628, -0.00867597, -0.0038518896, .... ], "num_candidates": 3 } } } } }

Steps to Reproduce

steps_to_reproduce.txt

The file consists of index mapping, test documents to be ingested and query to reproduce the issue.

The resulting scores are as follows -

doc1 :
innerHit1: 0.9055939
innerHit2: 0.8818512

doc2:
innerHit1: 0.90008545
innerHit2: 0.89920044

and as explained in the description, the score for top level docs remain the same irrespective of the score_mode being chosen.

Logs (if relevant)

No response

[benwtrent]

which tends to indicate that only max score_mode is working.

Max score is the only score mode allowed as we only search for nearest neighbors. Potentially new score mode support could be added.

What is your goal here?

Do you want to search by nearest neighbor (getting the overall nearest docs) and THEN order the inner hits based on a different mode?

Or do you want to search for parent docs based on their average or worst inner doc?

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[nit0906]

Hi @benwtrent

Max score is the only score mode allowed as we only search for nearest neighbors

Thanks for confirming this. I could not find this documented any place.

do you want to search for parent docs based on their average or worst inner doc?

Yes, this is what I want to do.

To explain a bit in detail, basically for my use case, the content of the parent doc would be chunked and then these chunks would be vectorised as a list of sub objects in the main document (vectorSpaces.semantic.vector in the example file). I am the doing a nested knn query on this vector field and would be able to want return the main documents based on the average, sum or max of the inner docs (the chunks). Returning the parent docs based on the max score of the inner hit (chunk) works for one of my use cases, but there are some cases that might need average or sum.

I hope this makes it more clear ? Thanks.

[benwtrent]

would be able to want return the main documents based on the average, sum or max of the inner docs (the chunks).

To do sum and max I think the simpliest thing would be to index two more vectors:

• One that is the sum of the inner-docs vector components
• One that is the average of the inner-docs vector components

Since dot-product is just the component product sum, you should be able to search over the "sum" components and the "avg" components and achieve what you want.

Tricky part with "sum" is that this would then take into account vector magnitude, if this is important to you, I would index that vector with max_inner_product instead of cosine or dot_product.