Adding MinScore support to Linear Retriever #124182

mridula-s109 · 2025-03-06T06:36:36Z

This PR introduces support for the min_score parameter in the linear retriever. The min_score filter ensures that only documents with a relevance score above a specified threshold are retrieved, improving result quality and filtering out low-relevance matches.

Added min_score support in the linear retriever.
Updated the retrieval logic to apply the min_score threshold.
Based on an example from the Elasticsearch Guide (8.18), extended it to demonstrate min_score usage.

mridula-s109 · 2025-03-06T06:37:22Z

WIP - The testcases are failing, i have been trying to fix it. Also need to write yaml tests.

kderusso

Nice start @mridula-s109

I left some comments. Additionally, it looks like you have some test compilation errors. I'd also love to see some yaml tests here, and we'll want to update our documentation to reflect the min score (I'm unsure of the state of our markdown documentation, but if we could do it in the same PR it would be ideal).

kderusso · 2025-03-06T13:36:16Z

server/src/main/java/org/elasticsearch/index/query/RankDocsQueryBuilder.java

        if (in.getTransportVersion().onOrAfter(TransportVersions.V_8_16_0)) {
            this.queryBuilders = in.readOptionalArray(c -> c.readNamedWriteable(QueryBuilder.class), QueryBuilder[]::new);
            this.onlyRankDocs = in.readBoolean();
+            this.minScore = in.readFloat();


As this is written, it will break transport serialization.

What you want to do, is register a new TransportVersion for your change in TransportVersions.java. You'll then reference this new transport version to serialize the min score - older transport versions will always just use a default value.

If i want to include min_score without transport versioning changes, then is the way it is currently written fine?

server/src/main/java/org/elasticsearch/index/query/RankDocsQueryBuilder.java

kderusso · 2025-03-06T13:37:25Z

server/src/main/java/org/elasticsearch/index/query/RankDocsQueryBuilder.java

        if (out.getTransportVersion().onOrAfter(TransportVersions.V_8_16_0)) {
            out.writeOptionalArray(StreamOutput::writeNamedWriteable, queryBuilders);
            out.writeBoolean(onlyRankDocs);
+            out.writeFloat(minScore);


You'll need to reference the new transport version you create here as well, to ensure that serialization is consistent.

server/src/main/java/org/elasticsearch/search/retriever/KnnRetrieverBuilder.java

...lugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverBuilder.java

...plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/MinMaxScoreNormalizer.java

...rf/src/test/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverBuilderParsingTests.java

…inear it

elasticsearchmachine · 2025-03-13T18:48:34Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-03-13T18:50:37Z

Hi @mridula-s109, I've created a changelog YAML for you.

Samiul-TheSoccerFan

Nice work, LGTM. Left some comments

server/src/main/java/org/elasticsearch/index/query/RankDocsQueryBuilder.java

server/src/main/java/org/elasticsearch/search/retriever/KnnRetrieverBuilder.java

...plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/linear/10_linear_retriever.yml

elasticsearchmachine · 2025-03-14T22:54:57Z

Pinging @elastic/search-eng (Team:SearchOrg)

elasticsearchmachine · 2025-03-14T22:54:57Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

server/src/main/java/org/elasticsearch/index/query/RankDocsQueryBuilder.java

server/src/main/java/org/elasticsearch/search/retriever/rankdoc/RankDocsQuery.java

...-rrf/src/internalClusterTest/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverIT.java

docs/changelog/124182.yaml

server/src/main/java/org/elasticsearch/index/query/RankDocsQueryBuilder.java

server/src/main/java/org/elasticsearch/search/retriever/KnnRetrieverBuilder.java

server/src/test/java/org/elasticsearch/index/query/RankDocsQueryBuilderTests.java

...lugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverBuilder.java

...plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/MinMaxScoreNormalizer.java

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/rrf/RRFRankDoc.java

...plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/linear/10_linear_retriever.yml

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilder.java

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRankDoc.java

mridula-s109 · 2025-04-03T17:57:37Z

Closing this PR in favor of a fresh one: #126238 — will be published shortly.

All feedback and comments from this PR will be carried forward into the new, cleaner version. Thanks for all the input so far!

kderusso

Stuff that triggered my spidey senses 😁

server/src/main/java/org/elasticsearch/search/retriever/RankDocsRetrieverBuilder.java

server/src/main/java/org/elasticsearch/search/retriever/rankdoc/RankDocsQuery.java

...ugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/IdentityScoreNormalizer.java

...lugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverBuilder.java

MinScore implementation in Linear retriever

a41dac9

elasticsearchmachine added the v9.1.0 label Mar 6, 2025

mridula-s109 requested a review from kderusso March 6, 2025 06:37

[CI] Auto commit changes from spotless

61dd8df

kderusso reviewed Mar 6, 2025

View reviewed changes

mridula-s109 added 2 commits March 12, 2025 17:42

Merge remote-tracking branch 'origin/minscore-linear' into minscore-l…

6be41de

…inear it

Resolving PR comments

a52628c

mridula-s109 added auto-backport Automatically create backport pull requests when merged :Search Relevance/Search Catch all for Search Relevance labels Mar 13, 2025

Fixed PR comments, added yaml and made changes to the markdown

a225c01

mridula-s109 marked this pull request as ready for review March 13, 2025 18:48

elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Mar 13, 2025

Merge branch 'main' into minscore-linear

429a620

mridula-s109 added the >enhancement label Mar 13, 2025

Update docs/changelog/124182.yaml

95710e6

mridula-s109 requested a review from a team March 13, 2025 18:52

[CI] Auto commit changes from spotless

a32f947

Samiul-TheSoccerFan reviewed Mar 14, 2025

View reviewed changes

Samiul-TheSoccerFan added the :SearchOrg/Relevance Label for the Search (solution/org) Relevance team label Mar 14, 2025

jimczi requested changes Mar 17, 2025

View reviewed changes

mridula-s109 and others added 3 commits March 20, 2025 09:46

Resolved on the PR comments

a424e77

[CI] Auto commit changes from spotless

38a9b50

Added changes wrt to yaml testing from PR comments

810d151

kderusso reviewed Mar 25, 2025

View reviewed changes

jimczi reviewed Mar 25, 2025

View reviewed changes

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilder.java Outdated Show resolved Hide resolved

jimczi reviewed Mar 25, 2025

View reviewed changes

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRankDoc.java Outdated Show resolved Hide resolved

mridula-s109 added 4 commits March 26, 2025 12:55

Worked on kathleen comments first half

88703fd

Reverted the integration test in line with the main branch

d9b44e2

Resolved comments in the PR and its in compiling state

46a1b94

Unit tests passing

73a9bad

mridula-s109 marked this pull request as draft March 26, 2025 18:35

elasticsearchmachine and others added 3 commits March 26, 2025 18:46

[CI] Auto commit changes from spotless

6f93d3d

reverted inclusion of pit in retrierver it file

1fd4a22

Removed transport versions

f53cd0a

mridula-s109 marked this pull request as ready for review March 27, 2025 12:57

mridula-s109 marked this pull request as draft March 27, 2025 13:13

mridula-s109 and others added 3 commits March 27, 2025 14:26

Modified rrfRank doc to the way main was

9be3783

Committing the changes done until now, will be doing a clean commit next

7e2f732

[CI] Auto commit changes from spotless

3feaa3f

mridula-s109 closed this Apr 3, 2025

kderusso reviewed Apr 8, 2025

View reviewed changes

kderusso mentioned this pull request Apr 8, 2025

Adding MinScore support to Linear Retriever #126238

Closed

Adding MinScore support to Linear Retriever #124182

Adding MinScore support to Linear Retriever #124182

Uh oh!

Conversation

mridula-s109 commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mridula-s109 commented Mar 6, 2025

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

kderusso Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

mridula-s109 Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kderusso Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 13, 2025

Uh oh!

elasticsearchmachine commented Mar 13, 2025

Uh oh!

Samiul-TheSoccerFan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 14, 2025

Uh oh!

elasticsearchmachine commented Mar 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mridula-s109 commented Apr 3, 2025

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mridula-s109 commented Mar 6, 2025 •

edited

Loading