refactor(test): add sparse vector pruning tests #132264

mromaios · 2025-07-31T12:11:19Z

This PR

enhances SparseVectorFieldMapperTests by adding documents with various frequency tokens in the mock index and accompanying query vector set so that we can test 3 pruning scenarios (NO_PRUNING, DEFAULT_PRUNING, STRICT_PRUNING) more effectively.
adds a case in the relevant yaml tests to cover the following pruning scenario: token is over the frequency ratio and over the weight threshold.

…omaios/elasticsearch into increase_sparse_pruning_test_coverage

mromaios · 2025-08-05T15:20:37Z

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java

+        NO_PRUNING,      // No pruning applied - all tokens preserved
+        DEFAULT_PRUNING, // Default pruning configuration
+        STRICT_PRUNING   // Stricter pruning with higher thresholds
+    }


💭 I started of with these 3 scenarios for index and query configs as well, but quickly realised that those have 2 core dimensions. prune and pruningConfig, which for index you need to have prune=true to set a pruningConfig but that's not the case for queries, where the user can send any combination. This lead to the creation of separate enums for query and index scenarios.

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java

elasticsearchmachine · 2025-08-05T16:08:25Z

Pinging @elastic/search-eng (Team:SearchOrg)

elasticsearchmachine · 2025-08-05T16:08:25Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java

Mikep86

Nice cleanup! I left some comments about where we have opportunities to streamline even further :)

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java

…omaios/elasticsearch into increase_sparse_pruning_test_coverage

mromaios · 2025-08-12T11:18:06Z

Nice cleanup! I left some comments about where we have opportunities to streamline even further :)

Thanks @Mikep86 🙏 , great suggestions!
Had a go at implementing them b22eb47. Ready for another (final?🤞) look when you get the chance. Thanks!

Mikep86

LGTM, thanks for iterating!

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java

Mikep86 · 2025-08-12T13:35:53Z

I configured the PR to backport these changes to 9.1 & 8.19. We'll need to do some adjustments to the index version checks in the 8.19 backport, but it shouldn't be too bad.

mromaios · 2025-08-12T14:53:09Z

LGTM, thanks for iterating!

Thanks for all the support throughout!

I configured the PR to backport these changes to 9.1 & 8.19. We'll need to do some adjustments to the index version checks in the 8.19 backport, but it shouldn't be too bad.

Sounds good!

elasticsearchmachine · 2025-08-13T09:16:10Z

💔 Backport failed

Status	Branch	Result
❌	9.1	Commit could not be cherrypicked due to conflicts
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 132264

(cherry picked from commit 6cd330c) # Conflicts: # server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java

mromaios · 2025-08-13T10:42:29Z

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	8.19

Questions ?

Please refer to the Backport tool documentation

mromaios · 2025-08-13T13:51:22Z

Oops we have some failures:

#132810 (comment)

Investigating

Mikep86 · 2025-08-13T14:17:07Z

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java

-        performTypeQueryFinalizationTest(mapperService, null, null, false);
-    }
+        if (shouldPrune == null) {
+            shouldPrune = indexVersion.onOrAfter(SPARSE_VECTOR_PRUNING_INDEX_OPTIONS_SUPPORT);


One thing I overlooked here is that we also need to handle 8.x index versions that support default token pruning. This check needs to change to something like:

shouldPrune = indexVersion.between(IndexVersions.SPARSE_VECTOR_PRUNING_INDEX_OPTIONS_SUPPORT_BACKPORT_8_X, IndexVersions.UPGRADE_TO_LUCENE_10_0_0) || indexVersion.onOrAfter(SPARSE_VECTOR_PRUNING_INDEX_OPTIONS_SUPPORT)

Good catch. Will add it, thanks!

…2800)

)

chore: add sparse vector pruning tests

fe0757c

elasticsearchmachine added the v9.2.0 label Jul 31, 2025

mromaios and others added 7 commits August 1, 2025 15:12

Merge branch 'main' into increase_sparse_pruning_test_coverage

fcb8ed9

refactor tests with pruning scenarios

127b6f0

Merge branch 'increase_sparse_pruning_test_coverage' of github.com:mr…

c49c67e

…omaios/elasticsearch into increase_sparse_pruning_test_coverage

Merge branch 'main' into increase_sparse_pruning_test_coverage

75bb40a

update yaml tests

3546602

[CI] Auto commit changes from spotless

ee00d63

move things around

64a8ed8

mromaios commented Aug 5, 2025

View reviewed changes

remove TODOs

6fea090

mromaios commented Aug 5, 2025

View reviewed changes

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java Outdated Show resolved Hide resolved

mromaios marked this pull request as ready for review August 5, 2025 16:03

Merge branch 'main' into increase_sparse_pruning_test_coverage

1c14f78

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Aug 5, 2025

mromaios self-assigned this Aug 5, 2025

mromaios marked this pull request as draft August 5, 2025 16:04

mromaios added :SearchOrg/Relevance Label for the Search (solution/org) Relevance team >test Issues or PRs that are addressing/adding tests and removed needs:triage Requires assignment of a team area label labels Aug 5, 2025

mromaios marked this pull request as ready for review August 5, 2025 16:07

elasticsearchmachine added Team:SearchOrg Meta label for the Search Org (Enterprise Search) Team:Search - Relevance The Search organization Search Relevance team labels Aug 5, 2025

[CI] Auto commit changes from spotless

a79f56f

Mikep86 requested review from a team and Mikep86 August 5, 2025 16:26

mromaios commented Aug 5, 2025

View reviewed changes

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java Outdated Show resolved Hide resolved

mromaios changed the title ~~chore: add sparse vector pruning tests~~ refactor(test): add sparse vector pruning tests Aug 7, 2025

[CI] Auto commit changes from spotless

a519058

mromaios commented Aug 11, 2025

View reviewed changes

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java Outdated Show resolved Hide resolved

mromaios commented Aug 11, 2025

View reviewed changes

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java Outdated Show resolved Hide resolved

mromaios commented Aug 11, 2025

View reviewed changes

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java Show resolved Hide resolved

Mikep86 reviewed Aug 11, 2025

View reviewed changes

mromaios and others added 3 commits August 12, 2025 10:50

Merge branch 'increase_sparse_pruning_test_coverage' of github.com:mr…

a2bed06

…omaios/elasticsearch into increase_sparse_pruning_test_coverage

further cleanup, include pruningConfig in enum

b22eb47

Merge branch 'main' into increase_sparse_pruning_test_coverage

676a764

[CI] Auto commit changes from spotless

976f947

Mikep86 approved these changes Aug 12, 2025

View reviewed changes

server/src/test/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapperTests.java Show resolved Hide resolved

Mikep86 added auto-backport Automatically create backport pull requests when merged v9.1.3 v8.19.3 labels Aug 12, 2025

mromaios merged commit 6cd330c into elastic:main Aug 13, 2025
33 checks passed

elasticsearchmachine added the backport pending label Aug 13, 2025

mromaios mentioned this pull request Aug 13, 2025

[9.1] refactor(test): add sparse vector pruning tests (#132264) #132799

Merged

mromaios mentioned this pull request Aug 13, 2025

[8.19] refactor(test): add sparse vector pruning tests (#132264) #132800

Merged

mromaios mentioned this pull request Aug 13, 2025

[CI] SparseVectorFieldMapperTests testPruningScenarios failing #132810

Closed

Mikep86 reviewed Aug 13, 2025

View reviewed changes

mromaios mentioned this pull request Aug 14, 2025

test: fix SparseVectorFieldMapperTests edge case #132937

Merged

mromaios added a commit that referenced this pull request Aug 19, 2025

[8.19] refactor(test): add sparse vector pruning tests (#132264) (#13…

441f3a6

…2800)

mromaios added a commit that referenced this pull request Aug 19, 2025

[9.1] refactor(test): add sparse vector pruning tests (#132264) (#132799

fc95457

)

refactor(test): add sparse vector pruning tests #132264

refactor(test): add sparse vector pruning tests #132264

Uh oh!

Conversation

mromaios commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mromaios Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 5, 2025

Uh oh!

elasticsearchmachine commented Aug 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mromaios commented Aug 12, 2025

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Mikep86 commented Aug 12, 2025

Uh oh!

mromaios commented Aug 12, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 13, 2025

💔 Backport failed

Uh oh!

mromaios commented Aug 13, 2025

💚 All backports created successfully

Questions ?

Uh oh!

mromaios commented Aug 13, 2025

Uh oh!

Mikep86 Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

mromaios Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mromaios commented Jul 31, 2025 •

edited

Loading