Skip to content

[DiskBBQ] Make centroid oversampling depend on the visit ratio #133945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Sep 2, 2025
Merged

[DiskBBQ] Make centroid oversampling depend on the visit ratio #133945

merged 8 commits into from
Sep 2, 2025

Conversation

@iverase
Copy link
Contributor

@iverase iverase commented Sep 1, 2025

We have currently a fixed oversampling of 20% of the total centroid. This works well for low / medium number of centroids but starts being quite slow when the number of centroids increases. Now that we introduced a visit ratio we can make it dependent on that an oversample the number of centroids we expect to visit by a constant.

Local experiment shows no lost of recall and a nice speed up when the number of centroids is high.

@elasticsearchmachine elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Sep 1, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search-relevance (Team:Search Relevance)

@benwtrent benwtrent self-requested a review September 2, 2025 13:04
@iverase iverase merged commit 0080d2c into elastic:main Sep 2, 2025
33 checks passed
@iverase iverase deleted the centroid_versample branch September 2, 2025 15:05
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Sep 11, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
6772b34 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Sep 11, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
14829cd 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Sep 16, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
bf09280 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 8, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
014b148 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 16, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
3493e57 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
2ce0cfa 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
5099fc7 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
0f45d01 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
97c3db2 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
0b98bf7 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
0ff70f7 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
6e9dc21 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
42ee7c8 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
0e29fee 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
5560496 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
4c57b8b 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 22, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
8a79fdf 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 23, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
d7ca611 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 23, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
8d17ee7 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 23, 2025
@phananh1010
Mirror upstream elastic#133945 as single snapshot commit for AI review
b1a0253 
BASE=3c264cff967750ff7cb581defa42b0d69bb31fc6
HEAD=0e24fc739f9daa5b003d20fb0fffaf197449db67
Branch=main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@benwtrent benwtrent benwtrent approved these changes

@john-wagster john-wagster Awaiting requested review from john-wagster

Assignees

No one assigned

Labels

>non-issue :Search Relevance/Vectors Vector search Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch v9.2.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@iverase @elasticsearchmachine @benwtrent