DiskBBQ Updates #4037
Open
DiskBBQ Updates #4037
+56
−6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…etter basis for diskbbq in the docs at all; added that
🔍 Preview links for changed docs |
Vale Linting ResultsSummary: 20 suggestions found 💡 Suggestions (20)
|
benwtrent
reviewed
Nov 21, 2025
|
|
||
| HNSW is a graph-based algorithm which only works efficiently when most vector data is held in memory. You should ensure that data nodes have at least enough RAM to hold the vector data and index structures. | ||
|
|
||
| DiskBBQ is a clustering algorithm which can scale effeciently on a fraction of the total memory. You can start with enough RAM to hold the vector data and index structures but it's likely you will be able to use signifigantly less than this and still maintain good performance. In testing we find this will be between 1-5% of the index structure size (centroids and quantized vector data) per unique query where unique queries access non-overlapping clusters. |
Member
There was a problem hiding this comment.
Let's be careful here, obviously, going to disk is always slower than just reading things in memory. We need to clarify that the performance degrades more linearly than with HNSW, which degrades exponentially.
I think calling about these percentages is OK.
Contributor
Author
There was a problem hiding this comment.
makes sense I'll reword to clarify
Contributor
Author
There was a problem hiding this comment.
reworded this a bit let me know what you think now
benwtrent
reviewed
Nov 21, 2025
|
|
||
| If utilizing HNSW, the graph must also be in memory, to estimate the required bytes use `num_vectors * 4 * HNSW.m`. The default value for `HNSW.m` is 16, so by default `num_vectors * 4 * 16`. | ||
|
|
||
| If utilizing DiskBBQ, a fraction of the clusters and centroids will need to be in memory. When doing this estimation it makes more sense to include both the index structure and the quantized vectors together as the structures are dependent. To estimate the total bytes we compute the cost of the centroids as `num_clusters * num_dimensions * 2 * 4 + num_clusters * (num_dimensions + 14)` plus the cost of the quantized vectors within the clusters as `num_vectors * ((num_dimensions/8 + 14 + 2) * 2)` where `num_clusters` is defined as `num_vectors / vectors_per_cluster` which by default will be `num_vectors / 384` |
Member
There was a problem hiding this comment.
This seems pretty complicated...
num_clusters * num_dimensions * 2 * 4 you need two floating point representations of the clusters?
Contributor
Author
There was a problem hiding this comment.
That 2 is definitely not right. Removing that.
Went back and double checked what bytes are there.
posting list w/o vectors = num_clusters * (centroid_bytes + dotProd(centroid,centroid) + clusterSize + encoding)
posting list w/o vectors = num_clusters * ((dims * 4) + 4 + 4 + 1)
I dont think it's worth it to describe the additional bytes. That would just make this even more complicated.
Suggestions for simplifying it are welcome. It probably seems more complex because it includes both the "index structure" for the centroids and the vectors because of SOAR. Maybe I can reference the bbq quantized calculation from above and somewhat simplify the centroids figures. So something like this instead. Thoughts?
the cost of the centroids as `num_clusters * (num_dimensions * 4 + (num_dimensions + 14))`
plus the cost of the quantized vectors within the clusters as `bbq_quantizated_vectors * 2`
The downside is this ignores the doc id (2 byte) cost per vector. I almost feel like it makes it more complicated to mention this rather than not if we break it up for instance like this:
plus the cost of the quantized vectors within the clusters as `bbq_quantizated_vectors * 2` + `num_vectors * 2 * 2`
I'll think about it some more.
… in the overhead calcs
…o diskbbq_updates
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added content around DiskBBQ to the ANN docs. Specifically needed to do this so we could address sizing for DiskBBQ.
I have a somewhat related PR here: elastic/elasticsearch#138433, which should probably go in first.