Adding new experimental bbq index types

[benwtrent]

new index types of bbq_hnsw and bbq_flat which utilize the better binary quantization formats. A 32x reduction in memory, with nice recall properties.

[mayya-sharipova]

We can use "k" instead of num_candidates to retrieve 20 results.

[benwtrent]

I can, but I would just set it to k. I don't see the value.

[mayya-sharipova]

Accept all the math in scorers which I don't completely follow, the rest LGTM.

Thanks @benwtrent. Great addition!

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @benwtrent, I've created a changelog YAML for you.

[benwtrent]

@elasticmachine update branch

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 114439

[benwtrent]

Did a larger scale benchmark over 134,705,698 1024 Coherev3 vectors on a single 64 GB node (about 32GB of off-heap allocated).

This was done with rally track: https://github.com/elastic/rally-tracks/tree/master/msmarco-v2-vector

I would expect rescoring per shard to have a slightly bigger latency hit, but better recall.

Took about 12hrs to ingest with aggressive merging enabled.

fmt: knn-recall-k-num_candidates-number_rescored

Here are some of the results:

Shows that oversampling and rescoring helps, but can hurt qps.
knn-recall-10-20_no_rescore
recall: 0.45
Avg Nodes Visited: 15,915.211
78% best ndgc
Single client latency: 9ms
Multi-client QPS: 1,134.649

knn-recall-10-100-50.
Its interesting that even at 5x oversampling, its latency isn't much worse and at 10x num candidates, it only visits 2x more vectors.
Recall: 0.704
Avg Nodes Visited: 36,079.801
90% of best NDGC
Multi-client QPS: 451.596 (30ms latency)
Single client latency: 18ms

knn-recall-10-1000-200: its pretty neat to see that even visiting >100k vectors over multiple segments has 42ms latency (we ain't even done optimizing this stuff yet).
recall: 0.895
AvgNodesVisited: 115,598.117
97% best ndgc
Single client latency: 42.534ms
Multi-client QPS: 167.806 (93ms)

[benwtrent]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

[flobernd]

Hi @benwtrent, would it be possible to add these types to the specification?

The types are currently missing in the client and there is no documentation available for them either.

Happy to add the new enum members myself, if somebody could provide a proper documentation string 🙂