Seeking Guidance on HNSW Index Comparison Between OpenSearch and Milvus

[dmitriivahrushev]

Dear Milvus Team,

Hello! I hope this message finds you well. I'm writing to seek your expert advice on some observations I've made while comparing HNSW implementations between OpenSearch and Milvus.

Context & Setup:
I have two vector databases: OpenSearch and Milvus.
I'm using COSINE metric in both systems.

Both contain 55 million vectors (face embeddings from photographs)
Both use HNSW indexes with identical parameters: m=16, ef_construction=1024
I'm performing vector similarity searches and comparing the results

Current Observations:
Top-1 match rate between OpenSearch and Milvus is approximately 70%
Example mismatch:

Mismatch! Query vector[vector] | Milvus result: people_222 | OpenSearch result: people_333

(Note: ID people_333 does exist in Milvus but doesn't appear as Top-1)

Interesting finding: When I increase the limit parameter in the search query (/v2/vectordb/entities/search), the match rate improves significantly.
Search parameters impact:

"searchParams": {
    "metricType": "COSINE",
    "params": {
        "radius": 0.1
    }
}

With low limit + radius: 0.1, I get completely different matches than without these parameters.

Questions for Your Expertise:
1)Validation of approach: Is it fair/correct to compare how Milvus HNSW vs OpenSearch HNSW identifies people in photos using this methodology?
2)Algorithm differences: If there are differences in HNSW search implementations between Milvus and OpenSearch, how does Milvus process and return results differently?
3)Parameter clarification: Could you help me understand how limit and radius parameters work together in Milvus?
Why might I not get the highest similarity match without radius: 0.1?
How does radius filter or reorder results?
Is there an optimal way to combine limit, radius, and possibly ef_search for most accurate Top-1 retrieval?

I truly appreciate any insights you can share! The Milvus documentation is excellent, but I believe your team's deep understanding could help clarify these subtle behaviors.
Thank you so much for your time and for building such an incredible product!

[dmitriivahrushev]

Could you please clarify if there are differences in the implementation of the HNSW algorithm between Milvus and OpenSearch? Thank you!

[yhmo]

• "Top-1 match rate between OpenSearch and Milvus is approximately 70%" ----------------------- I think you should compare the recall rate of Milvus and OpenSearch instead of comparing their match rate. Match rate is meaningless. You can use "FLAT" index to perform a full-scan on all vectors to get an actual result, and compare the results of Milvus and OpenSearch with this actual result.

• So far as I know, both milvus and opensearch manage data as segments, the default size of a segment is different. Milvus is 1GB per segment, opensearch is 5GB per segment. Each segment has an independent index. Although the M/ef_construction is the same, the hnsw graph is different between milvus segment and opensearch segment since the number of vectors are different.

• Theoretically, the result should not be much different between with and without radius. I need to do some tests to confirm.

[xiaofan-luan]

@dmitriivahrushev r u compare opensearch topk search with milvus range search?
it's better to do apple to apple compare, using topk search. Rnage search is usually more expensive and less accurate

[dmitriivahrushev]

Great tips, thanks a lot!