Clarify incremental HNSW reuse semantics

[sfmqrb]

When incremental HNSW building is enabled, we pick an old index and only insert new points. Does this reuse mean we literally keep the old graph’s links (after healing missing/deleted nodes) and just add links for new points, or is there additional rebuilding/rewiring beyond healing and new inserts?

[sfmqrb]

Hi there,

I'd really appreciate your response to this question.

Thanks!

[sfmqrb]

@timvisee @generall
Hey,

Will you please take a look at this?

[generall]

i saw your question, it requires a deep dive into the code, which I would recommend you do on your own. Please don't tag me

[sfmqrb]

Thanks for your response.
I already did the deep dive. I was just looking for confirmation.

[xzfc]

Does this reuse mean we literally keep the old graph’s links (after healing missing/deleted nodes) and just add links for new points, or is there additional rebuilding/rewiring beyond healing and new inserts?

• Main graph links are kept as is. (i.e. the first m/m0 links of each point). So, "we literally keep the old graph’s links" for them.
• Additional graph links (see docs) are dropped and rebuilt from scratch.
• Everything else is the same as when building from scratch.

Related files:
https://github.com/qdrant/qdrant/blob/e293ab1f08d3eb8fb8d9c3380c1de71b7a76570c/lib/segment/src/index/hnsw_index/hnsw.rs
https://github.com/qdrant/qdrant/blob/e293ab1f08d3eb8fb8d9c3380c1de71b7a76570c/lib/segment/src/index/hnsw_index/graph_layers_healer.rs

[sfmqrb]

Thank you, @xzfc!

Everything else is the same as when building from scratch.

I’m a bit confused about this part. Are the additional graph links only created and used for filtering?

So, if my vectors have no filterable properties, does the incremental HNSW logic essentially behave like incrementally inserting the new points into the existing (main) HNSW graph?

[xzfc]

I’m a bit confused about this part. Are the additional graph links only created and used for filtering?

Yes. They are created only when you have configured the payload indices.

So, if my vectors have no filterable properties, does the incremental HNSW logic essentially behave like incrementally inserting the new points into the existing (main) HNSW graph?

Yes.

[sfmqrb]

Thank you!