Stress test for replication

[skyhos]

Hi, I am new in Qdrant.

I have created Qdrant cluster and doing stress test on it. Details as following:

• Qdrant version: v1.7.3
• 1 collection (consist of 5,000,000 of randomly generated vectors each of which has 2048 length), created with replication_factor 1.
• The collection divided into 12 Shards.
• Started with 6 nodes. (6 Kubernetes pods)
• Thus each node contains 2 shards.
• Send traffic as many as possible (using Locust cluster) to /collections/{name}/points/search API.

---

Now I tried to scale out of replication as following steps:

1. Update replication_factor of the collection as 2 with the API:

curl -X 'PATCH' -H 'Content-Type: application/json' --data-raw '{"params": {"replication_factor": 2}}' http://qdrant_host/collections/test_collection

2. Scale out Kubernetes Qdrant statefulset so that we have 12 Qdrant nodes.
3. Replicate each shard with the API:

curl -X 'POST' -H 'Content-Type: application/json' --data-raw '{"replicate_shard": {"shard_id": <shard_id>, "from_peer_id": <from_peer_id>,"to_peer_id": <to_peer_id>}}' http://qdrant_host/collections/test_collection/cluster

4. Waiting for each replicating process completes.

---

Now I see that something worse, I don't see some performance improvement:

• Rate of total response count ( or RPS) is similar or less than before.
• CPU usage of each node did not decrease.
• Frequently, I see long-latency responses:
  • Before: most latency were less than 180ms
  • After: some latencies (p95) were more than 500ms
• It seems that whole system becomes unstable with replication.

Did I something wrong? Please, tell me what I missed.

[timvisee]

Please consider to upgrade to Qdrant 1.14.0 first. We've made a lot of improvements along the way which can have a significant effect on your results.

Also make sure that all data is fully indexed. To check this, ensure that each node reports a green collection status.

If you're using any kind of payload filtering, you must set up the appropriate payload index.

In Qdrant 1.14 we now have the 'query' API. I recommend to switch to that instead, it should require minimal changes.

Coul you also share the following:

• what kind of hardware are you running? (CPU, RAM, storage kind)
• what does your collection info look like? (GET /collections/{name})
• an example of a search query you're doing.

Would you mind to rerun your tests with the above suggestions.