How should I use MilvusClientV2Pool when accessing multiple databases?

[pipi-olo]

I am using MilvusClientV2Pool. Since I need to access multiple databases, I create a separate ClientPool for each database, using the collection name as the key.

My problem is that when I use ClientPool, the latency becomes significantly higher compared to when I don't use it.

Here are my questions:

Q1. Am I using ClientPool incorrectly? Please let me know the proper way to access multiple Milvus databases.

Q2. Should I use client.useDatabase within a single ClientPool? When I do this, the performance becomes very slow.

Q3. Is there a way to collect metrics for MilvusClient in a Java application?

[yhmo]

MilvusClientV2Pool uses a predefined ConnectConfig to create MilvusClient objects. All MilvusClient objects are using the same ConnectConfig:

ConnectConfig connectConfig = ConnectConfig.builder()
                .uri("http://localhost:19530")
                .dbName("aaa")
                .build();
MilvusClientV2Pool pool = new MilvusClientV2Pool(poolConfig, connectConfig);

In MilvusClientV2Pool, each key represents an auto-scaled pool of MilvusClient objects. When a thread want to get a client from the pool, the pool returns an idle client to the thread. If no idle client is available, the pool will create a new MilvusClient object with the predefined ConnectConfig. Only when the thread returns the client by returnClient(), the client is marked as "idle" again.

Pool before scaled:

Pool after scaled:

[yhmo]

For each pool, the max number of MilvusClient objects are defined by the PoolConfig.maxTotalPerKey. If all the client objects are used by getClient(), and the number of MilvusClient objects has reached the maxTotalPerKey, the next call of getClient() will have to wait no more than PoolConfig.maxBlockWaitDuration. If the getClient() could not get an idle client after PoolConfig.maxBlockWaitDuration, it will throw an exception "Timeout waiting for idle object".

[yhmo]

Use these methods to observe the number of MilvusClient objects in each pool:

    public int getIdleClientNumber(String key) 
    public int getActiveClientNumber(String key)
    public int getTotalIdleClientNumber()
    public int getTotalActiveClientNumber()

An example to demo the major functions of ClientPool: https://github.com/milvus-io/milvus-sdk-java/blob/master/examples/src/main/java/io/milvus/v2/ClientPoolExample.java

[yhmo]

For multi-database usage, one ClientPool currently can only handle one database since there is only one pre-defined ConnectConfig. One pool for one database:

 ConnectConfig connectConfig_A = ConnectConfig.builder()
                .uri("http://localhost:19530")
                .dbName("aaa")
                .build();
 MilvusClientV2Pool pool_A = new MilvusClientV2Pool(poolConfig, connectConfig_A);


ConnectConfig connectConfig_B = ConnectConfig.builder()
                .uri("http://localhost:19530")
                .dbName("bbb")
                .build();
 MilvusClientV2Pool pool_B = new MilvusClientV2Pool(poolConfig, connectConfig_B);

I will consider supporting multiple databases in one ClientPool.

[yhmo]

Typically, you can use one key for all collections in one database, no need to use different keys for different collections.

MilvusClientV2Pool pool_A = new MilvusClientV2Pool(poolConfig, connectConfig);

Thread t1 = new Thread(() -> {
    MilvusClientV2 client = pool_A.getClient("for_all_collections");
    client.search(SearchReq.builder().collectionName("coll_1").build());
});
t1.start();

Thread t2 = new Thread(() -> {
    MilvusClientV2 client = pool_A.getClient("for_all_collections");
    client.search(SearchReq.builder().collectionName("coll_2").build());
});
t2.start();

[xiaofan-luan]

For each pool, the max number of MilvusClient objects are defined by the PoolConfig.maxTotalPerKey. If all the client objects are used by getClient(), and the number of MilvusClient objects has reached the maxTotalPerKey, the next call of getClient() will have to wait no more than PoolConfig.maxBlockWaitDuration. If the getClient() could not get an idle client after PoolConfig.maxBlockWaitDuration, it will throw an exception "Timeout waiting for idle object".

Why do we need to wait?

We can create a new connection, and if when we returned the client number > maxTotalPerKey we destroy the client

[yhmo]

For each pool, the max number of MilvusClient objects are defined by the PoolConfig.maxTotalPerKey. If all the client objects are used by getClient(), and the number of MilvusClient objects has reached the maxTotalPerKey, the next call of getClient() will have to wait no more than PoolConfig.maxBlockWaitDuration. If the getClient() could not get an idle client after PoolConfig.maxBlockWaitDuration, it will throw an exception "Timeout waiting for idle object".

Why do we need to wait?

We can create a new connection, and if when we returned the client number > maxTotalPerKey we destroy the client

100 threads call getClient() at the same time, there will be maxTotalPerKey + 100 clients in use

[xiaofan-luan]

what is the default value of maxTotalPerKey?

we should at least set it to 300 or more.

[pipi-olo]

Typically, you can use one key for all collections in one database, no need to use different keys for different collections.

MilvusClientV2Pool pool_A = new MilvusClientV2Pool(poolConfig, connectConfig);

Thread t1 = new Thread(() -> {
    MilvusClientV2 client = pool_A.getClient("for_all_collections");
    client.search(SearchReq.builder().collectionName("coll_1").build());
});
t1.start();

Thread t2 = new Thread(() -> {
    MilvusClientV2 client = pool_A.getClient("for_all_collections");
    client.search(SearchReq.builder().collectionName("coll_2").build());
});
t2.start();

That's right.
We used milvus client pools, all using the same key.
It's faster than having a different key for each collection, but it's still slow.

[pipi-olo]

what is the default value of maxTotalPerKey?

we should at least set it to 300 or more.

My goal is QPS of over 200 with a Search P99 of 100ms. In the Milvus JDK library v2.5.12, the defaults are maxIdlePerKey: 5, minIdlePerKey: 0, maxTotalPerKey: 10, and maxTotal: 50. I've set all these values to 15 with a replica count of 4.
Is there a reason why maxTotal needs to be at least 300?

cf) I have removed the milvus client pool. I am using a milvus client with a replica count of 12. Through this, I have achieved the level I was aiming for.

Is the Milvus client implemented to be blocking after a search?
If two search requests come in for one client, does the second search only start after the first one has completely finished?

[yhmo]

• minIdlePerKey ---------------------- the pool will reserve at least "minIdlePerKey" of idle clients for per key, even if no one calls getClient(). If you set minIdlePerKey = 1, the pool will automatically create one client for when the sub-pool of the key is created. Since the first call of getClient(key) is to create a sub-pool and ask a client, it is no difference whether set minIdlePerKey to 0 or 1. If you set minIdlePerKey=5, the first call of getClient(key) will create 5 clients and return one of them, it might take 1 second(create 5 clients, each takes 200ms to connect milvus server) before returning this client.

• maxIdlePerKey --------------------- If the number of clients of a key is smaller than this value, and someone calls getClient(), the pool will create a new client for the caller. If the number of idle clients of a key exceeds maxIdlePerKey and no one calls getClient(), the pool will destroy some clients until the number of idle clients is equal to this value, which means maxIdlePerKey clients will remain in memory, waiting for the next call of getClient().

• maxTotalPerKey -------------------- the max total number of clients per key: idle clients + in-use clients, cannot exceed this value. If the number of in-use clients is equal this value, the pool will not allow to create new client for the next call of getClient(). The next call of getClient() will wait maxBlockWaitDuration, if no client becomes idle in maxBlockWaitDuration, getClient() will throw an exception.

• maxTotal ----------------------------- the max total number of clients of all keys: all idle clients + all in-use clients

[yhmo]

[yhmo]

Search request is blocking rpc call. If you call client.search(), the search() will be blocked until it receives results from milvus server. So, the answer is yes: if two search requests come in for one client, the second search only starts after the first one has finished.

You can use multiple clients to increase QPS, the ClientPool is mainly for this purpose. Increasing the maxTotalPerKey and maxTotal values can increase QPS, since there will be more clients maintained in the pool.

If you stop search actions, some of the clients will be destroyed to reduce the number to be smaller than maxIdlePerKey. Assume you start search actions again, the pool will create more clients until the number is equal to maxTotalPerKey.
Note that the maxTotal will limit the total number of clients. Let's say there are 3 keys, maxTotalPerKey=10, and you set maxTotal=20, then some of the keys will not have enough clients to hit the maxTotalPerKey. The number of each key could be 5 + 7 + 8 or 3 + 10 + 7, or even 0 + 10 + 10, etc.

@xiaofan-luan is right, set maxTotal to a large value is good practice. If you don't know how many keys will be created, it is ok to set maxTotal to 10000, just let the number of clients of each key be limited by maxTotalPerKey.

Each time a MilvusClient is created, it takes 100ms ~ 200ms to build the rpc channel between client and milvus server. So, the qps is much lower when the pool increases the number of clients from maxIdlePerKey to maxTotalPerKey. Once the number of clients reaches maxTotalPerKey, the qps is best. A larger maxIdlePerKey will get larger qps. The max qps will eventually depend on the server-side ability.