Performance getting slower when increase partitionCount

[kjlimxxxx]

Hi,

I am running performance test on increasing partitionCount to match to the worker available so that I can get faster processing speed.

However, the result I get is the speed is getting slower when I increase the partition count, the maximum is six workers with six partition count (3 initial workers/pod with auto scale up to 6).

I could see there were more occurrences on this message when I am increasing partitionCount. The maximum delay always close to 5 minutes.

Fetched [TaskCompleted#0] message from xxxxxxxx-control-02 (delay = 318670ms)

Can I know which area I could look at to solve the performance bottleneck? Thank You

Below is the speed test result:

1. partitionCount = 4, Average completion time = 5 seconds
2. partitionCount = 5, Average completion time = 7.2 seconds
3. partitionCount = 6, Average completion time = 7.8 seconds

[davidmrdavid]

Hi @kjlimxxxx:

Thanks for reaching out. Can you tell us a little bit more about your performance tests? It would help us to understand what kind of benchmark you're running, as that may help explain why increasing the partitionCount is making your app slower.

Thanks!

[kjlimxxxx]

Hi @davidmrdavid,

Thank you to follow up my question, below is the details setup of my load test:

The system setup:
- The ADF is inside the kubernetes, initial replica pod is 3.
- The auto scaling is set to scale up to 6 replica base on the CPU 75% threshold.

The test setup:
All the parameters keep the same except the partitionCount has changed to 4, 5, and 6 for testing
"extensions": {
"durableTask": {
"storageProvider": {
"controlQueueBatchSize": 32,
"controlQueueBufferThreshold": 256,
"controlQueueVisibilityTimeout": "00:05:00",
"maxQueuePollingInterval": "00:00:30",
"partitionCount": 4,
"workItemQueueVisibilityTimeout": "00:05:00",
"connectionStringName": "xxxxx"
},
"maxConcurrentActivityFunctions": 30,
"maxConcurrentOrchestratorFunctions": 30,
"hubName": "xxxxxxxx"
}
},

The test run:
1. I am using jmeter sending request to the system, about 7,000 request per minute
2. I capture the statistic from the table storage (the instances table) and calculate the average completion time.

[kjlimxxxx]

Hi @cgillum,

Yes I can see the partition activity thank you.

[kjlimxxxx]

Hi @cgillum,

When I run the load test, I found out some executions have strange behavior.

As the picture below, the message is inside control-00, but adf is deleting message at control-02 which cause not exists exception. It only perform the message deletion to the correct queue after 5 minutes.

[cgillum]

@kjlimxxxx are you changing the number of partitions in your app? Partitions are assigned using a hash of the orchestration instance ID, so it should never be the case that a message arrives on the wrong queue unless you are changing the number of partitions, which is not supported.

[kjlimxxxx]

Hi @cgillum,

Do you mean the number of partitions is regarding the partitionCount inside the host.json?
Yes I am changing it with different value 4,5 and 6 for test. I didn't change this value in the middle of test, I only change it once the test is completed. Then I start another test with different value. I did recreate all the pod once I update the value. I could see the partition was changed inside storage.
I did analyze the test result, the result seems fine. The instances could complete successfully. Only there are occurrence of deleting at the wrong partition message.

[davidmrdavid]

@kjlimxxxx: if you're able to replicate this "operation on wrong queue" error, we would appreciate if you could open a separate issue for us to investigate. Thanks!