Understanding python runtime on Azure VMs

[kanupriya15025]

I have been working on creating a python function app on azure durable functions. We have been running load tests and we are seeing performance issues while scaling the app. There are about 10k orchestrators + 10k activity functions being triggered simultaneously.
Here's my flow :

1. Eternal Orchestrator : Acts as a scheduler. It calculates the next time and sleeps till that time. After waking up, it calls an activity function "FireAndForgetSubOrchestrator" that calls a different orchestrator function "WorkflowOrchestrator" that parses the input and create another activity "HTTPOperator". This activity calls an endpoint over http.
2. For most of my questions, the reason I am pointed to is that because python is a single threaded runtime, we are facing latencies in terms of message dequeue rates. We have set the autoscaling to 100 VM instances but it only reaches to a max of 10-12 VMs only. There are 16 partitions on the control queue.
3. How can we increase the concurrency of my orchestrator functions?
4. Also, is it right to assume that there will only be any one instance of orchestrator/activity function running in my VM at any point in time? If yes, what's the priority between the two?

This is my host.json

{ "version": "2.0", "logging": { "applicationInsights": { "samplingSettings": { "isEnabled": true, "excludedTypes": "Request" } } }, "extensionBundle": { "id": "Microsoft.Azure.Functions.ExtensionBundle", "version": "[3.*, 4.0.0)" }, "extensions": { "durableTask": { "maxConcurrentActivityFunctions": 120, "maxConcurrentOrchestratorFunctions": 120, "storageProvider": { "partitionCount": 16 } } } }

[davidmrdavid]

Hi @kanupriya15025 , thanks for reaching out.

Let me reply to your questions one by one.

How can we increase the concurrency of my orchestrator functions?

To increase the concurrency of your of your orchestrator Functions, you can tweak the values of maxConcurrentOrchestratorFunctions and maxConcurrentActivityFunctions in your host.json. These concurrency knows are described here: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-perf-and-scale#configuration-of-throttles

For Python in particular, I recommend also reviewing this section of the docs: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-perf-and-scale#language-runtime-considerations

As you point out, Python is a single-threaded runtime and therefore configuring its concurrency settings can be particularly challenging. The documentation I linked above should link you to the general Azure Functions docs on Python concurrency, which I also recommend reviewing.

Also, is it right to assume that there will only be any one instance of orchestrator/activity function running in my VM at any point in time? If yes, what's the priority between the two?

I believe this isn't quite true, this depends on the concurrency settings I listed above. For example: if you set maxConcurrentOrchestratorFunctions and maxConcurrentActivityFunctions both to 1, then I believe you would get only 1 per VM as you describe.

Can you try lowering these two settings to see if that helps with scale out? Thanks!

[vigouredelaruse]

@cgillum do these python concurrency issues apply for kubernetes deployments? (1 python thread per 'vm')

because if not this is where i plan to start https://github.com/puthurr/python-azure-function-gpu

[cgillum]

@vigouredelaruse yes, absolutely. The python worker concurrency behavior is independent of where you run functions (it's a behavior of the Python worker), so the guidance applies on Azure Functions as well as Kubernetes.

[vigouredelaruse]

now i understand the stranglehold a lot better. thanks

[kanupriya15025]

@davidmrdavid In this doc here : https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-perf-and-scale#language-runtime-considerations
It says Durable Function apps written in Python or PowerShell may only support running a single function at a time on a single VM.
Does this not mean max concurrent activity/orchestrations is one for every VM?

Also, what does this line mean here :
For example, if you have a Python app that restricts concurrency to 4 functions (perhaps it's only configured with 4 threads on a single language worker process or 1 thread on 4 language worker processes) then you should configure both maxConcurrentOrchestratorFunctions and maxConcurrentActivityFunctions to 4.
Isn't this contradicting the above statement as it will never be able to achieve a concurrency of 4 on the same VM?

[davidmrdavid]

@kanupriya15025, let me try to respond to your questions one by one :-) .

It says Durable Function apps written in Python or PowerShell may only support running a single function at a time on a single VM.
Does this not mean max concurrent activity/orchestrations is one for every VM?

Not necessarily, let me try to explain.

In Azure Functions, Python code is invoked through a component called "the language worker", or just "worker" for short. By default, we only instantiate 1 worker process per VM. Single Python is a single-threaded language, this means that if you're working with the default Python concurrency settings (and assuming you're not using co-routines/async), then the language worker for Python will constraint your app to only be able to run 1 Function (whether it be an Activity or an Orchestrator or something else) at a time. Note that this is configurable, so you can change the setting FUNCTIONS_WORKER_PROCESS_COUNT to change this behavior and potentially have more workers per VM which in turn allow for concurrent Functions.

Now, the subtlety is that these Python concurrency settings are declared independently from the Durable Functions concurrency controls. Therefore, if your Python config only allows for you to run 1 Function at a time per VM but you were to set up maxConcurrentActivityFunctions. to 10, you would still only be able to run 1 Activity at a time per VM. Furthermore, since maxConcurrentActivityFunctions is set to 10, Durable Functions will not load balance those 10 Activities to other VMs because its configuration specifies that it should allocate 10 Activities per VM prior to finding another VM that can run them.

To summarize, these are the key points:

• Due to Python's single-threadedness, the concurrency of Function executions isn't guaranteed by default.
• The Python worker concurrency settings control the number of Functions that can run at the same time on a given VM
• The values of maxConcurrentActivityFunctions and maxConcurrentOrchestratorFunctions do not control the number of Functions that may run per VM. Instead, they specify how many Activities/Orchestrator may be allocated to a VM before Durable Functions load balances invocation requests to another VM.

I realize this is a subtle distinction, so please let me know if the explanation helped.

For example, if you have a Python app that restricts concurrency to 4 functions (perhaps it's only configured with 4 threads on a single language worker process or 1 thread on 4 language worker processes) then you should configure both maxConcurrentOrchestratorFunctions and maxConcurrentActivityFunctions to 4.

Isn't this contradicting the above statement as it will never be able to achieve a concurrency of 4 on the same VM?

As mentioned above, it is definitely possible for a Python app to run more than 1 Function per VM at a time. The prior statement just said that this isn't guaranteed and therefore users should review their Python worker concurrency settings (FUNCTIONS_WORKER_PROCESS_COUNT and PYTHON_THREADPOOL_THREAD_COUNT) as well as their usage of coroutines/async to ensure that their app is set up for concurrency.

The quoted statement also suggests that the maximum number of Orchestrators and Activities per VM should not exceed the maximum number of concurrent Function executions that your Python worker configuration actually allows. Not doing this can be detrimental to the ability of Durable Functions to load balance your Activity and Orchestrator invocations correctly.

[kanupriya15025]

According the doc here : https://learn.microsoft.com/en-us/azure/azure-functions/python-scale-performance-reference, it says that value for parameter FUNCTIONS_WORKER_PROCESS_COUNT>1 isn't supported for functions v2.

[davidmrdavid]

@kanupriya15025: That note refers to the V2 programming model, not V2 of the Azure Functions runtime.

I'll reach out to the Python team to let them know that the wording of that note is confusing.

Does that help answer your question?