Durable Function Using Sub Orchestrators for Fan Out Causing Out Of Memory Exception Depending on Message Size Sent to Work-Item Queue

[empowerJustinVocke]

I'm using an Isolated Worker Durable Function in .NET 7 on an EP3 plan that uses the fan out/fan in pattern to spawn 4 parallel sub-orchestrators. Each of those sub-orchestrators uses the fan out/fan in pattern to spawn a certain number of parallel activity functions depending on the amount of records we're testing. The goal is 600k records but it seems like ~30k messages to the work-item queue is the cutoff, after which we get Out of Memory exceptions when trying to create sub-orchestrators.

If I try 40k or 50k (they are split evenly across each sub-orchestrator, so for 50k each sub-orchestrator would create 12,500 activity functions) I get an Out Of Memory exception for any sub-orchestrator that would hit that limit. For example, when I tried testing 50k messages with 5 sub-orchestrators (10k per sub), 2 sub-orchestrators failed from an Out of Memory exception and 30k messages were still sent to the work-item queue/executed but the orchestrator ended in a failed state because of the two failed sub-orchestrators.

Is there a limit on how many messages can go in the work-item queue or a rate-limit on how many can be added? I tried to add a retry policy to the sub-orchestrators to no avail as well as a delay between generating each sub-orchestrator so the messages would be added to the queue in a staggered manner after reading this article on scalability targets for queues also to no avail.

Each message is 11kb. There's no setting for maxConcurrentActivityFunctions or maxConcurrentOrchestratorFunctions so it should fan out maximally. The partition size is default of 4, so there's 4 control queues. I'm not sure what to check/tweak to be able to add the necessary amount of messages without having to implement some sort of loop that waits on prior sub-orchestrators to complete as that seems like it'd be a bit tedious to implement. I'm open to any possible ideas as I've spent the weekend trying to figure out what the issue might be.