Stack Overflow during Fan Out

[richbigpoppa]

I have a very lengthy set of calculations that I am trying to complete. The way we have to complete these calculations there is a change that occurs in the system. This change causes us to need to update the values of some 500k values in our database. The calculations have to be completed in a hierarchy so I have to make multiple passes through our calculation logic in order to complete this calculation.

I have a ServiceBus triggered function which listens for a message on the bus and subsequently spawns an instance of an orchestration function. This orchestration function then loops through the iterations and for each iteration spawns a sub orchestration to handle that particular iteration. This is how I am spinning off each iteration:

                for (int i = 0; i <= necessarySteps; i++)
                {
                    calculationRequestMessage = await context.CallSubOrchestratorAsync<CalculationMessageContainer>(@"CalculateIteration", calculationRequestMessage);
                }

The sub iteration for the instance has alot of work to do, long story short, there are potentially multiple types of calculations that I have to do for each iteration. So I use the fan out pattern as such:

            var parallelExpressionTasks = new List<Task>();
            dataGroupsIds.ForEach(dg =>
            {
                var newMessage = new IterationDataGroupMessage
                {
                    //removed for simplicity
                };
                parallelExpressionTasks.Add(context.CallSubOrchestratorAsync(@"CalculateExpressionDataGroup", newMessage));
            });

            await Task.WhenAll(parallelExpressionTasks);

The CalculateExpressionDataGroup method can in some cases have a half million calculations it has to do. This is where I run into problems. I have tried doing the fan out pattern where I split the half million calculations into chunks and follow a similar pattern to the one above. However, if I split the 500k calculations into 5 parallel activity functions that handle 100k values, the individual functions time out. If I split it into say 100 parallel activity functions that each handle 50k calculations, then it runs for awhile but then out of nowhere I get a stack overflow with no stack trace or info on what happened.

What I am doing now is splitting my 500k calculations into 100 different activity function calls, but I am only calling them 5 at a time. Here is the code that I wrote for that:

                for (int i = 0; i <= messages.Count; i+=5)
                {
                    var currentIndex = i;
                    var targetIndex = i + 5;
                    log.LogInformation($"Working Iterations {i} - {targetIndex}");
                    var parallelTasks = new List<Task>();
                    while (currentIndex < targetIndex && currentIndex < messages.Count())
                    {
                        parallelTasks.Add(context.CallActivityAsync(@"CalculateRows", messages[currentIndex]));
                        currentIndex++;
                    }
                    await Task.WhenAll(parallelTasks);
                }

This code works, and it spins up 5 parallel activity functions and waits for them to execute. Then it picks up the next 5 and continues until it has executed them all. When completing a smaller set, say 100k records, it works fine. When I execute it locally from VS it gets about half way through and just dies with a stack overflow. When I deploy it as a premium function in Azure, it gets about a 1/4 way through and just dies with no messaging on what happened. No error in the console, no error in App Insights.

[cgillum]

Are you at least able to pinpoint which function the StackOverflow exception is coming from? For example, do you know which sub-orchestration or which activity might be causing it? How large are the inputs or outputs of your activity and or sub-orchestration functions? I'm wondering if data serialization might be a potential source of issues.

Your design seems perfectly reasonable to me so I don't have any obvious guesses as to why things wouldn't work.

[richbigpoppa]

I'm almost positive it is happening in the CalculateRows activity function which is the bottom function in the call. It returns void so shouldn't be an output issue. The inputs are rather large as it passes in a set of data that needs to be operated on. It's hard to tell as it just prints out "Stack Overflow" to the console and dies.

[cgillum]

If the inputs are large, there's a chance that the stackoverflow is happening in the activity payload deserialization process. If it's possible to test your scenario using smaller payloads, then I would try that to see if smaller payloads allow the scenario to run to completion. Otherwise, you'll likely need to closely inspect your activity code to see whether there's any path that could lead to deep recursion, resulting in a stackoverflow exception.

[richbigpoppa]

So I made some changes to greatly reduce the size of the payload. Long story short, I was fetching a large dataset of results, splitting it into smaller subsets, then passing those subsets to the activity functions. I made a change to just pass the ids into the activity function. Then the activity function fetches just the information that it needs. Didn't seem to make much of a difference. What I see now is that the function will be running along just fine and then start getting slower and slower until eventually it will take minutes between log statements on the console. Then I finally get a stack overflow.

I'm thinking it is not a deserialization issue as it fails with different sets of data each time. Is there anything else I can check? Like I mentioned earlier, this works fine when it doesn't have to do as many passes, or work on as large a set of data. The code is the same and it is working in the same size chunks of data, just more of them.

[richbigpoppa]

Is it at all possible that this is because I am calling sub orchestrator functions in the Task.WhenAll fan out?

[davidmrdavid]

What I see now is that the function will be running along just fine and then start getting slower and slower until eventually it will take minutes between log statements on the console. Then I finally get a stack overflow.

Huh, this is really interesting. I'm not sure how to interpret this, but it caught my eye.

Is it at all possible that this is because I am calling sub orchestrator functions in the Task.WhenAll fan out?

On top of mind, I don't think so. There's nothing too different about sub-orchestrator Tasks compared to other Durable Function APIs.

3 clarifying questions:

1. Long shot - but are you able to provide a minimal repro of any kind?
2. Can you show us what the stack trace looks like? Is it just really a 'stack overflow' message with nothing else?
3. Is this running on Azure? If so, an app name, orchestrator instanceId, and timerange of the incident would help us look at our telemetry for more clues.

Thanks!