Suspend/resume orchestration with (limited) fan out

[anel-al]

Hi,

I am trying to implement suspend and resume feature in my orchestrations.

So far, it works when using for each loop with awaited activity. Unfortunately this approach does not offer any parallelism.

Is it possible to use suspend and resume feature with pattern where small subset of N activities is executed (in parallel as limited fan-out) and then awaited? From my understanding, once activity is scheduled via fan-out, it cannot be suspended or resumed using existing API - it will be executed no matter if orchestration is suspended?

My goal is to use suspend and resume between chunks of N activities executed in parallel to improve performance. It would be acceptable for me that chunk of N activities is completed before orchestration is suspended.

Is this supported by durable framework at all? Would suspend and resume run into inconsistencies or break orchestration?

Thanks,
Anel

[cgillum]

I’m not sure I fully understand. Are you looking for a suspend/resume feature or a way to do limited fan-out of activities? We have a suspend/resume feature already, and if you schedule your activities in batches, you’ll be able to suspend execution between those batches.

[anel-al]

Thank you for the quick answer. I am trying to accomplish something similar to the sample code below:

`var partitions = await context.CallActivityAsync<List<List>>("someactivity", input);

            foreach (var partition in partitions) 
            {
                replayLogger.LogDebug($"[{context.InstanceId}] - Processing partition: {JsonConvert.SerializeObject(partition)}.");

                List<Task> tasks = new List<Task>();

                try
                {
                    foreach (int id in partition)
                    {
                        tasks.Add(context.CallActivityWithRetryAsync("someactivity", new RetryOptions(TimeSpan.FromSeconds(_retryDelay), _retryCount) { BackoffCoefficient = _retryCoefficient }, new Item()
                        {
                        }));
                    }

                    await Task.WhenAll(tasks);
                }
                catch (Exception ex)
                {
                    
                }
            }`

However, when playing with the suspend and resume, while orchestration is running, sometimes I get following error:

"output": "Non-Deterministic workflow detected: A previous execution of this orchestration scheduled an activity task with sequence ID 61 and name 'activityname' (version ''), but the current replay execution hasn't (yet?) scheduled this task. Was a change made to the orchestrator code after this instance had already started running?"

I am not able to consistently reproduce this error especially not when single orchestration is running. It usually happens when testing multiple orchestrations in parallel.

What would be correct pattern to ensure deterministic execution in this case?

With regards,
Anel

[anel-al]

Update: I get same error

"output": "Non-Deterministic workflow detected: A previous execution of this orchestration scheduled an activity task with sequence ID 3213 and name 'xxx' (version ''), but the current replay execution hasn't (yet?) scheduled this task. Was a change made to the orchestrator code after this instance had already started running?"

also when having single awaited call to the activity (chaining, loop) when under load and playing with suspend and resume API more frequently.

With regards,
Anel