Retry activity without throwing exception

[lxalln]

Is your feature request related to a problem? Please describe.
There are a number of occasions where an activity doesn't have a successful output, but hasn't thrown an exception. I would like to be able to retry these with CallActivityWithRetryAsync().

Describe the solution you'd like
I would like a callback HandleResult on RetryOptions with the signature Func<TResult, bool> (or Func<object, bool> if the generics cause too much of a problem);

I could then write code that looks like this:

var retryOptions = new RetryOptions(TimeSpan.FromSeconds(1), 100)
                       {
                           RetryTimeout = TimeSpan.FromDays(7),
                           HandleResult = result => 
                           {
                               // true will cause orchestrator to retry according to retry policy
                               return result.IsRetryRequired;
                           }
                       };

Describe alternatives you've considered
There are 2 alternatives at the moment from what I can see:

Throw exceptions

At the end of my activity, I check if the command succeeded, if not, then I throw an exception to cause the function to fail and for the orchestrator to retry. This is messy, it makes a mess of our application insights, meaning I can't see exceptions that need my attention because of all the infrastructure exceptions that are being used to control flow.

Custom retry logic

This is my next step, which is to remove the built in Retry work flow, and simply write my own. But I am introducing the chance of bugs and working around a feature that should exist.

If there is something I am missing, then I'd love to have it pointed out to me. This is how the Polly library implements retry, there doesn't always have to be an exception.

[lxalln]

Looks like this may actually be an issue that needs to exist in the durable task project too?

[cgillum]

Thanks for the suggestion. I agree this would be a convenient feature. Depending on how it’s designed, it may or may not require a change in the Durable Task framework.

[lxalln]

I implemented this as an extension method on IDurableOrchestrationContext for the time being. It works, but the assumptions made are only reasonable in a known code base. For example the retry options apply to both, but not together. So it has 10 maxium retries that are either results, or exceptions. So could potentially retry 20 times. But it works in theory :)

Heavily inspired by: Durable Task's RetryInterceptor.cs

public static async Task<TResult> CallActivityWithRetryAsync<TResult>(
            this IDurableOrchestrationContext context,
            string functionName,
            RetryOptions retryOptions,
            object input,
            Func<TResult, bool> handleResult = null)
        {
            var firstAttempt = context.CurrentUtcDateTime;

            for (var retryCount = 0; retryCount < retryOptions.MaxNumberOfAttempts; retryCount++)
            {
                var result = await context.CallActivityWithRetryAsync<TResult>(functionName, retryOptions, input);

                var nextDelay = ComputeNextDelay(retryCount, result);

                if (nextDelay == TimeSpan.Zero)
                {
                    return result;
                }

                var retryAt = context.CurrentUtcDateTime.Add(nextDelay);
                await context.CreateTimer(retryAt, CancellationToken.None);
            }

            /// this is our type, and lets us abort from the built-in exception retry logic
            throw new DurableRetryExhaustedException();

            TimeSpan ComputeNextDelay(int attempt, TResult result)
            {
                var nextDelay = TimeSpan.Zero;

                if (handleResult == null)
                {
                    return TimeSpan.Zero;
                }

                if (handleResult(result))
                {
                    var retryExpiration = (retryOptions.RetryTimeout != TimeSpan.MaxValue)
                                              ? firstAttempt.Add(retryOptions.RetryTimeout)
                                              : DateTime.MaxValue;

                    if (context.CurrentUtcDateTime < retryExpiration)
                    {
                        var nextDelayInMilliseconds = retryOptions.FirstRetryInterval.TotalMilliseconds
                                                      * Math.Pow(retryOptions.BackoffCoefficient, attempt);

                        nextDelay = nextDelayInMilliseconds < retryOptions.MaxRetryInterval.TotalMilliseconds
                                        ? TimeSpan.FromMilliseconds(nextDelayInMilliseconds)
                                        : retryOptions.MaxRetryInterval;
                    }
                }

                return nextDelay;
            }
        }

The advantage this solution has, is the generic TResult is already known, which makes life easier.
The downside, is the loop, and the ComputedNextDelay code is duplicated.