Actor execution cancellation - sidecar context cancellation

[VaclavK]

At the moment, the actor execution has no support to cancel the invocation in the backend runtime. When actor call is received from sidecar, the actor is executed. When the sidecar cancels the execution, the actor continues to run and retries of actor or any such actor invocation will logically break actor single thread contract.

There are other reasons for context cancellation - such as scheduler disconnects etc.

An example

• Actor is called
• timeout is reached or other reason for cancelling
• sidecar cancels the invocation by closing the connection
• actor continues to run
• sidecar will (under default retry policy) retry the actor and you have parallel executions of the actor id

We proposed and tested the generic solution of handling this by creating a cancellation token source per actor callback connection and cancelling the tokens via the source (standard .net cancellation token handling)

sample code for detection (we can only include routes that target actor handler)

namespace Frontgate.Jobs.Services;

public class HttpRequestCancellationTokenSourceFactory(IHttpContextAccessor httpContextAccessor, ILogger<HttpRequestCancellationTokenSourceFactory> logger)
{
    public CancellationTokenSource ObtainCancellationTokenSource()
    {
        var httpContext = httpContextAccessor.HttpContext;

        var ctxSource = new CancellationTokenSource();

        if (httpContext == null)
            return ctxSource;

        httpContext.RequestAborted.Register(() =>
        {
            try
            {
                logger.LogWarning(
                    "context cancelled for {RequestUrl}, will cancel underlying cancellation token",
                    $"{httpContext.Request.Scheme}://{httpContext.Request.Host}{httpContext.Request.PathBase}{httpContext.Request.Path}{httpContext.Request.QueryString}");
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "error retrieving request details");
            }

            ctxSource.Cancel();
        });

        return ctxSource;
    }
}

to pass the context to the execution method we delegate the main logic to a sub-method and the sub-method accepts cancellation token as a parameter

e.g.

ProcessAsync -> this is IActor contract method, calls
ProcessInternalAsync(CancellationToken)

this I suppose could be done by convention and automated at handler level