Akka.Actor: `ReceiveAsync` loses track of `ActorContext` following an `await` on `WaitAsync`, `Sender` throws

[Aaronontheweb]

Version Information
Version of Akka.NET? 1.4.46, 1.5.0-alpha3
Which Akka.NET Modules? Akka.Actor

Describe the bug

When using the following ReceiveAsync statement, the Sender.Tell crashes with an System.NotSupportedException:

ReceiveAsync<SubscribeToEnvironment>(async cmd =>
{
    // NOTE: can't throw here - one subscription failing shouldn't blow up any of the others

    // create a ChannelReader, from which we'll create an IAsyncEnumerable in the service
    var (reader, sink) = ChannelSink.AsReader<IEnvironmentEvent>(100, true, BoundedChannelFullMode.DropOldest)
        .PreMaterialize(Context.Materializer());
    var (sinkRefTask, killSwitch) = StreamRefs.SinkRef<IEnvironmentEvent>()
        .ViaMaterialized(KillSwitches.Single<IEnvironmentEvent>(), Keep.Both)
        .Via(cmd.CancellationToken.AsFlow<IEnvironmentEvent>())
        .To(sink)
        .Run(Context.Materializer());

    try
    {
        using var cts = new CancellationTokenSource(_subscriptionTimeout);
        var sinkRef = await sinkRefTask.WaitAsync(cts.Token).ConfigureAwait(false);

        var resp = await _clusterStateActor.ExecuteCommand(
            new SubscriptionCommands.SubscribeToClusterEvents(_subscriberId, cmd.EnvironmentId, sinkRef),
            cts.Token).ConfigureAwait(false);
        if (resp.Status == ResponseStatus.Success)
        {
            Sender.Tell(new SubscribeSuccess(cmd.EnvironmentId, reader));
        }
        else // no-ops are failures here - no guarantee that the new stream is going to run
        {
            // deliver SubscribeFailed to the client
            Sender.Tell(new SubscribeFailed(cmd.EnvironmentId, resp.Message));
            // kill the stream
            killSwitch.Shutdown();
        }
    }
    catch (Exception ex)
    {
        _log.Error(ex, "Failed to create subscription for [EnvironmentId: {0}]", cmd.EnvironmentId);
        Sender.Tell(new SubscribeFailed(cmd.EnvironmentId, ex.Message));
        killSwitch.Abort(ex);
    }
});

The exception thrown:

System.NotSupportedException: There is no active ActorContext, this is most likely due to use of async operations from within this actor.
   at Akka.Actor.ActorBase.get_Context()
   at Akka.Actor.ActorBase.get_Sender()
   at OpsCenter.ClusterState.Subscriptions.SubscriptionManager.<.ctor>b__9_0(SubscribeToEnvironment cmd) in D:\Repositories\Petabridge\OpsCenter\opscenter-blazor-prototype\src\OpsCenter.ClusterState\Subscriptions\SubscriptionManager.cs:line 94

To Reproduce

Working on this - not sure which part is causing this. await-ing inside ReceiveAsync is pretty common and shouldn't cause issues. I'm wondering if it's WaitAsync combined with ConfigureAwait(false) that is causing the antecedent Task to execute outside of the actor's built-in scheduler.

Expected behavior

Actor should be executing with a valid IActorContext and should be able to access the Sender, Self,, Parent, and Children properties without throwing.

Actual behavior

Actor crashes upon accessing the Sender property.

Additional context

We dealt with lots of issues like this one back in the 2015 era of Akka.NET - so this it's something with the way this code works that is causing it to throw.

[Aaronontheweb]

Work-around: I can work around this issue for the time being by closing over the Sender property.

[Aaronontheweb]

After doing some step-through debugging, I can confirm:

1. The issue is not WaitAsync - context is preserved just fine here;
2. It looks like the issue is calling ConfigureAwait(false) during an Ask operation - going to do some more digging.

[Aaronontheweb]

Yeah it's happening with the Ask<T> operation and it doesn't matter if I call ConfigureAwait or not on the Ask<T> operation itself.

I wonder if this is related to #5098 - I doubt it, but that's the last time we messed around with the await internals of Ask<T>.

[to11mtm]

Yeah it's happening with the Ask<T> operation and it doesn't matter if I call ConfigureAwait or not on the Ask<T> operation itself.

I wonder if this is related to #5098 - I doubt it, but that's the last time we messed around with the await internals of Ask<T>.

https://github.com/akkadotnet/akka.net/pull/5098/files#diff-e78b6b992cc8751569f2b6246e32669a37aafeba45e53948c313f6183ef55a66L122

That went away, which alone raises a brow

[Aaronontheweb]

Have a reproduction for this now:

public sealed class AsyncAwaitActor : ReceiveActor
{
    private readonly IActorRef _replyActor;
    private readonly bool _configureAwait;

    public AsyncAwaitActor(IActorRef replyActor, bool configureAwait)
    {
        _replyActor = replyActor;
        _configureAwait = configureAwait;

        ReceiveAsync<int>(async max =>
        {
            foreach (var i in Enumerable.Range(0, max))
            {
                var count = await _replyActor.Ask<int>(i, TimeSpan.FromSeconds(1)).ConfigureAwait(configureAwait);
                Sender.Tell(count);
            }
        });
    }
}

/// <summary>
/// Reproduction spec for https://github.com/akkadotnet/akka.net/issues/6279
/// </summary>
/// <param name="configureAwait">Input for actor's ConfigureAwait(bool)</param>
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Actor_should_Await_on_Ask_inside_receive_loop(bool configureAwait)
{
    var count = 3;
    var replyActor = Sys.ActorOf(act => { act.ReceiveAny((o, context) => context.Sender.Tell(o)); });
    // create AsyncAwaitActor
    var asyncAwaitActor = Sys.ActorOf(Props.Create(() => new AsyncAwaitActor(replyActor, configureAwait)));
    asyncAwaitActor.Tell(count);
    ReceiveN(count);
}

Test passes when ConfigureAwait(true); fails when ConfigureAwait(false) - with the same error as above.

[Aaronontheweb]

I'm thinking this might have to get bumped to v1.5 as it's going to involve touching some much nastier stuff inside the the ActorCell, Mailbox, and ActorTaskScheduler possibly.

[Aaronontheweb]

I think there are some older issues that indicate this has been a problem going back years: #3894