Akka.Actor: FutureActorRef<T> does not support Context.Watch and may cause memory leaks

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.37 and earlier
Which Akka.NET Modules? Akka

Describe the bug

Related to akkadotnet/Akka.Persistence.Sql#516 - basically, any actor that calls Context.Watch will never get a Terminated message back for a death-watched FutureActorRef<T> even once it completes its TaskCompletionSource<T>, which can lead to memory leaks and other undefined system behavior.

To Reproduce

[Fact]
    public async Task FutureActorRefShouldSupportDeathWatch()
    {
        // arrange
        var watcher = Sys.ActorOf(act =>
        {
            act.Receive<string>((_, context) =>
            {
                // complete the Ask
                context.Sender.Tell("hi");
                
                // deliver the IActorRef of the Ask-er to TestActor
                TestActor.Tell(context.Sender);
            });
        });

        // act
        await watcher.Ask<string>("boo", RemainingOrDefault);
        var futureActorRef = await ExpectMsgAsync<IActorRef>();
        await WatchAsync(futureActorRef); // Ask is finished - should immediately dead-letter
        
        // assert
        await ExpectTerminatedAsync(futureActorRef);
    }

Expected behavior

Either Context.Watch should no-op for the calling actor if the target is a FutureActorRef<T> or Context.Watch should work properly and the invoking actor should receive a Terminated message back from the FutureActorRef<T>.

Actual behavior

Actor invokes Context.Watch and stores the FutureActorRef<T> in its death watch collection and never releases it, which causes the TaskCompletionSource<T> / Task<T> / the result T all to get rooted and retained in-memory.

Additional context

As a general design rule, I don't think the caller should have to be aware or care what the implementation of an IActorRef is when calling Context.Watch - it should "just work."

[Aaronontheweb]

We've had some code for handling this scenario but it's never worked:

akka.net/src/core/Akka/Actor/ActorRef.cs

Lines 113 to 115 in 2ee5176

	case ISystemMessage msg:
	handled = _result.TrySetException(new InvalidOperationException($"system message of type '{msg.GetType().Name}' is invalid for {nameof(FutureActorRef<T>)}"));
	break;

I suspect this is because ISystemMessage instances go down a totally separate message-handling path.