ServiceChannel gets stuck in failed Open state when attempting to auto close #5803
Description
Describe the bug
ServiceChannel stays in Open state after a graceful server shutdown. It has _hasChannelStartedAutoClosing set to true which ends up throwing ProtocolException on any usage attempts.
Even subsequent killing of the server process doesn't help getting it unstuck.
I'm using CoreWCF 1.7.0, NetNamedPipe transport, but being centered around NetFramingBase it could be reproduceable in other transports as well.
The sequence is as follows:
I call app.StopAsync() in my host. This begins shutdown sequence. However, this is not recognized properly by the client.
The sequence starts as expected:
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.TransportDuplexSessionChannel.CloseOutputSessionAsync(System.TimeSpan timeout) Line 267 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.TransportDuplexSessionChannel.ConnectionDuplexSession.CloseOutputSession(System.TimeSpan timeout) Line 641 C#
System.ServiceModel.Primitives.dll!System.ServiceModel.Channels.ServiceChannel.DecrementActivity() Line 843 C#
System.ServiceModel.Primitives.dll!System.ServiceModel.Channels.ServiceChannel.HandleReceiveComplete(System.ServiceModel.Channels.RequestContext context) Line 907 C#
System.ServiceModel.Primitives.dll!System.ServiceModel.Dispatcher.ChannelHandler.HandleReceiveComplete(System.ServiceModel.Channels.RequestContext context) Line 638 C#
System.ServiceModel.Primitives.dll!System.ServiceModel.Dispatcher.ChannelHandler.EndTryReceive(System.IAsyncResult result, out System.ServiceModel.Channels.RequestContext requestContext) Line 327 C#
System.ServiceModel.Primitives.dll!System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult result) Line 191 C#
System.ServiceModel.Primitives.dll!System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(System.IAsyncResult result) Line 752 C#At this point, _hasChannelStartedAutoClosing is set to true so all further calls to the service will end up with
System.ServiceModel.ProtocolException: 'This channel can no longer be used to send messages as the output session was auto-closed due to a server-initiated shutdown. Either disable auto-close by setting the DispatchRuntime.AutomaticInputSessionShutdown to false, or consider modifying the shutdown protocol with the remote server.'
Next, CloseOutputSessionCoreAsync() is called, which succeeds.
But the following sequence is what baffles me: the ServiceChannel stays Open, there's no event notification to Closing or Closed events, there's no way for my code to check if it's alive (aside from receiving System.ServiceModel.ProtocolException on any call) while the connection is being pushed into the connection pool.
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.CommunicationPool<string, System.ServiceModel.Channels.IConnection>.EndpointConnectionPool.AddConnection(System.ServiceModel.Channels.IConnection connection, System.TimeSpan timeout) Line 407 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.CommunicationPool<string, System.ServiceModel.Channels.IConnection>.AddConnection(string key, System.ServiceModel.Channels.IConnection connection, System.TimeSpan timeout) Line 198 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.ConnectionPoolHelper.ReleaseConnection(bool abort, System.TimeSpan timeout) Line 206 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.ConnectionPoolHelper.Close(System.TimeSpan timeout) Line 167 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.ReturnConnectionIfNecessary(bool abort, System.TimeSpan timeout) Line 314 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.TransportDuplexSessionChannel.OnOutputSessionClosed(ref System.Runtime.TimeoutHelper timeoutHelper) Line 570 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.TransportDuplexSessionChannel.CloseOutputSessionAsync(System.TimeSpan timeout) Line 271 C#As an attempted fix, I've set on my Binding MaxConnections = 0 which forces IdleConnections.Add() to return false.
This leads to even more baffling situation:
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.CommunicationPool<string, System.ServiceModel.Channels.IConnection>.EndpointConnectionPool.CloseItemAsync(System.ServiceModel.Channels.IConnection item, System.TimeSpan timeout) Line 290 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.IdlingCommunicationPool<string, System.ServiceModel.Channels.IConnection>.IdleTimeoutEndpointConnectionPool.CloseItemAsync(System.ServiceModel.Channels.IConnection item, System.TimeSpan timeout) Line 69 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.CommunicationPool<string, System.ServiceModel.Channels.IConnection>.EndpointConnectionPool.CloseIdleConnection(System.ServiceModel.Channels.IConnection connection, System.TimeSpan timeout) Line 521 C#
System.ServiceModel.NetFramingBase.dll!System.ServiceModel.Channels.CommunicationPool<string, System.ServiceModel.Channels.IConnection>.EndpointConnectionPool.AddConnection(System.ServiceModel.Channels.IConnection connection, System.TimeSpan timeout) Line 409 C#It closes my connection... and then nothing. ServiceChannel still stays in a glitched Open state.
So, I would assume that OnOutputSessionClosed should transition the ServiceChannel into a Closed state first and return the connection to the pool later.
To Reproduce
Create a minimal NetNamedPipe based CoreWCF server
Call app.StopAsync()
Observe channel state on the client
Expected behavior
Channel sends Closing, then Closed event and changes its state to Closed so that client code could react to a server shutdown accordingly
Also reported for NetTcp connections here:
CoreWCF/CoreWCF#1402