error serializing large responses on client with remote pipe

[zachrybaker]

see issue 48 for context.

Granted this case is probably not the norm:

Remote piping (for now, going to be local soon for security reasons).
The server code is 100% synchronous re-platformed mainframe code, that I bridge the async gap via Task.FromResult(data);

When I replay two week's worth of API calls, 0.19% of the calls - 1283 of them - failed on the client side (server sent "normal data"). These calls can be replayed on a local pipe successfully between a .Net6.0 server and a .Net6.0 client. Replaying them in isolation using the same client, to a remote .Net6.0 server, however, they will consistently fail, regardless of load.

A sampling of these, when traced back to the request that originated the failure, all happened to be deserializing a large-ish payload, say around 200KB when serialized to JSON. When pulling back the same collection but with a smaller size, no problem deserializing.

So I think there is something that gets fouled with the bytes payload when it is a remote pipe that isn't happening when it is not remote. Seems unlikely that I have stumbled upon a framing issue, but I guess that is possible.

System.AggregateException
HResult=0x80131500
Message=One or more errors occurred. (Protocol message contained an invalid tag (zero).)
Source=System.Private.CoreLib
StackTrace:
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task1.get_Result()
at GrpcDotNetNamedPipes.NamedPipeChannel.BlockingUnaryCall[TRequest,TResponse](Method2 method, String host, CallOptions callOptions, TRequest request) in C:\code\patterns\GrpcDotNetNamedPipes\NamedPipeChannel.cs:line 93 at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext2 ctx)
at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext2 context, BlockingUnaryCallContinuation2 continuation)
at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method2 method, String host, CallOptions options, TRequest request) at Tamus.Famis.Protos.Shadow.Iasys.IasysSvc.IasysSvcClient.GetSysTableDictionaryCollection(GetSysTableDictionaryRqst request, CallOptions options) in C:\code\patterns\s\Tamus.Famis.Protos.Client\obj\Debug\netstandard2.0\protos\shadow\IasysGrpc.cs:line 1153 at Tamus.Famis.Protos.Shadow.Iasys.IasysSvc.IasysSvcClient.GetSysTableDictionaryCollection(GetSysTableDictionaryRqst request, Metadata headers, Nullable1 deadline, CancellationToken cancellationToken) in C:\code\patterns\s\Tamus.Famis.Protos.Client\obj\Debug\netstandard2.0\protos\shadow\IasysGrpc.cs:line 1142
at Tamus.Integration.Shadow.ShadowServerService.<>c__DisplayClass632_0.b__0() in C:\code\patterns\s\Tamus.Integration.Shadow.Client\ShadowServerService_Calls\ShadowServerService.Iasys.cs:line 57
at Tamus.Famis.Protos.Client.Helpers.ServiceExtensions.ThenRun[O](StartPipeFn InitializeFn, Func1 do) in C:\code\patterns\s\Tamus.Famis.Protos.Client\Helpers\ServiceExtensions.cs:line 52 at Tamus.Integration.Shadow.ShadowServerService.RunIasys[O](Func1 do) in C:\code\patterns\s\Tamus.Integration.Shadow.Client\ShadowServerService.cs:line 73
at Tamus.Integration.Shadow.ShadowServerService.GetSysTableDictionaryCollection(GetSysTableDictionaryRqst request) in C:\code\patterns\s\Tamus.Integration.Shadow.Client\ShadowServerService_Calls\ShadowServerService.Iasys.cs:line 57
at Famis.DataLayer.Sys.TableDictionaryCollection.Fill(String campusCd) in C:\code\patterns\s\Tamus.NaturalMigration.Iasys.Code.ShadowSubprograms\ZNNPDICT.TableDictionary.cs:line 86

This exception was originally thrown at this call stack:
[External Code]
Tamus.Famis.Protos.Shadow.Iasys.IasysSvc.__Helper_DeserializeMessage(Grpc.Core.DeserializationContext, Google.Protobuf.MessageParser) in IasysGrpc.cs
Tamus.Famis.Protos.Shadow.Iasys.IasysSvc..cctor.AnonymousMethod__136_23(Grpc.Core.DeserializationContext) in IasysGrpc.cs
GrpcDotNetNamedPipes.Internal.Helpers.SerializationHelpers.Deserialize(Grpc.Core.Marshaller, byte[]) in SerializationHelpers.cs
GrpcDotNetNamedPipes.Internal.MessageReader.Current.get() in MessageReader.cs
GrpcDotNetNamedPipes.Internal.MessageReader.ReadNextMessage.AnonymousMethod__0() in MessageReader.cs

Inner Exception 1:
InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).

[cyanfish]

If it's a message framing issue you could try setting TransmissionMode to Byte and see if that makes a difference.

[zachrybaker]

Using this branch, here's a bit more info:

2023-09-08 17:49:34.875 -05:00 [VRB] [CLIENT][80] Sending for '/shadow.Iasys.IasysSvc/GetSysTableDictionaryCollection'
2023-09-08 17:49:34.875 -05:00 [VRB] [CLIENT][80] Sending
2023-09-08 17:49:34.875 -05:00 [VRB] [CLIENT][80] Sending with 4 bytes
2023-09-08 17:49:34.991 -05:00 [VRB] [CLIENT][80] Received with 111033 bytes
2023-09-08 17:49:35.003 -05:00 [VRB] [CLIENT][80] Received with status '0'
2023-09-08 17:49:35.003 -05:00 [VRB] [CLIENT][80] Pipe disconnected
2023-09-08 17:49:35.095 -05:00 [ERR] Call to IPC failed after "00:00:00.2222476": System.AggregateException: One or more errors occurred. (Protocol message contained an invalid tag (zero).)
---> Google.Protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
at Google.Protobuf.ParsingPrimitives.ParseTag(ReadOnlySpan`1& buffer, ParserInternalState& state)
at Tamus.Famis.Protos.Shadow.Iasys.GetSysTableDictionaryResp.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in C:\code\patterns\s\Tamus.Famis.Protos.Client\obj\Debug\netstandard2.0\protos\shadow\Iasys.cs:line 10202

[zachrybaker]

@cyanfish the issue occurs regardless of whether or not the transmission mode is bytes or message. So framing is not the issue here.

And local-to-local pipe continues to work, bytes or message transmission mode.

Side note, it is scary how easy it is to hose a remote server when you have it in message transmission mode and have a client attempting to talk to it in bytes mode....it seems to go off the rails waiting for an end of message. I managed to make the server consume literally all the RAM, until a hard reboot was required of it. I couldn't even stop the thing.