Strange interaction with IIS/ClassicAsp+Response Flushing

[pinkfloydx33]

My YARP proxy, running on 1.0.0, is proxying data to an IIS backend(s). The sites in IIS are a mixture of Asp.Net and classic ASP. All has been going well except for one issue.

One of the pages we are proxying can return a response with a few megabytes of data. This is one of our classic asp pages. We've noticed that the request to the page "hangs"--up until the HttpRequest.ActivityTimeout--at which point we get a YARP exception, and a seemingly successful response in the browser (never any errors). If I extend the timeout, the request spins up until the new setting and behaves the same. I've tried reducing the value as well. It does make it "fail" sooner, but the shorter values don't seem to play nice with the rest of the application.

The exception is as follows:

System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
---> System.Net.Sockets.SocketException (125): Operation canceled
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory1 buffer)
at System.Net.Http.HttpConnection.ReadAsync(Memory1 destination)
at System.Net.Http.HttpConnection.ConnectionCloseReadStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.ConnectionCloseReadStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)
at Yarp.ReverseProxy.Forwarder.StreamCopier.CopyAsync(Boolean isRequest, Stream input, Stream output, IClock clock, ActivityCancellationTokenSource activityToken, CancellationToken cancellation)

Friendlier view:

{
   "ExceptionDetail":{
      "Type":"System.Threading.Tasks.TaskCanceledException",
      "HResult":-2146233029,
      "Message":"The operation was canceled.",
      "Source":"System.Net.Http",
      "TargetSite":"Void MoveNext()",
      "InnerException":{
         "Type":"System.IO.IOException",
         "HResult":-2146232800,
         "Message":"Unable to read data from the transport connection: Operation canceled.",
         "Source":"System.Net.Sockets",
         "TargetSite":"Void ThrowException(System.Net.Sockets.SocketError, System.Threading.CancellationToken)",
         "InnerException":{
            "Type":"System.Net.Sockets.SocketException",
            "HResult":-2147467259,
            "Message":"Operation canceled",
            "Source":null,
            "SocketErrorCode":"OperationAborted",
            "SocketErrorCodeMessage":"The overlapped operation was aborted due to the closure of the Socket."
         }
      },
      "CancellationToken":"CancellationRequested: true",
      "Task":"null"
   }
}

This particular file was last edited back in 2008! It writes out data from our database one line/record at a time, keeping a count. Every 1000 lines it calls Response.Flush likely as some sort of optimization required over a decade ago. On a hunch, I commented out the Flush. The page now loads near instantaneously and we've been unable to get Yarp to throw the above error.

Anyone have any ideas? The behavior does not exhibit when accessing the sites directly (no YARP). While it's a simple change, I'm not sure if it's the only place that would require fixing. It would also mean a production hotfix deployment across several hundred clients. We are using YARP to assist in our migration between cloud providers which was scheduled to begin next week. If possible, a fix to our proxy (which uses nothing other than the standard boilerplate configuration) would be acceptable and preferable to a patch across all clients.

If anyone has any ideas regarding something in YARP we can tweak, I'm all ears.

YARP: 1.0.0
.NET 6
Running on Azure App Service in Linux
Proxying to IIS (Windows 2019 and Windows 2012)
Edit: Windows 2012 traffic is over Azure Hybrid Connection

@Tratcher

[Tratcher]

@MihaZupan ?

[pinkfloydx33]

So I've actually left out an important detail. After a more thorough review, it turns out this is only happening for one particular cluster. We're using an Azure Hybrid connection to tunnel back to the other provider. The proxy is essentially sending the https requests over a service bus connection. If I remove that and instead make direct requests the issue disappears.

Our bandwidth with this provider is really slow (part of the reason we are moving). Yet somehow, connections made over an Azure Relay are several orders of magnitude faster. Sql is a good example--a "direct" connection over VPN takes about a minute to expand the db list in ssms, yet over the relay is near instantaneous. We've seen significant improvement using the relay for https traffic as well which is why we are using them here. Frankly it makes absolutely no sense to me.

Since customers are already used to the slowness, I think it's fine for us to skip the relays and just proxy back out over the public internet. As we move them to Azure they should begin seeing an improvement. The overall window here is only about a month anyways.

I'd prefer to keep the relay/hybrid connection if possible, due to the unexplained boost in speed everywhere else. I do accept that this is likely an unusual/unsupported setup though. So if there's nothing I can do, I'll just skip the hybrid connection and we can close this discussion.

[Tratcher]

That's an interesting situation. It seems like an issue with the service bus connection that you'd have to talk to the azure folks about.

Can you show how you're plugging that into YARP?

[pinkfloydx33]

Nothing special. The hybrid connection is a feature of App Service where it adds DNS entries to the container that point to the relay components (and then we install software on the remote). We just reference it as a cluster at https://whatever-the-connection-is-called/ as if it's any other, normal URL. Nothing special in YARP itself which is how I overlooked it to begin with. The other clusters are going straight to their target VMs in Azure where we have direct connectivity.

We've settled on just proxying back out the internet instead of using the hybrid connection. We considered adding a dedicated cluster and route for the offending page only. We tried it out and although it worked we decided against it in order to keep things simple.

Thanks for taking the time. Always appreciated.