High Request Volume Causes Timeout in Hybrid Connection Listener

[hultsk]

Description

When the Hybrid Connection Listene rexperiences a high volume of requests in a short period (e.g., 1,000 requests within a minute), it encounters a timeout exception.

Root Cause:

The issue seems to stem from how the HybridHttpConnection class in the Microsoft.Azure.Relay package handles incoming requests. Specifically, in the CreateAsync method, Task.Run is used to spawn a new task for each request.

During a period of request spikes, if the request handler takes a long (eg. 100ms) time to execute, this will quickly consume the readily available threads from the ThreadPool. A significant delay is thereafter introduced for the creation of each new thread, leading
to increasignly long response times.

Further adding to the problem is that Azure Relay itself has a maximum timeout of 60 seconds, causing the request to fail if a response has not been received in that time.

Normally a long running request handler in ASP.NET Core is created as an async method allowing I/O operations to not block the threads. However, HybridConnectionClient.RequestHandler is an Action<RelayedHttpListenerContext> preventing us from using async techniques to alleviate this problem.

Suggestion is to make HybridConnectionListener.RequestHandler async, and await it from within ProcessFirstRequestAsync. This would be a breaking change, so perhaps an alternative would be to introduce a new property Func<RelayedHttpListenerContext, Task> RequestHandlerAsync that is used if it is set instead of the current RequestHandler