Avoiding Thread Pool Starvation

[DejanPelzel]

We're currently considering using YARP as part of one of our projects. However, we have situations where we receive hundreds of thousands of concurrent connections within a few seconds. When this hits YARP or any other .NET software for that matter, it immediately obliterates the thread pool (thousands of threads) and results in an instant deadlock.

Does anyone have any recommendations on how to tackle this? We can't really do much rate or connection limiting here. I've tried to implement some sort of worker/event-driven system, but it seems Kestrel alone is part of the problem.

UnsafePreferInlineScheduling is also out of the question given that the requests do have async / long-running requests.

Seems like YARP is not really suitable for very high load environments, unfortunately.

[karelz]

Out of curiosity: Which .NET version do you run on? The ThreadPool has been rewritten in .NET 6 - @kouvel can provide more info.

On lower version of .NET, the ThreadPool (and YARP) can scale, it just takes incremental load increase to scale well. If there is spike, ThreadPool does not respond that well. There is a workaround to adjust ThreadPool.SetMinThreads to match the needs of your workload if you expect certain load.
Would that help your situation?

[DejanPelzel]

Thanks for the message. We use .NET 6.0 RC1.

We set the following:

ThreadPool.SetMaxThreads(20000, 20000);
ThreadPool.SetMinThreads(10000, 10000);

I did notice performance was significantly improved when going from 5.0 to 6.0, but it's these super large spikes that we can't quite work around. I've been thinking of using a concurrency control middleware to incrementally increase throughput if needed, but that seems like it's working around the problem.

We did also notice there is a part of code that is somewhat CPU intensive, so we're also optimizing that at the moment to see if it helps. Perhaps one is related to the other.

I guess we have a bit of an extreme environment. In rare cases, spikes of legitimate traffic reach up to a million connections in a few seconds. The issue is this then results in a lockup that usually only resolves itself with a restart of the app.

[samsp-msft]

@stephentoub

[stephentoub]

ThreadPool.SetMinThreads(10000, 10000);

This is dangerous :-) and will result in exactly the kind of behavior you're witnessing. Setting this is basically telling the ThreadPool that it should aim to have at least 10K threads available, such that if a work item is queued and there aren't already 10K threads, it should immediately create one. This is actively working against what the ThreadPool is there to achieve.

it's these super large spikes that we can't quite work around

What work is being done as part of handling each of these requests? Are you doing any synchronous blocking anywhere? Fundamentally if your requests are CPU bound and you're saturating the CPU by doing work, you're limited by hardware, e.g. number of cores, and adding more threads is actually likely to hurt rather than help. If you're IO bound, and you found you needed to add this many threads in order to enable forward progress, it seems like some of these threads are getting blocked waiting for IO, in which case the answer is to avoid said blocking and rely purely on asynchrony. You shouldn't need anywhere close to this many threads in order to handle a fully asynchronous workload.

The ThreadPool has been rewritten in .NET 6

Only kinda sorta. At a high level, the thread pool had two pieces previously, the C# portion that handled the dispatching loop, and a native portion that handled thread management. In .NET 6, the thread management portion was ported to also be in C#, but it's a fairly faithful port, so I wouldn't expect much improvement just from that... the primary benefits of that from a perf standpoint are that it's now easier to make other perf improvements. One such example of that in .NET 6 is that we now much more aggressively inject threads if threads synchronously block on a Task; if you noticed thread pool-related improvements in .NET 6, and you need 10K threads to handle bursty workloads, I suspect your problem might indeed be rooted in one of performing synchronous I/O or synchronously blocking waiting for asynchronous I/O to complete.

I suggest profiling the app and using that for next steps.