Allow larger write queues for large nodes

[Tim-Brooks]

With the rise of larger CPU count nodes our current write queue size
might be too conservative. Indexing pressure will still provide protect
against out of memories.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[github-actions]

🔍 Preview links for changed docs:

• docs/reference/elasticsearch/configuration-reference/thread-pool-settings.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

[nicpenning]

Curtis, how does this apply to current clusters that are heavy CPU allocated?

For example, 7 nodes at 16 CPUs each. Is 10k still the best or would this new setting open up that 10k limit and be a larger value?

Just trying to understand the logic here because we have been told over and over again that increasing this 10k write queue is not advisable, so wondering what has changed here.

[Tim-Brooks]

For example, 7 nodes at 16 CPUs each. Is 10k still the best or would this new setting open up that 10k limit and be a larger value?

This setting would make the queue be of size 12,000 for a 16 core CPU. max(10,0000, 750 * cores).

Just trying to understand the logic here because we have been told over and over again that increasing this 10k write queue is not advisable, so wondering what has changed here.

We encountered some internal benchmarks which showed 32 core machines exhausting the thread pool queue but still having manageable latency. Which led us to make a this tweak for larger machines.

We would only advise changing it if a user is advanced and has a clear understanding of their load. I'm sure there are workloads out there with many, many small tasks which exhaust the queue with the node not being backed up. So if some benchmark shows that scenario, then changing the queue might be a way to address that.

But the standard scenario where the queue is consistently exhausted is just under provisioning and increasing the queue will just lead to high write latencies, timeouts, retries, etc.

[nicpenning]

Sounds great, thank you! This does help and I appreciate that breakdown!