Add thread pool for write coordination

[Tim-Brooks]

This change adds a thread pool for write coordination to ensure that
bulk coordination does not get stuck on an overloaded primary node.

[Tim-Brooks]

WIP / opened for CI.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[henningandersen]

Thanks, this makes sense to me, left a comment, but otherwise think we can move forward with tests etc.

[henningandersen]

I am ok with this, but it seems like for any case where we have ingest processing, we would then still have the coordination happen behind any local write work. At least the PR then avoids some of the wait roundtrips.

We could also decide to have both a system-write-coordination pool and a write-coordination pool and use those here? We can look at this in follow-ups ofc.

[Tim-Brooks]

We could also decide to have both a system-write-coordination pool and a write-coordination pool and use those here? We can look at this in follow-ups ofc.

Yes I would like to move ingest work to a non-WRITE thread pool in a follow-up as I think there might be a few things to discuss.

[github-actions]

🔍 Preview links for changed docs:

• docs/reference/elasticsearch/configuration-reference/thread-pool-settings.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

[henningandersen]

LGTM.

This seems intuitive and simple enough that we can merge. But it seems worth keeping an eye on nightly benchmarks the following day or two, just to double check.

[henningandersen]

Should we rename to:

Suggested change

	private static void fillWriteQueue(ThreadPool threadPool) {
	private static void fillWriteCoordinationQueue(ThreadPool threadPool) {

[henningandersen]

Rename to:

Suggested change

	private static void blockWritePool(ThreadPool threadPool, CountDownLatch finishLatch) {
	private static void blockWriteCoordinationPool(ThreadPool threadPool, CountDownLatch finishLatch) {