Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

As a note to anyone reviewing this PR:

This is the fifth part of a series of PRs fixing how the FailedToCommitClusterStateException is used in ElasticSearch. As per #135017, FailedToCommitClusterStateException is defined as:

Thrown when a cluster state publication fails to commit the new cluster state. If publication fails then a new master is elected but the
update might or might not take effect, depending on whether the newly-elected master accepted the published state that failed to
be committed. This exception should only be used when there is <i>ambiguity</i> whether a state update took effect or not.

Currently, FailedToCommitClusterStateException is used as a 'catch-all' exception thrown at multiple places throughout the Coordinator and MasterService during the publication process. Semantically however, it doesn't make sense to throw this exception before the cluster state update is actually sent over the wire, since at this point, we know for certain that the cluster state update failed. FailedToCommitClusterStateException is intended to display ambiguity.

This work is a pre-requisite to #134213.

Changes

Replaces the FailedToCommitClusterStateException inside MasterService .BatchingTaskQueue.submitTask with a NotMasterException. This exception is thrown when the threadpools are shutting down, which occurs when the node itself is shutting down, hence it is no longer the current master. As a note, since there is an assert false, this cannot happen in practice.

Next Steps

The goal of this work is to fix up all erroneously used FailedToCommitClusterStateException.

Done:

• Update exception messages: Update exception messages #135017
• Update the FailedToCommitClusterStateException thrown inside MasterService.Batch.onResponse() when draining the queue after the threadpool has shut down - Change FailedToCommitClusterStateException to NotMasterException #135008
• Change a FailedToCommitClusterStateException to NotMasterException during the pre-publication process: Changes FailedToCommitClusterStateException to NotMasterException #135548
• Replace all FailedToCommitClusterStateExceptions prior to the publication of the cluster state update over the wire with FailedToPublishClusterStateException : Replace pre publication failed to commit cluster state exceptions #135706
• Removes a check on the FailedToCommitClusterStateException : Remove FailedToCommitClusterStateException Check #135846

Todo:

• Any final uses of FailedToCommitClusterStateException that need to be updated

Relates to: ES-13061