Fix race condition in LedgerHandle in drainPendingAddsAndAdjustLength & sendAddSuccessCallbacks

[lhotari]

Motivation

There's a race condition in LedgerHandle between drainPendingAddsAndAdjustLength & sendAddSuccessCallbacks method calls. The methods could get called simultaneously which would cause inconsistency.
The changingEnsemble field has a thread safety issue and the changes might not be visible to the reading thread.

Changes

• use synchronized (pendingAddOps) as the object monitor for serializing drainPendingAddsAndAdjustLength & sendAddSuccessCallbacks method calls.
• make changingEnsemble field volatile
• remove the too strict rule for pendingAddsSequenceHead which breaks things after failures

This rule is removed:

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java

Lines 1822 to 1829 in 13e7efa

	// Check if it is the next entry in the sequence.
	if (pendingAddOp.entryId != 0 && pendingAddOp.entryId != pendingAddsSequenceHead + 1) {
	if (LOG.isDebugEnabled()) {
	LOG.debug("Head of the queue entryId: {} is not the expected value: {}", pendingAddOp.entryId,
	pendingAddsSequenceHead + 1);
	}
	return;
	}

The current assumption is that this doesn't hold under failure conditions.

Additional Context

• this is an alternative fix for [Bug] Receives topic is temporarily unavailable error when bookies are rolling restart pulsar#21860
• the other fix is [fix][client] Fix race conditions of LedgerHandle in client #4171

[lhotari]

Closing in favor of the original PR #4171.