Log failure in internalSend

[DaveCTurner]

An exception here should be impossible, but we don't assert that, nor do
we emit a log message to prove it didn't happen in a production
environment. This commit adds the missing log and assert.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[mhl-b]

I'm suspecting other place that might throw error that we dont catch. Async connect call that we dont check for exceptions. Looking at netty's code path I cannot tell 100% that channel can be null with exception as we assume in the code.

ChannelFuture connectFuture = bootstrapWithHandler.connect(); -> connect().await(timeout);

elasticsearch/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4Transport.java

Lines 268 to 274 in e713f7c

	ChannelFuture connectFuture = bootstrapWithHandler.connect();
	Channel channel = connectFuture.channel();
	if (channel == null) {
	ExceptionsHelper.maybeDieOnAnotherThread(connectFuture.cause());
	throw new IOException(connectFuture.cause());
	}

[DaveCTurner]

Possibly, but that wouldn't cause a long-established channel to close. We can address that separately, no need to do so here.

[DaveCTurner]

I opened #131520

[mhl-b]

LGTM

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
✅	9.1
✅	8.19
❌	8.18	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 131418