[CI] EsqlClientYamlIT test {p0=esql/60_usage/Basic ESQL usage output (telemetry)} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #4182 / 8.16.0_bwc
• elasticsearch-intake #10128 / 8.16.0_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:plugin:esql:qa:server:mixed-cluster:v8.16.0#yamlRestTest" -Dtests.class="org.elasticsearch.xpack.esql.qa.mixed.EsqlClientYamlIT" -Dtests.method="test {p0=esql/60_usage/Basic ESQL usage output (telemetry)}" -Dtests.seed=721C143E54D2988C -Dtests.bwc=true -Dtests.locale=no-NO -Dtests.timezone=Asia/Hovd -Druntime.java=22

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [esql/60_usage:23]: expected [2xx] status code but api [xpack.usage] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"transport_serialization_exception","reason":"Failed to deserialize response from handler [ContextRestoreResponseHandler[UnregisterChildTransportResponseHandler[unregisterChildNode=releaseOnce[org.elasticsearch.tasks.TaskManager$$Lambda/0x00007fe0c438edc0@369963e8], handler=[org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@59c9c58c}{CancellableTask{Task{id=5043, type='transport', action='cluster:monitor/xpack/usage', description='', parentTask=unset, startTime=1727238352068, headers={}, startTimeNanos=1194615182022}, reason='null', isCancelled=false}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00007fe0c4300b08@23e57dc3/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda/0x00007fe0c4302558@44595e7d], action=cluster:monitor/xpack/usage, childRequest=TermOverridingMasterNodeRequest[newMasterTerm=1 in org.elasticsearch.protocol.xpack.XPackUsageRequest/lyiBRlKgQWeftTd6cHSjmQ:5043], childConnection=NodeChannels[{test-cluster-2}{ARcOX2BwTHKDJkoYo8SGFQ}{zdsv6rkJQuijPsRgp554oA}{test-cluster-2}{127.0.0.1}{127.0.0.1:41691}{cdfhilmrstw}{8.16.0}{7000099-8515000}], taskManager=org.elasticsearch.tasks.TaskManager@1045c79e]]]","stack_trace":"org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response from handler [ContextRestoreResponseHandler[UnregisterChildTransportResponseHandler[unregisterChildNode=releaseOnce[org.elasticsearch.tasks.TaskManager$$Lambda/0x00007fe0c438edc0@369963e8], handler=[org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@59c9c58c}{CancellableTask{Task{id=5043, type='transport', action='cluster:monitor/xpack/usage', description='', parentTask=unset, startTime=1727238352068, headers={}, startTimeNanos=1194615182022}, reason='null', isCancelled=false}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00007fe0c4300b08@23e57dc3/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda/0x00007fe0c4302558@44595e7d], action=cluster:monitor/xpack/usage, childRequest=TermOverridingMasterNodeRequest[newMasterTerm=1 in org.elasticsearch.protocol.xpack.XPackUsageRequest/lyiBRlKgQWeftTd6cHSjmQ:5043], childConnection=NodeChannels[{test-cluster-2}{ARcOX2BwTHKDJkoYo8SGFQ}{zdsv6rkJQuijPsRgp554oA}{test-cluster-2}{127.0.0.1}{127.0.0.1:41691}{cdfhilmrstw}{8.16.0}{7000099-8515000}], taskManager=org.elasticsearch.tasks.TaskManager@1045c79e]]]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:422)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:390)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:992)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: java.io.IOException: Can't read unknown type [111]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:887)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:741)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:726)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:862)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.ml.MachineLearningFeatureSetUsage.<init>(MachineLearningFeatureSetUsage.java:70)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteableCollectionAsList(NamedWriteableAwareStreamInput.java:46)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.action.XPackUsageResponse.<init>(XPackUsageResponse.java:27)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:64)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:26)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1785)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1754)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1489)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1476)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:419)\n\t... 6 more\n"}],"type":"transport_serialization_exception","reason":"Failed to deserialize response from handler [ContextRestoreResponseHandler[UnregisterChildTransportResponseHandler[unregisterChildNode=releaseOnce[org.elasticsearch.tasks.TaskManager$$Lambda/0x00007fe0c438edc0@369963e8], handler=[org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@59c9c58c}{CancellableTask{Task{id=5043, type='transport', action='cluster:monitor/xpack/usage', description='', parentTask=unset, startTime=1727238352068, headers={}, startTimeNanos=1194615182022}, reason='null', isCancelled=false}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00007fe0c4300b08@23e57dc3/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda/0x00007fe0c4302558@44595e7d], action=cluster:monitor/xpack/usage, childRequest=TermOverridingMasterNodeRequest[newMasterTerm=1 in org.elasticsearch.protocol.xpack.XPackUsageRequest/lyiBRlKgQWeftTd6cHSjmQ:5043], childConnection=NodeChannels[{test-cluster-2}{ARcOX2BwTHKDJkoYo8SGFQ}{zdsv6rkJQuijPsRgp554oA}{test-cluster-2}{127.0.0.1}{127.0.0.1:41691}{cdfhilmrstw}{8.16.0}{7000099-8515000}], taskManager=org.elasticsearch.tasks.TaskManager@1045c79e]]]","caused_by":{"type":"i_o_exception","reason":"Can't read unknown type [111]","stack_trace":"java.io.IOException: Can't read unknown type [111]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:887)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:741)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:726)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:862)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.ml.MachineLearningFeatureSetUsage.<init>(MachineLearningFeatureSetUsage.java:70)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteableCollectionAsList(NamedWriteableAwareStreamInput.java:46)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.action.XPackUsageResponse.<init>(XPackUsageResponse.java:27)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:64)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:26)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1785)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1754)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1489)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1476)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:419)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:390)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:992)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"},"stack_trace":"org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response from handler [ContextRestoreResponseHandler[UnregisterChildTransportResponseHandler[unregisterChildNode=releaseOnce[org.elasticsearch.tasks.TaskManager$$Lambda/0x00007fe0c438edc0@369963e8], handler=[org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@59c9c58c}{CancellableTask{Task{id=5043, type='transport', action='cluster:monitor/xpack/usage', description='', parentTask=unset, startTime=1727238352068, headers={}, startTimeNanos=1194615182022}, reason='null', isCancelled=false}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00007fe0c4300b08@23e57dc3/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda/0x00007fe0c4302558@44595e7d], action=cluster:monitor/xpack/usage, childRequest=TermOverridingMasterNodeRequest[newMasterTerm=1 in org.elasticsearch.protocol.xpack.XPackUsageRequest/lyiBRlKgQWeftTd6cHSjmQ:5043], childConnection=NodeChannels[{test-cluster-2}{ARcOX2BwTHKDJkoYo8SGFQ}{zdsv6rkJQuijPsRgp554oA}{test-cluster-2}{127.0.0.1}{127.0.0.1:41691}{cdfhilmrstw}{8.16.0}{7000099-8515000}], taskManager=org.elasticsearch.tasks.TaskManager@1045c79e]]]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:422)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:390)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:992)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: java.io.IOException: Can't read unknown type [111]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:887)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:741)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:726)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:862)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.ml.MachineLearningFeatureSetUsage.<init>(MachineLearningFeatureSetUsage.java:70)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteableCollectionAsList(NamedWriteableAwareStreamInput.java:46)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.action.XPackUsageResponse.<init>(XPackUsageResponse.java:27)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:64)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:26)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1785)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1754)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1489)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1476)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:419)\n\t... 6 more\n"},"status":500}]

Issue Reasons:

• [main] 2 failures in test test {p0=esql/60_usage/Basic ESQL usage output (telemetry)} (0.8% fail rate in 250 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test test {p0=esql/60_usage/Basic ESQL usage output (telemetry)} (0.8% fail rate in 250 executions)

Build Scans:

• elasticsearch-periodic #4182 / 8.16.0_bwc
• elasticsearch-intake #10128 / 8.16.0_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[iverase]

I had a quick look and the error is:

stack_trace":"java.io.IOException: Can't read unknown type [111]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:887)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:741)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:726)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:862)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT
org.elasticsearch.xpack.core.ml.MachineLearningFeatureSetUsage.<init>(MachineLearningFeatureSetUsage.java:70)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteableCollectionAsList(NamedWriteableAwareStreamInput.java:46)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT

It seems some BWC issues coming from ML feature set?

[iverase]

For what I see the TransportVersions are lying. For what I see we are serializing MachineLearningFeatureSetUsage with 8.16 and deserializing with 9.0 but the transport version is providing the latest one.

[thecoop]

Currently 8.16 and 9.0 have the same transport version, as we need to keep 8.x and main identical until 8.17

[iverase]

I added low risk as it will be fixed shortly.