Indexing a single doc to a data stream bulk endpoint trips assertion and returns class cast exception #120746
Description
Elasticsearch Version
Latest main branch
Installed Plugins
No response
Java Version
bundled
OS Version
OSX
Problem Description
Even though this is an invalid request, we shouldn't trip an assertion, and we shouldn't return a cryptic exception.
Steps to Reproduce
First, create a data stream:
POST /logs-foo-bar/_doc
{
"@timestamp": "2024-01-01",
"message": "aoeu aoeu aoeu"
}
Then, try to index a doc, but accidentally add …/_bulk to the endpoint:
POST /logs-foo-bar/_doc/_bulk
{
"@timestamp": "2024-01-01",
"message": "aoeu aoeu aoeu"
}
Logs (if relevant)
ES running in dev mode (assertions enabled) will then crash with:
[2025-01-23T09:44:59,734][ERROR][o.e.ExceptionsHelper ] [runTask-0] fatal error java.lang.AssertionError: null
at [email protected]/org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$4(ExceptionsHelper.java:448)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at [email protected]/org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:438)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.exceptionCaught(Netty4HttpPipeliningHandler.java:499)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4LeakDetectionHandler.channelRead(Netty4LeakDetectionHandler.java:38)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
at [email protected]/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpContentSizeHandler.handleRequest(Netty4HttpContentSizeHandler.java:150)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpContentSizeHandler.channelRead(Netty4HttpContentSizeHandler.java:106)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpAggregator.channelRead(Netty4HttpAggregator.java:52)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:213)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:156)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator$1.lambda$onResponse$0(Netty4HttpHeaderValidator.java:128)
at [email protected]/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at [email protected]/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:1583)
[2025-01-23T09:44:59,735][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [runTask-0] fatal error in thread [elasticsearch-error-rethrower], exiting java.lang.AssertionError
at [email protected]/org.elasticsearch.http.HttpBody.asFull(HttpBody.java:42)
at [email protected]/org.elasticsearch.rest.RestRequest.content(RestRequest.java:313)
at [email protected]/org.elasticsearch.rest.RestRequest.requiredContent(RestRequest.java:341)
at [email protected]/org.elasticsearch.rest.action.document.RestIndexAction.prepareRequest(RestIndexAction.java:109)
at [email protected]/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:99)
at [email protected]/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:466)
at [email protected]/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:460)
at [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)
at [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)
at [email protected]/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:257)
at [email protected]/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)
at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)
at [email protected]/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)
at [email protected]/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)
at [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)
at [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:460)
at [email protected]/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:624)
at [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:307)
at [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:488)
at [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:584)
at [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:461)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:170)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:149)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4LeakDetectionHandler.channelRead(Netty4LeakDetectionHandler.java:38)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
at [email protected]/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpContentSizeHandler.handleRequest(Netty4HttpContentSizeHandler.java:150)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpContentSizeHandler.channelRead(Netty4HttpContentSizeHandler.java:106)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpAggregator.channelRead(Netty4HttpAggregator.java:52)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:213)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:156)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator$1.lambda$onResponse$0(Netty4HttpHeaderValidator.java:128)
at [email protected]/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at [email protected]/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:1583)
[2025-01-23T09:44:59,735][WARN ][o.e.h.AbstractHttpServerTransport] [runTask-0] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9200, remoteAddress=/[0:0:0:0:0:0:0:1]:51276} java.lang.Exception: java.lang.AssertionError
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.exceptionCaught(Netty4HttpPipeliningHandler.java:504)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4LeakDetectionHandler.channelRead(Netty4LeakDetectionHandler.java:38)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
at [email protected]/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpContentSizeHandler.handleRequest(Netty4HttpContentSizeHandler.java:150)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpContentSizeHandler.channelRead(Netty4HttpContentSizeHandler.java:106)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpAggregator.channelRead(Netty4HttpAggregator.java:52)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:213)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:156)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator$1.lambda$onResponse$0(Netty4HttpHeaderValidator.java:128)
at [email protected]/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at [email protected]/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.AssertionError
at [email protected]/org.elasticsearch.http.HttpBody.asFull(HttpBody.java:42)
at [email protected]/org.elasticsearch.rest.RestRequest.content(RestRequest.java:313)
at [email protected]/org.elasticsearch.rest.RestRequest.requiredContent(RestRequest.java:341)
at [email protected]/org.elasticsearch.rest.action.document.RestIndexAction.prepareRequest(RestIndexAction.java:109)
at [email protected]/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:99)
at [email protected]/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:466)
at [email protected]/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:460)
at [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)
at [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)
at [email protected]/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:257)
at [email protected]/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)
at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)
at [email protected]/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)
at [email protected]/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)
at [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)
at [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:460)
at [email protected]/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:624)
at [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:307)
at [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:488)
at [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:584)
at [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:461)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:170)
at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:149)
at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
... 31 more
If you do NOT have assertions enabled, you get this response:
{
"error" : {
"root_cause" : [
{
"type" : "class_cast_exception",
"reason" : "class org.elasticsearch.http.netty4.Netty4HttpRequestBodyStream cannot be cast to class org.elasticsearch.http.HttpBody$Full (org.elasticsearch.http.netty4.Netty4HttpRequestBodyStream is in module [email protected] of loader jdk.internal.loader.Loader @3cd206b5; org.elasticsearch.http.HttpBody$Full is in module [email protected] of loader 'app')"
}
],
"type" : "class_cast_exception",
"reason" : "class org.elasticsearch.http.netty4.Netty4HttpRequestBodyStream cannot be cast to class org.elasticsearch.http.HttpBody$Full (org.elasticsearch.http.netty4.Netty4HttpRequestBodyStream is in module [email protected] of loader jdk.internal.loader.Loader @3cd206b5; org.elasticsearch.http.HttpBody$Full is in module [email protected] of loader 'app')"
},
"status" : 500
}
And this in the logs:
[2025-01-23T10:36:14,458][WARN ][r.suppressed ] [Pelennor] path: /logs-foo-bar/_doc/_bulk, params: {pretty=true, index=logs-foo-bar, id=_bulk}, status: 500java.lang.ClassCastException: class org.elasticsearch.http.netty4.Netty4HttpRequestBodyStream cannot be cast to class org.elasticsearch.http.HttpBody$Full (org.elasticsearch.http.netty4.Netty4HttpRequestBodyStream is in module [email protected] of loader jdk.internal.loader.Loader @3cd206b5; org.elasticsearch.http.HttpBody$Full is in module [email protected] of loader 'app')
at [email protected]/org.elasticsearch.http.HttpBody.asFull(HttpBody.java:43)
at [email protected]/org.elasticsearch.rest.RestRequest.content(RestRequest.java:313)
at [email protected]/org.elasticsearch.rest.RestRequest.requiredContent(RestRequest.java:341)
at [email protected]/org.elasticsearch.rest.action.document.RestIndexAction.prepareRequest(RestIndexAction.java:109)
at [email protected]/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:99)