Netty Buffer Leak when upgrading Java API client version 8.15.X #893
Description
Java API client version
8.15.2
Java version
OpenJDK Runtime Environment Temurin-21.0.2+13
Elasticsearch Version
8.13.4
Problem description
Hello,
We noticed after upgrading the Java API client version from 8.14.1 -> 8.15.0-8.15.2, ES version still 8.13.4 we started to monitor memory leaks in our logs for our component, trace is attached to this case. We saw that there was a problem regarding netty leaks that was fixed for the server side: elastic/elasticsearch#108369, however our issue resides on the client side. Is there a known issue for the Java API client versions 8.15.X regarding this?
LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:410) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116) org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:72) org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39) org.springframework.http.codec.json.AbstractJackson2Encoder.encodeValue(AbstractJackson2Encoder.java:266) org.springframework.http.codec.json.AbstractJackson2Encoder.lambda$encode$0(AbstractJackson2Encoder.java:158) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137) reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245) reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305) reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137) reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2097) reactor.core.publisher.MonoReduceSeed$ReduceSeedSubscriber.onComplete(MonoReduceSeed.java:163) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) reactor.core.publisher.FluxMergeSequential$MergeSequentialMain.drain(FluxMergeSequential.java:374) reactor.core.publisher.FluxMergeSequential$MergeSequentialMain.innerComplete(FluxMergeSequential.java:335) reactor.core.publisher.FluxMergeSequential$MergeSequentialInner.onComplete(FluxMergeSequential.java:591) reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:298) reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:478) reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:178) reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) io.github.resilience4j.reactor.circuitbreaker.operator.CircuitBreakerSubscriber.hookOnNext(CircuitBreakerSubscriber.java:59) reactor.core.publisher.BaseSubscriber.onNext(BaseSubscriber.java:160) reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) reactor.core.publisher.MonoCompletionStage$MonoCompletionStageSubscription.apply(MonoCompletionStage.java:121) reactor.core.publisher.MonoCompletionStage$MonoCompletionStageSubscription.apply(MonoCompletionStage.java:67) java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) co.elastic.clients.transport.ElasticsearchTransportBase.lambda$performRequestAsync$0(ElasticsearchTransportBase.java:213) java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) co.elastic.clients.transport.rest_client.RestClientHttpClient$1.onSuccess(RestClientHttpClient.java:115) datadog.trace.instrumentation.elasticsearch7.RestResponseListener.onSuccess(RestResponseListener.java:26) org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:680) org.elasticsearch.client.RestClient$1.completed(RestClient.java:403) org.elasticsearch.client.RestClient$1.completed(RestClient.java:397) datadog.trace.instrumentation.apachehttpasyncclient.TraceContinuedFutureCallback.completeDelegate(TraceContinuedFutureCallback.java:83) datadog.trace.instrumentation.apachehttpasyncclient.TraceContinuedFutureCallback.completed(TraceContinuedFutureCallback.java:43) org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182) org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) java.base/java.lang.Thread.run(Thread.java:1583)