Netty 4.2.0 support

[wendigo]

Describe the feature

Currently, async S3 client doesn't work correctly with Netty 4.2.0 (RC exceptions are raised).

Use Case

Dependency kept up to date

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS Java SDK version used

2.31.21

JDK version used

24

Operating System and version

MacOS/Linux

[debora-ito]

async S3 client doesn't work correctly with Netty 4.2.0

Can you give us more details on what you see?
Do you see any errors, if so can you share the stacktraces?

[wendigo]

Caused by: java.lang.IndexOutOfBoundsException: writerIndex(0) + minWritableBytes(5242880) exceeds maxCapacity(0): AdaptivePoolingAllocator$AdaptiveByteBuf(ridx: 0, widx: 0, cap: 0/0)
        at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:294)
        at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1103)
        at io.netty.channel.nio.AbstractNioChannel.newDirectBuffer(AbstractNioChannel.java:519)
        at io.netty.channel.nio.AbstractNioByteChannel.filterOutboundMessage(AbstractNioByteChannel.java:279)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:739)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1386)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.timeout.WriteTimeoutHandler.write(WriteTimeoutHandler.java:112)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:830)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:749)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.write(CombinedChannelDuplexHandler.java:521)
        at io.netty.handler.codec.http.HttpObjectEncoder.writePromiseCombiner(HttpObjectEncoder.java:146)
        at io.netty.handler.codec.http.HttpObjectEncoder.writeOutList(HttpObjectEncoder.java:128)
        at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:113)
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.logging.LoggingHandler.write(LoggingHandler.java:288)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:807)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.completeBody(HttpStreamsHandler.java:340)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.access$400(HttpStreamsHandler.java:46)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler$3$1.run(HttpStreamsHandler.java:318)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.executeInEventLoop(HttpStreamsHandler.java:384)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.access$300(HttpStreamsHandler.java:46)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler$3.complete(HttpStreamsHandler.java:315)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:274)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:271)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:603)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:596)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:572)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:505)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:649)
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:642)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:131)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:870)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:748)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1386)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.timeout.WriteTimeoutHandler.write(WriteTimeoutHandler.java:112)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:830)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.write(CombinedChannelDuplexHandler.java:526)
        at io.netty.handler.codec.http.HttpObjectEncoder.writeOutList(HttpObjectEncoder.java:121)
        at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:113)
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.logging.LoggingHandler.write(LoggingHandler.java:288)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.write(HttpStreamsHandler.java:281)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsClientHandler.write(HttpStreamsClientHandler.java:59)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:807)
        at software.amazon.awssdk.http.nio.netty.internal.utils.DelegatingChannelHandlerContext.writeAndFlush(DelegatingChannelHandlerContext.java:196)
        at software.amazon.awssdk.http.nio.netty.internal.utils.OrderedWriteChannelHandlerContext.lambda$writeAndFlush$2(OrderedWriteChannelHandlerContext.java:54)
        at software.amazon.awssdk.http.nio.netty.internal.utils.OrderedWriteChannelHandlerContext.lambda$doInOrder$4(OrderedWriteChannelHandlerContext.java:73)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:507)
        at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:180)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1073)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

[wendigo]

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
        at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
        at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:148)
        at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
        at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:746)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1386)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.timeout.WriteTimeoutHandler.write(WriteTimeoutHandler.java:112)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:830)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:749)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.write(CombinedChannelDuplexHandler.java:521)
        at io.netty.handler.codec.http.HttpObjectEncoder.writePromiseCombiner(HttpObjectEncoder.java:146)
        at io.netty.handler.codec.http.HttpObjectEncoder.writeOutList(HttpObjectEncoder.java:128)
        at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:113)
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.logging.LoggingHandler.write(LoggingHandler.java:288)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:807)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.completeBody(HttpStreamsHandler.java:340)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.access$400(HttpStreamsHandler.java:46)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler$3$1.run(HttpStreamsHandler.java:318)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.executeInEventLoop(HttpStreamsHandler.java:384)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.access$300(HttpStreamsHandler.java:46)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler$3.complete(HttpStreamsHandler.java:315)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:274)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:271)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:603)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:596)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:572)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:505)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:649)
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:642)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:131)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:870)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:748)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1386)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.timeout.WriteTimeoutHandler.write(WriteTimeoutHandler.java:112)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:830)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.write(CombinedChannelDuplexHandler.java:526)
        at io.netty.handler.codec.http.HttpObjectEncoder.writeOutList(HttpObjectEncoder.java:121)
        at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:113)
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at io.netty.handler.logging.LoggingHandler.write(LoggingHandler.java:288)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:755)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.write(HttpStreamsHandler.java:281)
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsClientHandler.write(HttpStreamsClientHandler.java:59)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:828)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:807)
        at software.amazon.awssdk.http.nio.netty.internal.utils.DelegatingChannelHandlerContext.writeAndFlush(DelegatingChannelHandlerContext.java:196)
        at software.amazon.awssdk.http.nio.netty.internal.utils.OrderedWriteChannelHandlerContext.lambda$writeAndFlush$2(OrderedWriteChannelHandlerContext.java:54)
        at software.amazon.awssdk.http.nio.netty.internal.utils.OrderedWriteChannelHandlerContext.lambda$doInOrder$4(OrderedWriteChannelHandlerContext.java:73)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:507)
        at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:180)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1073)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:1447)

[wendigo]

I can add that this only happens for async S3 client in our case

[wendigo]

Any news? Google Cloud SDK already moved to 4.2.0 so it can't be used with AWS SDK in the same project anymore.

[wendigo]

I've found that async S3 client works properly when allocator is the previous default (PooledByteBufAllocator) instead of a new adaptive one.

Setting .putChannelOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT) on the NettyNioAsyncHttpClient does the trick.