Issues with DynamoDbAsyncClient in an AWS lambda container image

[einarjohnson]

Hello all,
I am using the asynchronous DynamoDbAsyncClient in my Quarkus AWS Lambda project. This lambda is consuming events from a MSK topic and processing events and updating/inserting data into DynamoDB.
The project is compiled as a native container image and it runs pretty smoothly until after a while when I start seeing these errors pop up

I am using Quarkus version 2.13.3.Final and quarkus-amazon-services version 1.3.1

exception:{ refId:1 exceptionType:software.amazon.awssdk.core.exception.SdkClientException message: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate. Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate. Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout. If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests. frames:[ { class:software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl method:build line:102 } { class:software.amazon.awssdk.core.exception.SdkClientException method:create line:47 } { class:software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper method:setLastException line:204 } { class:software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper method:setLastException line:200 } { class:software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor method:maybeRetryExecute line:179 } { class:software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor method:lambda$attemptExecute$1 line:159 } { class:java.util.concurrent.CompletableFuture method:uniWhenComplete line:863 } { class:java.util.concurrent.CompletableFuture$UniWhenComplete method:tryFire line:841 } { class:java.util.concurrent.CompletableFuture method:postComplete line:510 } { class:java.util.concurrent.CompletableFuture method:completeExceptionally line:2162 } { class:software.amazon.awssdk.utils.CompletableFutureUtils method:lambda$forwardExceptionTo$0 line:76 } { class:java.util.concurrent.CompletableFuture method:uniWhenComplete line:863 } { class:java.util.concurrent.CompletableFuture$UniWhenComplete method:tryFire line:841 } { class:java.util.concurrent.CompletableFuture method:postComplete line:510 } { class:java.util.concurrent.CompletableFuture method:completeExceptionally line:2162 } { class:software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage method:lambda$null$0 line:103 } { class:java.util.concurrent.CompletableFuture method:uniWhenComplete line:863 } { class:java.util.concurrent.CompletableFuture$UniWhenComplete method:tryFire line:841 } { class:java.util.concurrent.CompletableFuture method:postComplete line:510 } { class:java.util.concurrent.CompletableFuture method:completeExceptionally line:2162 } { class:software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage method:lambda$executeHttpRequest$3 line:165 } { class:java.util.concurrent.CompletableFuture method:uniWhenComplete line:863 } { class:java.util.concurrent.CompletableFuture$UniWhenComplete method:tryFire line:841 } { class:java.util.concurrent.CompletableFuture$Completion method:run line:482 } { class:java.util.concurrent.ThreadPoolExecutor method:runWorker line:1136 } { class:java.util.concurrent.ThreadPoolExecutor$Worker method:run line:635 } { class:java.lang.Thread method:run line:833 } { class:com.oracle.svm.core.thread.PlatformThreads method:threadStartRoutine line:704 } { class:com.oracle.svm.core.posix.thread.PosixPlatformThreads method:pthreadStartRoutine line:202 } ]

I have tried modifying the documented netty parameters without luck. They are currently set to:
quarkus.dynamodb.async-client.max-concurrency=200
quarkus.dynamodb.async-client.connection-timeout=30S
quarkus.dynamodb.async-client.connection-acquisition-timeout=30S

According to my metrics I am only receiving a very low volume of events which I am processing and calling PutItem and UpdateItem.
The greatest number I have seen is about 20 events per lambda call so I cannot imagine I am making too many asynchronous requests that would consume up all the connection pool inside Netty. Is there a configuration detail I may be missing or some clean up operation that I need to make sure gets executed by the Quarkus framework?

[quarkus-bot[bot]]

/cc @matejvasek, @patriot1burke