SocketTimeoutException during SSL handshake when using StsWebIdentityCredentialsProvider in AWS Kotlin SDK

[vineetu]

Describe the bug

I’m encountering a SocketTimeoutException when using the AWS Kotlin SDK to connect to DynamoDB in a Kubernetes environment. The exception occurs during the SSL handshake when the StsWebIdentityCredentialsProvider attempts to retrieve credentials.

When I do the same using the java alternative software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider. It works just fine.

Here is the stack

Exception in thread "main" aws.smithy.kotlin.runtime.http.HttpException: java.net.SocketTimeoutException: Read timed out; HttpErrorCode(SOCKET_TIMEOUT)
at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine.roundTrip(OkHttpEngine.kt:168)
at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine$roundTrip$1.invokeSuspend(OkHttpEngine.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:99)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:589)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:832)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:720)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:707)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278)
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

Regression Issue

• Select this option if this issue appears to be a regression.

Expected behavior

The AWS Kotlin SDK should successfully use the StsWebIdentityCredentialsProvider to obtain temporary credentials and communicate with AWS services without encountering a socket timeout.

Current behavior

There is a socket timeout and the application throws a SocketTimeoutException during the SSL handshake when attempting to retrieve credentials using the StsWebIdentityCredentialsProvider.

Steps to Reproduce

dynamoDB = DynamoDbClient {
region = dynamoConfig.region
credentialsProvider = StsWebIdentityCredentialsProvider.fromEnvironment(roleArn = "arn", roleSessionName = "name",
webIdentityTokenFilePath= "token-location", region = "us-west-2")
}

Possible Solution

No response

Context

I am no longer able to call dynamo-db. I am moving my service from java to kotlin and wanted to use the kotlin sdk.
Now this is blocker.

AWS SDK for Kotlin version

1.3.57

Platform (JVM/JS/Native)

JVM

Operating system and version

runs in an kubernetes pod in an ec2 container

[ianbotsf]

Hi @vineetu, I'm sorry you're seeing socket timeout exceptions connecting to STS. I'm unable to reproduce this on a regular (i.e., non-Kubernetes) EC2 instance. I'm working on setting up EKS now but, in the meantime, can you please provide some additional info for debugging:

• Do any AWS SDK calls (using non-STS credentials) succeed in your environment? That is, can you use local credentials to access DynamoDB directly?
• Does your environment use a VPC? Is that VPC configured to allow secure access to STS endpoints?
• Can you share the Java AWS SDK code which succeeds?

[github-actions]

It looks like this issue has not been active for more than 5 days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.