EOFException (\n not found: limit=0) occurs when calling getObject function

[john-levvels]

Describe the bug

When calling the getObject function of the sdk, EOFException exception occurs occasionally. If I retried calling the function once or twice when the exception occurred, it worked normally. I couldn't find a way to reproduce it.

The exception log is as follows:

aws.smithy.kotlin.runtime.http.HttpException: java.io.IOException: unexpected end of stream on https:/....amazonaws.com/...; HttpErrorCode(CONNECTION_CLOSED)
	at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine.roundTrip(OkHttpEngine.kt:166)
	at aws.smithy.kotlin.runtime.http.engine.okhttp.OkHttpEngine$roundTrip$1.invokeSuspend(OkHttpEngine.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:102)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:811)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:715)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:702)
Caused by: java.io.IOException: unexpected end of stream on https://.....amazonaws.com/...
	at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:220)
	at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:114)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:72)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
	at datadog.trace.instrumentation.okhttp3.TracingInterceptor.intercept(TracingInterceptor.java:39)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
	at aws.smithy.kotlin.runtime.http.engine.okhttp.MetricsInterceptor.intercept(MetricsInterceptor.kt:30)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:203)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:527)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:339)
	at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
	at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:188)
	... 20 more

Regression Issue

• Select this option if this issue appears to be a regression.

Expected behavior

There is no the exception.

Current behavior

SDK version is 1.3.57. The getObject function is suspend function and I referred to the following example.
https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/kotlin_s3_code_examples.html.
When the function is called, the exception occurred occasionally. I will attach example code in Steps to Reproduce.

Steps to Reproduce

Example code is as follows:

private val s3client: aws.sdk.kotlin.services.s3.S3Client = aws.sdk.kotlin.services.s3.S3Client {
    region = properties.regionName
}

suspend fun getObject(keyName: String): Pair<String?, ByteArray?> {
    val request = GetObjectRequest {
        key = keyName
        bucket = bucketName
    }
    val response = s3client.let { s3 ->
        s3.getObject(request) { resp ->
            Pair(resp.contentType, resp.body?.toByteArray())
        }
    }
    return response
}

Possible Solution

No response

Context

No response

AWS SDK for Kotlin version

1.3.57

Platform (JVM/JS/Native)

amazon-corretto-21.jdk

Operating system and version

amazonlinux:2023 (docker image)

[ianbotsf]

Hi @john-levvels, this is a known issue being tracked and actively worked in #1214. That issue also lists the current workarounds while we implement a bigger fix:

• Reduce the SDK's idle timeout to below what S3's service-side timeout appears to be:

  S3Client.fromEnvironment {
      httpEngine {
          connectionIdleTimeout = 3.seconds
      }
  }

• Use the CRT HTTP engine instead of OkHttp:

  S3Client.fromEnvironment {
      httpClient(CrtHttpEngine)
  }

I'm doing to resolve this issue as a duplicate for now. If you have any additional context to add, please contribute on #1214.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.