putObject with ByteStream fails for some files

[sbakhtiarov]

Describe the bug

I am using putObject to upload small files (< 5Mb) to S3 bucket:

            putObject {
                bucket = credentials.bucketName
                key = "$path/$fileName"
                metadata = metadataVal
                body = path.toFile().asByteStream()
            }

some files are uploaded without any issues for others I get 503 error:

<Error>
    <Code>XMinioBackendDown</Code>
    <Message>Object storage backend is unreachable</Message>
    <Key>wire-cells-android/test_image_1.png</Key>
    <BucketName>io</BucketName>
    <Resource>/io/wire-cells-android/test_image_1.png</Resource>
    <RequestId>181F882CC38AD42D</RequestId>
    <HostId>f2096f02-5516-4920-8eff-0ae12239852a</HostId>
</Error>

Observation 1:

            putObject {
                bucket = credentials.bucketName
                key = "$path/$fileName"
                metadata = metadataVal
                body = ByteStream.fromBytes(path.toFile().readBytes())
            }

If I am reading file in memory before sending and construct ByteBuffer from ByteArray it works for all files without any issues:

Observation 2:

            putObject {
                bucket = credentials.bucketName
                key = "$cellName/$fileName"
                metadata = metadataVal
                body = path.toFile().readBytes().inputStream().asByteStream()
            }

If I am reading file in memory but create ByteBuffer via inputStream I get same errors again:

Observation 3:
I created a custom ProgressListener wrapping stream content to support upload progress indication.
For successfully uploaded files progress listener was reporting the amount of bytes sent equal to the file size.
For files that could not be uploaded putObject operation tried to send more than the file size.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected behavior

putObject operation with stream body should work consistently for all files

putObject {
                bucket = credentials.bucketName
                key = "$path/$fileName"
                metadata = metadataVal
                body = path.toFile().asByteStream()
            }

Current behavior

putObject operation with stream body fails to upload some files.

Steps to Reproduce

        with(s3Client) {
            putObject {
                bucket = credentials.bucketName
                key = "$cellName/$fileName"
                metadata = metadataVal
                body = path.toFile().asByteStream()
            }
        }

Possible Solution

No response

Context

No response

AWS SDK for Kotlin version

1.3.112

Platform (JVM/JS/Native)

Jvm

Operating system and version

Android

[ianbotsf]

Hi @sbakhtiarov, from the error message it appears you're using Minio from the AWS SDK, possibly as an intermediary cache. We are unable to provide direct support for Minio-specific issues and you may have better luck at Minio's repo.

That said, the observations you've posted likely indicate a gap in Minio's compatibility with particular S3 APIs/features. S3 recently announced some significant changes to object integrity protections which required updates from Minio to retain compatibility (see minio/minio#20849 and minio/minio#20845 for more details/context). I recommend updating your AWS SDK for Kotlin version to v1.4.10 and updating any Minio components to their latest versions.

Reading a file into memory before sending it (i.e., ByteStream.fromBytes(path.toFile().readBytes())) will cause the SDK to send the body in a standard payload with the content checksum precalculated. Creating the byte stream directly from the file (i.e., path.toFile().asByteStream()) or from a JVM InputStream (i.e., path.toFile().readBytes().inputStream().asByteStream()) with a size over 1MB causes the SDK to send the body as a chunked payload with the checksum calculated during transmission and included as an HTTP trailer. This behavior allows the SDK to avoid loading large data into memory and impacting performance. That distinction may be relevant to debugging this issue from the Minio side.

If you wish, you can disable chunked encoding by customizing the S3 client:

S3Client.fromEnvironment {
    enableAwsChunked = false
}

This will cause the S3 client to fully read payloads into memory and calculate their checksum before sending, potentially impacting memory usage by your application.

Does that address your issue?

[sbakhtiarov]

Thank you, I understand now what is causing the issue. Disabling the enableAwsChunked feature also solved the problem.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.