S3 Transfer Manager Performance Issues: Single Thread Execution and Long Completion Times on uploading 30MB file

[3umarG]

Describe the bug

I'm experiencing two significant performance issues with the AWS S3 Transfer Manager when uploading ONLY 30MB files using AsyncRequestBody.fromInputStream(). Despite configuring multiple threads using Executors Service and proper concurrency settings, uploads are not utilizing multiple threads effectively, and there are unexpectedly long delays after the upload progress reaches 100%.

• I want to know if uploading 30MB file using Transfer Manager capabilities takes up to 2 minutes as a normal rate or not ??
• Finally, please provide to me what are the best solution or approach(s) to upload files with maximum size 100MB length as Multipart file from REST API.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

1. Multiple threads should be utilized for multipart uploads when using AsyncRequestBody.fromInputStream() with configured executors setter in the config.
2. Completion phase should not take significantly longer than the actual data transfer phase.

Current Behavior

1. Only one thread (pool-4-thread-1) is handling the upload process, despite having multiple parts and sufficient concurrency configuration.

2025-06-28T17:51:10.166+03:00  INFO 25086 --- [pool-4-thread-1] s.a.a.t.s.p.LoggingTransferListener      : |=                   | 5.0%
2025-06-28T17:51:10.170+03:00  INFO 25086 --- [pool-4-thread-1] s.a.a.t.s.p.LoggingTransferListener      : |==                  | 10.0%
2025-06-28T17:51:10.174+03:00  INFO 25086 --- [pool-4-thread-1] s.a.a.t.s.p.LoggingTransferListener      : |===                 | 15.0%
// ... all progress updates show pool-4-thread-1

2. There's a significant delay (almost 2 minutes) between reaching 100% progress and actual completion.

2025-06-28T17:51:10.206+03:00  INFO 25086 --- [ AwsEventLoop 3] s.a.a.t.s.p.LoggingTransferListener      : |====================| 100.0%
// ... long delay here ...
2025-06-28T17:53:00.921+03:00  INFO 25086 --- [nio-8080-exec-1] o.e.s.services.TransferManagerService    : Transfer Manager upload completed successfully in 116372 ms
2025-06-28T17:53:00.922+03:00  INFO 25086 --- [nc-response-0-0] s.a.a.t.s.p.LoggingTransferListener      : Transfer complete!

Reproduction Steps

S3 maven dependencies

  <properties>
        <java.version>17</java.version>
        <aws.sdk.version>2.21.29</aws.sdk.version>
        <amazon.awssdk.crt.version>0.29.9</amazon.awssdk.crt.version>
    </properties>
<dependencies>
        <!-- AWS SDK v2 -->
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>${aws.sdk.version}</version>
        </dependency>

        <!-- AWS SDK v2 Transfer Manager -->
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3-transfer-manager</artifactId>
            <version>${aws.sdk.version}</version>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>${amazon.awssdk.crt.version}</version>
        </dependency>
</dependencies>

S3 Client and Transfer Manager config

package org.example.s3streamingpoc;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.transfer.s3.S3TransferManager;

import java.net.URI;
import java.time.Duration;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

@Configuration
public class S3Config {

    @Value("${aws.s3.region}")
    private String region;

    @Value("${aws.s3.access-key}")
    private String accessKey;

    @Value("${aws.s3.secret-key}")
    private String secretKey;

    @Value("${aws.s3.endpoint}")
    private String endpoint;


    @Bean
    public S3Client s3Client() {
        AwsBasicCredentials awsCredentials = AwsBasicCredentials.create(accessKey, secretKey);
        return S3Client
                .builder()
                .credentialsProvider(StaticCredentialsProvider.create(awsCredentials))
                .endpointOverride(URI.create(endpoint))
                .forcePathStyle(true)
                .region(Region.of(region))
                .overrideConfiguration(ClientOverrideConfiguration.builder()
                        .apiCallTimeout(Duration.ofMinutes(30))
                        .apiCallAttemptTimeout(Duration.ofMinutes(5))
                        .build())
                .build();
    }



    @Bean
    public S3AsyncClient s3AsyncClient() {
        AwsBasicCredentials awsCredentials = AwsBasicCredentials.create(accessKey, secretKey);
        return S3AsyncClient
                .crtBuilder()
                .credentialsProvider(StaticCredentialsProvider.create(awsCredentials))
                .endpointOverride(URI.create(endpoint))
                .forcePathStyle(true)
                .region(Region.of(region))
                .minimumPartSizeInBytes(10L * 1024 * 1024) // 10MB
                .maxConcurrency(5)
                .build();
    }

    @Bean
    public S3TransferManager s3TransferManager(S3AsyncClient s3AsyncClient) {
        return S3TransferManager.builder()
                .s3Client(s3AsyncClient)
                .executor(executorService())
                .build();
    }

    @Bean
    public ExecutorService executorService(){
        return Executors.newFixedThreadPool(5);
    }

}

Actual Service

 public void uploadLargeFile(MultipartFile file) {
        long startTime = System.currentTimeMillis();
        String key = generateKey(file.getOriginalFilename());

        log.info("Starting upload for file: {} (size: {} bytes)", file.getOriginalFilename(), file.getSize());

        try {
            UploadRequest uploadRequest = UploadRequest.builder()
                    .putObjectRequest(request -> request
                            .bucket(bucketName)
                            .key(key)
                            .contentType(file.getContentType())
                            .contentLength(file.getSize()))
                    .requestBody(AsyncRequestBody.fromInputStream(
                            file.getInputStream(),
                            file.getSize(),
                            executorService
                    ))
                    .addTransferListener(LoggingTransferListener.create())
                    .build();

            transferManager.upload(uploadRequest).completionFuture().join();

            long totalTime = System.currentTimeMillis() - startTime;
            log.info("Total upload completed in {} ms", totalTime);

        } catch (Exception e) {
            log.error("Upload failed", e);
        }
    }

    private String generateKey(String originalFilename) {
        return "uploads/" + UUID.randomUUID() + "_" + originalFilename;
    }

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.21.29

JDK version used

17

Operating System and version

MacOS M2 15.3.1 16GB Memory

[debora-ito]

Hi @3umarG, thank you for reaching out.

The odd long delay TransferListener is taking to show "Transfer complete" was reported already, here: #3670. Please track this issue for updates.

As for Only one thread (pool-4-thread-1) is handling the upload process, I believe [pool-4-thread-1] is the thread handling the TransferListener logging, not the actual PUT request process. If you generate the CRT logs (instructions here) you will see metrics on the number of requests in-flight and they'll match the maxConcurrency configured.

Let us know if this helps.

[3umarG]

Hi @debora-ito ,
First thanks for your reply I will follow the mentioned issue for any future help. The actual question after on some debugging and reading for many days in v2 docs for aws-sdk; what is the best setup/approach to upload input stream from multipart file that maximum size = 100MB ??. Based on docs using TransferManager and any multipart upload approach is useful when file above 100MB. I need a simple approach; first of all to be memory safer and can scale to some concurrent uploads.
Thanks in advance for your help.