Uploading using multiple threads

[vadimkantorov]

Tell us more about this new feature.

We found that s5cmd is several times faster and has higher throughput because of usage of multiple threads. Can s3-connector-for-pytorch also benefit from multiple-thread-uploading?

This can lead to much faster checkpoint uploads, given that checkpoints can now easily measure in dozens or hundreds of gigabytes

The underlying mountpoint-s3 seems to support multiple-thread-uploading, but how could I enable/configure it using from s3-connector-for-pytorch API surface?

[jet-tong]

Hi @vadimkantorov, thank you for raising this performance question.

To configure concurrent uploading:

• You can adjust S3ClientConfig's throughput_target_gbps parameter, which influences the number of active connections in AWS CRT. You can pass a configured S3ClientConfig object via s3client_config when initializing any of our provided primitives (e.g. S3Checkpoint(region, s3client_config=S3ClientConfig(throughput_target_gbps=20)))
• For larger checkpoints, we recommend using our distributed checkpointing primitives with S3StorageWriter, which enables parallel uploads from multiple training processes. You can control concurrency via the number of PyTorch workers and the number of writer threads (e.g. S3StorageWriter(region, uri, thread_count=thread_count))

Would you mind clarifying a few things about your use case?

• Are you using S3Checkpoint or our distributed checkpointing S3StorageWriter primitive?
• When comparing performance with s5cmd, could you share more details about your benchmark setup and results? Specifically, what's your typical checkpoint size, any s5cmd configuration settings, and what timing measurements are you including? This helps us understand if the comparison includes just the file transfer time or also accounts for PyTorch model serialization time, which is part of the checkpointing process.