Skip to content

[CELEBORN-2262] Improve performance of S3 client#4

Merged
eolivelli merged 1 commit intomainfrom
CELEBORN-2262-qbeast
Feb 13, 2026
Merged

[CELEBORN-2262] Improve performance of S3 client#4
eolivelli merged 1 commit intomainfrom
CELEBORN-2262-qbeast

Conversation

@eolivelli
Copy link
Collaborator

What changes were proposed in this pull request?

  • Create only one S3 client for all MultiPartUploaders
  • Create S3 worker directory only once and not for every slot

Why are the changes needed?

  • Because on S3 AWS creating connections is slow (due to credentials handshaking and TLS handshaking)
  • Because "mkdirs" in S3 AWS is very slow (and it needs several S3 calls)

Does this PR resolve a correctness bug?

No

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual testing on real S3 AWS

@eolivelli eolivelli merged commit a17995c into main Feb 13, 2026
77 of 110 checks passed
@eolivelli eolivelli deleted the CELEBORN-2262-qbeast branch February 13, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant