Skip to content

Fix bug in MultipartS3AsyncClient GetObject #6320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 7, 2025

Conversation

davidh44
Copy link
Contributor

@davidh44 davidh44 commented Aug 7, 2025

Motivation and Context

This PR fixes a bug in the Java-based multipart S3 client (MultipartS3AsyncClient) that causes duplicate request logging when a retryable error occurs during many concurrent GetObject operations

Currently, when a retryable error, e.g., 503 Slowdown, is returned by S3, the SDK may do one of two things:

  1. Fail the request right away instead of retrying
  2. Retry the error, but incorrectly process subsequent responses. A successful 200 response returned by the sever will be ignored. Instead, the SDK will log the initial error, and continue to retry until retry attempts are exhausted. This may happen instead of scenario 1 (failing right away), if many concurrent requests are in progress and a race condition occurs.

This PR fixes 2), a follow up PR will fix 1)

Modifications

SplittingTransformer

  • When completing the individualFuture exceptionally in the IndividualTransformer, wrap the exception with NonRetryableException

MultipartDownloaderSubscriber

  • Keep track of each part GET request and cancel all in-flight ones if onError() is invoked

Testing

Added mock tests

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

@davidh44 davidh44 requested a review from a team as a code owner August 7, 2025 17:33
@davidh44 davidh44 added this pull request to the merge queue Aug 7, 2025
Merged via the queue into master with commit 19b969e Aug 7, 2025
35 of 37 checks passed
Copy link

github-actions bot commented Aug 7, 2025

This pull request has been closed and the conversation has been locked. Comments on closed PRs are hard for our team to see. If you need more assistance, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 7, 2025
@davidh44 davidh44 deleted the hdavidh/multipart-download-cancel-futures branch August 7, 2025 23:03
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants