[client] Introduce batchDeliveryTimeout in client to expire batches that have been stuck for a long time #686
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…hat have been stuck for a long time
swuferhong
force-pushed
the
reset-writerId
branch
from
4d66d10 to
873f771
Compare
wuchong
reviewed
Mar 31, 2025
Member
wuchong
left a comment
wuchong
left a comment
There was a problem hiding this comment.
Could you add an IT case for this in FlussFailServerTableITCase (mock all servers are down and the request timeout)?
fluss-client/src/main/java/com/alibaba/fluss/client/write/WriterClient.java
Show resolved
Hide resolved
| // TODO add logic for batch expire. | ||
| // check and expire batches if they have reached batch delivery timeout. This can avoid | ||
| // the client to wait for a long time while the batch is forever retry. | ||
| checkAndExpireBatches(now); |
Member
There was a problem hiding this comment.
What happens if the batch is expired here but is still in the sendWriteRequests(batches)? Will the expired batch still be sent to servers? What happens if the batches fail or success from the request?
polyzos
force-pushed
the
main
branch
3 times, most recently
from
d88c76c to
434a4f4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: #685
This pr is aims to introduce
batchDeliveryTimeoutin client to expire batches that have been stuck for a long time. The purpose of this approach is to enable our Flink job can be recover the write operation by restarting the client via failover.Brief change log
Tests
API and Format
Documentation