perf: optimize shuffle writer with buffered I/O and fix file size bug #1386
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Add BufWriter for buffered file I/O in shuffle writer Wrapping File with BufWriter reduces syscalls when writing multiple small batches to shuffle files. This affects both the hash-partitioned shuffle path and the non-partitioned write_stream_to_disk utility. 2. Fix file size read before writer finish Previously, fs::metadata() was called before writer.finish(), which could report incorrect file sizes since data may not have been fully flushed to disk. This is especially important now that BufWriter is used, as buffered data would not be reflected in the file size until after finish() flushes it. Co-Authored-By: Claude Opus 4.5 <[email protected]>
andygrove
force-pushed
the
shuffle-writer-optimizations
branch
from
5b3e9ea to
53c71b6
Compare
milenkovicm
reviewed
Jan 17, 2026
| if let Some(w) = w { | ||
| let num_bytes = fs::metadata(&w.path)?.len(); | ||
| w.writer.finish()?; | ||
| let num_bytes = fs::metadata(&w.path)?.len(); |
Contributor
There was a problem hiding this comment.
took me a moment to understand whats changed
Member
Author
|
Thanks for the review @milenkovicm |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Two improvements to the shuffle writer:
Add BufWriter for buffered file I/O
FilewithBufWriterreduces syscalls when writing multiple small batches to shuffle filesShuffleWriterExecand thewrite_stream_to_diskutilityFix file size read before writer finish (bug fix)
fs::metadata()was called beforewriter.finish(), which could report incorrect file sizesBufWriteris usedfinish()first, then read the file sizeNote
This PR was created with AI assistance using Claude Code. All changes were reviewed and approved by a human maintainer.
Test plan
cargo test -p ballista-core shuffle_writer)