fix: prevent OOM on large upload/download #291
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mcxiv
force-pushed
the
fix/oom-large-file-upload-download
branch
from
8e1472a to
0a25a56
Compare
- Refactored Upload function to stream file content instead of loading it all into memory (fixes OOM issue for large files) - Added log.IsVerbose() to expose verbose state from log package - Updated Download function to dump HTTP response only when verbose mode is enabled, to avoid potential OOM on large responses Changelog: None Ticket: None Signed-off-by: Quentin Dufournet <[email protected]>
mcxiv
force-pushed
the
fix/oom-large-file-upload-download
branch
from
0a25a56 to
10ddf50
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
In my work, I occasionally need to upload and download large files (several GBs), such as video recordings. While using
mender-clifor this purpose, I encountered OOM issues during uploads, and realized the same risk could occur when dumping large HTTP responses during downloads.This PR includes the following fixes:
Refactors the Upload function to stream file content using
io.Pipe, avoiding loading the entire file into memory.Adds
log.IsVerbose()to expose the verbose mode flag.Updates the Download function to conditionally dump HTTP responses only when verbose mode is enabled, to avoid loading it in memory when there is no need.
These changes aim to improve memory efficiency during large file transfers, reducing the risk of OOM errors.
This is my first PR to this project, so please be indulgent if I missed anything — happy to make adjustments! 🙌
Quentin