Refactored restore to better use network resources #62
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 17 commits
September 26, 2017 10:09
S3 has a chance of randomly closing the connection before the download is finished. This makes restoring from large files impossible. This is a hack, to download the file quickly, then do the much-slower restore.
most of the time was spent in node (CPU bound). Timing only how long the request took failed to acount for overhead, and thus throttled down to 20% of the target.
This is 60% faster on my test tables
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Staggers write requests in order to reduce number of unprocessed items.
Combines unprocessed items into new batches (no more batches of only a few items).
Allows restoring from local file, because s3 likes to close long-running connections.
Implementation
It now has two completely separate loops:
This separation allows _sendBatch to call itself after a certain amount of time has passed (every (1000 / concurrency) milliseconds). The previous implementation allowed a certain number of concurrent requests regardless of speed; on a fast network (a large EC2 instance), even 1 concurrent request was equivalent to 2500 writes per second.