Skip to content

Refactored restore to better use network resources #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Refactored restore to better use network resources #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
0089b93
basic refactor of restore
Sep 20, 2017
d972da9
Added timeout between _sendBatch when concurrency exceeded
Sep 20, 2017
e8cefa4
moved send-batch emit
Sep 20, 2017
e1ab3ae
attempt to time requests
Sep 20, 2017
1ec2844
purged the unclean
Sep 20, 2017
4b3650c
redid timing math
Sep 20, 2017
d1fa68d
Revert "attempt to time requests"
Sep 20, 2017
f6bec0b
decrease concurrency when running fast
Sep 20, 2017
753bc3a
debug printing
Sep 20, 2017
7af2552
added option to read from local file rather than s3
Sep 21, 2017
d952d33
removed timer based throttling
Sep 21, 2017
424e25e
fixed for testing
Sep 21, 2017
a461672
updated package.js for release
Sep 21, 2017
c9aaeca
Separated read and send loops; no unprocessed items when requests < 1s
Sep 25, 2017
d75a0ac
Minor fixes
Sep 25, 2017
86df1e0
covered some edge cases
Sep 25, 2017
1f2b39d
I did not understand this
Sep 26, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions bin/dynamo-restore-from-s3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ program
.option('-s, --source [path]', 'Full S3 path to a JSON backup file (Required)')
.option('-t, --table [name]', 'Name of the Dynamo Table to restore to (Required)')
.option('-o, --overwrite', 'Table already exists, skip auto-create. Default is false.')
.option('-c, --concurrency <requestcount>', 'Number of concurrent requests & dynamo capacity units. Defaults to 200.')
.option('-c, --concurrency <requestcount>', 'Number of concurrent requests & dynamo capacity units. Due to default partitioning, is useless above 1000. Defaults to 500.')
.option('-pk, --partitionkey [columnname]', 'Name of Primary Partition Key. If not provided will try determine from backup.')
.option('-sk, --sortkey [columnname]', 'Name of Secondary Sort Key. Ignored unless --partitionkey is provided.')
.option('-rc, --readcapacity <units>', 'Read Units for new table (when finished). Default is 5.')
Expand Down Expand Up @@ -55,9 +55,9 @@ function translate(contentLength) {
var kb = contentLength / 1024,
mb = kb / 1024,
gb = mb / 1024;
return gb > 5 ? gb.toFixed(0) + ' Gb' :
(mb > 5 ? mb.toFixed(0) + 'Mb' :
kb.toFixed(0) + 'Kb');
return gb > 5 ? gb.toFixed(1) + ' GiB' :
(mb > 5 ? mb.toFixed(1) + 'MiB' :
kb.toFixed(0) + 'KiB');
}

// Define events
Expand All @@ -75,8 +75,8 @@ dynamoRestore.on('start-download', function(streamMeta) {
console.log('Starting download. %s remaining...', translate(streamMeta.ContentLength));
});

dynamoRestore.on('send-batch', function(batches, requests, streamMeta) {
console.log('Batch sent. %d in flight. %s remaining to download...', requests, translate(streamMeta.RemainingLength));
dynamoRestore.on('send-batch', function(batches, queue, streamMeta) {
console.log('%d batches in flight. %d items in queue. %s remaining to download...', batches, queue, translate(streamMeta.RemainingLength));
});

// Start Process
Expand All @@ -87,4 +87,4 @@ dynamoRestore.run(function() {
seconds = Math.floor((diff % 1000 * 60) / 1000);
console.log('Done! Process completed in %s minutes %s seconds.', minutes, seconds);
process.exit(0);
});
});
Loading