Skip to content

Directory key is not filtered out from versionList  #1

@dixia

Description

@dixia

The object versions retrieved from s3.listObjectVersions (buildS3VersionList.js) will contain a directory key as well. For example, if you have put incremental object records in:

backup/

then a object version of backup directory itself will also be pass to s3ToDynamo.js:

Therefore, the following exception will be throw:

exception in /Users/han/repo/dynamo-incremental-restore/lib/s3ToDynamo.js:40
SyntaxError: Unexpected end of input

 38                 console.timeEnd('RFS3' + version.Key);
 39                 try {
>40                     deferred.resolve([version, JSON.parse(data)]);
 41                 } catch (e) {
 42                     deferred.reject('Failed to retrieve file from S3 - Params: ' + JSON.stringify(s3Params));

Because the data will be empty so it will be an invalid JSON file. Just wondering is this issue because my prefix setting? I have the following map pass to logbuildList function.


{
    Prefix: "backup/registrations/",
    //Adjust the InstID as needed. OR you can not provide an Inst ID and you get all the forms.
    // e.g. Formsmith/Formsmith.Green.FormData/543
    DestinationTableName: "registrations" ,
    //The table to backup. 1 table at a time e.g. Formsmith.Green.FormData
    Bucket: "blanco-han-test",
    //The bucket where the data is stored. e.g. pageup-dynamo-backup
    restoreToPointInTime: new Date('2017-01-12T18:34:34.128Z'),
    //the last point in time you want to update a version
    //restoreToPointInTime: new Date('2016-11-14T23:50:32.000Z'),
}

If so I am curious how do your guys set up the incremental backup so that the directory will never be included.

BTW, I also made some fix for the signature version 4 signing which will follow later.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions