Can't cope with large number of images sync

[amanhanzra]

Hi,
This solution is working well with small number of images to sync from one bucket to other, but I have more than 1million images, Lambda taking a long time to create the list and then step function timing out. Any solution??
Thanks.

[glez-aws]

Hi,

thanks for trying out this solution and for your question.

As a general rule, whenever you are running into a timeout, try breaking down the task into smaller chunks. In this case, you can modify the list creation function (list_bucket.py) to use a smaller list size. Actually, the code does that already, but the default values have been chosen based on simple experimenting, so tweaking the MAX_KEYS constant to a smaller number like 128 or so could help.

This will lead to the Lambda function only listing MAX_KEYS or less items at a time which should complete within the 60 second Lambda execution timeout.

Another thing you can try is to modify the Timeout: 60 comment in list_bucket.py to use a timeout of 300 seconds (the maximum for Lambda functions).

Together, you should be able to find a good setting for listing a useful number of items per step.

Keep in mind that larger object counts may require more modifications. AWS Step Functions currently support a maximum execution history size of 25000 so you cannot run an execution that runs for more than 25000 steps. This will not work for millions of objects, but you can coordinate multiple runs of this state machine by writing another state machine that breaks down the overall problem into multiple executions of this state machine (i.e. one for each subfolder, or use some form of prefix partitioning, etc.).

If all you want is to have a one-time-copy of your bucket with a million of images, it may be better to use an approach based on Amazon EMR and s3distcp: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html

Hope this helps,
Constantin