-
Notifications
You must be signed in to change notification settings - Fork 10
Open
Description
Currently the most manual part of the process is migrating files to and from S3. In the docker case, this happens mostly automatically. On AWS, however, it requires several manual steps which become increasingly time consuming with growing file size.
A background process and/or task manager (prefect?) would be ideal to handle these migrations:
- Create two AWS instances ("A" and "B")
- Generate all files on instance A
- Upload all files to S3 from instance A, measuring the upload speeds
- Download all files from S3 to instance B, measuring the download speeds
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels