Skip to content

Automate upload to / download from S3 #28

@joshmoore

Description

@joshmoore

Currently the most manual part of the process is migrating files to and from S3. In the docker case, this happens mostly automatically. On AWS, however, it requires several manual steps which become increasingly time consuming with growing file size.

A background process and/or task manager (prefect?) would be ideal to handle these migrations:

  • Create two AWS instances ("A" and "B")
  • Generate all files on instance A
  • Upload all files to S3 from instance A, measuring the upload speeds
  • Download all files from S3 to instance B, measuring the download speeds

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions