S3 bucket to bucket transfer

[IgorBalog-Eng]

S3-to-S3 bucket transfer
From ASW documentation:

This pattern describes how to migrate data from an Amazon Simple Storage Service (Amazon S3) bucket in an AWS source account to a destination S3 bucket in another AWS account, either in the same AWS Region or in a different Region.

• source S3 bucket allows AWS Identity and Access Management (IAM) access by using an attached resource policy
• user in the destination account has to assume a role that has PutObject and GetObject permissions for the source bucket
• copy and sync commands to transfer data from the source S3 bucket to the destination S3 bucket
• destination account has ownership of the copied objects
• remove these users when they are no longer needed

Create an IAM user and role in the destination AWS account

• Create and attach an S3 bucket policy for source account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::<destination_account>:role/<RoleName>"},
            "Action": ["s3:ListBucket",
                       "s3:GetObject",
                       "s3:GetObjectTagging",
                       "s3:GetObjectVersion",
                       "s3:GetObjectVersionTagging"
                       ],
            "Resource": [
                "arn:aws:s3:::source-bucket/*",
                "arn:aws:s3:::source-bucket"
            ]
        }
    ]
}

• Create a destination S3 bucket. This operation should be done by source client.

Phase 1:

• Substitute Mongo as file/artifact storage and use S3 (MinIO)
• Use HTTP as DataPlane to transfer files
• On HTTP-Pull, presigned URL will be generated on provider side and artifact transfer will be delegated to S3 bucket service; this will solve situation of file size restriction. Actual transfer will NOT pass through connector and Spring components.
• On consumer side, when artifact is downloaded, connector will upload it to it's S3 bucket storage, to enforce policy enforcement when artifact is consumed.

Phase 2:

• Implement bucket-to-bucket transfer (support S3 as DataPlane)

TODO add part on java SDK implementation (real or pseudo-code)

[marest94]

This will be a long one