Cross Account Support?

[jwdinkel]

Hello,

Wondering if this method supports cross-account synchronization because I don't see a way to specify two profiles in order that the proper rights are setup for the buckets and lambda functions. Is it working this way or can I only sync between two buckets in the same account in the same region?

Thank you

[glez-aws]

Hi,

yes, this should also work cross-account if the role/policy that the Lambda functions are running under have permission to access cross-account buckets and if the bucket policies allow for it.

Cross-region should also work, I just haven’t tested it so far. I hope to get some time allocated for it so I can at least test cross-region and make sure it works well and update the code.

Thanks,
Constantin

[jwdinkel]

Thank you Constantin.

I haven't been able to get back to this perhaps somebody will have time to build out and document an example cross account. I didn't see immediately how I could configure the two accounts and make them work from a security standpoint. I started to document the code, but got no further than this: https://realtimeboard.com/app/board/o9J_kzGtZjg=/

Best Regards,
Jason

[ericfrederich]

I have tested and this code works with cross-account. No modifications needed.

Here is an article which shows how to set up access.
https://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html
The article omits a key part though. It only shows how to give access to the bucket, not objects under the bucket. See my complete example below where I grant access to both arn:aws:s3:::src-bucket-123 and arn:aws:s3:::src-bucket-123/*

We are transforming this code into Terraform instead of fabfile/cloud formation. Once this is done, we will likely create a single role for these lambdas to use instead of separate ones. This will greatly cleanup the policy by only having to give access to a single role, not 4.

In testing this cross-account I ran into some issues that were happening within a thread and weren't propagated. I have submitted a pull request which does the propagation.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Example permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::AAAAAAAAAAAA:role/sync-buckets-state-machine-s-GetBucketLocationRole-SOMEHASH123"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::src-bucket-123",
                "arn:aws:s3:::src-bucket-123/*"
            ]
        },
        {
            "Sid": "Example permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::AAAAAAAAAAAA:role/sync-buckets-state-machine-stack-ListBucketRole-SOMEHASH123"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::src-bucket-123",
                "arn:aws:s3:::src-bucket-123/*"
            ]
        },
        {
            "Sid": "Example permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::AAAAAAAAAAAA:role/sync-buckets-state-machine-stack-CopyKeysRole-SOMEHASH123"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::src-bucket-123",
                "arn:aws:s3:::src-bucket-123/*"
            ]
        },
        {
            "Sid": "Example permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::AAAAAAAAAAAA:role/sync-buckets-state-machine-DeleteOrphanedKeysRole-SOMEHASH123"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::src-bucket-123",
                "arn:aws:s3:::src-bucket-123/*"
            ]
        }
    ]
}

[ericfrederich]

The code, as-is, works with cross-account access if you give the proper access in a bucket policy as outlined in my above comment. If that is good enough, I'd say close this issue.

I could imagine another solution which utilized assumed roles in a different account. I don't think you'd be able to use s3.copy_object there though. I think you'd have to work with two sessions (one default lambda, the other assumed) and read from one and put to the other. Less efficient probably.