Share custom build to other aws account got error: Snapshots encrypted with the AWS Managed CMK can't be shared

[NaomiLYJ]

I follow the BUILDING.md and successfully build my own bottlerocket ami. But I'm not able to share it with another account because of the error:
An error occurred (InvalidParameter) when calling the ModifyImageAttribute operation: Snapshots encrypted with the AWS Managed CMK can't be shared. Specify another snapshot

I'm not familiar with the snapshot encryption part, is it related to the settings "signing_keys = { kms = { key_id = "abc-def-123" } }" in Infra.toml?

I'm sharing with python sdk boto3:

boto3.client('ec2').modify_image_attribute(
        ImageId=ami_id,
        LaunchPermission={
            'Add': user_ids
        }
    )

And this is similar to the rust implementation of grant ami in PUBLISHING_AWS.md

I would be grateful if someone could help here!

Update:
I've also tried the grant_ami, but also got error:

[INFO] Waiting for AMI to be available before changing its permissions
[INFO] Found ami-xxxxxxx available in eu-central-1
[INFO] Updating all snapshot permissions before changing any AMI permissions - granting access
[ERROR] Failed to modify permissions in eu-central-1 for snapshots [snap-xxxx, snap-xxxx]: "OperationNotPermitted"
Error during publish-ami command: No AMI permissions were updated: Failed to modify permissions of 1 of 1 snapshots
Error while executing command, exit code: 1
Error: Command was unsuccessful, exit code 105
Error while executing command, exit code: 1

IAM permission ec2:ModifySnapshotAttribute is attached, so should not be IAM issue

[NaomiLYJ]

I checked that the snapshots (os and data) are encrypted with the default kms key aws/ebs

I went through the source code and it's using coldsnap package to upload the image to snapshot. From the code, the kms key for encryption is not configured for the ebs_client.start_snapshot, so the default key is used

    /// Start a new snapshot and return the ID and block size for subsequent puts.
    async fn start_snapshot(&self, volume_size: i64, description: String) -> Result<(String, i32)> {
        let start_response = self
            .ebs_client
            .start_snapshot()
            .volume_size(volume_size)
            .set_description(Some(description))
            .set_timeout(Some(SNAPSHOT_TIMEOUT_MINUTES))
            .send()
            .await
            .context(error::StartSnapshotSnafu)?;

        let snapshot_id = start_response
            .snapshot_id
            .context(error::FindSnapshotIdSnafu)?;
        let block_size = start_response
            .block_size
            .context(error::FindSnapshotBlockSizeSnafu)?;

        Ok((snapshot_id, block_size))
    }

How could we encrypt with custom kms key? Or how could we share it with other accounts? Thank you very much!

[bcressey]

coldsnap would first need to support a custom KMS key ARN, and then pubsys (part of twoliter) would need to allow that to be specified in Infra.toml, and use it if present when registering snapshots.

The code changes involved should be relatively small if you want to contribute the functionality. Otherwise, you can open an issue and I can try to find a few cycles.

[NaomiLYJ]

Hi @bcressey , thanks a lot for your reply! I actually do not know Rust, but I'll see if I could contribute, otherwise I'll create an issue.

[larvacea]

In the past, I have been able to share one of my private AMIs with another AWS account using the EC2 web console. On EC2/AMIs, after selecting an individual AMI, the Actions pulldown has an Edit AMI Permissions selection. On the edit-permissions page, you can add either an account ID to share with a particular account, or an organization/OU by ARN. Will this work for you?

[yeazelm]

Can you confirm if you have default EBS encryption enabled? There might be a setting at the account level that is encrypting these volumes. By default, the grant ami calls you referred to should work, but depending on this account setting, that could be the problem.

[NaomiLYJ]

Hi @yeazelm, thank you for your reply! I could confirm that default EBS encryption is enabled for every created volumes out of security. Does it mean that we are not able to share bottlerocket ami to other accounts when default encryption is enabled. Would you support snapshot encryption with custom kms key? Thank you very much!

[yeazelm]

If you are using encrypted volumes, you probably need to follow a guide similar to this blog post or look at the AWS docs on using encrypted EBS volumes for AMIs. I do believe you can solve this with a customer owned KMS key but would need to share that key so that the volume could be decrypted.

[NaomiLYJ]

Thanks again for your @yeazelm reply! I think there is misunderstanding. The problem is that there isn’t a way to configure custom kms key when registering bottlerocket ami. Registering process would create snapshot in the account. Since my account has default encryption enabled, without providing a custom kms key, the default kms key is used for encryption when creating snapshot. Therefore the ami could not be shared.

So to be able to share ami to other accounts, either we disable the default ebs encryption, or we are able to configure the custom kms key for the registering. Would the configuration of custom kms key be supported when registering ami?

[bcressey]

Creating a snapshot with a custom key would be a useful feature, it's just not implemented yet. I don't recall if it was supported back when I originally wrote coldsnap, but it looks like it should work now barring any surprises.

I've hit this snag before when testing encrypted snapshots (by turning on the account level setting) and then forgetting it was on and trying to share AMIs with another testing account. It's annoying for sure.