DC1: Encrypt storage gateway EC2 AMI root(non-EBS) /dev/xvda volume to adhere AWS SEA deny rule on encryption: false

[obriensystems]

This is one of the encryption flags - the other root (non-EBS) volume is hidden because it comes with the unencrypted snapshot backing the AMI
see
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html

https://github.com/aws-quickstart/quickstart-uipath-orchestrator/blob/main/templates/storage.template.yaml#L296

API: ec2:RunInstances You are not authorized to perform this operation.
"DecodedMessage": "{"allowed":false,"explicitDeny":true,"matchedStatements":
{"items":[{"statementId":
"PreventEc2MountUnencryptedVolume"
,"effect":"DENY","principals":
{"items":[{"value":"ARO....HWH"}]},"principalGroups":{"items":[]},"actions":
{"items":[{"value":"ec2:RunInstances"}]},"resources":{"items"
:[{"value":"arn:aws:ec2:::volume/*"}]},"conditions":{"items":
[{"key":"ec2:Encrypted","values":{"items":
[{"value":"false"
}]}}]}}]},

other fixes - ha template
add to ha template
HAMaster:
Type: 'AWS::EC2::Instance'
Properties:

  BlockDeviceMappings:
    - DeviceName: /dev/sda1
      Ebs:
        DeleteOnTermination: true
        Encrypted: true
        VolumeSize: 100
        VolumeType: gp2

[obriensystems]

the unencrypted snapshot backing the AMI see (my personal uipath deployment has the ec2 volume encryption DENY rule set to false - to bring up the system in a less secure environment)
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html

"In addition, when you launch an instance from an AMI backed by unencrypted EBS snapshots, you can encrypt some or all of the volumes during launch."

fix to check
Turn on at account level
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html#encryption-by-default

EC2
Settings - EBS encryption
Modify EBS encryption
EBS encryption Info
Set the default encryption status of all new EBS volumes and copies of snapshots created in your account.
Always encrypt new EBS volumes
Enables encryption by default for newly created EBS volumes and snapshots.
Enable (default false)

never mind - the account in question with the DENY rule on the unencrypted EBS volume already has it set to true - so the issue is flipping an AMI volume to encrypted

fix 2 to check
Instance with both Encrypted and KmsKeyId set
An unencrypted snapshot is restored to an EBS volume encrypted by the specified KMS key.

ami-081419a3f54890191

Volume ID	Device name	Volume size (GiB)	Attachment status	Attachment time	Encrypted	KMS key ID	Delete on termination
vol-03a8688d236eed4fb	/dev/xvda	80	Attached	Tue Oct 26 2021 11:11:21 GMT-0400 (Eastern Daylight Time)	No	–	Yes
vol-05ac349c22fda6756	/dev/sdb	150	Attached	Tue Oct 26 2021 11:11:21 GMT-0400 (Eastern Daylight Time)	No	–

[obriensystems]

experiment: ami existing ec2 - turn on encryption by default - restore ami to new vm (although this should have worked during the cf deployment on the DENY account)
Noticed that the offending drive is greyed on encrypting via AMI creation

launching of ami - allows for encryption (although these should be auto) with the enable global flag on

default encryption does not kick in - but the selection did

I will re- ami this one and try to use in the cloudformation ami search override

[obriensystems]

Weird, they are already encrypted - in the 60 day old version before the deny rule - looks like the global flag was flipped in the middle

also the root drive is now ebs

testing by raising the original ami to check encryption status
worst case we save the existing encrypted ami for later use in the cf script

tried ami
does not work

2021-11-07 00:56:21 UTC-0400	uipath306-michael-OrchestratorStack-18PNLJIMURS7R-StorageStack-UGUCW08YRU9M	CREATE_FAILED	The following resource(s) failed to create: [ActivationKey].
2021-11-07 00:56:20 UTC-0400	ActivationKey	CREATE_FAILED	Received response status [FAILED] from custom resource. Message returned: See the details in CloudWatch Log Stream: 2021/11/07/[$LATEST]9f726642b7674601bd2e0cee503bcebe (RequestId: ff40fe28-c852-4218-9889-5d4727d88415)

	2021-11-07T00:56:18.532-04:00	Described instances: [{'AvailabilityZone': 'ca-central-1a', 'InstanceId': 'i-0aaf7a32cef8316da', 'InstanceState': {'Code': 16, 'Name': 'running'}, 'InstanceStatus': {'Details': [{'Name': 'reachability', 'Status': 'passed'}], 'Status': 'ok'}, 'SystemStatus': {'Details': [{'Name': 'reachability', 'Status': 'passed'}], 'Status': 'ok'}}]
	2021-11-07T00:56:18.532-04:00	Retrieving activation key ...
	2021-11-07T00:56:18.743-04:00	curl: (7) Failed to connect to 10.200.138.150 port 80: Connection refused
	2021-11-07T00:56:18.764-04:00	Exception: No redirect url returned for ip: 10.200.138.150

[obriensystems]

Anyway both drives are EBS and encrypted (as per the ami).
It looks like the account DENY rule only blocks cf ec2 create - but if we allowed it through - then the account level encryption=true override would encrypt all volumes anyway.

will reuse the previous activation key from 145

ActivationKey	J635P-LII2P-FHF20-32FBR-DCTUV	Custom::ActivationKey	CREATE_COMPLETE

309

Exception: An error occurred (InvalidGatewayRequestException) when calling the ActivateGateway operation: The specified activation key was not found.

rerun with normal ami search shows

ami-04109bbae95017363

which is different than uipath145's ami
ami-05b96677f6afd44be

312 use ami-05 hardcoded
and turn off default encryption

however we are failing later on activation key

313 - theory - try encryption on the yaml - not the default ec2 setting
# uipath313 try with default ami - global encryption off
ImageId: !Ref StorageGatewayAMI
# copy of uipath145 double encryped existing ami copy
#ImageId: ami-05b96677f6afd44be
# uipath145 original - passed in the past
# used in 312 with default encyrption off - ec2 came in double encrypted but activation failed later
#ImageId: ami-05b96677f6afd44be
# uipath310 this dev original - failing
# ami-04109bbae95017363
InstanceType: m4.xlarge

encoded ec2 again using
ami-04109bbae95017363

1604 noticed account encryption back on - turned off, reran with all defaults
315 verified, encoded error message

316 turn back on default account encryption - run defaults
result - encryption error

317: encryption global on and add below
Gateway:
Type: 'AWS::EC2::Instance'
Properties:
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
DeleteOnTermination: true
Encrypted: true
VolumeSize: 80
VolumeType: gp3

2021-11-07 17:31:52 UTC-0500	FileShare	CREATE_FAILED	Received response status [FAILED] from custom resource. Message returned: See the details in CloudWatch Log Stream: 2021/11/07/[$LATEST]e32e8ed09d464796ae9ab130b6dc0326 (RequestId: ffab79a8-932d-4cf7-ac48-bc54b8543bf7)

2021-11-07T17:31:51.505-05:00	Creating NFS File Share for Gateway arn:aws:storagegateway:ca-central-1:045590688108:gateway/sgw-F792729E ...
	2021-11-07T17:31:51.505-05:00

good partial news - the drives are now encrypted but we get the above fileshare error

activation key good

ActivationKey	GG1LF-C989U-DRJUI-M3MQH-495LR	Custom::ActivationKey	CREATE_COMPLETE

using ami

StorageGatewayAMI	ami-04109bbae95017363	Custom::StorageGatewayAMI	CREATE_COMPLETE

3xxx keep default encryption off - re ami 04 with ebs encryption on - hardcode ami to copy

318: pending

[obriensystems]

Summary:
When the account is set for EC2 | Dashboard | Encryption | encryption = True (for new volumes during runinstance ops) - all EBS volumes are encrypted - even those that came in through any AMI that had un-encrypted volumes
The IAM DENY rule on EC2:createinstance where the encryption flag is set to false - has no effect when the global encrypt flag is on - all volumes are encrypted anyway.

I see however there would be case where the regional encryption global flag is turned off and and EC2 was created (indirectly via ec2 api via CloudFormation) - that would now come in un-encrypted.

For the purposes of the dev account as long as the regional ec2 encryption flag is always false - we can relax the rule on the EC2 deny during development of the storage gateway AMI retrofit.

The remaining issue is that the root drive on the AMI used for the SG is unencrypted - however the latest run of 317 flipped the encryption to true as soon as the AMI was used via the EC2 global flag. If you check all the volumes for the entire account you will see that except for 2 test instances from another user last quarter are encrypted - specifically the pre-prod 145 from 6 weeks ago actually already had encrypted volumes across all the EC2 VMs

reference

[ubikusss]

Hey, thank you very much for reporting this.
If I understand correctly you need a feature where the storage gateway will have encryption enabled for the root volume. Is this correct?