OSDOCS-3633 Enable Azure DiskEncryptionSets at install time

Author: bscott-rh

_topic_maps/_topic_map.yml

@@ -182,6 +182,8 @@ Topics:
     File: installing-azure-account
   - Name: Manually creating IAM
     File: manually-creating-iam-azure
+  - Name: Enabling user-managed encryption on Azure
+    File: enabling-user-managed-encryption-azure
   - Name: Installing a cluster quickly on Azure
     File: installing-azure-default
   - Name: Installing a cluster on Azure with customizations

installing/installing_azure/enabling-user-managed-encryption-azure.adoc

@@ -0,0 +1,21 @@
+:_content-type: ASSEMBLY
+[id="enabling-user-managed-encryption-azure"]
+= Enabling user-managed encryption for Azure
+include::_attributes/common-attributes.adoc[]
+:context: enabling-user-managed-encryption-azure
+
+toc::[]
+
+In {product-title} version {product-version}, you can install a cluster with a user-managed encryption key in Azure. To enable this feature, you can prepare an Azure DiskEncryptionSet before installation, modify the `install-config.yaml` file, and then perform post-installation steps.
+
+include::modules/installation-azure-preparing-diskencryptionsets.adoc[leveloffset=+1]
+
+[id="enabling-disk-encryption-sets-azure-next-steps"]
+== Next steps
+
+* Install an {product-title} cluster:
+** xref:../../installing/installing_azure/installing-azure-customizations.adoc#installing-azure-customizations[Install a cluster with customizations on installer-provisioned infrastructure]
+** xref:../../installing/installing_azure/installing-azure-network-customizations.adoc#installing-azure-network-customizations[Install a cluster with network customizations on installer-provisioned infrastructure]
+** xref:../../installing/installing_azure/installing-azure-vnet.adoc#installing-azure-vnet[Install a cluster into an existing VNet on installer-provisioned infrastructure]
+** xref:../../installing/installing_azure/installing-azure-private.adoc#installing-azure-private[Install a private cluster on installer-provisioned infrastructure]
+** xref:../../installing/installing_azure/installing-azure-government-region.adoc#installing-azure-government-region[Install a cluster into an government region on installer-provisioned infrastructure]

installing/installing_azure/installing-azure-customizations.adoc

@@ -18,6 +18,7 @@ parameters in the `install-config.yaml` file before you install the cluster.
 * You xref:../../installing/installing_azure/installing-azure-account.adoc#installing-azure-account[configured an Azure account] to host the cluster and determined the tested and validated region to deploy the cluster to.
 * If you use a firewall, you xref:../../installing/install_config/configuring-firewall.adoc#configuring-firewall[configured it to allow the sites] that your cluster requires access to.
 * If the cloud identity and access management (IAM) APIs are not accessible in your environment, or if you do not want to store an administrator-level credential secret in the `kube-system` namespace, you can xref:../../installing/installing_azure/manually-creating-iam-azure.adoc#manually-creating-iam-azure[manually create and maintain IAM credentials].
+* If you use customer-managed encryption keys, you xref:../../installing/installing_azure/enabling-user-managed-encryption-azure.adoc#enabling-user-managed-encryption-azure[prepared your Azure environment for encryption].
 
 include::modules/cluster-entitlements.adoc[leveloffset=+1]
 
@@ -45,6 +46,8 @@ include::modules/machineset-azure-enabling-accelerated-networking-new-install.ad
 
 include::modules/installation-launching-installer.adoc[leveloffset=+1]
 
+include::modules/installation-azure-finalizing-encryption.adoc[leveloffset=+1]
+
 include::modules/cli-installing-cli.adoc[leveloffset=+1]
 
 include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1]

installing/installing_azure/installing-azure-government-region.adoc

@@ -18,6 +18,7 @@ cluster.
 * You xref:../../installing/installing_azure/installing-azure-account.adoc#installing-azure-account[configured an Azure account] to host the cluster and determined the tested and validated government region to deploy the cluster to.
 * If you use a firewall, you xref:../../installing/install_config/configuring-firewall.adoc#configuring-firewall[configured it to allow the sites] that your cluster requires access to.
 * If the cloud identity and access management (IAM) APIs are not accessible in your environment, or if you do not want to store an administrator-level credential secret in the `kube-system` namespace, you can xref:../../installing/installing_azure/manually-creating-iam-azure.adoc#manually-creating-iam-azure[manually create and maintain IAM credentials].
+* If you use customer-managed encryption keys, you xref:../../installing/installing_azure/enabling-user-managed-encryption-azure.adoc#enabling-user-managed-encryption-azure[prepared your Azure environment for encryption].
 
 include::modules/installation-azure-about-government-region.adoc[leveloffset=+1]
 
@@ -55,6 +56,8 @@ include::modules/machineset-azure-enabling-accelerated-networking-new-install.ad
 
 include::modules/installation-launching-installer.adoc[leveloffset=+1]
 
+include::modules/installation-azure-finalizing-encryption.adoc[leveloffset=+1]
+
 include::modules/cli-installing-cli.adoc[leveloffset=+1]
 
 include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1]

installing/installing_azure/installing-azure-network-customizations.adoc

@@ -23,6 +23,7 @@ cluster.
 * You xref:../../installing/installing_azure/installing-azure-account.adoc#installing-azure-account[configured an Azure account] to host the cluster and determined the tested and validated region to deploy the cluster to.
 * If you use a firewall, you xref:../../installing/install_config/configuring-firewall.adoc#configuring-firewall[configured it to allow the sites] that your cluster requires access to.
 * If the cloud identity and access management (IAM) APIs are not accessible in your environment, or if you do not want to store an administrator-level credential secret in the `kube-system` namespace, you can xref:../../installing/installing_azure/manually-creating-iam-azure.adoc#manually-creating-iam-azure[manually create and maintain IAM credentials]. Manual mode can also be used in environments where the cloud IAM APIs are not reachable.
+* If you use customer-managed encryption keys, you xref:../../installing/installing_azure/enabling-user-managed-encryption-azure.adoc#enabling-user-managed-encryption-azure[prepared your Azure environment for encryption].
 
 include::modules/cluster-entitlements.adoc[leveloffset=+1]
 
@@ -63,6 +64,8 @@ include::modules/machineset-azure-enabling-accelerated-networking-new-install.ad
 
 include::modules/installation-launching-installer.adoc[leveloffset=+1]
 
+include::modules/installation-azure-finalizing-encryption.adoc[leveloffset=+1]
+
 include::modules/cli-installing-cli.adoc[leveloffset=+1]
 
 include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1]

installing/installing_azure/installing-azure-private.adoc

@@ -15,6 +15,7 @@ In {product-title} version {product-version}, you can install a private cluster
 * You xref:../../installing/installing_azure/installing-azure-account.adoc#installing-azure-account[configured an Azure account] to host the cluster and determined the tested and validated region to deploy the cluster to.
 * If you use a firewall, you xref:../../installing/install_config/configuring-firewall.adoc#configuring-firewall[configured it to allow the sites] that your cluster requires access to.
 * If the cloud identity and access management (IAM) APIs are not accessible in your environment, or if you do not want to store an administrator-level credential secret in the `kube-system` namespace, you can xref:../../installing/installing_azure/manually-creating-iam-azure.adoc#manually-creating-iam-azure[manually create and maintain IAM credentials].
+* If you use customer-managed encryption keys, you xref:../../installing/installing_azure/enabling-user-managed-encryption-azure.adoc#enabling-user-managed-encryption-azure[prepared your Azure environment for encryption].
 
 include::modules/private-clusters-default.adoc[leveloffset=+1]
 
@@ -50,6 +51,8 @@ include::modules/machineset-azure-enabling-accelerated-networking-new-install.ad
 
 include::modules/installation-launching-installer.adoc[leveloffset=+1]
 
+include::modules/installation-azure-finalizing-encryption.adoc[leveloffset=+1]
+
 include::modules/cli-installing-cli.adoc[leveloffset=+1]
 
 include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1]

installing/installing_azure/installing-azure-vnet.adoc

@@ -15,6 +15,7 @@ In {product-title} version {product-version}, you can install a cluster into an
 * You xref:../../installing/installing_azure/installing-azure-account.adoc#installing-azure-account[configured an Azure account] to host the cluster and determined the tested and validated region to deploy the cluster to.
 * If you use a firewall, you xref:../../installing/install_config/configuring-firewall.adoc#configuring-firewall[configured it to allow the sites] that your cluster requires access to.
 * If the cloud identity and access management (IAM) APIs are not accessible in your environment, or if you do not want to store an administrator-level credential secret in the `kube-system` namespace, you can xref:../../installing/installing_azure/manually-creating-iam-azure.adoc#manually-creating-iam-azure[manually create and maintain IAM credentials].
+* If you use customer-managed encryption keys, you xref:../../installing/installing_azure/enabling-user-managed-encryption-azure.adoc#enabling-user-managed-encryption-azure[prepared your Azure environment for encryption].
 
 include::modules/installation-about-custom-azure-vnet.adoc[leveloffset=+1]
 
@@ -44,6 +45,8 @@ include::modules/machineset-azure-enabling-accelerated-networking-new-install.ad
 
 include::modules/installation-launching-installer.adoc[leveloffset=+1]
 
+include::modules/installation-azure-finalizing-encryption.adoc[leveloffset=+1]
+
 include::modules/cli-installing-cli.adoc[leveloffset=+1]
 
 include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1]

modules/installation-azure-config-yaml.adoc

@@ -41,10 +41,15 @@ controlPlane: <2>
   name: master
   platform:
     azure:
+      encryptionAtHost: true
       ultraSSDCapability: Enabled
       osDisk:
         diskSizeGB: 1024 <5>
         diskType: Premium_LRS
+        diskEncryptionSet:
+          resourceGroup: disk_encryption_set_resource_group
+          name: disk_encryption_set_name
+          subscriptionId: secondary_subscription_id
       type: Standard_D8s_v3
   replicas: 3
 compute: <2>
@@ -54,9 +59,14 @@ compute: <2>
     azure:
       ultraSSDCapability: Enabled
       type: Standard_D2s_v3
+      encryptionAtHost: true
       osDisk:
         diskSizeGB: 512 <5>
         diskType: Standard_LRS
+        diskEncryptionSet:
+          resourceGroup: disk_encryption_set_resource_group
+          name: disk_encryption_set_name
+          subscriptionId: secondary_subscription_id
       zones: <6>
       - "1"
       - "2"

modules/installation-azure-finalizing-encryption.adoc

@@ -0,0 +1,71 @@
+//Module included in the following assemblies:
+//
+// * installing/installing_azure/installing-azure-customizations.adoc
+// * installing/installing_azure/installing-azure-government-region.adoc
+// * installing/installing_azure/installing-azure-network-customizations.adoc
+// * installing/installing_azure/installing-azure-private.adoc
+// * installing/installing_azure/installing-azure-vnet.adoc
+
+:_content-type: PROCEDURE
+[id="finalizing-encryption_{context}"]
+= Finalizing user-managed encryption after installation
+If you installed {product-title} using a user-managed encryption key, you can complete the installation by creating a new storage class and granting write permissions to the Azure cluster resource group.
+
+.Procedure
+. Obtain the identity of the cluster resource group used by the installer:
+.. If you specified an existing resource group in `install-config.yaml`, obtain its Azure identity by running the following command:
++
+[source,terminal]
+----
+$ az identity list --resource-group "<existing_resource_group>"
+----
+.. If you did not specify a existing resource group in `install-config.yaml`, locate the resource group that the installer created, and then obtain its Azure identity by running the following commands:
++
+[source,terminal]
+----
+$ az group list
+----
++
+[source,terminal]
+----
+$ az identity list --resource-group "<installer_created_resource_group>"
+----
++
+. Grant a role assignment to the cluster resource group so that it can write to the Disk Encryption Set by running the following command:
++
+[source,terminal]
+----
+$ az role assignment create --role "<privileged_role>" \// <1>
+    --assignee "<resource_group_identity>" <2>
+----
+<1> Specifies an Azure role that has read/write permissions to the disk encryption set. You can use the `Owner` role or a custom role with the necessary permissions.
+<2> Specifies the identity of the cluster resource group.
++
+. Create a storage class that uses the user-managed disk encryption set:
+.. Save the following storage class definition to a file, for example `storage-class-definition.yaml`:
++
+[source,yaml]
+----
+kind: StorageClass
+apiVersion: storage.k8s.io/v1
+metadata:
+  name: managed-premium
+provisioner: kubernetes.io/azure-disk
+parameters:
+  skuname: Premium_LRS
+  kind: Managed
+  diskEncryptionSetID: "<disk_encryption_set_ID>" <1>
+  resourceGroup: <resource_group_name> <2>
+reclaimPolicy: Delete
+allowVolumeExpansion: true
+volumeBindingMode: WaitForFirstConsumer
+----
+<1> Specifies the ID of the disk encryption set that you created in the prerequisite steps, for example `"/subscriptions/xxxxxx-xxxxx-xxxxx/resourceGroups/test-encryption/providers/Microsoft.Compute/diskEncryptionSets/disk-encryption-set-xxxxxx"`.
+<2> Specifies the name of the resource group used by the installer. This is the same resource group from the first step.
+.. Create the storage class `managed-premium` from the file you created by running the following command:
++
+[source,terminal]
+----
+$ oc create -f storage-class-definition.yaml
+----
+. Select the `managed-premium` storage class when you create persistent volumes to use encrypted storage.

modules/installation-azure-preparing-diskencryptionsets.adoc

@@ -0,0 +1,132 @@
+//Module included in the following assemblies:
+//
+// * installing/installing_azure/enabling-disk-encryption-sets-azure.adoc
+
+:_content-type: PROCEDURE
+[id="preparing-disk-encryption-sets"]
+= Preparing an Azure Disk Encryption Set
+The {product-title} installer can use an existing Disk Encryption Set with a user-managed key. To enable this feature, you can create a Disk Encryption Set in Azure and provide the key to the installer.
+
+.Procedure
+
+. Set the following environment variables for the Azure resource group by running the following command:
++
+[source,terminal]
+----
+$ export RESOURCEGROUP="<resource_group>" \// <1>
+    LOCATION="<location>" <2>
+----
+<1> Specifies the name of the Azure resource group where you will create the Disk Encryption Set and encryption key. To avoid losing access to your keys after destroying the cluster, you should create the Disk Encryption Set in a different resource group than the resource group where you install the cluster.
+<2> Specifies the Azure location where you will create the resource group.
++
+. Set the following environment variables for the Azure Key Vault and Disk Encryption Set by running the following command:
++
+[source,terminal]
+----
+$ export KEYVAULT_NAME="<keyvault_name>" \// <1>
+    KEYVAULT_KEY_NAME="<keyvault_key_name>" \// <2>
+    DISK_ENCRYPTION_SET_NAME="<disk_encryption_set_name>" <3>
+----
+<1> Specifies the name of the Azure Key Vault you will create.
+<2> Specifies the name of the encryption key you will create.
+<3> Specifies the name of the disk encryption set you will create.
++
+. Set the environment variable for the ID of your Azure Service Principal by running the following command:
++
+[source,terminal]
+----
+$ export CLUSTER_SP_ID="<service_principal_id>" <1>
+----
+<1> Specifies the ID of the service principal you will use for this installation.
++
+. Enable host-level encryption in Azure by running the following commands:
++
+[source,terminal]
+----
+$ az feature register --namespace "Microsoft.Compute" --name "EncryptionAtHost"
+----
++
+[source,terminal]
+----
+$ az feature show --namespace Microsoft.Compute --name EncryptionAtHost
+----
++
+[source,terminal]
+----
+$ az provider register -n Microsoft.Compute
+----
++
+. Create an Azure Resource Group to hold the disk encryption set and associated resources by running the following command:
++
+[source,terminal]
+----
+$ az group create --name $RESOURCEGROUP --location $LOCATION
+----
++
+. Create an Azure key vault by running the following command:
++
+[source,terminal]
+----
+$ az keyvault create -n $KEYVAULT_NAME -g $RESOURCEGROUP -l $LOCATION \
+    --enable-purge-protection true --enable-soft-delete true
+----
++
+. Create an encryption key in the key vault by running the following command:
++
+[source,terminal]
+----
+$ az keyvault key create --vault-name $KEYVAULT_NAME -n $KEYVAULT_KEY_NAME \
+    --protection software
+----
++
+. Capture the ID of the key vault by running the following command:
++
+[source,terminal]
+----
+$ KEYVAULT_ID=$(az keyvault show --name $KEYVAULT_NAME --query "[id]" -o tsv)
+----
++
+. Capture the key URL in the key vault by running the following command:
++
+[source,terminal]
+----
+$ KEYVAULT_KEY_URL=$(az keyvault key show --vault-name $KEYVAULT_NAME --name \
+    $KEYVAULT_KEY_NAME --query "[key.kid]" -o tsv)
+----
++
+. Create a disk encryption set by running the following command:
++
+[source,terminal]
+----
+$ az disk-encryption-set create -n $DISK_ENCRYPTION_SET_NAME -l $LOCATION -g \
+    $RESOURCEGROUP --source-vault $KEYVAULT_ID --key-url $KEYVAULT_KEY_URL
+----
++
+. Grant the DiskEncryptionSet resource access to the key vault by running the following commands:
++
+[source,terminal]
+----
+$ DES_IDENTITY=$(az disk-encryption-set show -n $DISK_ENCRYPTION_SET_NAME -g \
+    $RESOURCEGROUP --query "[identity.principalId]" -o tsv)
+----
++
+[source,terminal]
+----
+$ az keyvault set-policy -n $KEYVAULT_NAME -g $RESOURCEGROUP --object-id \
+    $DES_IDENTITY --key-permissions wrapkey unwrapkey get
+----
++
+. Grant the Azure Service Principal permission to read the DiskEncryptionSet by running the following commands:
++
+[source,terminal]
+----
+$ DES_RESOURCE_ID=$(az disk-encryption-set show -n $DISK_ENCRYPTION_SET_NAME -g \
+    $RESOURCEGROUP --query "[id]" -o tsv)
+----
++
+[source,terminal]
+----
+$ az role assignment create --assignee $CLUSTER_SP_ID --role "<reader_role>" \// <1>
+    --scope $DES_RESOURCE_ID -o jsonc
+----
+<1> Specifies an Azure role with read permissions to the disk encryption set. You can use the `Owner` role or a custom role with the necessary permissions.