feat: use block storage for crunchy and rabbit in Azure (PSKD-1025) (#601)

Signed-off-by: John Boone <[email protected]>

Author: joboon

docs/CONFIG-VARS.md

@@ -74,6 +74,7 @@ Supported configuration variables are listed in the table below.  All variables
 | V4_CFG_CLOUD_SERVICE_ACCOUNT_AUTH | Full path to service account credentials file | string | | false | See [Ansible Cloud Authentication](user/AnsibleCloudAuthentication.md) for more information. | viya |
 
 ## Jump Server
+
 Viya4-deployment uses the jump server to interact with the RWX filestore, which must be pre-mounted to `JUMP_SVR_RWX_FILESTORE_PATH` when `V4_CFG_MANAGE_STORAGE` is set to `true`.
 
 | Name | Description | Type | Default | Required | Notes | Tasks |
@@ -83,13 +84,27 @@ Viya4-deployment uses the jump server to interact with the RWX filestore, which
 | JUMP_SVR_PRIVATE_KEY | Path to the SSH user's private key to access the jump server host | string |  | true | | baseline, viya |
 | JUMP_SVR_RWX_FILESTORE_PATH | Path on the jump server to the NFS mount | string | /viya-share | false | | viya |
 
-## Storage for AWS
+## Storage
+
+### Storage for AWS
+
 When `V4_CFG_MANAGE_STORAGE` is set to `true`, viya4-deployment uses the [EBS CSI driver](#ebs-csi-driver) to create two elastic block storage based storage classes with the default names of `io2-vol-mq` and `io2-vol-pg`. The volume type for both storage classes defaults to `io2`. For EKS clusters, RabbitMQ makes PVC requests to create block storage persistent volumes using the `io2-vol-mq` storage class while Crunchy Postgres makes PVC requests to create block storage persistent volumes using the `io2-vol-pg` storage class. Viya4-deployment also creates the `sas` storage class using the nfs-subdir-external-provisioner Helm chart. If a jump server is used, viya4-deployment uses that server to create the folders for the `astores`, `bin`, `data` and `homes` RWX Filestore NFS paths that are outlined below in the [RWX Filestore](#rwx-filestore) section.
 
+### Storage for Azure
+
+By default, viya4-deployment uses the [Azure managed disks CSI driver](#azure-managed-disk-csi-driver) to create two elastic block storage based storage classes with the default names of `managed-csi-premium-v2-mq` and `managed-csi-premium-v2-pg`. The disk SKU for both storage classes defaults to `PremiumV2_LRS`. For AKS clusters, RabbitMQ makes PVC requests to create block storage persistent volumes using the `managed-csi-premium-v2-mq` storage class while Crunchy Postgres makes PVC requests to create block storage persistent volumes using the `managed-csi-premium-v2-pg` storage class. To use a different StorageClass for RabbitMQ, set the `V4_CFG_RABBITMQ_STORAGECLASS` property to the name of the StorageClass to use. To use a different StorageClass for Crunchy Postgres, set the `V4_CFG_CRUNCHY_STORAGECLASS` property to the name of the StorageClass to use.
 
-## Storage for Azure and Google Cloud
+**NOTE**: The Azure managed disk CSI Driver can only be included at AKS cluster creation time. It is included in all AKS clusters by default, and any AKS clusters created with viya4-iac-azure will have the driver installed. If you did not use the viya4-iac-azure project to create your AKS cluster, ensure that you have enabled the Azure disk CSI driver prior to using this project or disable the creation of the StorageClasses.
+
+viya4-deployment also creates the `sas` storage class using the nfs-subdir-external-provisioner Helm chart. If a jump server is used, viya4-deployment uses that server to create the folders for the `astores`, `bin`, `data` and `homes` RWX Filestore NFS paths that are outlined below in the [RWX Filestore](#rwx-filestore) section.
+
+### Storage for Google Cloud
 When `V4_CFG_MANAGE_STORAGE` is set to `true`, viya4-deployment creates the `sas` and `pg-storage` storage classes using the nfs-subdir-external-provisioner Helm chart. If a jump server is used, viya4-deployment uses that server to create the folders for the `astores`, `bin`, `data` and `homes` RWX Filestore NFS paths that are outlined below in the [RWX Filestore](#rwx-filestore) section.
 
+### NFS Storage
+
+When `V4_CFG_MANAGE_STORAGE` is set to `true`, viya4-deployment creates NFS-based storage classes using the nfs-subdir-external-provisioner Helm chart.
+
 When `V4_CFG_MANAGE_STORAGE` is set to `false`, viya4-deployment does not create the `sas` or `pg-storage` storage classes for you. In addition, viya4-deployment does not create or manage the RWX Filestore NFS paths. Before you run the SAS Viya deployment, you must set the values for `V4_CFG_RWX_FILESTORE_DATA_PATH` and `V4_CFG_RWX_FILESTORE_HOMES_PATH` to specify existing NFS folder locations. The viya4-deployment user can create the required NFS folders from the jump server before starting the deployment. Recommended attribute settings for each folder are as follows:
 - **filemode**: `0777`
 - **group**: the equivalent of `nogroup` for your operating system
@@ -100,7 +115,7 @@ When `V4_CFG_MANAGE_STORAGE` is set to `false`, viya4-deployment does not create
 | V4_CFG_MANAGE_STORAGE | Whether viya4-deployment should manage the StorageClass | bool | true | false | Set to false if you want to manage the StorageClass yourself. | all |
 | V4_CFG_STORAGECLASS | StorageClass name | string | "sas" | false | When V4_CFG_MANAGE_STORAGE is false, set to the name of your preexisting StorageClass that supports ReadWriteMany. | baseline, viya |
 
-### RWX Filestore
+#### RWX Filestore
 
 | Name | Description | Type | Default | Required | Notes | Tasks |
 | :--- | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -109,15 +124,15 @@ When `V4_CFG_MANAGE_STORAGE` is set to `false`, viya4-deployment does not create
 | V4_CFG_RWX_FILESTORE_DATA_PATH | NFS path to data directory | string | <V4_CFG_RWX_FILESTORE_PATH>/\<NAMESPACE>/data | false | | viya |
 | V4_CFG_RWX_FILESTORE_HOMES_PATH | NFS path to homes directory | string | <V4_CFG_RWX_FILESTORE_PATH>/\<NAMESPACE>/homes | false | | viya |
 
-### Azure
+#### Azure
 
-When V4_CFG_MANAGE_STORAGE is set to `true`, the `sas` and `pg-storage` storage classes are created (Azure NetApp or NFS).
+When V4_CFG_MANAGE_STORAGE is set to `true`, the `sas` storage class is created (Azure NetApp or NFS).
 
-### AWS
+#### AWS
 
-When V4_CFG_MANAGE_STORAGE is set to `true`, the efs-provisioner is deployed, the `sas` and `pg-storage` storage classes are created (EFS or NFS).
+When V4_CFG_MANAGE_STORAGE is set to `true`, the efs-provisioner is deployed, and the `sas` storage class is created (EFS or NFS).
 
-### Google Cloud
+#### Google Cloud
 
 When V4_CFG_MANAGE_STORAGE is set to `true`, the `sas` and `pg-storage` storage classes are created (Google Filestore or NFS).
 
@@ -161,7 +176,6 @@ When V4_CFG_MANAGE_STORAGE is set to `true`, the `sas` and `pg-storage` storage
 | :--- | ---: | ---: | ---: | ---: | ---: | ---: |
 | V4_CFG_AWS_LB_SUBNETS | The AWS subnets and by association the AWS availability zones to deploy the load balancing service to. This variable sets an ingress-nginx annotation which interacts with the [Cloud Controller Manager](https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/) to set the subnets used by the AWS load balancer. Specifying a subnet value or values for this variable takes precedence over the Subnet Discovery method described in [AWS docs](https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html) that relies on the tags applied to AWS subnets documented in scenario 2 of this [table.](https://github.com/sassoftware/viya4-iac-aws/blob/main/docs/user/BYOnetwork.md#supported-scenarios-and-requirements-for-using-existing-network-resources) This variable can be set with [BYO network scenarios 0-3](https://github.com/sassoftware/viya4-iac-aws/blob/main/docs/user/BYOnetwork.md#supported-scenarios-and-requirements-for-using-existing-network-resources). | string | | false | The value is either a comma separated list of subnet IDs, or a comma separated list of subnet names. Does not affect the subnets used for load balancers enabled with  `V4_CFG_CAS_ENABLE_LOADBALANCER`, `V4_CFG_CONNECT_ENABLE_LOADBALANCER`, or `V4_CFG_CONSUL_ENABLE_LOADBALANCER`. | baseline |
 
-
 ## TLS
 
 The SAS Viya platform supports two certificate generators: cert-manager and openssl.
@@ -347,7 +361,6 @@ Cluster-autoscaler is currently only used for AWS EKS clusters. Google GKE and A
 
 If you used [viya4-iac-aws:5.6.0](https://github.com/sassoftware/viya4-iac-aws/releases) or newer to create your infrastructure, a cluster autoscaler account should have been created for you with a policy that is compatible with both our default versions for the `CLUSTER_AUTOSCALER_CHART_VERSION` variable. If you choose an alternative version ensure that your autoscaler account has a policy that matches the recommendation from the [kubernetes/autoscaler documentation](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#iam-policy). This note is only applicable for EKS clusters.
 
-
 ### EBS CSI Driver
 
 The EBS CSI driver is only used for kubernetes v1.23 or later AWS EKS clusters.
@@ -370,6 +383,27 @@ The EBS CSI driver is only used for kubernetes v1.23 or later AWS EKS clusters.
 |EBS_CSI_CRUNCHY_STORAGE_CLASS_THROUGHPUT | Maximum volume throughput in MiB/s for the `EBS_CSI_CRUNCHY_STORAGE_CLASS_NAME` storage class | string| 400 | false | The maximum value for io2, io1 and gp3 volume types is 1000.| baseline |
 |EBS_CSI_CRUNCHY_STORAGE_CLASS_RECLAIM_POLICY | The ReclaimPolicy for the `EBS_CSI_CRUNCHY_STORAGE_CLASS_NAME` storage class. | string | Delete | false | Supported values: [`Delete`, `Retain`] **Note**: If set to `Retain`, manual deletion of the Crunchy Persistent Volumes is required after deleting the PostgresCluster.| baseline |
 
+### Azure managed disk CSI Driver
+
+The Azure managed disk CSI Driver can only be included at AKS cluster creation time. It is included in all AKS clusters by default, and any AKS clusters created with viya4-iac-azure will have the driver installed. If you did not use the viya4-iac-azure project to create your AKS cluster, ensure that you have enabled the Azure disk CSI driver prior to using this project or disable the creation of the StorageClasses.
+
+By default, two block storage StorageClasses are created using the driver, one for RabbitMQ and one for Crunchy Postgres. The defaults for these StorageClasses are listed below. 
+
+**Note**: The StorageClasses created by viya4-deployment are intended for the Premium SSD v2 or Ultra Disk types. If you would like to use the Premium SSD v1 type or lower, disable creation of the StorageClasses in this project and use one of the default StorageClasses provided by the CSI driver.
+
+| Name | Description | Type | Default | Required | Notes | Tasks |
+| :--- | ---: | ---: | ---: | ---: | ---: | ---: |
+|CREATE_AZURE_RABBITMQ_STORAGE_CLASS| Whether to create an Azure files StorageClass for RabbitMQ | bool | true | false | | baseline |
+|AZURE_RABBITMQ_STORAGE_CLASS_NAME| The StorageClass name for RabbitMQ | string | managed-csi-premium-v2-mq | false | | baseline |
+|AZURE_RABBITMQ_STORAGE_CLASS_SKU_NAME| The disk type SKU name to use for RabbitMQ persistent volumes | string | PremiumV2_LRS | false | Supported values: [`PremiumV2_LRS`, `UltraSSD_LRS`]  | baseline |
+|AZURE_RABBITMQ_STORAGE_CLASS_DISKIOPS | Disk total IOPS parameter for the `AZURE_RABBITMQ_STORAGE_CLASS_NAME` storage class|string|3000|false | Refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types) for IOPS limits considerations | baseline |
+|AZURE_RABBITMQ_STORAGE_CLASS_THROUGHPUT| Maximum volume throughput in MiB/s for the `AZURE_RABBITMQ_STORAGE_CLASS_NAME` storage class| string| 400 | false | Refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types) for throughput limits considerations | baseline |
+|CREATE_AZURE_CRUNCHY_STORAGE_CLASS| Whether to create an Azure files StorageClass for Crunchy Postgres | bool | true | false | | baseline |
+|AZURE_CRUNCHY_STORAGE_CLASS_NAME| The StorageClass name for Crunchy Postgres | string| managed-csi-premium-v2-pg | false | | baseline |
+|AZURE_CRUNCHY_STORAGE_CLASS_SKU_NAME| The disk type SKU name to use for Crunchy Postgres persistent volumes | string | PremiumV2_LRS | false | Supported values: [`PremiumV2_LRS`, `UltraSSD_LRS`] | baseline |
+|AZURE_CRUNCHY_STORAGE_CLASS_DISKIOPS | Disk total IOPS parameter for the `AZURE_CRUNCHY_STORAGE_CLASS_NAME` storage class | string | 5000 | false | Refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types) for IOPS limits considerations | baseline |
+|AZURE_CRUNCHY_STORAGE_CLASS_THROUGHPUT | Maximum volume throughput in MiB/s for the `AZURE_CRUNCHY_STORAGE_CLASS_NAME` storage class | string| 400 | false | Refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types) for throughput limits considerations | baseline |
+|AZURE_CRUNCHY_STORAGE_CLASS_RECLAIM_POLICY | The ReclaimPolicy for the `AZURE_CRUNCHY_STORAGE_CLASS_NAME` storage class | string | Delete | false | Supported values: [`Delete`, `Retain`] **Note**: If set to `Retain`, manual deletion of the Crunchy Persistent Volumes is required after deleting the PostgresCluster. | baseline |
 
 ### Ingress-nginx
 

roles/baseline/defaults/main.yml

@@ -259,3 +259,45 @@ private_ingress:
 
 ## NIST Features
 V4_CFG_NIST_FEATURES_ENABLED: false
+
+## Azure StorageClass config
+CREATE_AZURE_RABBITMQ_STORAGE_CLASS: true
+AZURE_RABBITMQ_STORAGE_CLASS_NAME: managed-csi-premium-v2-mq
+AZURE_RABBITMQ_STORAGE_CLASS_SKU_NAME: PremiumV2_LRS
+AZURE_RABBITMQ_STORAGE_CLASS_DISKIOPS: 3000
+AZURE_RABBITMQ_STORAGE_CLASS_THROUGHPUT: 400
+AZURE_RABBITMQ_STORAGE_CLASS:
+  allowVolumeExpansion: true
+  apiVersion: storage.k8s.io/v1
+  kind: StorageClass
+  metadata:
+    name: "{{ AZURE_RABBITMQ_STORAGE_CLASS_NAME }}"
+  parameters:
+    skuname: "{{ AZURE_RABBITMQ_STORAGE_CLASS_SKU_NAME }}"
+    fstype: ext4
+    DiskIOPSReadWrite: "{{ AZURE_RABBITMQ_STORAGE_CLASS_DISKIOPS | int }}"
+    DiskMBpsReadWrite: "{{ AZURE_RABBITMQ_STORAGE_CLASS_THROUGHPUT | int }}"
+  provisioner: disk.csi.azure.com
+  reclaimPolicy: Delete
+  volumeBindingMode: WaitForFirstConsumer
+
+CREATE_AZURE_CRUNCHY_STORAGE_CLASS: true
+AZURE_CRUNCHY_STORAGE_CLASS_NAME: managed-csi-premium-v2-pg
+AZURE_CRUNCHY_STORAGE_CLASS_SKU_NAME: PremiumV2_LRS
+AZURE_CRUNCHY_STORAGE_CLASS_DISKIOPS: 5000
+AZURE_CRUNCHY_STORAGE_CLASS_THROUGHPUT: 400
+AZURE_CRUNCHY_STORAGE_CLASS_RECLAIM_POLICY: Delete
+AZURE_CRUNCHY_STORAGE_CLASS:
+  allowVolumeExpansion: true
+  apiVersion: storage.k8s.io/v1
+  kind: StorageClass
+  metadata:
+    name: "{{ AZURE_CRUNCHY_STORAGE_CLASS_NAME }}"
+  parameters:
+    skuname: "{{ AZURE_CRUNCHY_STORAGE_CLASS_SKU_NAME }}"
+    fstype: ext4
+    DiskIOPSReadWrite: "{{ AZURE_CRUNCHY_STORAGE_CLASS_DISKIOPS | int }}"
+    DiskMBpsReadWrite: "{{ AZURE_CRUNCHY_STORAGE_CLASS_THROUGHPUT | int }}"
+  provisioner: disk.csi.azure.com
+  reclaimPolicy: "{{ AZURE_CRUNCHY_STORAGE_CLASS_RECLAIM_POLICY }}"
+  volumeBindingMode: WaitForFirstConsumer

roles/baseline/tasks/main.yaml

@@ -71,6 +71,14 @@
   tags:
     - baseline
 
+- name: Include StorageClasses
+  include_tasks:
+    file: storage-classes.yaml
+  when:
+    - PROVIDER == "azure"
+  tags:
+    - baseline
+
 - name: Include cert manager
   include_tasks:
     file: cert-manager.yaml

roles/baseline/tasks/nfs-subdir-external-provisioner.yaml

@@ -66,6 +66,7 @@
 - name: Deploy nfs-subdir-external-provisioner-pg-storage
   when:
     - PROVIDER != "aws"
+    - PROVIDER != "azure"
   kubernetes.core.helm:
     name: "{{ PG_NFS_CLIENT_NAME }}"
     namespace: "{{ PG_NFS_CLIENT_NAMESPACE }}"
@@ -93,6 +94,7 @@
 - name: Remove nfs-subdir-external-provisioner-pg-storage
   when:
     - PROVIDER != "aws"
+    - PROVIDER != "azure"
   kubernetes.core.helm:
     name: "{{ PG_NFS_CLIENT_NAME }}"
     namespace: "{{ NFS_CLIENT_NAMESPACE }}"

roles/baseline/tasks/storage-classes.yaml

@@ -0,0 +1,55 @@
+- name: Create Azure storage class for RabbitMQ
+  kubernetes.core.k8s:
+    api_version: v1
+    definition: "{{ AZURE_RABBITMQ_STORAGE_CLASS }}"
+    wait: true
+    wait_timeout: 600
+    state: present
+    kubeconfig: "{{ KUBECONFIG }}"
+  when:
+    - PROVIDER == "azure"
+    - CREATE_AZURE_RABBITMQ_STORAGE_CLASS
+  tags:
+    - install
+
+- name: Create Azure storage class for Postgres
+  kubernetes.core.k8s:
+    api_version: v1
+    definition: "{{ AZURE_CRUNCHY_STORAGE_CLASS }}"
+    wait: true
+    wait_timeout: 600
+    state: present
+    kubeconfig: "{{ KUBECONFIG }}"
+  when:
+    - PROVIDER == "azure"
+    - CREATE_AZURE_CRUNCHY_STORAGE_CLASS
+  tags:
+    - install
+
+- name: Remove Azure storage class for RabbitMQ
+  kubernetes.core.k8s:
+    api_version: v1
+    definition: "{{ AZURE_RABBITMQ_STORAGE_CLASS }}"
+    wait: true
+    wait_timeout: 600
+    state: absent
+    kubeconfig: "{{ KUBECONFIG }}"
+  when:
+    - PROVIDER == "azure"
+    - CREATE_AZURE_RABBITMQ_STORAGE_CLASS
+  tags:
+    - uninstall
+
+- name: Remove Azure storage class for Postgres
+  kubernetes.core.k8s:
+    api_version: v1
+    definition: "{{ AZURE_CRUNCHY_STORAGE_CLASS }}"
+    wait: true
+    wait_timeout: 600
+    state: absent
+    kubeconfig: "{{ KUBECONFIG }}"
+  when:
+    - PROVIDER == "azure"
+    - CREATE_AZURE_CRUNCHY_STORAGE_CLASS
+  tags:
+    - uninstall

roles/vdm/defaults/main.yaml

@@ -32,6 +32,8 @@ V4_CFG_RWX_FILESTORE_ASTORES_PATH: "{{ V4_CFG_RWX_FILESTORE_PATH | replace('/$',
 V4_CFG_RWX_FILESTORE_BIN_PATH: "{{ V4_CFG_RWX_FILESTORE_PATH | replace('/$', '') }}/{{ NAMESPACE }}/bin"
 
 V4_CFG_STORAGECLASS: sas
+V4_CFG_RABBITMQ_STORAGECLASS: "{{ 'io2-vol-mq' if PROVIDER == 'aws' else ('managed-csi-premium-v2-mq' if PROVIDER == 'azure' else V4_CFG_STORAGECLASS) }}"
+V4_CFG_CRUNCHY_STORAGECLASS: "{{ 'io2-vol-pg' if PROVIDER == 'aws' else ('managed-csi-premium-v2-pg' if PROVIDER == 'azure' else 'pg-storage') }}"
 V4_CFG_MANAGE_STORAGE: true
 
 V4_CFG_DEPLOYMENT_URL_PREFIX: null