feat: Use block storage for Crunchy Postgres and RabbitMQ for AWS (PSKD-1282) (#594)

dhoucgitter · web-flow · commit 7503f2965974 · 2025-02-27T16:18:05.000-05:00
Signed-off-by: David.Houck &lt;David.Houck@sas.com&gt;
diff --git a/docs/CONFIG-VARS.md b/docs/CONFIG-VARS.md
@@ -7,7 +7,8 @@ Supported configuration variables are listed in the table below.  All variables
   - [Cloud](#cloud)
     - [Authentication](#authentication)
   - [Jump Server](#jump-server)
-  - [Storage](#storage)
+  - [Storage for AWS](#storage-for-aws)
+  - [Storage for Azure and Google Cloud](#storage-for-azure-and-google-cloud)
     - [RWX Filestore](#rwx-filestore)
     - [Azure](#azure)
     - [AWS](#aws)
@@ -85,7 +86,11 @@ Viya4-deployment uses the jump server to interact with the RWX filestore, which
 | JUMP_SVR_PRIVATE_KEY | Path to the SSH user's private key to access the jump server host | string |  | true | | baseline, viya |
 | JUMP_SVR_RWX_FILESTORE_PATH | Path on the jump server to the NFS mount | string | /viya-share | false | | viya |
 
-## Storage
+## Storage for AWS
+When `V4_CFG_MANAGE_STORAGE` is set to `true`, viya4-deployment uses the [EBS CSI driver](#ebs-csi-driver) to create two elastic block storage based storage classes with the default names of `io2-vol-mq` and `io2-vol-pg`. The volume type for both storage classes defaults to `io2`. For EKS clusters, RabbitMQ makes PVC requests to create block storage persistent volumes using the `io2-vol-mq` storage class while Crunchy Postgres makes PVC requests to create block storage persistent volumes using the `io2-vol-pg` storage class. Viya4-deployment also creates the `sas` storage class using the nfs-subdir-external-provisioner Helm chart. If a jump server is used, viya4-deployment uses that server to create the folders for the `astores`, `bin`, `data` and `homes` RWX Filestore NFS paths that are outlined below in the [RWX Filestore](#rwx-filestore) section.
+
+
+## Storage for Azure and Google Cloud
 When `V4_CFG_MANAGE_STORAGE` is set to `true`, viya4-deployment creates the `sas` and `pg-storage` storage classes using the nfs-subdir-external-provisioner Helm chart. If a jump server is used, viya4-deployment uses that server to create the folders for the `astores`, `bin`, `data` and `homes` RWX Filestore NFS paths that are outlined below in the [RWX Filestore](#rwx-filestore) section.
 
 When `V4_CFG_MANAGE_STORAGE` is set to `false`, viya4-deployment does not create the `sas` or `pg-storage` storage classes for you. In addition, viya4-deployment does not create or manage the RWX Filestore NFS paths. Before you run the SAS Viya deployment, you must set the values for `V4_CFG_RWX_FILESTORE_DATA_PATH` and `V4_CFG_RWX_FILESTORE_HOMES_PATH` to specify existing NFS folder locations. The viya4-deployment user can create the required NFS folders from the jump server before starting the deployment. Recommended attribute settings for each folder are as follows:
@@ -401,7 +406,7 @@ If you used [viya4-iac-aws:5.6.0](https://github.com/sassoftware/viya4-iac-aws/r
 
 ### EBS CSI Driver
 
-The EBS CSI driver is currently only used for kubernetes v1.23 or later AWS EKS clusters.
+The EBS CSI driver is only used for kubernetes v1.23 or later AWS EKS clusters.
 
 | Name | Description | Type | Default | Required | Notes | Tasks |
 | :--- | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -411,6 +416,16 @@ The EBS CSI driver is currently only used for kubernetes v1.23 or later AWS EKS
 | EBS_CSI_DRIVER_CONFIG | aws ebs csi driver helm values | string | see [here](../roles/baseline/defaults/main.yml) | false | | baseline |
 | EBS_CSI_DRIVER_ACCOUNT | cluster autoscaler aws role arn | string | | false | Required to enable the aws ebs csi driver on AWS | baseline |
 | EBS_CSI_DRIVER_LOCATION | aws region where kubernetes cluster resides | string | us-east-1 | false | | baseline |
+|EBS_CSI_RABBITMQ_STORAGE_CLASS_NAME| The EBS CSI storage class name for RabbitMQ | string | io2-vol-mq | false | | baseline |
+|EBS_CSI_RABBITMQ_STORAGE_CLASS_VOLUME_TYPE| The EBS CSI volume type to use for RabbitMQ persistent volumes| string | io2 | false | Supported values: [`io2`, `io1`, `gp3`]  | baseline |
+|EBS_CSI_RABBITMQ_STORAGE_CLASS_IOPSPERGB | IOPs per GB parameter for the `EBS_CSI_RABBITMQ_STORAGE_CLASS_NAME` storage class|string|1250|false |Multiply this value by the volume size in GiB to obtain total IOPS per volume  | baseline |
+|EBS_CSI_RABBITMQ_STORAGE_CLASS_THROUGHPUT| Maximum volume throughput in MiB/s for the `EBS_CSI_RABBITMQ_STORAGE_CLASS_NAME` storage class| string| 400 | false | The maximum value for io2, io1 and gp3 volume types is 1000.| baseline |
+|EBS_CSI_CRUNCHY_STORAGE_CLASS_NAME| The EBS CSI storage class name for Crunchy Postgres use| string| io2-vol-pg | false | | baseline |
+|EBS_CSI_CRUNCHY_STORAGE_CLASS_VOLUME_TYPE| The EBS CSI volume type to use for Crunchy Postgres persistent volumes | string | io2 | false | Supported values: [`io2`, `io1`, `gp3`] | baseline |
+|EBS_CSI_CRUNCHY_STORAGE_CLASS_IOPSPERGB | IOPs per GB parameter for the `EBS_CSI_CRUNCHY_STORAGE_CLASS_NAME` storage class | string | 40 | false |Multiply this value by the volume size in GiB to obtain total IOPS per volume | baseline |
+|EBS_CSI_CRUNCHY_STORAGE_CLASS_THROUGHPUT | Maximum volume throughput in MiB/s for the `EBS_CSI_CRUNCHY_STORAGE_CLASS_NAME` storage class | string| 400 | false | The maximum value for io2, io1 and gp3 volume types is 1000.| baseline |
+|EBS_CSI_CRUNCHY_STORAGE_CLASS_RECLAIM_POLICY | The ReclaimPolicy for the `EBS_CSI_CRUNCHY_STORAGE_CLASS_NAME` storage class | string | Delete | false | Supported values: [`Delete`, `Retain`] | baseline |
+
 
 ### Ingress-nginx
 
diff --git a/roles/baseline/defaults/main.yml b/roles/baseline/defaults/main.yml
@@ -199,6 +199,17 @@ EBS_CSI_DRIVER_CHART_URL: https://kubernetes-sigs.github.io/aws-ebs-csi-driver
 EBS_CSI_DRIVER_CHART_VERSION: 2.38.1
 EBS_CSI_DRIVER_ACCOUNT: null
 EBS_CSI_DRIVER_LOCATION: us-east-1
+EBS_CSI_RABBITMQ_STORAGE_CLASS_NAME: io2-vol-mq
+EBS_CSI_RABBITMQ_STORAGE_CLASS_VOLUME_TYPE: io2
+EBS_CSI_RABBITMQ_STORAGE_CLASS_IOPSPERGB: 1250
+EBS_CSI_RABBITMQ_STORAGE_CLASS_THROUGHPUT: 400
+
+EBS_CSI_CRUNCHY_STORAGE_CLASS_NAME: io2-vol-pg
+EBS_CSI_CRUNCHY_STORAGE_CLASS_VOLUME_TYPE: io2
+EBS_CSI_CRUNCHY_STORAGE_CLASS_IOPSPERGB: 40
+EBS_CSI_CRUNCHY_STORAGE_CLASS_THROUGHPUT: 400
+EBS_CSI_CRUNCHY_STORAGE_CLASS_RECLAIM_POLICY: Delete
+
 EBS_CSI_DRIVER_CONFIG:
   controller:
     k8sTagClusterId: "{{ CLUSTER_NAME }}"
@@ -208,6 +219,25 @@ EBS_CSI_DRIVER_CONFIG:
       name: ebs-csi-controller-sa
       annotations:
         eks.amazonaws.com/role-arn: "{{ EBS_CSI_DRIVER_ACCOUNT }}"
+  storageClasses:
+    - name: "{{ EBS_CSI_RABBITMQ_STORAGE_CLASS_NAME }}"
+      parameters:
+        type: "{{ EBS_CSI_RABBITMQ_STORAGE_CLASS_VOLUME_TYPE }}"
+        fstype: ext4
+        iopsPerGB: "{{ EBS_CSI_RABBITMQ_STORAGE_CLASS_IOPSPERGB | int }}"
+        throughput: "{{ EBS_CSI_RABBITMQ_STORAGE_CLASS_THROUGHPUT | int }}"
+      reclaimPolicy: Delete
+      volumeBindingMode: WaitForFirstConsumer
+      allowVolumeExpansion: true
+    - name: "{{ EBS_CSI_CRUNCHY_STORAGE_CLASS_NAME }}"
+      parameters:
+        type: "{{ EBS_CSI_CRUNCHY_STORAGE_CLASS_VOLUME_TYPE }}"
+        fstype: ext4
+        iopsPerGB: "{{ EBS_CSI_CRUNCHY_STORAGE_CLASS_IOPSPERGB | int }}"
+        throughput: "{{ EBS_CSI_CRUNCHY_STORAGE_CLASS_THROUGHPUT | int }}"
+      reclaimPolicy: "{{ EBS_CSI_CRUNCHY_STORAGE_CLASS_RECLAIM_POLICY }}"
+      volumeBindingMode: WaitForFirstConsumer
+      allowVolumeExpansion: true
 
 private_ingress:
   aws:
diff --git a/roles/baseline/tasks/nfs-subdir-external-provisioner.yaml b/roles/baseline/tasks/nfs-subdir-external-provisioner.yaml
@@ -64,6 +64,8 @@
     - update
 
 - name: Deploy nfs-subdir-external-provisioner-pg-storage
+  when:
+    - PROVIDER != "aws"
   kubernetes.core.helm:
     name: "{{ PG_NFS_CLIENT_NAME }}"
     namespace: "{{ PG_NFS_CLIENT_NAMESPACE }}"
@@ -89,6 +91,8 @@
     - uninstall
 
 - name: Remove nfs-subdir-external-provisioner-pg-storage
+  when:
+    - PROVIDER != "aws"
   kubernetes.core.helm:
     name: "{{ PG_NFS_CLIENT_NAME }}"
     namespace: "{{ NFS_CLIENT_NAMESPACE }}"
diff --git a/roles/vdm/tasks/postgres/postgres-instance.yaml b/roles/vdm/tasks/postgres/postgres-instance.yaml
@@ -206,10 +206,20 @@
           block:
             - name: Postgres instance - save default role map values
               set_fact:
-                default_postgres_storage_class: "{{ settings.postgres_storage_class | default('pg-storage', true) }}"
                 default_postgres_pvc_access_mode: "{{ settings.postgres_pvc_access_mode | default('ReadWriteOnce', true) }}"
-                default_backrest_storage_class: "{{ settings.backrest_storage_class | default('pg-storage', true) }}"
                 default_backrest_pvc_access_mode: "{{ settings.backrest_pvc_access_mode | default('ReadWriteOnce', true) }}"
+            - name: Postgres instance - save default role map values for AWS
+              when:
+                - PROVIDER == "aws"
+              set_fact:
+                default_postgres_storage_class: "{{ settings.postgres_storage_class | default('io2-vol-pg', true) }}"
+                default_backrest_storage_class: "{{ settings.backrest_storage_class | default('io2-vol-pg', true) }}"
+            - name: Postgres instance - save default role map values for non-AWS
+              when:
+                - PROVIDER != "aws"
+              set_fact:
+                default_postgres_storage_class: "{{ settings.postgres_storage_class | default('pg-storage', true) }}"
+                default_backrest_storage_class: "{{ settings.backrest_storage_class | default('pg-storage', true) }}"
             - name: Postgres instance - Create deployment namespace
               kubernetes.core.k8s:
                 kubeconfig: "{{ KUBECONFIG }}"
@@ -240,10 +250,20 @@
           block:
             - name: Postgres instance - save cds role map values
               set_fact:
-                cds_postgres_storage_class: "{{ settings.postgres_storage_class | default('pg-storage', true) }}"
                 cds_postgres_pvc_access_mode: "{{ settings.postgres_pvc_access_mode | default('ReadWriteOnce', true) }}"
-                cds_backrest_storage_class: "{{ settings.backrest_storage_class | default('pg-storage', true) }}"
                 cds_backrest_pvc_access_mode: "{{ settings.backrest_pvc_access_mode | default('ReadWriteOnce', true) }}"
+            - name: Postgres instance - save cds role map values for AWS
+              when:
+                - PROVIDER == "aws"
+              set_fact:
+                cds_postgres_storage_class: "{{ settings.postgres_storage_class | default('io2-vol-pg', true) }}"
+                cds_backrest_storage_class: "{{ settings.backrest_storage_class | default('io2-vol-pg', true) }}"
+            - name: Postgres instance - save cds role map values for non-AWS
+              when:
+                - PROVIDER != "aws"
+              set_fact:
+                cds_postgres_storage_class: "{{ settings.postgres_storage_class | default('pg-storage', true) }}"
+                cds_backrest_storage_class: "{{ settings.backrest_storage_class | default('pg-storage', true) }}"
             - name: Postgres instance - Update ConfigMap
               kubernetes.core.k8s:
                 kubeconfig: "{{ KUBECONFIG }}"
diff --git a/roles/vdm/templates/transformers/crunchy-storage-transformer.yaml b/roles/vdm/templates/transformers/crunchy-storage-transformer.yaml
@@ -29,6 +29,18 @@ patch: |-
     value:
       - {{ default_backrest_pvc_access_mode if (default_backrest_pvc_access_mode is defined) else settings.backrest_pvc_access_mode|default('ReadWriteOnce', true) }}
 
+  {% if PROVIDER == "aws" %}
+  # This value cannot be changed after the initial deployment
+  - op: replace
+    path: /spec/instances/0/dataVolumeClaimSpec/storageClassName
+    value: {{ default_postgres_storage_class if (default_postgres_storage_class is defined) else settings.postgres_storage_class|default('io2-vol-pg', true) }}
+
+  # This value cannot be changed after the initial deployment
+  - op: replace
+    path: /spec/backups/pgbackrest/repos/0/volume/volumeClaimSpec/storageClassName
+    value: {{ default_backrest_storage_class if (default_backrest_storage_class is defined) else settings.backrest_storage_class|default('io2-vol-pg', true) }}
+
+  {% else %}
   # This value cannot be changed after the initial deployment
   - op: replace
     path: /spec/instances/0/dataVolumeClaimSpec/storageClassName
@@ -38,6 +50,7 @@ patch: |-
   - op: replace
     path: /spec/backups/pgbackrest/repos/0/volume/volumeClaimSpec/storageClassName
     value: {{ default_backrest_storage_class if (default_backrest_storage_class is defined) else settings.backrest_storage_class|default('pg-storage', true) }}
+  {% endif %}
 
 {% elif role == "cds-postgres" %}
   # This value cannot be changed after the initial deployment
@@ -52,6 +65,18 @@ patch: |-
     value:
       - {{ cds_backrest_pvc_access_mode if (cds_backrest_pvc_access_mode is defined) else settings.backrest_pvc_access_mode|default('ReadWriteOnce', true) }}
 
+  {% if PROVIDER == "aws" %}
+  # This value cannot be changed after the initial deployment
+  - op: replace
+    path: /spec/instances/0/dataVolumeClaimSpec/storageClassName
+    value: {{ cds_postgres_storage_class if (cds_postgres_storage_class is defined) else settings.postgres_storage_class|default('io2-vol-pg', true) }}
+
+  # This value cannot be changed after the initial deployment
+  - op: replace
+    path: /spec/backups/pgbackrest/repos/0/volume/volumeClaimSpec/storageClassName
+    value: {{ cds_backrest_storage_class if (cds_backrest_storage_class is defined) else settings.backrest_storage_class|default('io2-vol-pg', true) }}
+
+  {% else %}
   # This value cannot be changed after the initial deployment
   - op: replace
     path: /spec/instances/0/dataVolumeClaimSpec/storageClassName
@@ -61,6 +86,7 @@ patch: |-
   - op: replace
     path: /spec/backups/pgbackrest/repos/0/volume/volumeClaimSpec/storageClassName
     value: {{ cds_backrest_storage_class if (cds_backrest_storage_class is defined) else settings.backrest_storage_class|default('pg-storage', true) }}
+  {% endif %}
 
 {% endif %}
 target:
diff --git a/roles/vdm/templates/transformers/sas-storageclass.yaml b/roles/vdm/templates/transformers/sas-storageclass.yaml
@@ -11,6 +11,37 @@ patch: |-
     storageClassName: "{{ V4_CFG_STORAGECLASS }}"
 target:
   kind: PersistentVolumeClaim
+{% if PROVIDER == "aws" %}
+---
+apiVersion: builtin
+kind: PatchTransformer
+metadata:
+  name: sas-ss-storageclass
+patch: |-
+  - op: add
+    path: /spec/volumeClaimTemplates/0/spec/storageClassName
+    value: {{ V4_CFG_STORAGECLASS }}
+target:
+  group: apps
+  kind: StatefulSet
+  annotationSelector: sas.com/component-name notin (sas-rabbitmq-server,sas-airflow,sas-risk-cirrus-search,sas-workload-orchestrator,sas-data-agent-server-colocated)
+---
+apiVersion: builtin
+kind: PatchTransformer
+metadata:
+  name: sas-ebs-storageclass
+patch: |-
+  - op: add
+    path: /spec/volumeClaimTemplates/0/spec/storageClassName
+    value: io2-vol-mq
+  - op: replace
+    path: /spec/volumeClaimTemplates/0/spec/resources/requests/storage
+    value: 4Gi
+target:
+  group: apps
+  kind: StatefulSet
+  annotationSelector: sas.com/component-name in (sas-rabbitmq-server)
+{% else %}
 ---
 apiVersion: builtin
 kind: PatchTransformer
@@ -24,3 +55,4 @@ target:
   group: apps
   kind: StatefulSet
   annotationSelector: sas.com/component-name notin (sas-airflow,sas-risk-cirrus-search,sas-workload-orchestrator,sas-data-agent-server-colocated)
+{% endif %}
