Updated the README.md file to be clearer.

Author: kcantrel

Solutions/FSxN-as-PVC-for-EKS/README.md

@@ -14,8 +14,11 @@ A Unix based system with the following installed:
 	- Deploy an EKS cluster
 	- Deploy an FSx for Netapp ONTAP File System
 	- Create security groups
+  - Create policies and roles
+  - Create secrtets in AWS SecretsManager
 	- Create a VPC and subnets
 	- Create a NAT Gateway
+  - Create a Internet Gateway
 	- Create an EC2 instance
 
 ## Installation Overview
@@ -27,16 +30,18 @@ The overall process is as follows:
 - Run 'terraform apply -auto-approve' to:
   - Create a new VPC with public and private subnets.
   - Deploy a FSx for NetApp ONTAP File System.
+  - Create a secret in AWS SecretsManager to hold the FSxN password.
   - Deploy an EKS cluster.
   - Deploy an EC2 Linux based instance.
+  - Create policies, roles and security groups to protect the new environement.
 - SSH to the Linux based instance to complete the setup:
   - Install the FSx for NetApp ONTAP Trident CSI driver.
   - Configure the Trident CSI driver.
   - Create a Kubernetes storage class.
 - Deploy a sample application to test the storage with.
 
 ## Detailed Instructions
-## Clone the "NetApp/FSx-ONTAP-samples-scripts" repo from GitHub
+### Clone the "NetApp/FSx-ONTAP-samples-scripts" repo from GitHub
 Execute the following commands to clone the repo and change into the directory where the
 terraform files are located:
 ```bash
@@ -46,6 +51,7 @@ cd FSx-ONTAP-samples-scripts/Solutions/FSxN-as-PVC-for-EKS/terraform
 ### Make any desired changes to the variables.tf file.
 Variables that can be changed include:
 - aws_region - The AWS region where you want to deploy the resources.
+- aws_secrets_region - The region where the fsx_password_secret will be created.
 - fsx_name - The name you want applied to the FSx for NetApp ONTAP File System. Must not already exist.
 - fsx_password_secret_name - A base name of the AWS SecretsManager secret that will hold the FSxN password.
 A random string will be appended to this name to ensure uniqueness.
@@ -73,14 +79,15 @@ the following is an example of last part of the output of a successful deploymen
 ```bash
 Outputs:
 
-eks-cluster-name = "fsx-eks-mWFem72Z"
-eks-jump-server = "Instance ID: i-0bcf0ed9adeb55814, Public IP: 35.92.238.240"
-fsx-id = "fs-04794c394fa5a85de"
-fsx-password-secret-arn = "arn:aws:secretsmanager:us-west-2:759995470648:secret:fsx-eks-secret20240618170506480900000001-u8IQEp"
-fsx-password-secret-name = "fsx-eks-secrete-3f55084"
+eks-cluster-name = "fsx-eks-DB0H69vL"
+eks-jump-server = "Instance ID: i-0e99a61431a39d327, Public IP: 54.244.16.198"
+fsx-id = "fs-0887a493cXXXXXXXX"
+fsx-management-ip = "198.19.255.174"
+fsx-password-secret-arn = "arn:aws:secretsmanager:us-west-2:759995400000:secret:fsx-eks-secret-3b8bde97-Fst5rj"
+fsx-password-secret-name = "fsx-eks-secret-3b8bde97"
 fsx-svm-name = "ekssvm"
 region = "us-west-2"
-vpc-id = "vpc-043a3d602b64e2f56"
+vpc-id = "vpc-03ed6b1867d76e1a9"
 ```
 You will use the values in the commands below, so probably a good idea to copy the output somewhere
 so you can easily reference it later.
@@ -89,7 +96,7 @@ Note that an FSxN File System was created, with a vserver (a.k.a. SVM). The defa
 for the FSxN File System is 'fsxadmin'. And the default username for the vserver is 'vsadmin'. The
 password for both of these users is the same and is what is stored in the AWS SecretsManager secret
 shown above. Note that since Terraform was used to create the secret, the password is stored in
-plain text therefore it is **HIGHLY** recommended that you change the password to something else
+plain text and therefore it is **HIGHLY** recommended that you change the password to something else
 by first changing the passwords via the AWS Management Console and then updating the password in
 the AWS SecretsManager secret. You can update the 'username' key in the secret if you want, but
 it must be a vserver admin user, not a system level user. This secret is used by Astra
@@ -113,11 +120,9 @@ There are various ways to configure the AWS cli. If you are unsure how to do it,
 refer to this the AWS documentation for instructions:
 [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)
 
-**NOTE:** When asked for a default region, use the region you specified in the variables.tf file.
-
 ### Allow access to the EKS cluster for your user id
 AWS's EKS clusters have a secondary form for permissions. As such, you have to add an "access-entry"
-to your EKS configuration and associate it with Cluster Admin policy to be able to view and
+to your EKS configuration and associate it with the Cluster Admin policy to be able to view and
 configure the EKS cluster. The first step to do this is to find out your IAM ARN.
 You can do that via this command:
 ```bash
@@ -186,8 +191,8 @@ trident-operator-67d6fd899b-jrnt2     1/1     Running   0          20h
 For the example below we are going to set up an iSCSI LUN for a MySQL
 database. To help facilicate that, we are going to setup Astra Trident as a backend provider.
 Since we are going to be creating an iSCSI LUN, we are going to use its `ontap-san` driver.
-Astra Trident has varioius different drivers that you might be interested in. You can read
-more about the drivers it supports in the
+Astra Trident has serveral different drivers to choose from. You can read more about the
+drivers it supports in the
 [Astra Trident documentation.](https://docs.netapp.com/us-en/trident/trident-use/trident-fsx.html#fsx-for-ontap-driver-details)
 
 As you go through the steps below, you will notice that most of the files have "-san" in their
@@ -214,7 +219,7 @@ After making the following substitutions in the commands below:
 
 Execute the following commands to configure Trident to use the FSxN file system that was
 created earlier using the `terraform --apply` command:
-```bash
+```
 cd ~/FSx-ONTAP-samples-scripts/Solutions/FSxN-as-PVC-for-EKS
 mkdir temp
 export FSX_ID=<fsx-id>
@@ -244,6 +249,20 @@ The output should look similar to this:
 NAME                    BACKEND NAME            BACKEND UUID                           PHASE   STATUS
 backend-fsx-ontap-san   backend-fsx-ontap-san   7a551921-997c-4c37-a1d1-f2f4c87fa629   Bound   Success
 ```
+If the status is Failed, then you can add the "--output=json" flag to the `kubectl get tridentbackendconfig`
+command to get more information as to why it failed. Specifically, look at the "message" field in the output.
+The following command will get just the status messages:
+```bash
+kubectl get tridentbackendconfig -n trident --output=json | jq '.items[] | .status.message'
+```
+Once you have resolved any issues, you can remove the failed backend by running:
+```bash
+kubectl delete -n trident -f temp/backend-tbc-ontap-san.yaml
+```
+Then, you can re-run the `kubectl create -n trident -f temp/backend-tbc-ontap-san.yaml` command.
+If the issues was with one of the variables that was substituted in, then you will need to
+rerun the `envsubst` command to create a new `temp/backend-tbc-ontap-san.yaml` file
+before running the `kubectl create -n trident -f temp/backend-tbc-ontap-san.yaml` command.
 
 ### Create a Kubernetes storage class
 The next step is to create a Kubernetes storage class by executing:
@@ -283,7 +302,31 @@ kubectl get pvc
 The output should look similar to this:
 ```bash
 NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
-mysql-volume-san   Bound    pvc-15e834eb-daf8-4a96-a2d5-4044442fbe90   50Gi       RWO            fsx-basic-san   <unset>                 13s
+mysql-volume-san   Bound    pvc-1aae479e-4b27-4310-8bb2-71255134edf0   50Gi       RWO            fsx-basic-san   <unset>                 114m
+```
+
+If you want to see what was created on the FSxN file system, you can log into it and take a look.
+You will want to login as the 'fsxadmin' user, using the password stored in the AWS SecretsManager secret.
+You can find the IP address of the FSxN file system in the output from the `terraform apply` command.
+Here is an example of logging in and listing all the LUNs on the system:
+```bash
+ubuntu@ip-10-0-4-125:~/FSx-ONTAP-samples-scripts/Solutions/FSxN-as-PVC-for-EKS$ ssh -l fsxadmin 198.19.255.174
+([email protected]) Password:
+
+Last login time: 6/21/2024 15:30:27
+FsxId0887a493c777c5122::> lun show
+Vserver   Path                            State   Mapped   Type        Size
+--------- ------------------------------- ------- -------- -------- --------
+ekssvm    /vol/trident_pvc_1aae479e_4b27_4310_8bb2_71255134edf0/lun0
+                                          online  mapped   linux        50GB
+
+FsxId0887a493c777c5122::> volume show
+Vserver   Volume       Aggregate    State      Type       Size  Available Used%
+--------- ------------ ------------ ---------- ---- ---------- ---------- -----
+ekssvm    ekssvm_root  aggr1        online     RW          1GB    972.4MB    0%
+ekssvm    trident_pvc_1aae479e_4b27_4310_8bb2_71255134edf0
+                       aggr1        online     RW         55GB    54.90GB    0%
+3 entries were displayed.
 ```
 
 ### Deploy a MySQL database using the storage created above
@@ -297,10 +340,11 @@ kubectl get pods
 ```
 The output should look similar to this:
 ```bash
-NAME                         READY   STATUS    RESTARTS   AGE
-csi-snapshotter-0            3/3     Running   0          22h
-mysql-fsx-695b497757-pvn7q   1/1     Running   0          20h
+NAME                             READY   STATUS    RESTARTS   AGE
+mysql-fsx-san-79cdb57b58-m2lgr   1/1     Running   0          31s
 ```
+Note that it might take a minute or two for the pod to get to the Running status.
+
 To see how the MySQL was configured, check out the `manifests/mysql-san.yaml` file.
 
 ### Populate the MySQL database with data
@@ -316,7 +360,7 @@ kubectl exec -it $(kubectl get pod -l "app=mysql-fsx-san" --namespace=default -o
 
 After you have logged in, here is a session showing an example of creating a database, then creating a table, then inserting
 some values into the table:
-```sql
+```
 mysql> create database fsxdatabase; 
 Query OK, 1 row affected (0.01 sec)
 
@@ -347,6 +391,9 @@ mysql> select * from fsx;
 | netapp04   | 1024GB   | us-west-1 |
 +------------+----------+-----------+
 6 rows in set (0.00 sec)
+
+mysql> quit
+Bye
 ```
 
 ## Create a snapshot of the MySQL data
@@ -355,7 +402,7 @@ These snapshots take almost no additional space on the backend storage and pose
 
 ### Install the Kubernetes Snapshot CRDs and Snapshot Controller:
 The first step is to install the Snapshot CRDs and the Snapshot Controller.
-To do that run the following commands:
+To do that by running these commands:
 ```bash
 git clone https://github.com/kubernetes-csi/external-snapshotter 
 cd external-snapshotter/ 
@@ -364,9 +411,8 @@ kubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl
 kubectl kustomize deploy/kubernetes/csi-snapshotter | kubectl create -f - 
 cd ..
 ```
-
-### Create a snapshot of the MySQL data
-Now, add the volume snapshot class by executing:
+### Create a snapshot class based on the CRD instsalled
+Create a snapshot class by executing:
 ```bash
 kubectl create -f manifests/volume-snapshot-class.yaml 
 ```
@@ -377,7 +423,8 @@ volumesnapshotclass.snapshot.storage.k8s.io/fsx-snapclass created
 Note, that this storage class works for both LUNs and NFS volumes, so there aren't different versions
 of this file based on the storage type you are testing with.
 
-The final step is to create a snapshot of the data by executing:
+### Create a snapshot of the MySQL data
+Now you can create a snapshot by executing:
 ```bash
 kubectl create -f manifests/volume-snapshot-san.yaml
 ```
@@ -392,9 +439,19 @@ kubectl get volumesnapshot
 The output should look like:
 ```bash
 NAME                       READYTOUSE   SOURCEPVC          SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                    CREATIONTIME   AGE
-mysql-volume-san-snap-01   true         mysql-volume-san                           50Gi          fsx-snapclass   snapcontent-b3491f26-47e3-484c-aae0-69d45087d6c7   4s             5s
+mysql-volume-san-snap-01   true         mysql-volume-san                           50Gi          fsx-snapclass   snapcontent-bdce9310-9698-4b37-9f9b-d1d802e44f17   2m18s          2m18s
 ```
 
+You can log onto the FSxN file system to see that the snapshot was created there:
+```bash
+FsxId0887a493c777c5122::> snapshot show -volume trident_pvc_*
+                                                                 ---Blocks---
+Vserver  Volume   Snapshot                                  Size Total% Used%
+-------- -------- ------------------------------------- -------- ------ -----
+ekssvm   trident_pvc_1aae479e_4b27_4310_8bb2_71255134edf0
+                  snapshot-bdce9310-9698-4b37-9f9b-d1d802e44f17
+                                                           140KB     0%    0%
+
 ## Clone the MySQL data to a new storage persistent volume
 Now that you have a snapshot of the data, you can use it to create a read/write version
 of it. This can be used as a new storage volume for another mysql database. This operation
@@ -406,6 +463,29 @@ The first step is to create a Persistent Volume Claim from the snapshot by execu
 ```bash
 kubectl create -f manifests/pvc-from-san-snapshot.yaml
 ```
+To check that it worked, execute:
+```bash
+kubectl get pvc
+```
+The output should look similar to this:
+```bash
+$ kubectl get pvc
+NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
+mysql-volume-san         Bound    pvc-1aae479e-4b27-4310-8bb2-71255134edf0   50Gi       RWO            fsx-basic-san   <unset>                 125m
+mysql-volume-san-clone   Bound    pvc-ceb1b2c2-de35-4011-8d6e-682b6844bf02   50Gi       RWO            fsx-basic-san   <unset>                 2m22s
+```
+
+To check it on the FSxN side, you can run:
+```bash
+FsxId0887a493c777c5122::> volume clone show
+                      Parent  Parent        Parent
+Vserver FlexClone     Vserver Volume        Snapshot             State     Type
+------- ------------- ------- ------------- -------------------- --------- ----
+ekssvm  trident_pvc_ceb1b2c2_de35_4011_8d6e_682b6844bf02
+                      ekssvm  trident_pvc_1aae479e_4b27_4310_8bb2_71255134edf0
+                                            snapshot-bdce9310-9698-4b37-9f9b-d1d802e44f17
+                                                                 online    RW
+```
 ## Create a new MySQL database using the cloned volume
 Now that you have a new storage volume, you can create a new MySQL database that uses it by executing:
 ```bash

Solutions/FSxN-as-PVC-for-EKS/terraform/variables.tf

@@ -3,24 +3,19 @@ variable "aws_region" {
   description = "aws region where you want the resources deployed."
 }
 
+variable "aws_secrets_region" {
+  default     = "us-west-2"
+  description = "The region where you want the FSxN secret stored within AWS Secrets Manager."
+}
+
 variable "fsx_name" {
   default     = "eksfs"
   description = "The name you want assigned to the FSxN file system."
 }
 
 variable "fsx_password_secret_name" {
   default     = "fsx-eks-secret"
-  description = "The basename of the secret to create within the AWS Secrets Manager that will contain the FSxN password. A random string will be appended to the end of the secreate name to ensure no name conflict."
-}
-
-variable "aws_secrets_region" {
-  default     = "us-west-2"
-  description = "The region where you want the secret stored within AWS Secrets Manager."
-}
-
-variable "trident_version" {
-  default     = "v24.2.0-eksbuild.1"
-  description = "The version of Astra Trident to 'add-on' to the EKS cluster."
+  description = "The base name of the secret to create within the AWS Secrets Manager that will contain the FSxN password. A random string will be appended to the end of the secreate name to ensure no name conflict."
 }
 
 variable "fsxn_throughput_capacity" {
@@ -48,6 +43,11 @@ variable "secure_ips" {
 # Don't change any variables below this line.
 ################################################################################
 
+variable "trident_version" {
+  default     = "v24.2.0-eksbuild.1"
+  description = "The version of Astra Trident to 'add-on' to the EKS cluster."
+}
+
 variable "kubernetes_version" {
   default     = 1.29
   description = "kubernetes version"