Added a CloudFormation template to auto-add-cw-alarms.

Author: kcantrel

.github/workflows/update-CloudformationTemplate-auto-add-cw-alarms.yml

@@ -0,0 +1,37 @@
+---
+# Copyright (c) NetApp, Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+name: "Update Cloudformation Template"
+
+on:
+  pull_request:
+    paths:
+      - 'Monitoring/auto-add-cw-alarms/auto_add_cw_alarms.py'
+  push:
+    paths:
+      - 'Monitoring/auto-add-cw-alarms/auto_add_cw_alarms.py'
+    branches:
+      - main
+
+jobs:
+  update-Cloudformation-Template:
+    runs-on: ubuntu-latest
+    permissions:
+      # Give the default GITHUB_TOKEN write permission to commit and push the
+      # added or changed files to the repository.
+      contents: write
+
+    steps:
+      - name: Checkout pull request
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.ref }}
+
+      - name: Update the Cloudformation Template
+        shell: bash
+        working-directory: Monitoring/auto-add-cw-alarms
+        run: ./update-auto-add-cw-alarms-CF-Template
+
+      - name: Commit the changes
+        uses: stefanzweifel/git-auto-commit-action@v5

Monitoring/auto-add-cw-alarms/README.md

@@ -10,19 +10,32 @@ to monitor the CPU utilization of the file system. And if a volume or file syste
 
 To implement this, you might think to just create EventTail filters to trigger on the creation or deletion of an FSx Volume.
 This would kind of work, but since you have command line access to the FSx for ONTAP file system, you can create
-and delete volumes without creating CloudTrail events. So, this method would not be reliable. Therefore, instead
+and delete volumes without generating any CloudTrail events. So, this method would not be reliable. Therefore, instead
 of relying on those events, this script will scan all the file systems and volumes in all the regions then create and delete alarms as needed.
 
 ## Invocation
-There are two ways you can invoke this script (Python program). Either from a computer that has Python installed, or you could upload it
-as a Lambda function.
+There are two ways you can invoke this script (Python program). Either from a computer that has Python installed, or you could install it
+as a Lambda function. If you want to run it as a Lambda function, a CloudFormation template is included in the repo that will:
+- Create a role that will allow the Lambda function to:
+    - List AWS regions. So it can scan all regions for FSx for ONTAP file systems and volumes.
+    - List the FSx for ONTAP file systems.
+    - List the FSx volume.
+    - List the CloudWatch alarms.
+    - List tags for the resources. This is so you can customize the thresholds for the alarms.
+    - Create CloudWatch alarms.
+    - Delete CloudWatch alarms that it has created (based on alarm names).
+- Create a Lambda function with the Python program.
+- Create a EventBridge schedule that will run the Lambda function on a user defined basis.
+- Create a role that will allow the EventBridge schedule to trigger the Lambda function.
 
 ### Configuring the program
 Before you can run the program you will need to configure it. You can configure it a few ways:
 * By editing the top part of the program itself where there are the following variable definitions.
-* By setting environment variables.
+* By setting environment variables with the same names as the variables in the program.
 * If running it as a standalone program, via some command line options.
 
+:bulb: **NOTE:** The CloudFormation template will prompt for these values when you create the stack and will set the appropriate environment variables for you.
+
 Here is the list of variables, and what they define:
 
 | Variable | Description |Command Line Option|
@@ -78,19 +91,20 @@ You can run the program in "Dry Run" mode by specifying the `-d` (or `--dryRun`)
 messages showing what it would have done, and not really create or delete any CloudWatch alarms.
 
 ### Running as a Lambda function
-If you run the program as a Lambda function, you will want to set the timeout to at least two minutes since some of the API calls
+A CloudFormation template is included in the repo that will do the steps below. Otherwise, here are the steps required to install the program as a Lambda function.
+Create a Lambda function and upload the program as the function code. Set the set the timeout to at least five minutes since some of the API calls
 can take a significant amount of "clock time" to run, especially in distant regions.
 
 Once you have installed the Lambda function it is recommended to set up a scheduled type EventBridge rule so the function will run on a regular basis.
 
 The appropriate permissions will need to be assigned to the Lambda function in order for it to run correctly.
 It doesn't need many permissions. It just needs to be able to:
-* List the FSx for ONTAP file systems
-* List the FSx volume names
-* List the CloudWatch alarms
-* Create CloudWatch alarms
-* Delete CloudWatch alarms
-* Create CloudWatch Log Groups and Log Streams in case you need to diagnose an issue
+* List the FSx for ONTAP file systems.
+* List the FSx volume names.
+* List the CloudWatch alarms.
+* Create CloudWatch alarms.
+* Delete CloudWatch alarms. You can set resource to "arn:aws:cloudwatch:*:${AWS::AccountId}:alarm:FSx-ONTAP-Auto*" to limit the deletion to only the alarms that it created.
+* Create CloudWatch Log Groups and Log Streams in case you need to diagnose an issue.
 
 The following permissions are required to run the script (although you could narrow the "Resource" specification to suit your needs.)
 ```JSON
@@ -105,7 +119,6 @@ The following permissions are required to run the script (although you could nar
                 "fsx:ListTagsForResource",
                 "fsx:DescribeVolumes",
                 "fsx:DescribeFilesystems",
-                "cloudwatch:DeleteAlarms",
                 "cloudwatch:DescribeAlarmsForMetric",
                 "ec2:DescribeRegions",
                 "cloudwatch:DescribeAlarms"
@@ -115,14 +128,22 @@ The following permissions are required to run the script (although you could nar
         {
             "Sid": "VisualEditor1",
             "Effect": "Allow",
+            "Action": [
+                "cloudwatch:DeleteAlarms"
+            ],
+            "Resource": "arn:aws:cloudwatch:*:*:alarm:FSx-ONTAP-Auto*"
+        },
+        {
+            "Sid": "VisualEditor2",
+            "Effect": "Allow",
             "Action": [
                 "logs:CreateLogStream",
                 "logs:PutLogEvents"
             ],
             "Resource": "arn:aws:logs:*:*:log-group:*:log-stream:*"
         },
         {
-            "Sid": "VisualEditor2",
+            "Sid": "VisualEditor3",
             "Effect": "Allow",
             "Action": "logs:CreateLogGroup",
             "Resource": "arn:aws:logs:*:*:log-group:*"
@@ -133,7 +154,7 @@ The following permissions are required to run the script (although you could nar
 
 ### Expected Action
 Once the script has been configured and invoked, it will:
-* Scan for every FSx for ONTAP file systems in every region. For every file system it finds it will:
+* Scan for every FSx for ONTAP file systems in every region. For every file system that it finds it will:
     * Create a CPU utilization CloudWatch alarm, unless the threshold value is set to 100 for the specific alarm.
     * Create a SSD utilization CloudWatch alarm, unless the threshold value is set to 100 for the specific alarm.
 * Scan for every FSx for ONTAP volume in every region. For every volume it finds it will:

Monitoring/auto-add-cw-alarms/auto_add_cw_alarms.py

@@ -4,15 +4,18 @@
 # ONTAP volumes, that don't already have one, that will trigger when the
 # utilization of the volume gets above the threshold defined below. It will
 # also create an alarm that will trigger when the file system reach
-# an average CPU utilization greater than what is specified below.
+# an average CPU utilization greater than what is specified below as well
+# an alarm that will trigger when the SSD utilization is greater than what
+# is specified below.
 #
 # It can either be run as a standalone script, or uploaded as a Lambda
 # function with the thought being that you will create a EventBridge schedule
 # to invoke it periodically.
 #
-# It will scan all regions looking for FSxN volumes, and since CloudWatch
-# can't send SNS messages across regions, it assumes that the specified
-# SNS topic exist in each region for the specified account ID.
+# It will scan all regions looking for FSxN volumes and file systems
+# and since CloudWatch can't send SNS messages across regions, it assumes
+# that the specified SNS topic exist in each region for the specified
+# account ID.
 #
 # Finally, a default volume threshold is defined below. It sets the volume
 # utilization threshold that will cause CloudWatch to send the alarm event
@@ -24,6 +27,9 @@
 # Lastly, you can create an override for the SSD alarm, by creating a tag
 # with the name "SSD_Alarm_Threshold" on the file system resource.
 #
+# Version: %%VERSION%%
+# Date: %%DATE%%
+#
 ################################################################################
 #
 # The following variables effect the behavior of the script. They can be
@@ -64,14 +70,20 @@
 # what you are doing.
 ################################################################################
 #
+# The following is put in front of all alarms so an IAM policy can be create
+# that will allow this script to only be able to delete the alarms it creates.
+# If you change this, you must also change the IAM policy. Note that the
+# Cloudfomration template also assume the value of this variable.
+basePrefix="FSx-ONTAP-Auto"
+#
 # Define the prefix for the volume utilization alarm name for the CloudWatch alarms.
-alarmPrefixVolume="Volume_Utilization_for_volume_"
+alarmPrefixVolume=f"{basePrefix}-Volume_Utilization_for_volume_"
 #
 # Define the prefix for the CPU utilization alarm name for the CloudWatch alarms.
-alarmPrefixCPU="CPU_Utilization_for_fs_"
+alarmPrefixCPU=f"{basePrefix}-CPU_Utilization_for_fs_"
 #
 # Define the prefix for the SSD utilization alarm name for the CloudWatch alarms.
-alarmPrefixSSD="SSD_Utilization_for_fs_"
+alarmPrefixSSD=f"{basePrefix}-SSD_Utilization_for_fs_"
 
 ################################################################################
 # You shouldn't have to modify anything below here.
@@ -531,7 +543,7 @@ def lambda_handler(event, context):
 # This function is used to print out the usage of the script.
 ################################################################################
 def usage():
-    print('Usage: add_cw_alarm [-h|--help] [-d|--dryRun] [[-c|--customerID customerID] [[-a|--accountID aws_account_id] [[-s|--SNSTopic SNS_Topic_Name] [[-r|--region region] [[-C|--CPUThreshold threshold] [[-S|--SSDThreshold threshold] [[-V|--VolumeThreshold threshold] [-F|--FileSystemID FileSystemID]')
+    print('Usage: auto_add_cw_alarms [-h|--help] [-d|--dryRun] [[-c|--customerID customerID] [[-a|--accountID aws_account_id] [[-s|--SNSTopic SNS_Topic_Name] [[-r|--region region] [[-C|--CPUThreshold threshold] [[-S|--SSDThreshold threshold] [[-V|--VolumeThreshold threshold] [-F|--FileSystemID FileSystemID]')
 
 ################################################################################
 # Main logic starts here.