Added the option of copying the audit log files to the S3 bucket.

Author: kcantrel

Monitoring/ingest_nas_audit_logs_into_cloudwatch/README-MANUAL.md

@@ -14,9 +14,12 @@ or by following the manual instructions found in the this file.
 
 ## Prerequisites
 - An FSx for Data ONTAP file system.
-- An S3 bucket to store the "stats" file and a Lambda layer zip file.
-    - You will need to download the [Lambda layer zip file](https://raw.githubusercontent.com/NetApp/FSx-ONTAP-samples-scripts/main/Monitoring/ingest_nas_audit_logs_into_cloudwatch/lambda_layer.zip) from this repo and upload it to the S3 bucket. Be sure to perserve the name `lambda_layer.zip`.
-    - The "stats" file is maintained by the program. It is used to keep track of the last time the Lambda function successfully ingested audit logs from each SVM. Its size will be small (i.e. less than a few megabytes).
+- An S3 bucket to store the "stats" file and optionally a copy of all the raw NAS audit log files. It will also
+hold a Lambda layer file needed to be able to an add Lambda Layer from a CloudFormation script.
+    - You will need to download the [Lambda layer zip file](https://raw.githubusercontent.com/NetApp/FSx-ONTAP-samples-scripts/main/Monitoring/ingest_nas_audit_logs_into_cloudwatch/lambda_layer.zip)
+    from this repo and upload it to the S3 bucket. Be sure to perserve the name `lambda_layer.zip`.
+    - The "stats" file is maintained by the program. It is used to keep track of the last time the Lambda function
+    successfully ingested audit logs from each SVM. Its size will be small (i.e. less than a few megabytes).
 - A CloudWatch log group to ingest the audit logs into. Each audit log file with get its own log stream within the log group.
 - Have NAS auditing configured and enabled on the SVM within a FSx for Data ONTAP file system. **Ensure you have selected the XML format for the audit logs.** Also,
 ensure you have set up a rotation schedule. The program will only act on audit log files that have been finalized, and not the "active" one. You can read this
@@ -79,7 +82,7 @@ and `DeleteNetworkInterface` actions. The correct resource line is `arn:aws:ec2:
 `zip -r ingest_nas_audit_logs.zip .`<br>
 
 2. Within the AWS console, or using the AWS API, create a Lambda function with:
-    1. Python 3.10, or higher, as the runtime.
+    1. Python 3.11, or higher, as the runtime.
     1. Set the permissions to the role created above.
     1. Under `Additional Configurations` select `Enable VPC` and select a VPC and Subnet that will have access to all the FSx for ONTAP
 file system management endpoints that you want to gather audit logs from. Also, select a Security Group that allows TCP port 443 outbound.
@@ -96,6 +99,7 @@ process a lot of audit entries and/or process a lot of SVMs.
 | secretArn | The ARN of the secret that contains the credentials for all the FSx for ONTAP file systems you want to gather audit logs from. |
 | s3BucketRegion | The region of the S3 bucket where the stats file is stored. |
 | s3BucketName | The name of the S3 bucket where the stats file is stored. |
+| copyToS3 | Set to `true` if you want to copy the raw audit log files to the S3 bucket.|
 | statsName | The name you want to use as the stats file. |
 | logGroupName | The name of the CloudWatch log group to ingest the audit logs into. |
 | volumeName | The name of the volume, on all the FSx for ONTAP file systems, where the audit logs are stored. |
@@ -122,4 +126,4 @@ Unless required by applicable law or agreed to in writing, software distributed
 
 See the License for the specific language governing permissions and limitations under the License.
 
-© 2024 NetApp, Inc. All Rights Reserved.
+© 2025 NetApp, Inc. All Rights Reserved.

Monitoring/ingest_nas_audit_logs_into_cloudwatch/README.md

@@ -15,9 +15,12 @@ or by following the manual instructions found in the [README-MANUEL.md](README-M
 
 ## Prerequisites
 - An FSx for Data ONTAP file system.
-- An S3 bucket to store the "stats" file and a Lambda layer zip file.
-    - You will need to download the [Lambda layer zip file](https://raw.githubusercontent.com/NetApp/FSx-ONTAP-samples-scripts/main/Monitoring/ingest_nas_audit_logs_into_cloudwatch/lambda_layer.zip) from this repo and upload it to the S3 bucket. Be sure to preserve the name `lambda_layer.zip`.
-    - The "stats" file is maintained by the program. It is used to keep track of the last time the Lambda function successfully ingested audit logs from each SVM. Its size will be small (i.e. less than a few megabytes).
+- An S3 bucket to store the "stats" file and optionally a copy of all the raw NAS audit log files. It will also
+hold a Lambda layer file needed to be able to an add Lambda Layer from a CloudFormation script.
+    - You will need to download the [Lambda layer zip file](https://raw.githubusercontent.com/NetApp/FSx-ONTAP-samples-scripts/main/Monitoring/ingest_nas_audit_logs_into_cloudwatch/lambda_layer.zip)
+     from this repo and upload it to the S3 bucket. Be sure to preserve the name `lambda_layer.zip`.
+    - The "stats" file is maintained by the program. It is used to keep track of the last time the Lambda function successfully
+    ingested audit logs from each SVM. Its size will be small (i.e. less than a few megabytes).
 - A CloudWatch log group to ingest the audit logs into. Each audit log file will get its own log stream within the log group.
 - Have NAS auditing configured and enabled on the SVM within a FSx for Data ONTAP file system. **Ensure you have selected the XML format for the audit logs.** Also,
 ensure you have set up a rotation schedule. The program will only act on audit log files that have been finalized, and not the "active" one. You can read this
@@ -92,11 +95,12 @@ and `DeleteNetworkInterface` actions. The correct resource line is `arn:aws:ec2:
     |lambdaSecruityGroupsIds|Yes|Select the security groups that you want the Lambda function associated with. The security group must allow outbound traffic on TCP port 443. Inbound rules don't matter since the Lambda function is not accessible from a network.|
     |s3BucketName|Yes|The name of the S3 bucket where the stats file is stored. This bucket must already exist.|
     |s3BucketRegion|Yes|The region of the S3 bucket resides.|
+    |copyToS3|No|If set to `true` it will copy the audit logs to the S3 bucket specified in `s3BucketName`.|
     |secretArn|Yes|The ARN to the secret that contains the credentials for the FSxN file systems that you want to ingest audit logs from.|
     |createWatchdogAlarm|No|If set to `true` it will create a CloudWatch alarm that will alert you if the Lambda function throws in error.|
     |snsTopicArn|No|The ARN of the SNS topic to send the alarm to. This is required if `createWatchdogAlarm` is set to `true`.|
     |lambdaRoleArn|No|The ARN of the role that the Lambda function will use. If not provided, the CloudFormation script will create a role for you.|
-    |schedulreRoleArn|No|The ARN of the role that the EventBridge scheduler will run as. If not provided, the CloudFormation script will create a role for you.|
+    |schedulerRoleArn|No|The ARN of the role that the EventBridge scheduler will run as. If not provided, the CloudFormation script will create a role for you.|
     |createFsxEndpoint|No|If set to `true` it will create the VPC endpoints for the FSx service|
     |createCloudWatchLogsEndpoint|No|If set to `true` it will create the VPC endpoints for the CloudWatch Logs service|
     |createSecretsManagerEndpoint|No|If set to `true` it will create the VPC endpoints for the Secrets Manager service|

Monitoring/ingest_nas_audit_logs_into_cloudwatch/cloudformation-template.yaml

@@ -14,6 +14,7 @@ Metadata:
           - lambdaSecurityGroupIds
           - s3BucketName
           - s3BucketRegion
+          - copyToS3
           - secretArn
           - createWatchdogAlarm
           - snsTopicArn
@@ -57,6 +58,12 @@ Parameters:
     Description: "The AWS region where the s3 bucket resides."
     Type: String
 
+  copyToS3:
+    Description: "Set to 'true' if you to copy the audit log files to the S3 bucket as well as sending the individual events to the CloudWatch log stream."
+    Type: String
+    Default: "false"
+    AllowedValues: ["true", "false"]
+
   secretArn:
     Description: "The ARN of the secret that holds the FSxN credentials to use."
     Type: String
@@ -301,6 +308,7 @@ Resources:
           logGroupName: !Ref logGroupName
           s3BucketName: !Ref s3BucketName
           s3BucketRegion: !Ref s3BucketRegion
+          copyToS3: !Ref copyToS3
           secretArn: !Ref secretArn
           statsName: "lastFileRead"
           volumeName: !Ref volumeName
@@ -397,7 +405,7 @@ Resources:
           # APIs, and then calls the ingestAuditFile function to upload the audit
           # log entires to the CloudWatch log group.
           ################################################################################
-          def processFile(ontapAdminServer, headers, volumeUUID, filePath):
+          def readFile(ontapAdminServer, headers, volumeUUID, filePath):
               global http
               #
               # Create the tempoary file to hold the contents from the ONTAP/FSxN file.
@@ -407,6 +415,7 @@ Resources:
               # Number of bytes to read for each API call.
               blockSize=1024*1024
           
+              failed = False
               bytesRead = 0
               requestSize = 1   # Set to > 0 to start the loop.
               while requestSize > 0:
@@ -433,12 +442,17 @@ Resources:
                               f.write(part.content)
                   else:
                       print(f'Warning: API call to {endpoint} failed. HTTP status code: {response.status}.')
+                      filed = True
                       break
           
               f.close()
-              #
-              # Upload the audit events to CloudWatch.
-              ingestAuditFile(tmpFileName, filePath)
+              if failed:
+                  os.remove(tmpFileName)
+                  return None
+              else:
+                  return tmpFileName
+                  #
+                  # Upload the audit events to CloudWatch.
           
           ################################################################################
           # This functions converts the timestamp from the XML file to a timestamp in
@@ -592,14 +606,21 @@ Resources:
                   'secretArn': secretArn if 'secretArn' in globals() else None,                  # pylint: disable=E0602
                   's3BucketRegion': s3BucketRegion if 's3BucketRegion' in globals() else None,   # pylint: disable=E0602
                   's3BucketName': s3BucketName if 's3BucketName' in globals() else None,         # pylint: disable=E0602
-                  'statsName': statsName if 'statsName' in globals() else None                   # pylint: disable=E0602
+                  'statsName': statsName if 'statsName' in globals() else None,                  # pylint: disable=E0602
+                  'copyToS3': copyToS3 if 'copyToS3' in globals() else None                      # pylint: disable=E0602
               }
+              optionalConfig = ['copyToS3']
           
               for item in config:
                   if config[item] == None:
                       config[item] = os.environ.get(item)
-                  if config[item] == None:
+                  if item not in optionalConfig and config[item] == None:
                       raise Exception(f"{item} is not set.")
+          
+              if config['copyToS3'] == None:
+                  config['copyToS3'] = False
+              else:
+                  config['copyToS3'] = config['copyToS3'].lower() == 'true'
               #
               # To be backwards compatible, load the vserverName.
               config['vserverName'] = vserverName if 'vserverName' in globals() else os.environ.get('vserverName')  # pylint: disable=E0602
@@ -729,14 +750,17 @@ Resources:
                               for file in data['records']:
                                   filePath = file['name']
                                   if lastFileRead.get(fsxn) is None or lastFileRead[fsxn].get(vserverName) is None or getEpoch(filePath) > lastFileRead[fsxn][vserverName]:
-                                      #
-                                      # Process the file.
-                                      processFile(fsxn, headersDownload, volumeUUID, filePath)
-                                      if lastFileRead.get(fsxn) is None:
-                                          lastFileRead[fsxn] = {vserverName: getEpoch(filePath)}
-                                      else:
-                                          lastFileRead[fsxn][vserverName] = getEpoch(filePath)
-                                      s3Client.put_object(Key=config['statsName'], Bucket=config['s3BucketName'], Body=json.dumps(lastFileRead).encode('UTF-8'))
+                                      localFileName = readFile(fsxn, headersDownload, volumeUUID, filePath)
+                                      if localFileName is not None:
+                                          if config['copyToS3']:
+                                              s3Client.upload_file(localFileName, config['s3BucketName'], filePath)
+                                          ingestAuditFile(localFileName, filePath)
+                                          if lastFileRead.get(fsxn) is None:
+                                              lastFileRead[fsxn] = {vserverName: getEpoch(filePath)}
+                                          else:
+                                              lastFileRead[fsxn][vserverName] = getEpoch(filePath)
+                                          s3Client.put_object(Key=config['statsName'], Bucket=config['s3BucketName'], Body=json.dumps(lastFileRead).encode('UTF-8'))
+                                          os.remove(localFileName)
                           #
                           # Get the next set of SVMs.
                           if svmsData['_links'].get('next') != None:

Monitoring/ingest_nas_audit_logs_into_cloudwatch/ingest_audit_log.py

@@ -89,7 +89,7 @@ def getEpoch(filename):
 # APIs, and then calls the ingestAuditFile function to upload the audit
 # log entires to the CloudWatch log group.
 ################################################################################
-def processFile(ontapAdminServer, headers, volumeUUID, filePath):
+def readFile(ontapAdminServer, headers, volumeUUID, filePath):
     global http
     #
     # Create the tempoary file to hold the contents from the ONTAP/FSxN file.
@@ -99,6 +99,7 @@ def processFile(ontapAdminServer, headers, volumeUUID, filePath):
     # Number of bytes to read for each API call.
     blockSize=1024*1024
 
+    failed = False
     bytesRead = 0
     requestSize = 1   # Set to > 0 to start the loop.
     while requestSize > 0:
@@ -125,12 +126,17 @@ def processFile(ontapAdminServer, headers, volumeUUID, filePath):
                     f.write(part.content)
         else:
             print(f'Warning: API call to {endpoint} failed. HTTP status code: {response.status}.')
+            filed = True
             break
 
     f.close()
-    #
-    # Upload the audit events to CloudWatch.
-    ingestAuditFile(tmpFileName, filePath)
+    if failed:
+        os.remove(tmpFileName)
+        return None
+    else:
+        return tmpFileName
+        #
+        # Upload the audit events to CloudWatch.
 
 ################################################################################
 # This functions converts the timestamp from the XML file to a timestamp in
@@ -284,14 +290,21 @@ def checkConfig():
         'secretArn': secretArn if 'secretArn' in globals() else None,                  # pylint: disable=E0602
         's3BucketRegion': s3BucketRegion if 's3BucketRegion' in globals() else None,   # pylint: disable=E0602
         's3BucketName': s3BucketName if 's3BucketName' in globals() else None,         # pylint: disable=E0602
-        'statsName': statsName if 'statsName' in globals() else None                   # pylint: disable=E0602
+        'statsName': statsName if 'statsName' in globals() else None,                  # pylint: disable=E0602
+        'copyToS3': copyToS3 if 'copyToS3' in globals() else None                      # pylint: disable=E0602
     }
+    optionalConfig = ['copyToS3']
 
     for item in config:
         if config[item] == None:
             config[item] = os.environ.get(item)
-        if config[item] == None:
+        if item not in optionalConfig and config[item] == None:
             raise Exception(f"{item} is not set.")
+
+    if config['copyToS3'] == None:
+        config['copyToS3'] = False
+    else:
+        config['copyToS3'] = config['copyToS3'].lower() == 'true'
     #
     # To be backwards compatible, load the vserverName.
     config['vserverName'] = vserverName if 'vserverName' in globals() else os.environ.get('vserverName')  # pylint: disable=E0602
@@ -421,14 +434,17 @@ def lambda_handler(event, context):     # pylint: disable=W0613
                     for file in data['records']:
                         filePath = file['name']
                         if lastFileRead.get(fsxn) is None or lastFileRead[fsxn].get(vserverName) is None or getEpoch(filePath) > lastFileRead[fsxn][vserverName]:
-                            #
-                            # Process the file.
-                            processFile(fsxn, headersDownload, volumeUUID, filePath)
-                            if lastFileRead.get(fsxn) is None:
-                                lastFileRead[fsxn] = {vserverName: getEpoch(filePath)}
-                            else:
-                                lastFileRead[fsxn][vserverName] = getEpoch(filePath)
-                            s3Client.put_object(Key=config['statsName'], Bucket=config['s3BucketName'], Body=json.dumps(lastFileRead).encode('UTF-8'))
+                            localFileName = readFile(fsxn, headersDownload, volumeUUID, filePath)
+                            if localFileName is not None:
+                                if config['copyToS3']:
+                                    s3Client.upload_file(localFileName, config['s3BucketName'], filePath)
+                                ingestAuditFile(localFileName, filePath)
+                                if lastFileRead.get(fsxn) is None:
+                                    lastFileRead[fsxn] = {vserverName: getEpoch(filePath)}
+                                else:
+                                    lastFileRead[fsxn][vserverName] = getEpoch(filePath)
+                                s3Client.put_object(Key=config['statsName'], Bucket=config['s3BucketName'], Body=json.dumps(lastFileRead).encode('UTF-8'))
+                                os.remove(localFileName)
                 #
                 # Get the next set of SVMs.
                 if svmsData['_links'].get('next') != None: