Added some new fields audit fields to accept.

Author: kcantrel

Monitoring/ingest_nas_audit_logs_into_cloudwatch/README.md

@@ -26,20 +26,28 @@ systems that you want to ingest the audit logs from.
       }
 ```
 - You have applied the necessary SACLs to the files you want to audit. The knowledge base article linked above provides guidance on how to do this.
+- Since the Lambda function runs within your VPC it will not have access to the Internet, even if you can access the Internet from the Subnet it run from.
+Therefore, there needs to be an VPC endpoint for all the AWS services that the Lambda function uses. Specifically, the Lambda function needs to be able to access the following services:
+  - FSx.
+  - Secrets Manager.
+  - CloudWatch Logs.
+  - S3 - Note that typically there is a Gateway type VPC endpoint for S3, so you should not need to create a VPC endpoint for S3.
+  - EC2.
 - You have created a role with the necessary permissions to allow the Lambda function to do the following:
 
 <table>
 <tr><th>Service</td><th>Actions</td><th>Resources</th></tr>
-<tr><td>fsx</td><td>fsx:DescribeFileSystems</td><td>*</td></tr>
-<tr><td rowspan="3">ec2</td><td>DescribeNetworkInterfaces</td><td>*</td></tr>
-<tr><td>CreateNetworkInterface</td><td>arn:aws:ec2:*:&lt;accountID&gt;:*</td></tr>
-<tr><td>DeleteNetworkInterface</td><td>arn:aws:ec2:*:&lt;accountID&gt;:*</td></tr>
-<tr><td rowspan="2">logs</td><td>CreateLogStream        </td><td> arn:aws:logs:&lt;region&gt;:&lt;accountID&gt;:log-group:&lt;logGroupName&gt;:* </td></tr>
-<tr><td>PutLogEvents           </td><td> arn:aws:logs:&lt;region&gt;:&lt;accountID&gt;:log-group:&lt;logGroupName&gt;:* </td></tr>
-<tr><td rowspan="3"> s3  </td><td> ListBucket             </td><td> arn:aws:s3:&lt;region&gt;:&lt;accountID&gt;:* </td></tr>
-<tr><td>GetObject              </td><td> arn:aws:s3:&lt;region>:&lt;accountID&gt;:*/* </td></tr>
-<tr><td>PutObject              </td><td> arn:aws:s3:&lt;region>:&lt;accountID&gt;:*/* </td></tr>
-<tr><td>secretsmanager </td><td> GetSecretValue </td><td> arn:aws:secretsmanager:&lt;region&gt;:&lt;accountID&gt;:secret:&lt;secretName&gt;</td></tr>
+<tr><td>Fsx</td><td>fsx:DescribeFileSystems</td><td>\*</td></tr>
+<tr><td rowspan="3">ec2</td><td>DescribeNetworkInterfaces</td><td>\*</td></tr>
+<tr><td>CreateNetworkInterface</td><td>arn:aws:ec2:&lt;region&gt;:&lt;accountID&gt;:\*</td></tr>
+<tr><td>DeleteNetworkInterface</td><td>arn:aws:ec2:&lt;region&gt;:&lt;accountID&gt;:\*</td></tr>
+<tr><td rowspan="3">CloudWatch Logs</td><td>CreateLogGroup</td><td rowspan="3">arn:aws:logs:&lt;region&gt;:&lt;accountID&gt;:log-group:\* </td></tr>
+<tr><td>CreateLogStream</td></tr>
+<tr><td>PutLogEvents</td></tr>
+<tr><td rowspan="3">s3</td><td> ListBucket</td><td> arn:aws:s3:&lt;region&gt;:&lt;accountID&gt;:* </td></tr>
+<tr><td>GetObject</td><td rowspan="2">arn:aws:s3:&lt;region>:&lt;accountID&gt;:*/* </td></tr>
+<tr><td>PutObject</td></tr>
+<tr><td>Secrets Manager</td><td> GetSecretValue </td><td>arn:aws:secretsmanager:&lt;region&gt;:&lt;accountID&gt;:secret:&lt;secretName&gt\*;</td></tr>
 </table>
 Where:
 
@@ -48,6 +56,11 @@ Where:
 - &lt;logGroupName&gt; - is the name of the CloudWatch log group where the audit logs will be ingested.
 - &lt;secretName&gt; - is the name of the secret that contains the credentials for the fsxadmin accounts.
 
+Notes:
+- Since the Lambda function runs within your VPC it needs to be able to create an delete network interfaces.
+- It needs to be able to create a log groups so it can create a log group for the diagnostic output from the Lambda function.
+- Since the ARN of any Secrets Manager secret has random characters at the end of it, you must add the `*` at the end.
+
 ## Deployment
 1. Create a Lambda deployment package by:
     1. Downloading the `ingest_fsx_audit_logs.py` file from this repository and placing it in an empty directory.
@@ -68,14 +81,19 @@ Where:
 
 | Variable | Description |
 | --- | --- |
+| fsxRegion | The region where the FSx for ONTAP file systems are located. |
 | secretArn | The ARN of the secret that contains the credentials for all the FSx for ONTAP file systems you want to gather audit logs from. |
 | secretRegion | The region where the secret is stored. |
 | s3BucketRegion | The region of the S3 bucket where the stats file is stored. |
 | s3BucketName | The name of the S3 bucket where the stats file is stored. |
 | statsName | The name you want to use as the stats file. |
 | logGroupName | The name of the CloudWatch log group to ingest the audit logs into. |
+| volumeName | The name of the volume, on all the FSx for ONTAP file systems, where the audit logs are stored. |
+
+4. Test the Lambda function by clicking on the `Test` tab and then clicking on the `Test` button. You should see "Executing function: succeeded".
+If not, click on the "Details" button to see what errors there are.
 
-4. After you have tested that the Ladmba function is running correctly, add an EventBridge trigger to have it run periodically.
+5. After you have tested that the Ladmba function is running correctly, add an EventBridge trigger to have it run periodically.
 You can do this by clicking on the `Add Trigger` button within the AWS console and selecting `EventBridge (CloudWatch Events)`
 from the dropdown. You can then configure the schedule to run as often as you want. How often depends on how often you have
 set up your FSx for ONTAP file systems to generate audit logs, and how up-to-date you want the CloudWatch logs to be.

Monitoring/ingest_nas_audit_logs_into_cloudwatch/ingest_audit_log.py

@@ -88,7 +88,7 @@ def processFile(ontapAdminServer, headers, volumeUUID, filePath):
     #
     # Number of bytes to read for each API call.
     blockSize=1024*1024
-    
+
     bytesRead = 0
     requestSize = 1   # Set to > 0 to start the loop.
     while requestSize > 0:
@@ -116,7 +116,7 @@ def processFile(ontapAdminServer, headers, volumeUUID, filePath):
         else:
             print(f'API call to {endpoint} failed. HTTP status code: {response.status}.')
             break
-    
+
     f.close()
     #
     # Upload the audit events to CloudWatch.
@@ -135,8 +135,10 @@ def createCWEvent(event):
     # Attributes: A verbose list of strings representing the attributes.
     # DirHandleID: A string of numbers that I'm not sure what they represent.
     # SearchFilter: Always seems to be null.
-    # SearchPattern: Always seems to be set to "Not Present"
-    ignoredDataFields = ["ObjectServer", "HandleID", "InformationRequested", "AccessList", "AccessMask", "DesiredAccess", "Attributes", "DirHandleID", "SearchFilter", "SearchPattern"]
+    # SearchPattern: Always seems to be set to "Not Present".
+    # SubjectPort: Just the TCP port that the user came in on.
+    # OldDirHandle and NewDirHandle: Are the UUIDs of the directory. The OldPath and NewPath are human readable.
+    ignoredDataFields = ["ObjectServer", "HandleID", "InformationRequested", "AccessList", "AccessMask", "DesiredAccess", "Attributes", "DirHandleID", "SearchFilter", "SearchPattern", "SubjectPort", "OldDirHandle", "NewDirHandle"]
     #
     # Convert the timestamp from the XML file to a timestamp in milliseconds.
     # An example format of the time is: 2024-09-22T21:05:27.263864000Z
@@ -180,7 +182,7 @@ def createCWEvent(event):
                     str += ", InformationSet=Null"
                 else:
                     str += f", InformationSet={data['#text']}"
-            elif data['@Name'] in ['ObjectType', 'WriteOffset', 'WriteCount', 'NewSD', 'OldSD']: # These don't require special handling.
+            elif data['@Name'] in ['ObjectType', 'WriteOffset', 'WriteCount', 'NewSD', 'OldSD', 'SubjectUserIsLocal', 'OldPath', 'NewPath', 'OldRotateLimit', 'NewRotateLimit', 'OldLogFormat', 'NewLogFormat', 'OldRetentionDuration', 'NewRetentionDuration', 'AuditGuarantee', 'OldDestinationPath', 'NewDestinationPath']: # These don't require special handling.
                 str += f", {data['@Name']}={data['#text']}"
             else:
                 print(f"Unknown data type: {data['@Name']}")
@@ -201,7 +203,7 @@ def ingestAuditFile(auditLogPath, auditLogName):
 
     if dict.get('Events') == None or dict['Events'].get('Event') == None:
         print(f"No events found in {auditLogName}")
-        return  
+        return
     #
     # Ensure the logstream exists.
     try:
@@ -243,7 +245,8 @@ def checkConfig():
         'secretArn': secretArn if 'secretArn' in globals() else None,
         's3BucketRegion': s3BucketRegion if 's3BucketRegion' in globals() else None,
         's3BucketName': s3BucketName if 's3BucketName' in globals() else None,
-        'statsName': statsName if 'statsName' in globals() else None
+        'statsName': statsName if 'statsName' in globals() else None,
+        'vserverName': vserverName if 'vserverName' in globals() else None
     }
 
     for item in config:
@@ -315,7 +318,7 @@ def lambda_handler(event, context):
     lastFileReadChanged = False
     #
     # Process each FSxN.
-    for fsxn in fsxNs: 
+    for fsxn in fsxNs:
         fsId = fsxn.split('.')[1]
         #
         # Get the password
@@ -331,7 +334,7 @@ def lambda_handler(event, context):
         #
         # Get the volume UUID for the audit_logs volume.
         volumeUUID = None
-        endpoint = f"https://{fsxn}/api/storage/volumes?name={config['volumeName']}"
+        endpoint = f"https://{fsxn}/api/storage/volumes?name={config['volumeName']}&svm={config['vserverName']}"
         response = http.request('GET', endpoint, headers=headersQuery, timeout=5.0)
         if response.status == 200:
             data = json.loads(response.data.decode('utf-8'))
@@ -343,8 +346,7 @@ def lambda_handler(event, context):
             continue
         #
         # Get all the files in the volume that match the audit file pattern.
-        # Since the vserver is part of the filename, it assumes the vserver is 'fsx'.
-        endpoint = f'https://{fsxn}/api/storage/volumes/{volumeUUID}/files?name=audit_fsx_D*&order_by=name%20asc&fields=name'
+        endpoint = f'https://{fsxn}/api/storage/volumes/{volumeUUID}/files?name=audit_{config['vserverName']}_D*&order_by=name%20asc&fields=name'
         response = http.request('GET', endpoint, headers=headersQuery, timeout=5.0)
         data = json.loads(response.data.decode('utf-8'))
         for file in data['records']: