feat: de-dupe audit logs in collection pipeline

This change adjusts the NATS stream configuration to support a 10 minute
de-duplication window. The NATS message ID has been set to the audit log
ID since the ID will be unique across all audit logs.

Author: scotwells

config/components/nats-streams/audit-stream.yaml

@@ -34,7 +34,11 @@ spec:
   allowDirect: true
 
   # Deduplication window - prevents duplicate messages
-  duplicateWindow: 2m
+  # Extended to 10 minutes to handle webhook retries with exponential backoff
+  # and Vector restarts. NATS uses message IDs (set to Kubernetes auditID) to
+  # detect duplicates within this window, providing pipeline-level de-duplication
+  # before events reach ClickHouse.
+  duplicateWindow: 10m
 
   # Maximum number of consumers
   maxConsumers: 10

config/components/vector-sidecar/vector-sidecar-hr.yaml

@@ -121,6 +121,17 @@ spec:
           healthcheck:
             enabled: true
 
+          # NATS message ID for de-duplication
+          # Uses the Kubernetes auditID as the NATS message ID to enable
+          # JetStream's duplicate detection within the duplicateWindow (10m)
+          # This prevents duplicate events from webhook retries or Vector restarts
+          message_id:
+            type: vrl
+            source: |
+              # Use the Kubernetes audit ID as the NATS message ID
+              # NATS will reject duplicates within the duplicateWindow
+              to_string!(.auditID)
+
           # Buffer for durability
           # 10GB disk buffer to survive NATS outages
           buffer: