[sysdig] Add support for security event datastream (elastic#13626)

This security events logs provides an overview of your infrastructure,
and allows you to deep-dive into specific security events, distinguish,
false positives, and configure policies to enhance performance.

Sanitized test case inputs were obtained from live Sysdig Secure instance
using the Sysdig Next Gen API.

Author: brijesh-elastic

packages/sysdig/_dev/build/docs/README.md

@@ -1,15 +1,18 @@
 # Sysdig Integration
-This integration allows for the shipping of [Sysdig](https://sysdig.com/) alerts to Elastic for observability and organizational awareness. Alerts can then be analyzed by using either the dashboard included with the integration or via the creation of custom dashboards within Kibana.
+This integration allows for the shipping of [Sysdig](https://sysdig.com/) logs to Elastic for security, observability and organizational awareness. Logs can then be analyzed by using either the dashboard included with the integration or via the creation of custom dashboards within Kibana.
 
 ## Data Streams
-The Sysdig integration collects one type of data stream: alerts.
+The Sysdig integration collects two type of logs:
 
 **Alerts** The Alerts data stream collected by the Sysdig integration is comprised of Sysdig Alerts. See more details about Sysdig Alerts in [Sysdig's Alerts Documentation](https://docs.sysdig.com/en/docs/sysdig-monitor/alerts/). A complete list of potential fields used by this integration can be found in the [Logs reference](#logs-reference)
 
+**Event** The event data stream collected through the Sysdig integration consists of Sysdig Security Events. See more details about Security Events in [Sysdig's Events Feed Documentation](https://docs.sysdig.com/en/docs/sysdig-secure/threats/activity/events-feed/).
+
 ## Requirements
 
-You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
-You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
+### Agent-based installation
+
+Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md).
 
 Sysdig must be configured to output alerts to a supported output channel as defined in [Setup](#setup). The system will only receive common fields output by Sysdig's rules, meaning that if a rule does not include a desired field the rule must be edited in Sysdig to add the field.
 
@@ -25,14 +28,41 @@ The HTTP input allows the Elastic Agent to receive Sysdig Alerts via HTTP webhoo
 
 **Required:** To configure Sysdig to output JSON, you must set up as webhook notification channel as outlined in the [Sysdig Documentation](https://docs.sysdig.com/en/docs/administration/administration-settings/outbound-integrations/notifications-management/set-up-notification-channels/configure-a-webhook-channel/).
 
+### To collect data from the Sysdig Next Gen API:
+
+- Retrieve the API Token by following [Sysdig's API Token Guide](https://docs.sysdig.com/en/retrieve-the-sysdig-api-token).
+
+### Enabling the integration in Elastic:
+
+1. In Kibana navigate to Management > Integrations.
+2. In "Search for integrations" top bar, search for `Sysdig`.
+3. Select the "Sysdig" integration from the search results.
+4. Select "Add Sysdig" to add the integration.
+5. Add all the required integration configuration parameters, including the URL, API Token, Interval, and Initial Interval, to enable data collection.
+6. Select "Save and continue" to save the integration.
+
+**Note**:
+  - The URL may vary depending on your region. Please refer to the [Documentation](https://docs.sysdig.com/en/developer-tools/sysdig-api/#access-the-sysdig-api-using-the-regional-endpoints) to find the correct URL for your region.
+  - If you see an error saying `exceeded maximum number of CEL executions` during data ingestion, it usually means a large volume of data is being processed for the selected time interval. To fix this, try increasing the `Maximum Pages Per Interval` setting in the configuration.
+
 ## Logs Reference
 
-### alerts
+### Alerts
 
 Sysdig alerts can contain a multitude of various fields pertaining to the type of activity on the host machine.
 
+#### Example
+
+{{ event "alerts" }}
+
 {{ fields "alerts" }}
 
-**Example event**
+### Event
+
+This is the `event` dataset.
+
+#### Example
+
+{{event "event"}}
 
-{{ event "alerts" }}
+{{fields "event"}}

packages/sysdig/_dev/deploy/docker/docker-compose.yml

@@ -7,3 +7,16 @@ services:
       - STREAM_PROTOCOL=webhook
       - STREAM_ADDR=http://elastic-agent:9035/
     command: log --start-signal=SIGHUP --delay=5s /sample_logs/sysdig.log
+  sysdig:
+    image: docker.elastic.co/observability/stream:v0.17.1
+    hostname: sysdig
+    ports:
+      - 8090
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: '8090'
+    command:
+      - http-server
+      - --addr=:8090
+      - --config=/files/config.yml

packages/sysdig/_dev/deploy/docker/files/config.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.4.0"
+  changes:
+    - description: Add support for security event datastream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/13626
 - version: "0.3.0"
   changes:
     - description: Update Kibana constraint to support 9.0.0.

packages/sysdig/changelog.yml

@@ -1706,4 +1706,4 @@
             }
         }
     ]
-}
+}

packages/sysdig/data_stream/alerts/_dev/test/pipeline/test-sysdig.log-expected.json

@@ -1,9 +1,9 @@
 {
-    "@timestamp": "2024-09-12T13:06:12.675Z",
+    "@timestamp": "2025-05-15T20:55:10.950Z",
     "agent": {
-        "ephemeral_id": "fe172d2f-7b14-4b87-bc5a-acc14684e4c5",
+        "ephemeral_id": "d1edefb2-dd7d-40f4-bc12-f3e8e0e8a0c8",
         "id": "58014837",
-        "name": "docker-fleet-agent",
+        "name": "elastic-agent-68303",
         "type": "filebeat",
         "version": "8.14.1"
     },
@@ -31,22 +31,22 @@
     },
     "data_stream": {
         "dataset": "sysdig.alerts",
-        "namespace": "15372",
+        "namespace": "85290",
         "type": "logs"
     },
     "ecs": {
         "version": "8.0.0"
     },
     "elastic_agent": {
-        "id": "a2d71da8-f67f-43fa-a895-0251c4a68bb0",
+        "id": "e5c61bf4-097f-42fe-90df-25e8ef080bd8",
         "snapshot": false,
         "version": "8.14.1"
     },
     "event": {
         "agent_id_status": "mismatch",
         "dataset": "sysdig.alerts",
         "id": "17dec715376910362c8c3f62a4ceda2e",
-        "ingested": "2024-09-12T13:06:22Z",
+        "ingested": "2025-05-15T20:55:12Z",
         "kind": "alert",
         "provider": "syscall",
         "severity": 7,
@@ -197,4 +197,4 @@
     "threat.technique.id": [
         "T1136"
     ]
-}
+}

packages/sysdig/data_stream/alerts/sample_event.json

@@ -0,0 +1,3 @@
+fields:
+  tags:
+    - preserve_duplicate_custom_fields