Add new AWS Config datastream.

.github/CODEOWNERS

@@ -38,6 +38,7 @@
 /packages/aws/data_stream/cloudtrail @elastic/obs-infraobs-integrations
 /packages/aws/data_stream/cloudwatch_logs @elastic/obs-ds-hosted-services
 /packages/aws/data_stream/cloudwatch_metrics @elastic/obs-ds-hosted-services
+/packages/aws/data_stream/config @elastic/security-service-integrations
 /packages/aws/data_stream/dynamodb @elastic/obs-infraobs-integrations
 /packages/aws/data_stream/ebs @elastic/obs-ds-hosted-services
 /packages/aws/data_stream/ec2_logs @elastic/obs-ds-hosted-services

packages/aws/_dev/build/docs/config.md

@@ -0,0 +1,79 @@
+# Config
+
+[AWS Config](https://docs.aws.amazon.com/config/) provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time.
+
+Use this integration to collect and parse data from your AWS Config APIs. Visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
+
+**IMPORTANT: Extra AWS charges on API requests will be generated by this integration. Check [API Requests](https://www.elastic.co/docs/current/integrations/aws#api-requests) for more details.**
+
+## Data streams
+
+The AWS Config integration collects one type of data: logs.
+
+**Logs** help you keep a record of the findings in AWS Config, allowing you to track and audit compliance status of your resources.
+
+Within the `config` data stream, we first retrieve all config rules using the [DescribeConfigRules](https://docs.aws.amazon.com/config/latest/APIReference/API_DescribeConfigRules.html) action. Then, for each specific config rule, we fetch its evaluation results using the [GetComplianceDetailsByConfigRule](https://docs.aws.amazon.com/config/latest/APIReference/API_GetComplianceDetailsByConfigRule.html) action. These evaluation results enrich their respective config rules, ultimately producing a finding log.
+
+See more details in the [Logs reference](#logs-reference).
+
+## Requirements
+
+### Agentless Enabled Integration
+Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and the [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html).
+Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features.
+
+### Agent Based Installation
+- Elastic Agent must be installed
+- You can install only one Elastic Agent per host.
+- Elastic Agent is required to stream data from the REST API and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.
+
+Before using any AWS integration you will need:
+
+* **AWS Credentials** to connect with your AWS account.
+* **AWS Permissions** to make sure the user you're using to connect has permission to share the relevant data.
+
+For more details about these requirements, please take a look at the [AWS integration documentation](https://docs.elastic.co/integrations/aws#requirements).
+
+## Setup
+
+Use this integration if you only need to collect data from the AWS Config service.
+
+### To collect data from AWS Config APIs, users must have an Access Key and a Secret Key. To create API token follow below steps:
+
+1. Login to https://console.aws.amazon.com/.
+2. Go to https://console.aws.amazon.com/iam/ to access the IAM console.
+3. On the navigation menu, choose Users.
+4. Choose your IAM user name.
+5. Select Create access key from the Security Credentials tab.
+6. To see the new access key, choose Show.
+
+### Enabling the integration in Elastic:
+
+1. In Kibana navigate to Management > Integrations.
+2. In "Search for integrations" top bar, search for `AWS Config`.
+3. Select the "AWS Config" integration from the search results.
+4. Select "Add AWS Config" to add the integration.
+5. Add all the required integration configuration parameters, including the aws_region to enable data collection.
+6. Select "Save and continue" to save the integration.
+
+**Note** : For the current integration package, it is compulsory to add Secret Access Key and Access Key ID.
+
+## Logs reference
+
+### Config
+
+This is the `config` dataset.
+
+#### Example
+
+An example event for `config` looks as following:
+
+{{event "config"}}
+
+**ECS Field Reference**
+
+Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
+
+#### Exported fields
+
+{{fields "config"}}

packages/aws/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "3.4.0"
+  changes:
+    - description: Add new AWS Config datastream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/13830
 - version: "3.3.3"
   changes:
     - description: Update README - Document ingested log types of AWS Network Firewall.

packages/aws/data_stream/config/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,17 @@
+version: '2.3'
+services:
+  config:
+    image: docker.elastic.co/observability/stream:v0.17.0
+    hostname: config.xxxx.amazonaws.com
+    ports:
+      - 443
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: "443"
+    command:
+      - http-server
+      - --addr=:443
+      - --config=/files/config.yml
+      - --tls-cert=/files/certificate.crt
+      - --tls-key=/files/private.key

packages/aws/data_stream/config/_dev/deploy/docker/files/certificate.crt

@@ -0,0 +1,23 @@
+-----BEGIN CERTIFICATE-----
+MIID4TCCAsmgAwIBAgIUBdbnNWnUUMxH4YR2GEfqbZN60m8wDQYJKoZIhvcNAQEL
+BQAwgYUxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQH
+DA1TYW4gRnJhbmNpc2NvMRMwEQYDVQQKDApNeSBDb21wYW55MRAwDgYDVQQLDAdN
+eSBVbml0MSIwIAYDVQQDDBljb25maWcueHh4eC5hbWF6b25hd3MuY29tMB4XDTI1
+MDUwNjA2Mjc0M1oXDTI2MDUwNjA2Mjc0M1owgYUxCzAJBgNVBAYTAlVTMRMwEQYD
+VQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMRMwEQYDVQQK
+DApNeSBDb21wYW55MRAwDgYDVQQLDAdNeSBVbml0MSIwIAYDVQQDDBljb25maWcu
+eHh4eC5hbWF6b25hd3MuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
+AQEAwzflKp5qNhAy07KZDXq0cZ0w6HEPWGuPh+1qK+ZxDqbyPBYXtNJv4XXOKmJw
+nVH+XriwL9PA6T/R96zIr5GR7mT3lKa4QGdlOLAqFQjDs8HGNpePDLJImPp4Kktw
+svRXrvfgsNVGy7qejT2ufK0OgszpVDSH4NaXXdwpGOuXF0e5qLox1DFiUj4N9ntA
+Zqw/A9VLDvwuLveO1X4aI3a9xfTuSrLiRvED57rqW+3YJvOEru0SZn7F1pY9+V9j
+kPgyTlzK6sv3xbXkt17lK4wzUvPzi0wxDRYuBNmlhZ4oq2ysMNAVbcEMzebOQ1jf
+u9LTKeBn4cmltXTH9y9RfMsOkwIDAQABo0cwRTAkBgNVHREEHTAbghljb25maWcu
+eHh4eC5hbWF6b25hd3MuY29tMB0GA1UdDgQWBBSc5srLUGJ3wzkdpurLxqRL8dQQ
+RDANBgkqhkiG9w0BAQsFAAOCAQEAAz/TkbmDvstJg6Fc0AUWdR4YDN9N4pQXBCJ/
+C4aB+JVHoJfWD3tmXZ1y7or9/q/UXxfutUzSpXzFOq5gG3mlduaDDfgz54tr7Fzf
+FjMJMjNwuIxBILi2e6uJAwxuJRn7SmMtNv46PswR8N3XvM4kyTt/11nEB1YE2yr6
+46XFW+1db4ds8lnwmdRYM0j6gCe3jswZ6M3mhF5SNCrp+LCb70LUnsLSnh7LdPp+
+xR+OxIwWBtgT2iL5ArdWJr219Ey40G0bSVPZmtlED4Hi2oue5KIt3MnVzpxIsu8p
+UrA2ofnvUjhhk6CKjFBTE7BnkH9u6NAZseQLA42vtHvgm8tu5g==
+-----END CERTIFICATE-----

packages/aws/data_stream/config/_dev/deploy/docker/files/config.yml

@@ -0,0 +1,217 @@
+rules:
+  - path: /
+    methods: ["POST"]
+    request_headers:
+      Content-Type:
+        - "application/x-amz-json-1.1"
+      X-Amz-Target:
+        - "StarlingDoveService.DescribeConfigRules"
+    request_body: '{"NextToken":"page2"}'
+    responses:
+      - status_code: 200
+        body: |-
+          {{ minify_json `
+          {
+            "ConfigRules": [
+              {
+                "ConfigRuleArn": "arn:aws:config:us-east-1:11223344556:config-rule/config-rule-id3",
+                "ConfigRuleId": "config-rule-id3",
+                "ConfigRuleName": "required-tags",
+                "ConfigRuleState": "ACTIVE",
+                "Description": "Checks whether your resources have the tags that you specify.",
+                "EvaluationModes": [
+                  {
+                    "Mode": "DETECTIVE"
+                  }
+                ],
+                "InputParameters": "{\"tag1Key\":\"k1\",\"tag1Value\":\"v1\"}",
+                "Scope": {
+                  "ComplianceResourceTypes": [
+                    "AWS::EC2::Instance"
+                  ]
+                },
+                "Source": {
+                  "Owner": "AWS",
+                  "SourceIdentifier": "REQUIRED_TAGS"
+                }
+              }
+            ]
+          }
+          `}}
+  - path: /
+    methods: ["POST"]
+    request_headers:
+      Content-Type:
+        - "application/x-amz-json-1.1"
+      X-Amz-Target:
+        - "StarlingDoveService.DescribeConfigRules"
+    responses:
+      - status_code: 200
+        body: |-
+          {{ minify_json `
+          {
+            "ConfigRules": [
+              {
+                "ConfigRuleArn": "arn:aws:config:us-east-1:11223344556:config-rule/config-rule-id1",
+                "ConfigRuleId": "config-rule-id1",
+                "ConfigRuleName": "access-keys-rotated",
+                "ConfigRuleState": "ACTIVE",
+                "Description": "Checks whether the active access keys are rotated within the number of days specified in maxAccessKeyAge. The rule is non-compliant if the access keys have not been rotated for more than maxAccessKeyAge number of days.",
+                "EvaluationModes": [
+                  {
+                    "Mode": "DETECTIVE"
+                  }
+                ],
+                "InputParameters": "{\"maxAccessKeyAge\":\"90\"}",
+                "MaximumExecutionFrequency": "TwentyFour_Hours",
+                "Source": {
+                  "Owner": "AWS",
+                  "SourceIdentifier": "ACCESS_KEYS_ROTATED"
+                }
+              },
+              {
+                "ConfigRuleArn": "arn:aws:config:us-east-1:11223344556:config-rule/config-rule-id2",
+                "ConfigRuleId": "config-rule-id2",
+                "ConfigRuleName": "account-part-of-organizations",
+                "ConfigRuleState": "ACTIVE",
+                "Description": "Rule checks whether AWS account is part of AWS Organizations. The rule is NON_COMPLIANT if the AWS account is not part of AWS Organizations or AWS Organizations master account ID does not match rule parameter MasterAccountId.",
+                "EvaluationModes": [
+                  {
+                    "Mode": "DETECTIVE"
+                  }
+                ],
+                "InputParameters": "{}",
+                "MaximumExecutionFrequency": "TwentyFour_Hours",
+                "Source": {
+                  "Owner": "AWS",
+                  "SourceIdentifier": "ACCOUNT_PART_OF_ORGANIZATIONS"
+                }
+              }
+            ],
+            "NextToken": "page2"
+          }
+          `}}
+  - path: /
+    methods: ["POST"]
+    request_headers:
+      Content-Type:
+        - "application/x-amz-json-1.1"
+      X-Amz-Target:
+        - "StarlingDoveService.GetComplianceDetailsByConfigRule"
+    request_body: '{"ConfigRuleName":"access-keys-rotated","Limit":2}'
+    responses:
+      - status_code: 200
+        body: |-
+          {{ minify_json `
+          {
+            "EvaluationResults": [
+              {
+                "ComplianceType": "COMPLIANT",
+                "ConfigRuleInvokedTime": 1444799479.852,
+                "EvaluationResultIdentifier": {
+                  "EvaluationResultQualifier": {
+                    "ConfigRuleName": "access-keys-rotated",
+                    "EvaluationMode": "DETECTIVE",
+                    "ResourceId": "i-0a4468fbfafeeg20h",
+                    "ResourceType": "AWS::EC2::Instance"
+                  },
+                  "OrderingTimestamp": 1443541951.883
+                },
+                "ResultRecordedTime": 1444799480.061
+              },
+              {
+                "ComplianceType": "COMPLIANT",
+                "ConfigRuleInvokedTime": 1544799479.852,
+                "EvaluationResultIdentifier": {
+                  "EvaluationResultQualifier": {
+                    "ConfigRuleName": "access-keys-rotated",
+                    "EvaluationMode": "DETECTIVE",
+                    "ResourceId": "i-0a4468fbfafeeg30h",
+                    "ResourceType": "AWS::EC2::Instance"
+                  },
+                  "OrderingTimestamp": 1543541951.883
+                },
+                "ResultRecordedTime": 1544799480.061
+              }
+            ],
+            "NextToken": "page2"
+          }
+          `}}
+  - path: /
+    methods: ["POST"]
+    request_headers:
+      Content-Type:
+        - "application/x-amz-json-1.1"
+      X-Amz-Target:
+        - "StarlingDoveService.GetComplianceDetailsByConfigRule"
+    request_body: '{"ConfigRuleName":"access-keys-rotated","Limit":2,"NextToken":"page2"}'
+    responses:
+      - status_code: 200
+        body: |-
+          {{ minify_json `
+          {
+            "EvaluationResults": [
+              {
+                "ComplianceType": "NON_COMPLIANT",
+                "ConfigRuleInvokedTime": 1644799479.852,
+                "EvaluationResultIdentifier": {
+                  "EvaluationResultQualifier": {
+                    "ConfigRuleName": "access-keys-rotated",
+                    "EvaluationMode": "DETECTIVE",
+                    "ResourceId": "i-0a4468fbfafeeg30h",
+                    "ResourceType": "AWS::EC2::Instance"
+                  },
+                  "OrderingTimestamp": 1643541951.883
+                },
+                "ResultRecordedTime": 1644799480.061
+              }
+            ]
+          }
+          `}}
+  - path: /
+    methods: ["POST"]
+    request_headers:
+      Content-Type:
+        - "application/x-amz-json-1.1"
+      X-Amz-Target:
+        - "StarlingDoveService.GetComplianceDetailsByConfigRule"
+    request_body: '{"ConfigRuleName":"account-part-of-organizations","Limit":2}'
+    responses:
+      - status_code: 200
+        body: |-
+          {{ minify_json `
+          {
+            "EvaluationResults": []
+          }
+          `}}
+  - path: /
+    methods: ["POST"]
+    request_headers:
+      Content-Type:
+        - "application/x-amz-json-1.1"
+      X-Amz-Target:
+        - "StarlingDoveService.GetComplianceDetailsByConfigRule"
+    request_body: '{"ConfigRuleName":"required-tags","Limit":2}'
+    responses:
+      - status_code: 200
+        body: |-
+          {{ minify_json `
+          {
+            "EvaluationResults": [
+              {
+                "ComplianceType": "NON_COMPLIANT",
+                "ConfigRuleInvokedTime": 1844799479.852,
+                "EvaluationResultIdentifier": {
+                  "EvaluationResultQualifier": {
+                    "ConfigRuleName": "required-tags",
+                    "EvaluationMode": "PROACTIVE",
+                    "ResourceId": "i-0a4468fbfafeeg41h",
+                    "ResourceType": "AWS::EC2::Instance"
+                  },
+                  "OrderingTimestamp": 1843541951.883
+                },
+                "ResultRecordedTime": 1844799480.061
+              }
+            ]
+          }
+          `}}

packages/aws/data_stream/config/_dev/deploy/docker/files/private.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDDN+Uqnmo2EDLT
+spkNerRxnTDocQ9Ya4+H7Wor5nEOpvI8Fhe00m/hdc4qYnCdUf5euLAv08DpP9H3
+rMivkZHuZPeUprhAZ2U4sCoVCMOzwcY2l48MskiY+ngqS3Cy9Feu9+Cw1UbLup6N
+Pa58rQ6CzOlUNIfg1pdd3CkY65cXR7moujHUMWJSPg32e0BmrD8D1UsO/C4u947V
+fhojdr3F9O5KsuJG8QPnuupb7dgm84Su7RJmfsXWlj35X2OQ+DJOXMrqy/fFteS3
+XuUrjDNS8/OLTDENFi4E2aWFniirbKww0BVtwQzN5s5DWN+70tMp4GfhyaW1dMf3
+L1F8yw6TAgMBAAECggEABVqznOMia6AvHLJR01ZRu6oBjOaI8rZkhehjmJel8y5u
+B2rdtJZu/iKSiIQRrabxkJyFLJKkwGEBO8dP68zU0VKQndGizRVo59ChHtmSMIx4
+iMfIYyNCrXt1L0fJbAcanpBq5765xd20+o++COpgMwM2xRn5vhd0qFzg/a98geVn
+9TpZs2a2usDwqINw0S1W0v87zaia6ZQk/oUoYljF9Vhbd3GJAoZU9Xc/PiZH/qud
+1/7tPn2v8X+Ox8KkPIMgskKQc+2hUX75lES/XeJz3/CQ5YroCiNwgAL2VYMpof3W
+APJzJ2ilhVSvZf51OX1Rd4qQRwdKSU7cnudQ/qZu8QKBgQDhiJG5rXb+acD/vCMp
+iw8GnxZ9N9OGRIMayzXhZ2bS1YSFV06ugqeHuKKrZlpNAztA/2pJKJ3qgVCQZmJ4
+WASEsX0nsaGISQgvbc6fyWUvVnpo/wMayi/rRZ/KPQCv5l4qO6vitffG+985nKEf
+z0sK09rSBmb4d8V5D+uFk8RrIwKBgQDdlva0JT6uCtvZQRhPKw/wfqhUg5JAEMH9
+mC6ZgiVhM8JB0EjdPPB5k/hQa6Odza6mJy0oUZ0Aw1zaUenDiV4JRqwfV+qq27PA
+E5xtvZyeGZuskicbqroFA85+be39V0FaLp1P3OpAxIqlSfObL+OUFn+F0lUH4Jea
+/TTbXhr90QKBgAkpKPInz5uJ5CL/G1aGpXeZYqp3aAoeIk0mT+v17UFHFvjrkPCZ
+sgBbSZA4uhZCuVdsiH6sPa3WztTus7U7rgNNyk2gc3U7si9rAGeRIKEJnDNDmHaw
+G74st87ZJ3v9mXmRruuohIX6mRiX+ht2qg+oh0zcobYZ91VxhhmI5QONAoGBAMc8
+q4mCS39ViCMpYmAcifJlD5kdy+wKpUINCSlBWbayQSHH0xwJZPcL0qMMhUqn2zbN
+1s5/wzkib2RlblhANOsGPlDYTcleTZqQh4Askpuczto1dzBrK2LC73HCCdBWGg6q
+Bwv9yCqADWFcwspwHqHSMMr0OTwh9m6G6HWtgXthAoGARN+NgbT8aTMjpPGEMebi
+mCsSCQJ+nGnxyLXPaBlcxe0N43MEcfOPU1g0BZkXhzu+gAkRh92zkf67jZwVYBUi
+4p3lMblvbDi+/nYdKuF1XK5OhN/Y+WwqMYmdYdO0l6NEF+H5ljL60e4+bKoKhyv2
+XEbVW0ymp4YZQs8jznxgBlo=
+-----END PRIVATE KEY-----

packages/aws/data_stream/config/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,3 @@
+fields:
+  tags:
+    - preserve_duplicate_custom_fields

packages/aws/data_stream/config/_dev/test/pipeline/test-event.log

@@ -0,0 +1 @@
+{"ComplianceType":"COMPLIANT","ConfigRuleInvokedTime":1742799479.852,"EvaluationResultIdentifier":{"EvaluationResultQualifier":{"ConfigRuleName":"required-tags","EvaluationMode":"DETECTIVE","ResourceId":"i-0a4468fbfafee6a8f","ResourceType":"AWS::EC2::Instance"},"OrderingTimestamp":1742541951.883,"ResourceEvaluationId":"string"},"ResultRecordedTime":1742799480.061,"Annotation":"string","ResultToken":"string","ConfigRuleInfo":{"CreatedBy":"string","Scope":{"ComplianceResourceId":"string","ComplianceResourceTypes":["string"],"TagKey":"string","TagValue":"string"},"Source":{"CustomPolicyDetails":{"EnableDebugLogDelivery":false,"PolicyRuntime":"string","PolicyText":"string"},"Owner":"AWS","SourceDetails":[{"EventSource":"string","MaximumExecutionFrequency":"string","MessageType":"string"}],"SourceIdentifier":"ACCESS_KEYS_ROTATED"},"ConfigRuleArn":"arn:aws:config:us-east-1:329599655752:config-rule/config-rule-rwpvuz","ConfigRuleId":"config-rule-rwpvuz","ConfigRuleName":"access-keys-rotated","ConfigRuleState":"ACTIVE","Description":"Checks whether the active access keys are rotated within the number of days specified in maxAccessKeyAge. The rule is non-compliant if the access keys have not been rotated for more than maxAccessKeyAge number of days.","EvaluationModes":[{"Mode":"DETECTIVE"}],"InputParameters":"{\"maxAccessKeyAge\":\"90\"}","MaximumExecutionFrequency":"TwentyFour_Hours"}}