aws: Add Config and Inspector transforms for extended protections (CDR) workflow

packages/aws/_dev/build/docs/config.md

@@ -60,6 +60,27 @@ Use this integration if you only need to collect data from the AWS Config servic
 1. For the current integration package, it is compulsory to add Secret Access Key and Access Key ID.
 2. The AWS Config integration performs a full ingestion of all findings during each interval.
 
+## Troubleshooting
+
+### Breaking Changes
+
+#### Support for Elastic Misconfiguration Findings page.
+
+Version `4.0.0` of the AWS Config integration adds support for [Elastic Cloud Security workflow](https://www.elastic.co/docs/solutions/security/cloud/ingest-third-party-cloud-security-data#_ingest_third_party_security_posture_and_vulnerability_data). The enhancement enables the users of the AWS Config integration to ingest misconfiguration findings from the AWS Config platform into Elastic and get insights directly from the [Misconfiguration Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page).
+Version `4.0.0` adds [Elastic Latest Transform](https://www.elastic.co/docs/explore-analyze/transforms/transform-overview#latest-transform-overview) which copies the latest findings from source indices matching the pattern `logs-aws.config-*` into new destination indices matching the pattern `security_solution-awsconfig.misconfiguration_latest-*`. The Elastic Findings pages will display findings based on the destination indices.
+
+For existing users of the AWS Config integration, before upgrading to version `4.0.0` please ensure the following requirements are met:
+
+1. Users need [Elastic Security solution](https://www.elastic.co/docs/solutions/security) which has requirements documented [here](https://www.elastic.co/docs/solutions/security/get-started/elastic-security-requirements).
+2. To use transforms, users must have:
+   - at least one [transform node](https://www.elastic.co/docs/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#transform-node-role),
+   - management features visible in the Kibana space, and
+   - security privileges that:
+     - grant use of transforms, and
+     - grant access to source and destination indices
+   For more details on Transform Setup, refer to the link [here](https://www.elastic.co/docs/explore-analyze/transforms/transform-setup)
+3. Because the latest copy of findings is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly.
+
 ## Logs reference
 
 ### Config

packages/aws/_dev/build/docs/inspector.md

@@ -28,6 +28,27 @@ Agentless deployments are only supported in Elastic Serverless and Elastic Cloud
   - This data stream doesn't support setting a Role ARN.
   - Ensure your IAM has the `inspector2:ListFindings` permission granted. Without this permission, API requests will be denied.
 
+## Troubleshooting
+
+### Breaking Changes
+
+#### Support for Elastic Vulnerability Findings page.
+
+Version `4.0.0` of the AWS integration adds support for [Elastic Cloud Security workflow](https://www.elastic.co/docs/solutions/security/cloud/ingest-third-party-cloud-security-data#_ingest_third_party_security_posture_and_vulnerability_data). The enhancement enables the users of the AWS Inspector integration to ingest their enriched vulnerabilities from the Amazon Inspector platform into Elastic and get insights directly from the Elastic [Vulnerability Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page-3).
+This update adds [Elastic Latest Transform](https://www.elastic.co/docs/explore-analyze/transforms/transform-overview#latest-transform-overview) which copies the latest vulnerability findings from source indices matching the pattern `logs-aws.inspector-*` into new destination indices matching the pattern `security_solution-aws.vulnerability_latest-*`. The Elastic Vulnerability Findings page will display vulnerabilities based on the destination indices.
+
+For existing users of the AWS integration, before upgrading to `4.0.0` please ensure the following requirements are met:
+
+1. Users need [Elastic Security solution](https://www.elastic.co/docs/solutions/security) which has requirements documented [here](https://www.elastic.co/docs/solutions/security/get-started/elastic-security-requirements).
+2. To use transforms, users must have:
+   - at least one [transform node](https://www.elastic.co/docs/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#transform-node-role),
+   - management features visible in the Kibana space, and
+   - security privileges that:
+     - grant use of transforms, and
+     - grant access to source and destination indices
+   For more details on Transform Setup, refer to the link [here](https://www.elastic.co/docs/explore-analyze/transforms/transform-setup)
+3. Because the latest copy of vulnerabilities is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly.
+
 ## Logs
 
 ### Inspector

packages/aws/changelog.yml

@@ -1,4 +1,17 @@
 # newer versions go on top
+- version: "4.0.0"
+  changes:
+    - description: |
+        Add latest transform to `AWS Config` and `AWS Inspector`.
+        This enables support for extended protections for `AWS Config` and `AWS Inspector`.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/15230
+    - description: |
+        The latest transforms requires `transform` node and necessary permissions to use the transform.
+        The transforms stores the latest copy of vulnerabilities and misconfigurations in the destination indices, which will require additional storage.
+        Due to change in fingeprinting on source indices, duplicates occur on source indices of `AWS Config` and `AWS Inspector`.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/15230
 - version: "3.17.0"
   changes:
     - description: Enable Agentless deployment for AWS GuardDuty.
@@ -14,6 +27,11 @@
     - description: Add support for VPC Flow logs versions 6, 7, and 8.
       type: enhancement
       link: https://github.com/elastic/integrations/pull/15077
+- version: "3.14.2"
+  changes:
+    - description: Remove unused agent files.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/14995
 - version: "3.14.1"
   changes:
     - description: Fixed issue where empty DescribeConfigRules responses caused 'index out of bounds' errors in AWS Config integration.

packages/aws/data_stream/config/_dev/test/pipeline/test-event.log-expected.json

@@ -105,7 +105,8 @@
                 "id": "config-rule-rwpvuz",
                 "name": "access-keys-rotated",
                 "reference": "arn:aws:config:us-east-1:329599655752:config-rule/config-rule-rwpvuz",
-                "tags": "string"
+                "tags": "string",
+                "uuid": "chfNCTELmFlMeMhp21DvcEjdkK0="
             },
             "tags": [
                 "preserve_duplicate_custom_fields"

packages/aws/data_stream/config/elasticsearch/ilm/default_policy.json

@@ -0,0 +1,20 @@
+{
+	"policy": {
+		"phases": {
+			"hot": {
+				"actions": {
+					"rollover": {
+						"max_age": "7d",
+						"max_primary_shard_size": "50gb"
+					}
+				}
+			},
+			"delete": {
+				"min_age": "7d",
+				"actions": {
+					"delete": {}
+				}
+			}
+		}
+	}
+}

packages/aws/data_stream/config/elasticsearch/ingest_pipeline/default.yml

@@ -41,14 +41,6 @@ processors:
         - append:
             field: error.message
             value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
-  - fingerprint:
-      fields:
-        - json.EvaluationResultIdentifier.EvaluationResultQualifier.ResourceId
-        - json.ConfigRuleInvokedTime
-        - json.ConfigRuleInfo.ConfigRuleId
-      tag: fingerprint_aws_config
-      target_field: _id
-      ignore_missing: true
   - set:
       field: cloud.provider
       tag: set_cloud_provider
@@ -94,6 +86,12 @@ processors:
       tag: set_rule_reference_from_config_config_rule_info_config_rule_arn
       copy_from: aws.config.rule_info.config_rule_arn
       ignore_empty_value: true
+  - fingerprint:
+      fields:
+        - rule.reference
+      tag: fingerprint_rule_uuid
+      target_field: rule.uuid
+      ignore_missing: true
   - rename:
       field: json.ConfigRuleInfo.ConfigRuleId
       tag: rename_ConfigRuleInfo_ConfigRuleId

packages/aws/data_stream/config/fields/base-fields.yml

@@ -1,20 +1,16 @@
 - name: data_stream.type
-  type: constant_keyword
-  description: Data stream type.
+  external: ecs
 - name: data_stream.dataset
-  type: constant_keyword
-  description: Data stream dataset.
+  external: ecs
 - name: data_stream.namespace
-  type: constant_keyword
-  description: Data stream namespace.
+  external: ecs
 - name: event.module
   type: constant_keyword
-  description: Event module.
+  external: ecs
   value: aws
 - name: event.dataset
   type: constant_keyword
   value: aws.config
-  description: Event dataset.
+  external: ecs
 - name: '@timestamp'
-  type: date
-  description: Event timestamp.
+  external: ecs

packages/aws/data_stream/config/lifecycle.yml

@@ -0,0 +1 @@
+data_retention: "7d"

packages/aws/data_stream/config/manifest.yml

@@ -1,5 +1,6 @@
 title: Collect AWS Config Findings logs via API
 type: logs
+ilm_policy: logs-aws.config-default_policy
 streams:
   - input: cel
     title: Collect AWS Config Findings from AWS

packages/aws/data_stream/inspector/elasticsearch/ilm/default_policy.json

@@ -0,0 +1,20 @@
+{
+	"policy": {
+		"phases": {
+			"hot": {
+				"actions": {
+					"rollover": {
+						"max_age": "7d",
+						"max_primary_shard_size": "50gb"
+					}
+				}
+			},
+			"delete": {
+				"min_age": "7d",
+				"actions": {
+					"delete": {}
+				}
+			}
+		}
+	}
+}

packages/aws/data_stream/inspector/lifecycle.yml

@@ -0,0 +1 @@
+data_retention: "7d"

packages/aws/data_stream/inspector/manifest.yml

@@ -1,5 +1,6 @@
 title: Collect Amazon Inspector logs from AWS
 type: logs
+ilm_policy: logs-aws.inspector-default_policy
 streams:
   - input: httpjson
     title: Collect Amazon Inspector Findings from AWS

packages/aws/data_stream/securityhub_findings/_dev/test/system/test-default-config.yml

@@ -34,3 +34,6 @@ data_stream:
           NZJwli2WcEIuvEP2btR3aq3DSZiJwsgh3YaqA9GFv0e3A7rG5lUwaFFIhSFmNTUo
           QitGeqCxiwvdjD4d/jkyeG84779ewQQeYyxgOgvQaiS56a4DijLYkIU=
           -----END CERTIFICATE-----
+skip:
+  reason: "The fleet health status changes to degraded when the HTTPJSON template's value evaluation comes up empty, which leads to system test failures but does not interrupt the data flow."
+  link: https://github.com/elastic/beats/issues/45664

packages/aws/data_stream/securityhub_findings_full_posture/_dev/test/system/test-default-config.yml

@@ -34,3 +34,6 @@ data_stream:
           NZJwli2WcEIuvEP2btR3aq3DSZiJwsgh3YaqA9GFv0e3A7rG5lUwaFFIhSFmNTUo
           QitGeqCxiwvdjD4d/jkyeG84779ewQQeYyxgOgvQaiS56a4DijLYkIU=
           -----END CERTIFICATE-----
+skip:
+  reason: "The fleet health status changes to degraded when the HTTPJSON template's value evaluation comes up empty, which leads to system test failures but does not interrupt the data flow."
+  link: https://github.com/elastic/beats/issues/45664

packages/aws/data_stream/securityhub_insights/_dev/test/system/test-default-config.yml

@@ -33,3 +33,6 @@ data_stream:
           8gqQdAH8DCmCSwT/6JRLbDCCM7njqzGLb3d/hGdZYxVp+Bu0vbuE4BnifTvo79az
           IqZhWKmJamAm8bHDYVR+QPo7JWkPf117I3YORE3NSC1dfvXk1jOCl+zA7A==
           -----END CERTIFICATE-----
+skip:
+  reason: "The fleet health status changes to degraded when the HTTPJSON template's value evaluation comes up empty, which leads to system test failures but does not interrupt the data flow."
+  link: https://github.com/elastic/beats/issues/45664

packages/aws/docs/config.md

@@ -60,6 +60,27 @@ Use this integration if you only need to collect data from the AWS Config servic
 1. For the current integration package, it is compulsory to add Secret Access Key and Access Key ID.
 2. The AWS Config integration performs a full ingestion of all findings during each interval.
 
+## Troubleshooting
+
+### Breaking Changes
+
+#### Support for Elastic Misconfiguration Findings page.
+
+Version `4.0.0` of the AWS Config integration adds support for [Elastic Cloud Security workflow](https://www.elastic.co/docs/solutions/security/cloud/ingest-third-party-cloud-security-data#_ingest_third_party_security_posture_and_vulnerability_data). The enhancement enables the users of the AWS Config integration to ingest misconfiguration findings from the AWS Config platform into Elastic and get insights directly from the [Misconfiguration Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page).
+Version `4.0.0` adds [Elastic Latest Transform](https://www.elastic.co/docs/explore-analyze/transforms/transform-overview#latest-transform-overview) which copies the latest findings from source indices matching the pattern `logs-aws.config-*` into new destination indices matching the pattern `security_solution-awsconfig.misconfiguration_latest-*`. The Elastic Findings pages will display findings based on the destination indices.
+
+For existing users of the AWS Config integration, before upgrading to version `4.0.0` please ensure the following requirements are met:
+
+1. Users need [Elastic Security solution](https://www.elastic.co/docs/solutions/security) which has requirements documented [here](https://www.elastic.co/docs/solutions/security/get-started/elastic-security-requirements).
+2. To use transforms, users must have:
+   - at least one [transform node](https://www.elastic.co/docs/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#transform-node-role),
+   - management features visible in the Kibana space, and
+   - security privileges that:
+     - grant use of transforms, and
+     - grant access to source and destination indices
+   For more details on Transform Setup, refer to the link [here](https://www.elastic.co/docs/explore-analyze/transforms/transform-setup)
+3. Because the latest copy of findings is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly.
+
 ## Logs reference
 
 ### Config
@@ -192,7 +213,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 
 | Field | Description | Type |
 |---|---|---|
-| @timestamp | Event timestamp. | date |
+| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
 | aws.config.annotation | Supplementary information about how the evaluation determined the compliance. | keyword |
 | aws.config.compliance_type | Indicates whether the AWS resource complies with the AWS Config rule that evaluated it. | keyword |
 | aws.config.config_rule_invoked_time | The time when the AWS Config rule evaluated the AWS resource. | date |
@@ -226,11 +247,11 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 | aws.config.rule_info.source.source_details.message_type | The type of notification that triggers AWS Config to run an evaluation for a rule. | keyword |
 | aws.config.rule_info.source.source_identifier | For AWS Config Managed rules, a predefined identifier from a list. | keyword |
 | cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | constant_keyword |
-| data_stream.dataset | Data stream dataset. | constant_keyword |
-| data_stream.namespace | Data stream namespace. | constant_keyword |
-| data_stream.type | Data stream type. | constant_keyword |
-| event.dataset | Event dataset. | constant_keyword |
-| event.module | Event module. | constant_keyword |
+| data_stream.dataset | The field can contain anything that makes sense to signify the source of the data. Examples include `nginx.access`, `prometheus`, `endpoint` etc. For data streams that otherwise fit, but that do not have dataset set we use the value "generic" for the dataset value. `event.dataset` should have the same value as `data_stream.dataset`. Beyond the Elasticsearch data stream naming criteria noted above, the `dataset` value has additional restrictions:   \* Must not contain `-`   \* No longer than 100 characters | constant_keyword |
+| data_stream.namespace | A user defined namespace. Namespaces are useful to allow grouping of data. Many users already organize their indices this way, and the data stream naming scheme now provides this best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`. Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions:   \* Must not contain `-`   \* No longer than 100 characters | constant_keyword |
+| data_stream.type | An overarching type for the data stream. Currently allowed values are "logs" and "metrics". We expect to also add "traces" and "synthetics" in the near future. | constant_keyword |
+| event.dataset | Name of the dataset. If an event source publishes more than one type of log or events (e.g. access log, error log), the dataset is used to specify which one the event comes from. It's recommended but not required to start the dataset name with the module name, followed by a dot, then the dataset name. | constant_keyword |
+| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | constant_keyword |
 | input.type | Type of Filebeat input. | keyword |
 | log.offset | Log offset. | long |
 | observer.vendor | Vendor name of the observer. | constant_keyword |

packages/aws/docs/inspector.md

@@ -28,6 +28,27 @@ Agentless deployments are only supported in Elastic Serverless and Elastic Cloud
   - This data stream doesn't support setting a Role ARN.
   - Ensure your IAM has the `inspector2:ListFindings` permission granted. Without this permission, API requests will be denied.
 
+## Troubleshooting
+
+### Breaking Changes
+
+#### Support for Elastic Vulnerability Findings page.
+
+Version `4.0.0` of the AWS integration adds support for [Elastic Cloud Security workflow](https://www.elastic.co/docs/solutions/security/cloud/ingest-third-party-cloud-security-data#_ingest_third_party_security_posture_and_vulnerability_data). The enhancement enables the users of the AWS Inspector integration to ingest their enriched vulnerabilities from the Amazon Inspector platform into Elastic and get insights directly from the Elastic [Vulnerability Findings page](https://www.elastic.co/docs/solutions/security/cloud/findings-page-3).
+This update adds [Elastic Latest Transform](https://www.elastic.co/docs/explore-analyze/transforms/transform-overview#latest-transform-overview) which copies the latest vulnerability findings from source indices matching the pattern `logs-aws.inspector-*` into new destination indices matching the pattern `security_solution-aws.vulnerability_latest-*`. The Elastic Vulnerability Findings page will display vulnerabilities based on the destination indices.
+
+For existing users of the AWS integration, before upgrading to `4.0.0` please ensure the following requirements are met:
+
+1. Users need [Elastic Security solution](https://www.elastic.co/docs/solutions/security) which has requirements documented [here](https://www.elastic.co/docs/solutions/security/get-started/elastic-security-requirements).
+2. To use transforms, users must have:
+   - at least one [transform node](https://www.elastic.co/docs/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#transform-node-role),
+   - management features visible in the Kibana space, and
+   - security privileges that:
+     - grant use of transforms, and
+     - grant access to source and destination indices
+   For more details on Transform Setup, refer to the link [here](https://www.elastic.co/docs/explore-analyze/transforms/transform-setup)
+3. Because the latest copy of vulnerabilities is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly.
+
 ## Logs
 
 ### Inspector

packages/aws/elasticsearch/transform/latest_cdr_misconfigurations_awsconfig/fields/base-fields.yml

@@ -0,0 +1,17 @@
+- name: data_stream.type
+  external: ecs
+- name: data_stream.dataset
+  external: ecs
+- name: data_stream.namespace
+  type: keyword
+  external: ecs
+- name: event.module
+  type: constant_keyword
+  external: ecs
+  value: aws
+- name: event.dataset
+  type: constant_keyword
+  value: aws.config
+  external: ecs
+- name: '@timestamp'
+  external: ecs

packages/aws/elasticsearch/transform/latest_cdr_misconfigurations_awsconfig/fields/beats.yml

@@ -0,0 +1,6 @@
+- name: input.type
+  type: keyword
+  description: Type of Filebeat input.
+- name: log.offset
+  type: long
+  description: Log offset.

packages/aws/elasticsearch/transform/latest_cdr_misconfigurations_awsconfig/fields/ecs.yml

@@ -0,0 +1,6 @@
+- name: cloud.provider
+  type: constant_keyword
+  external: ecs
+- name: observer.vendor
+  type: constant_keyword
+  external: ecs