Merge branch 'main' into autoops-cc-troubleshoot-firewall

Author: wajihaparvez

deploy-manage/monitor/autoops/cc-cloud-connect-autoops-troubleshooting.md

@@ -168,4 +168,4 @@ The following table shows the errors you might encounter if something goes wrong
 | `LICENSE_USED_BY_ANOTHER_ACCOUNT` | License key connected to another account | A license key can only be connected to one {{ecloud}} organization. Contact [Elastic support](https://support.elastic.co/) for help. |
 | `VERSION_MISMATCH` | {{es}} version is unsupported | Upgrade your cluster to a [supported version](https://www.elastic.co/support/eol). |
 | `UNKNOWN_ERROR` | Installation failed | {{agent}} couldn't be installed due to an unknown issue. Consult the troubleshooting guide or contact [Elastic support](https://support.elastic.co/) for more help. |
-| | Failed to register Cloud Connected Mode: cluster license type is not supported | The cluster you are trying to connect doesn't have the required license to connect to AutoOps. For more information, refer to the [prerequisites](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md#prerequisites). |
+| | Failed to register Cloud Connected Mode: cluster license type is not supported | The cluster you are trying to connect doesn't have the required license to connect to AutoOps. For more information, refer to the [prerequisites](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md#prerequisites). |

deploy-manage/monitor/autoops/cc-manage-users.md

@@ -44,4 +44,4 @@ Assign the following roles to new or existing users based on levels of access to
 | Role | Allowed actions in AutoOps |
 | --- | --- |
 | **Organization owner** | View events and metrics reports <br> Add or edit customizations and notification preferences <br> Connect and disconnect clusters |
-| **Connected cluster access** | **Viewer**: <br> View events and metrics reports <br><br>  **Admin** for all connected clusters: <br> View events and metrics reports <br> Add or edit customizations and notification preferences <br> Connect and disconnect clusters <br><br>  **Admin** for selected clusters: <br> View events and metrics reports <br> Add or edit customizations and notification preferences <br> Connect clusters |
+| **Connected cluster access** | **Viewer**: <br> View events and metrics reports <br><br>  **Admin** for all connected clusters: <br> View events and metrics reports <br> Add or edit customizations and notification preferences <br> Connect and disconnect clusters <br><br>  **Admin** for selected clusters: <br> View events and metrics reports <br> Connect clusters |

manage-data/ingest/transform-enrich/ingest-pipelines.md

@@ -388,7 +388,7 @@ PUT _ingest/pipeline/my-pipeline
 Use dot notation to access object fields.
 
 ::::{important}
-If your document contains flattened objects, use the [`dot_expander`](elasticsearch://reference/enrich-processor/dot-expand-processor.md) processor to expand them first. Other ingest processors cannot access flattened objects.
+If your document contains flattened objects, use the [`dot_expander`](elasticsearch://reference/enrich-processor/dot-expand-processor.md) processor to expand them. If you wish to maintain your document structure, use the [`flexible`](ingest-pipelines.md#access-source-pattern-flexible) access pattern in your pipeline definition. Otherwise Ingest processors cannot access dotted field names.
 ::::
 
 
@@ -431,6 +431,232 @@ PUT _ingest/pipeline/my-pipeline
 }
 ```
 
+## Ingest field access pattern [access-source-pattern]
+```{applies_to}
+serverless: ga
+stack: ga 9.2
+```
+
+The default ingest pipeline access pattern does not recognize dotted field names in documents. Retrieving flattened and dotted field names from an ingest document requires a different field retrieval algorithm that does not have this limitation. We know that some pipelines have come to rely on these dotted field name limitations in their logic. In order to continue supporting the original behavior while still adding support for dotted field names, ingest pipelines now support configuring an access pattern to use for all processors in the pipeline.
+
+The `field_access_pattern` property on an ingest pipeline defines how ingest document fields are read and written for all processors in the current pipeline. It accepts two values: `classic` (which is the default) and `flexible`.
+
+```console
+PUT _ingest/pipeline/my-pipeline
+{
+  "field_access_pattern": "classic", <1>
+  "processors": [
+    {
+      "set": {
+        "description": "Set some searchable tags in our document's flattened field",
+        "field": "event.tags.ingest.processed_by", <2>
+        "value": "my-pipeline"
+      }
+    }
+  ]
+}
+```
+1. All processors in this pipeline will use the `classic` access pattern.
+2. The logic for resolving field paths used by processors to read and write values to ingest documents is based on the access pattern. 
+
+### Classic field access pattern [access-source-pattern-classic]
+
+The `classic` access pattern is the default access pattern that has been around since ingest node first released. Field paths given to processors (e.g. `event.tags.ingest.processed_by`) are split on the dot character (`.`). The processor then uses the resulting field names to traverse the document until a value is found. When writing a value to a document, if its parent fields do not exist in the source, the processor will create nested objects for the missing fields.
+
+```console
+POST /_ingest/pipeline/_simulate
+{
+  "pipeline" : {
+    "description": "example pipeline",
+    "field_access_pattern": "classic", <1>
+    "processors": [
+      {
+        "set" : {
+          "description" : "Copy the foo.bar field into the a.b.c.d field if it exists",
+          "copy_from" : "foo.bar", <2>
+          "field" : "a.b.c.d", <3>
+          "ignore_empty_value": true
+        }
+      }
+    ]
+  },
+  "docs": [
+    {
+      "_index": "index",
+      "_id": "id",
+      "_source": {
+        "foo": {
+          "bar": "baz" <4>
+        }
+      }
+    },
+    {
+      "_index": "index",
+      "_id": "id",
+      "_source": {
+        "foo.bar": "baz" <5>
+      }
+    }
+  ]
+}
+```
+1. Explicitly declaring to use the `classic` access pattern in the pipeline. This is the default value.
+2. We are reading a value from the field `foo.bar`.
+3. We are writing its value to the field `a.b.c.d`.
+4. This document uses nested json objects in its structure. 
+5. This document uses dotted field names in its structure.
+
+```console-result
+{
+   "docs": [
+      {
+         "doc": {
+            "_id": "id",
+            "_index": "index",
+            "_version": "-3",
+            "_source": { 
+              "foo": {
+                "bar": "baz" <1>
+              },
+              "a": {
+                "b": {
+                  "c": {
+                    "d": "baz" <2>
+                  }
+                }
+              }
+            },
+            "_ingest": {
+               "timestamp": "2017-05-04T22:30:03.187Z"
+            }
+         }
+      },
+      {
+         "doc": {
+            "_id": "id",
+            "_index": "index",
+            "_version": "-3",
+            "_source": {
+               "foo.bar": "baz" <3>
+            },
+            "_ingest": {
+               "timestamp": "2017-05-04T22:30:03.188Z"
+            }
+         }
+      }
+   ]
+}
+```
+1. The first document's `foo.bar` field is located, because it uses nested json. The processor looks for a `foo` field, and then a `bar` field.
+2. The value from the `foo.bar` field is written to a nested json structure at field `a.b.c.d`. The processor creates objects for each field in the path.
+3. The second document uses a dotted field name for `foo.bar`. The `classic` access pattern does not recognize dotted field names, and so nothing is copied.
+
+If the documents you are ingesting contain dotted field names, to read them with the `classic` access pattern, you must use the [`dot_expander`](elasticsearch://reference/enrich-processor/dot-expand-processor.md) processor. This approach is not always reasonable though. Consider the following document: 
+
+```json
+{
+  "event": {
+    "tags": {
+      "http.host": "localhost:9200",
+      "http.host.name": "localhost", 
+      "http.host.port": 9200
+    }
+  }
+}
+```
+If the `event.tags` field was processed with the [`dot_expander`](elasticsearch://reference/enrich-processor/dot-expand-processor.md) processor, the field values would collide. The `http.host` field cannot be a text value and an object value at the same time.
+
+### Flexible field access pattern [access-source-pattern-flexible]
+
+The `flexible` access pattern allows for ingest pipelines to access both nested and dotted field names without using the [`dot_expander`](elasticsearch://reference/enrich-processor/dot-expand-processor.md) processor. Additionally, when writing a value to a field that does not exist, any parent fields that are missing are concatenated to the start of the new key. Use the `flexible` access pattern if your documents have dotted field names, and also if you prefer to write missing fields to the document with dotted names.
+
+```console
+POST /_ingest/pipeline/_simulate
+{
+  "pipeline" : {
+    "description": "example pipeline",
+    "field_access_pattern": "flexible", <1>
+    "processors": [
+      {
+        "set" : {
+          "description" : "Copy the foo.bar field into the a.b.c.d field if it exists",
+          "copy_from" : "foo.bar", <2>
+          "field" : "a.b.c.d", <3>
+          "ignore_empty_value": true
+        }
+      }
+    ]
+  },
+  "docs": [
+    {
+      "_index": "index",
+      "_id": "id",
+      "_source": {
+        "foo": {
+          "bar": "baz" <4>
+        },
+        "a": {} <5>
+      }
+    },
+    {
+      "_index": "index",
+      "_id": "id",
+      "_source": {
+        "foo.bar": "baz", <6>
+      }
+    }
+  ]
+}
+```
+1. Using the `flexible` access pattern in the pipeline.
+2. We are reading a value from the field `foo.bar`.
+3. We are writing its value to the field `a.b.c.d`.
+4. The first document uses nested json objects in its structure.
+5. The first document has an existing `a` field in the root.
+6. The second document uses a dotted field name.
+
+```console-result
+{
+   "docs": [
+      {
+         "doc": {
+            "_id": "id",
+            "_index": "index",
+            "_version": "-3",
+            "_source": { 
+              "foo": {
+                "bar": "baz" <1>
+              },
+              "a": {
+                "b.c.d": "baz" <2>
+              }
+            },
+            "_ingest": {
+               "timestamp": "2017-05-04T22:30:03.187Z"
+            }
+         }
+      },
+      {
+         "doc": {
+            "_id": "id",
+            "_index": "index",
+            "_version": "-3",
+            "_source": {
+               "foo.bar": "baz", <3>
+               "a.b.c.d": "baz" <4>
+            },
+            "_ingest": {
+               "timestamp": "2017-05-04T22:30:03.188Z"
+            }
+         }
+      }
+   ]
+}
+```
+1. The `flexible` access pattern supports nested object fields. The processor looks for a `foo` field, and then a `bar` field.
+2. The value from the `foo.bar` field is written to the dotted field name `b.c.d` underneath the field `a`. The processor concatenates the missing field names together as a prefix on the key.
+3. The `flexible` access pattern also supports dotted field names. The processor looks for a field named `foo`, and after not finding it, looks for a field named `foo.bar`.
+4. The value from the `foo.bar` field is written to the dotted field name `a.b.c.d`. Since none of those fields exist in the document yet, they are concatenated together into a dotted field name.
 
 ## Access metadata fields in a processor [access-metadata-fields]
 

solutions/observability/apm/opentelemetry/index.md

@@ -78,6 +78,6 @@ Find more details about how to use an OpenTelemetry API or SDK with an Elastic A
 
 AWS Lambda functions can be instrumented with OpenTelemetry and monitored with Elastic {{observability}} or {{obs-serverless}}.
 
-To get started, follow the official AWS Distribution for OpenTelemetry Lambda documentation, and configure the OpenTelemetry Collector to output traces and metrics to your Elastic cluster:
+To get started, follow the official AWS Distribution for OpenTelemetry Lambda documentation, and [configure the EDOT Collector in Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode) to send traces and metrics to your Elastic cluster:
 
-[**Get started with the AWS Distro for OpenTelemetry Lambda**](https://aws-otel.github.io/docs/getting-started/lambda)
+[**Get started with the AWS Distro for OpenTelemetry Lambda**](https://aws-otel.github.io/docs/getting-started/lambda)

solutions/observability/get-started/logs-essentials.md

@@ -34,12 +34,12 @@ The **Admin** role or higher is required to create projects. Refer to [Assign us
 1. Select **Create serverless project**.
 1. Under **Elastic for Observability**, select **Next**.
 1. Enter a name for your project.
+1. Under **Product features**, select **Observability Logs Essentials**.
 1. (Optional) Under **Settings** you can change the following:
 
     * **Cloud provider**: The cloud platform where you’ll deploy your project. We currently support Amazon Web Services (AWS).
     * **Region**: The [region](/deploy-manage/deploy/elastic-cloud/regions.md) where your project will live.
 
-1. Select **Edit settings**, and select **Observability Logs Essentials**.
 1. Select **Create serverless project**. It takes a few minutes to create your project.
 1. When the project is ready, click **Continue**.
 

solutions/security/detect-and-alert/create-detection-rule.md

@@ -152,7 +152,19 @@ To filter noisy {{ml}} rules, use [rule exceptions](/solutions/security/detect-a
         You can also leave the **Group by** field undefined. The rule then creates an alert when the number of search results is equal to or greater than the threshold value. If you set **Count** to limit the results by `process.name` >= 2, an alert will only be generated for source/destination IP pairs that appear with at least 2 unique process names across all events.
 
         ::::{important}
-        Alerts created by threshold rules are synthetic alerts that do not resemble the source documents. The alert itself only contains data about the fields that were aggregated over (the **Group by** fields). Other fields are omitted, because they can vary across all source documents that were counted toward the threshold. Additionally, you can reference the actual count of documents that exceeded the threshold from the `kibana.alert.threshold_result.count` field.
+        Alerts created by threshold rules are synthetic alerts that do not resemble the source documents:
+        
+          - The alert itself only contains data about the fields that were aggregated over (the **Group by** fields specified in the rule).
+          - All other fields are omitted and aren't available in the alert. This is because these fields can vary across all source documents that were counted toward the threshold. 
+          - You can reference the actual count of documents that exceeded the threshold from the `kibana.alert.threshold_result.count` field. 
+          - `context.alerts.kibana.alert.threshold_result.terms` contains fields and values from any **Group by** fields specified in the rule. For example:
+        ```
+          {{#context.alerts}}
+            {{#kibana.alert.threshold_result.terms}}
+              {{field}}: {{value}}
+            {{/kibana.alert.threshold_result.terms}}
+         {{/context.alerts}}
+       ```
         ::::
 
 3. (Optional) Select **Suppress alerts** to reduce the number of repeated or duplicate alerts created by the rule. Refer to [Suppress detection alerts](/solutions/security/detect-and-alert/suppress-detection-alerts.md) for more information.

troubleshoot/ingest/opentelemetry/edot-collector/enable-debug-logging.md

@@ -86,3 +86,6 @@ Debug logging for the Collector is not currently configurable through {{fleet}}.
 :::
 
 
+## Resources
+
+To learn how to enable debug logging for the EDOT SDKs, refer to [Enable debug logging for EDOT SDKs](../edot-sdks/enable-debug-logging.md).

troubleshoot/ingest/opentelemetry/edot-sdks/enable-debug-logging.md

@@ -27,7 +27,8 @@ Enabling debug logging can help surface common problems such as:
 
 ## Verify you're looking at the right logs
 
-* Ensure you’re checking logs for the same process that starts your app (systemd service, container entrypoint, IIS worker, etc.).
+Ensure you’re checking logs for the same process that starts your app (systemd service, container entrypoint, IIS worker, and so on):
+
 * For containerized environments such as Kubernetes/Docker:  
   * `kubectl logs <pod> -c <container>` (correct container name matters if there are sidecars)  
   * Check the new Pod after a rollout, as old Pods may show stale environment without your debug flags.