[Observability] Add failure store documentation (#699)

This PR add failure store info to the Data Set Quality docs and closes
[Issue 4354](https://github.com/elastic/observability-docs/issues/4354)
and [Issue
4812](https://github.com/elastic/observability-docs/issues/4812).

Author: mdbirnstiehl

solutions/observability/data-set-quality-monitoring.md

@@ -11,19 +11,19 @@ products:
   - id: cloud-serverless
 ---
 
-# Data set quality monitoring [observability-monitor-datasets]
+# Data set quality [observability-monitor-datasets]
 
-The **Data Set Quality** page provides an overview of your log, metric, trace, and synthetic data sets. Use this information to get an idea of your overall data set quality and find data sets that contain incorrectly parsed documents.
+The **Data Set Quality** page provides an overview of your log, metric, trace, and synthetic data sets. You can then use this information to get an idea of your overall data set quality and find data sets that contain incorrectly parsed documents.
 
-To open **Data Set Quality**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). By default, the page only shows log data sets. To see other data set types, select them from the **Type** menu.
+To open the **Data Set Quality** page, find it under **Stack Management** in the main menu, or search for "data set quality" using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
 
-::::{admonition} Requirements
-:class: note
+By default, the page only shows log data sets. To see other data set types, select them from the **Type** menu.
 
-Users with the `viewer` role can view the Data Sets Quality summary. To view the Active Data Sets and Estimated Data summaries, users need the `monitor` [index privilege](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) for the `logs-*-*` index.
+## Required roles and privileges
 
-::::
+Users with the `viewer` [role](../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md) can only view the **Data Set Quality** summary. To view the **Active Data Sets** and **Estimated Data** summaries, you need the `monitor` [index privilege](../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) for the `logs-*-*` index.
 
+## Monitor data sets
 
 The quality of your data sets is based on the percentage of degraded documents in each data set. A degraded document in a data set contains the [`_ignored`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-ignored-field.md) property because one or more of its fields were ignored during indexing. Fields are ignored for a variety of reasons. For example, when the [`ignore_malformed`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-ignored-field.md) parameter is set to true, if a document field contains the wrong data type, the malformed field is ignored and the rest of the document is indexed.
 
@@ -35,35 +35,58 @@ From the data set table, you’ll find information for each data set such as its
 
 Opening the details of a specific data set shows the degraded documents history, a summary for the data set, and other details that can help you determine if you need to investigate any issues.
 
-
 ## Investigate issues [observability-monitor-datasets-investigate-issues]
 
-The Data Set Quality page has a couple of different ways to help you find ignored fields and investigate issues. From the data set table, you can open the data set’s details page, and view commonly ignored fields and information about those fields. Open a logs data set in Discover or other data set types in Discover to find ignored fields in individual documents.
+The Data Set Quality page provides several ways to help you investigate issues. From the data set table, you can open the data set’s details page, open failed docs sent to the failure store in Discover (serverless only), and view ignored fields.
 
+### Find failed documents with failure store
+```{applies_to}
+serverless: ga
+```
 
-### Find ignored fields in data sets [observability-monitor-datasets-find-ignored-fields-in-data-sets]
+To help diagnose issues with ingestion or mapping, documents that are rejected during ingestion are sent to a dedicated data stream called failure store. On the **Data Set Quality** page, data streams with documents in the failure store will show a percentage in the **Failed docs (%)** column. The failed docs percentage gives you a quick look at the magnitude of potential problems in your ingestion process.
+
+#### Required privileges
+
+Accessing failure store requires the `read_failure_store` or `all` [index privilege](../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices).
+
+#### Find failed documents
+
+Select the percentage in the **Failed docs (%)** column for a specific data stream to open Discover and see the raw documents that were sent to failure store.
 
-To open the details page for a data set with poor or degraded quality and view ignored fields:
+To understand how persistent an issue is, refer to **Document trends** for the number of failed documents over a selected time range:
+
+1. Select the data set name from the main table.
+1. Select the **Failed docs** tab under **Document trends**.
+
+To help diagnose what's causing an issue, refer to **Quality issues** for error messages and failure types related to your documents:
+
+1. From the data set table, select a data set name.
+1. Scroll down to **Quality issues**.
+1. Click the expand icon to open a summary of why your document failed.
+
+### Find ignored fields in data sets [observability-monitor-datasets-find-ignored-fields-in-data-sets]
 
-1. From the data set table, click ![expand icon](/solutions/images/serverless-expand.svg "") next to a data set with poor or degraded quality.
-2. From the details, scroll down to **Quality issues**.
+To open the details page for a data set with poor or degraded quality and view ignored fields and failed documents:
 
-The **Quality issues** section shows fields that have been ignored, the number of documents that contain ignored fields, and the timestamp of last occurrence of the field being ignored.
+1. From the data set table, select a data set name.
+1. Scroll down to **Quality issues**.
 
+The **Quality issues** section shows fields that have been ignored, the number of documents that contain ignored fields, the timestamp of last occurrence of the field being ignored, and failed documents (serverless only).
 
 ### Find ignored fields in individual logs [observability-monitor-datasets-find-ignored-fields-in-individual-logs]
 
 To use Discover to find ignored fields in individual logs:
 
-1. Find data sets with degraded documents using the **Degraded Docs** column of the data sets table.
-2. Click the percentage in the **Degraded Docs** column to open the data set in Discover.
+1. From the Data Set Quality page, use the **Degraded Docs** column to find data sets with degraded documents.
+1. Select the percentage in the **Degraded Docs** column to open the data set in Discover.
 
-The **Documents** table in Discover is automatically filtered to show documents that were not parsed correctly. Under the **actions** column, you’ll find the degraded document icon (![degraded document icon](../images/serverless-indexClose.svg "")).
+The **Documents** table in Discover is automatically filtered to show documents that were not parsed correctly. You’ll find the degraded document icon (![degraded document icon](/solutions/images/serverless-indexClose.svg "")) next to documents that weren't parsed correctly. You can also go directly to Discover and look for this icon to find documents that weren't parsed correctly.
 
 Now that you know which documents contain ignored fields, examine them more closely to find the origin of the issue:
 
 1. Under the **actions** column, click ![expand icon](/solutions/images/serverless-expand.svg "") to open the document details.
-2. Select the **JSON** tab.
-3. Scroll towards the end of the JSON to find the `ignored_field_values`.
+1. Select the **JSON** tab.
+1. Scroll towards the end of the JSON to find the `ignored_field_values`.
 
 Here, you’ll find all of the `_ignored` fields in the document and their values, which should provide some clues as to why the fields were ignored.