add Streams 9.2 updates

solutions/observability/streams/management/advanced.md

@@ -1,11 +1,40 @@
 ---
 applies_to:
-  serverless: preview
-  stack: preview 9.1
+  serverless: ga
+  stack: preview 9.1, ga 9.2
 navigation_title: Configure advanced settings
 ---
 # Configure advanced settings for streams [streams-advanced-settings]
 
-The **Advanced** tab on the **Manage stream** page shows the underlying configuration details of your stream. While Streams simplifies many configurations, it doesn't support modifying all pipelines and templates. From the **Advanced** tab, you can manually interact with the index or component templates or modify other ingest pipelines that used by the stream.
+The **Advanced** tab shows the underlying {{es}} configuration details and advanced configuration options for your stream.
 
-This UI is intended for advanced users.
+You can use the **Advanced** tab to manually configure the index or component templates or modify other ingest pipelines used by the stream.
+
+## Stream description
+
+% can we provide more information on what users should put here.
+
+Describe the data in the stream. AI features like system identification and significant events use this description when generating suggestions.
+
+## Stream system configuration
+
+% Why do users want to add systems here?
+
+Streams analyzes your stream and identifies systems. Then, you can select the ones you want to add to your stream.
+
+## Index configuration
+
+% Can we add use cases of when it makes sense to modify shards/replicas/refresh interval
+
+For classic streams, you can manually configure the stream's:
+
+- [index template](../../../../manage-data/data-store/templates.md#index-templates)
+- [component templates](../../../../manage-data/data-store/templates.md#component-templates)
+- [pipeline](../../../../manage-data/ingest/transform-enrich.md)
+- [data stream](../../../../manage-data/data-store/data-streams.md).
+
+For both wired ({applies_to}`stack: preview 9.2`) and classic streams, you can manually configure:
+
+- **Shards:** Control how the index is split across nodes. More shards can improve parallelism but may increase overhead.
+- **Replicas:** Define how many copies of the data exist. More replicas improve resilience and read performance but increase storage usage.
+- **Refresh interval:** Control how frequently new data becomes visible for search. A longer interval reduces resource usage; a short one makes data searchable sooner.

solutions/observability/streams/management/data-quality.md

@@ -0,0 +1,29 @@
+---
+applies_to:
+  serverless: preview
+  stack: preview 9.1, ga 9.2
+---
+
+# Manage data quality [streams-data-retention]
+
+Use the **Data quality** tab to find failed and degraded documents in your stream. Use the following components to monitor the health of your data and identify and fix issues:
+
+- **Degraded documents:** Documents with the `ignored` property usually because of malformed fields or exceeding the limit of total fields when `ignore_above:false`. This component shows the total number of degraded documents, the percentage, and status (**Good**, **Degraded**, **Poor**).
+- **Failed documents:**: Documents that were rejected during ingestion because of mapping conflicts or pipeline failures.
+- **Quality score:** Streams calculates the overall quality score (Good, Degraded, Poor) based on the percentage of degraded and failed documents.
+- **Trends over time:** A time-series chart so you can track how degraded and failed documents are accumulating over time. Use the date picker to zoom into a specific range and understand when problems are spiking.
+- **Issues:**: {applies_to}`stack: preview 9.2`Find issues with specific fields, how often they've occurred, and when they've occurred.
+
+## Failure store
+
+A [failure store](../../../../manage-data/data-store/data-streams/failure-store.md) is a secondary set of indices inside a data stream, dedicated to storing failed documents. Instead of losing documents that are rejected during ingestion, a failure store retains it in a `::failures` index, so you can review failed documents to understand what went wrong and how to fix it.
+
+### Required permissions
+To view and modify failure store in {{stack}}, you need the following data stream level privileges:
+- `read_failure_store`
+- `manage_failure_store`
+
+
+In Streams, you need to turn on failure stores to see failed documents. To do this, select **Enable failure store*. From here you can set your failure store retention period.
+
+For more information on data quality, refer to the [data set quality](../../data-set-quality-monitoring.md) documentation.

solutions/observability/streams/management/extract.md

@@ -1,45 +1,68 @@
 ---
 applies_to:
-  serverless: preview
-  stack: preview 9.1
+  serverless: ga
+  stack: preview 9.1, ga 9.2
 ---
 # Extract fields [streams-extract-fields]
 
-Unstructured log messages must be parsed into meaningful fields before you can filter and analyze them effectively. Commonly extracted fields include `@timestamp` and the `log.level`, but you can also extract information like IP addresses, usernames, and ports.
+Use the **Processing** tab to add [processors](#streams-extract-processors) that extract meaningful fields from your log messages. These fields let you filter and analyze your data more effectively.
 
-Use the **Processing** tab on the **Manage stream** page to process your data. The UI simulates your changes and provides an immediate preview that's tested end-to-end.
+For example, in [Discover](../../../../explore-analyze/discover.md), extracted fields might let you filter for log messages with an `ERROR` log level that occurred during a specific time period to help diagnose an issue. Without extracting the log level and timestamp fields from your messages, those filters wouldn't return meaningful results.
 
-The UI also shows indexing problems, such as mapping conflicts, so you can address them before applying changes.
+The processing tab also:
+
+- Simulates your processors and provides an immediate [preview](#streams-preview-changes) that's tested end to end.
+- Flags indexing issues, like [mapping conflicts](#streams-processing-mapping-conflicts), so you can address them before applying changes.
+
+After creating your processor, all future data ingested ingested into the stream will be parsed into structured fields accordingly.
 
 :::{note}
 Applied changes aren't retroactive and only affect *future ingested data*.
 :::
 
+## Supported processors [streams-extract-processors]
+
+Streams supports the following processors:
+
+- [Date](./extract/date.md): Converts date strings into timestamps, with options for timezone, locale, and output formatting.
+- [Dissect](./extract/dissect.md): Extracts fields from structured log messages using defined delimiters instead of patterns, making it faster than Grok and ideal for consistently formatted logs.
+- [Grok](./extract/grok.md): Extracts fields from unstructured log messages using predefined or custom patterns, supports multiple match attempts in sequence, and can automatically generate patterns with an LLM connector.
+- [Set](./extract/set.md): Assigns a specific value to a field, creating the field if it doesn’t exist or overwriting its value if it does.
+- [Rename](./extract/rename.md): Changes the name of a field, moving its value to a new field name and removing the original.
+- [Append](./extract/append.md): Adds a value to an existing array field, or creates the field as an array if it doesn’t exist.
+
 ## Add a processor [streams-add-processors]
 
-Streams uses {{es}} ingest pipelines to process your data. Ingest pipelines are made up of processors that transform your data.
+Streams uses [{{es}} ingest pipelines](../../../../manage-data/ingest/transform-enrich/ingest-pipelines.md) made up of processors to transform your data, without requiring you to switch interfaces and manually update pipelines.
+
+To add a processor from the **Processing** tab:
 
-To add a processor:
+1. Select **Create** → **Create processor** to open a list of supported processors.
+1. Select a processor from the **Processor** menu.
+1. Configure the processor and select **Create** to save the processor.
 
-1. Select **Add processor** to open a list of supported processors.
-1. Select a processor from the list:
-    - [Date](./extract/date.md)
-    - [Dissect](./extract/dissect.md)
-    - [Grok](./extract/grok.md)
-    - GeoIP
-    - Rename
-    - Set
-    - URL Decode
-1. Select **Add Processor** to save the processor.
+After adding all desired processors and conditions, make sure to **Save changes**.
+
+Refer to individual [supported processors](#streams-extract-processors) for more on configuring specific processors.
 
 :::{note}
 Editing processors with JSON is planned for a future release, and additional processors may be supported over time.
 :::
 
 ### Add conditions to processors [streams-add-processor-conditions]
 
-You can provide a condition for each processor under **Optional fields**. Conditions are boolean expressions that are evaluated for each document. Provide a field, a value, and a comparator.
-Processors support these comparators:
+You can add conditions to processors so they only run on data that meets those conditions. Each condition is a boolean expression evaluated for every document.
+
+To add a condition:
+
+1. Select **Create** → **Create condition**.
+1. Provide a **Field**, a **Value**, and a comparator.
+1. Select **Create condition**.
+1. Select **Save changes**.
+
+:::{dropdown} Supported comparators
+Streams processors support the following comparators:
+
 - equals
 - not equals
 - less than
@@ -51,80 +74,85 @@ Processors support these comparators:
 - ends with
 - exists
 - not exists
+:::
 
 ### Preview changes [streams-preview-changes]
 
-Under **Processors for field extraction**, when you set pipeline processors to modify your documents, **Data preview** shows you a preview of the results with additional filtering options depending on the outcome of the simulation.
+After you create processors, the **Data preview** tab simulates processor results with additional filtering options depending on the outcome of the simulation.
 
 When you add or edit processors, the **Data preview** updates automatically.
 
 :::{note}
-To avoid unexpected results, we recommend adding processors rather than removing or reordering existing processors.
+To avoid unexpected results, it's best to add processors rather than remove or reorder existing ones.
 :::
 
-**Data preview** loads 100 documents from your existing data and runs your changes using them.
-For any newly added processors, this simulation is reliable. You can save individual processors during the preview, and even reorder them.
-Selecting **Save changes** applies your changes to the data stream.
+**Data preview** loads 100 documents from your existing data and runs your changes against them.
+For any newly created processors and conditions, the preview results are reliable, and you can freely create and reorder during the preview.
 
-If you edit the stream again, note the following:
-- Adding more processors to the end of the list will work as expected.
-- Changing existing processors or re-ordering them may cause unexpected results. Because the pipeline may have already processed the documents used for sampling, the UI cannot accurately simulate changes to existing data.
-- Adding a new processor and moving it before an existing processor may cause unexpected results. The UI only simulates the new processor, not the existing ones, so the simulation may not accurately reflect changes to existing data.
+Select **Save changes** to apply your changes to the data stream.
 
-![Screenshot of the Grok processor UI](<../../../images/logs-streams-grok.png>)
+If you edit the stream after saving your changes, keep the following in mind:
+
+- Adding processors to the end of the list will work as expected.
+- Editing or reordering existing processors can cause inaccurate results. Because the pipeline may have already processed the documents used for sampling, **Data preview** cannot accurately simulate changes to existing data.
+- Adding a new processor and moving it before an existing processor may cause inaccurate results. **Data preview** only simulates the new processor, not the existing ones, so the simulation may not accurately reflect changes to existing data.
 
 ### Ignore failures [streams-ignore-failures]
 
-Turn on **Ignore failure** to ignore the processor if it fails. This is useful if you want to continue processing the document even if the processor fails.
+Each processor has the option to **Ignore failures**. When enabled, processing of the document continues when the processor fails.
 
 ### Ignore missing fields [streams-ignore-missing-fields]
 
-Turn on **Ignore missing fields** to ignore the processor if the field is not present. This is useful if you want to continue processing the document even if the field is not present.
+Dissect, grok, and rename processors include the **Ignore missing fields** option. When enabled, processing of the document continues when a source field is missing.
 
-## Detect and handle failures [streams-detect-failures]
+## Detect and resolve failures [streams-detect-failures]
 
-Documents fail processing for different reasons. Streams helps you to easily find and handle failures before deploying changes.
+Documents can fail processing for various reasons. Streams helps you identify and resolve these issues before deploying changes.
 
-In the following screenshot, the **Failed** percentage shows that not all messages matched the provided Grok pattern:
+In the following screenshot, the **Failed** percentage indicates that not all messages matched the provided grok pattern:
 
 ![Screenshot showing some failed documents](<../../../images/logs-streams-parsed.png>)
 
-You can filter your documents by selecting **Parsed** or **Failed** at the top of the table. Select **Failed** to see the documents that weren't parsed correctly:
+You can filter your documents by selecting **Parsed** or **Failed** at the top of the table.
+Selecting **Failed** shows the documents that weren't parsed correctly:
 
 ![Screenshot showing the documents UI with Failed selected](<../../../images/logs-streams-failures.png>)
 
 Failures are displayed at the bottom of the process editor:
 
 ![Screenshot showing failure notifications](<../../../images/logs-streams-processor-failures.png>)
 
-These failures may require action, but in some cases, they serve more as warnings.
+Some failures may require fixes, while others simply serve as a warning.
 
-### Mapping conflicts
+### Mapping conflicts [streams-processing-mapping-conflicts]
 
-As part of processing, Streams also checks for mapping conflicts by simulating the change end to end. If a mapping conflict is detected, Streams marks the processor as failed and displays a failure message like the following:
+As part of processing, Streams simulates your changes end to end to check for mapping conflicts. If a conflict is detected, Streams marks the processor as failed and displays a message like the following:
 
 ![Screenshot showing mapping conflict notifications](<../../../images/logs-streams-mapping-conflicts.png>)
 
-You can then use the information in the failure message to find and troubleshoot mapping issues going forward.
+Use the information in the failure message to find and troubleshoot the mapping issues.
 
 ## Processor statistics and detected fields [streams-stats-and-detected-fields]
 
-Once saved, the processor provides a quick look at the processor's success rate and the fields that it added.
+Once saved, the processor displays its success rate and the fields it added.
 
 ![Screenshot showing field stats](<../../../images/logs-streams-field-stats.png>)
 
-## Advanced: How and where do these changes get applied to the underlying datastream? [streams-applied-changes]
+## Advanced: How and where do these changes get applied to the underlying data stream? [streams-applied-changes]
+
+% make sure this is all still accurate.
+
+When you save processors, Streams modifies the best-matching ingest pipeline for the data stream. In short, it either chooses the best-matching pipeline ending in `@custom` in your data stream, or it adds one for you.
 
-When you save processors, Streams modifies the "best matching" ingest pipeline for the data stream. In short, Streams either chooses the best matching pipeline ending in `@custom` that is already part of your data stream, or it adds one for you.
+Streams identifies the appropriate `@custom` pipeline (for example, `logs-myintegration@custom` or `logs@custom`) by checking the `default_pipeline` that is set on the data stream. You can view the default pipeline on the **Advanced** tab under **Ingest pipeline**.
 
-Streams identifies the appropriate @custom pipeline (for example, `logs-myintegration@custom` or `logs@custom`).
-It checks the default_pipeline that is set on the datastream.
+In this default pipeline, Streams locates the last processor that calls a pipeline ending in `@custom`.
+- For integrations, this would result in a pipeline name like `logs-myintegration@custom`.
+- Without an integration, the only `@custom` pipeline available may be `logs@custom`.
 
-You can view the default pipeline at **Manage stream** → **Advanced** under **Ingest pipeline**.
-In this default pipeline, we locate the last processor that calls a pipeline ending in `@custom`. For integrations, this would result in a pipeline name like `logs-myintegration@custom`. Without an integration, the only `@custom` pipeline available may be `logs@custom`.
+If no default pipeline is detected, Streams adds a default pipeline to the data stream by updating the index templates.
 
-- If no default pipeline is detected, Streams adds a default pipeline to the data stream by updating the index templates.
-- If a default pipeline is detected, but it does not contain a custom pipeline, Streams adds the pipeline processor directly to the pipeline.
+If a default pipeline is detected, but it does not contain a custom pipeline, Streams adds the pipeline processor directly to the pipeline.
 
 Streams then adds a pipeline processor to the end of that `@custom` pipeline. This processor definition directs matching documents to a dedicated pipeline managed by Streams called `<data_stream_name>@stream.processing`:
 
@@ -150,7 +178,4 @@ You can still add your own processors manually to the `@custom` pipeline if need
 ## Known limitations [streams-known-limitations]
 
 - Streams does not support all processors. We are working on adding more processors in the future.
-- Streams does not support all processor options. We are working on adding more options in the future.
-- The data preview simulation may not accurately reflect the changes to the existing data when editing existing processors or re-ordering them.
-- Dots in field names are not supported. You can use the dot expand processor in the `@custom` pipeline as a workaround. You need to manually add the dot expand processor.
-- Providing any arbitrary JSON in the Streams UI is not supported. We are working on adding this in the future.
+- The data preview simulation may not accurately reflect the changes to the existing data when editing existing processors or re-ordering them. We will allow proper simulations using original documents in a future version.

solutions/observability/streams/management/extract/append.md

@@ -0,0 +1,18 @@
+---
+applies_to:
+  serverless: ga
+  stack: preview 9.1, ga 9.2
+---
+# Append processor [streams-append-processor]
+% Need use cases
+
+Use the append processor to add a value to an existing array field, or create the field as an array if it doesn’t exist.
+
+To use an append processor:
+
+1. Select **Create** → **Create processor**.
+1. Select **Append** from the **Processor** menu.
+1. Set **Source Field** to the field you want append values to.
+1. Set **Target field** to the values you want to append to the **Source Field**.
+
+This functionality uses the {{es}} rename pipeline processor. Refer to the [rename processor](elasticsearch://reference/enrich-processor/rename-processor.md) {{es}} documentation for more information.