Skip to content
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
a59928c
add Streams 9.2 updates
mdbirnstiehl Oct 6, 2025
010e19b
fix toc
mdbirnstiehl Oct 6, 2025
4c07374
update processing docs
mdbirnstiehl Oct 7, 2025
5f385b0
update processing docs
mdbirnstiehl Oct 7, 2025
fca203d
update known limitations
mdbirnstiehl Oct 7, 2025
925ef8b
fix limitations
mdbirnstiehl Oct 7, 2025
41968fe
add significant events
mdbirnstiehl Oct 7, 2025
16a72af
add title
mdbirnstiehl Oct 7, 2025
17974fb
update field mapping
mdbirnstiehl Oct 8, 2025
54995f0
update manual config
mdbirnstiehl Oct 9, 2025
9e03b09
update manual config
mdbirnstiehl Oct 9, 2025
e33ef58
Merge branch 'main' into streams-9.2-updates
mdbirnstiehl Oct 9, 2025
288097e
fix applies to
mdbirnstiehl Oct 9, 2025
463e295
update advanced tab
mdbirnstiehl Oct 9, 2025
faeb463
fix applies to tag
mdbirnstiehl Oct 9, 2025
260be1f
fix tag
mdbirnstiehl Oct 9, 2025
a5d13d4
update schema wording
mdbirnstiehl Oct 9, 2025
858ac32
add partitioning
mdbirnstiehl Oct 12, 2025
909158d
fix toc
mdbirnstiehl Oct 12, 2025
7306ee8
add notes
mdbirnstiehl Oct 13, 2025
2cf5c51
update partitions docs
mdbirnstiehl Oct 13, 2025
4fd50d0
fix note
mdbirnstiehl Oct 13, 2025
f033662
fix grok processor details
mdbirnstiehl Oct 13, 2025
194766e
fix typo
mdbirnstiehl Oct 13, 2025
2c5cf56
add info on sending data to wired streams
mdbirnstiehl Oct 13, 2025
7d7930b
add failure store info and link
mdbirnstiehl Oct 14, 2025
2258846
update retention
mdbirnstiehl Oct 14, 2025
e690dba
clean up
mdbirnstiehl Oct 14, 2025
8742a71
fix links
mdbirnstiehl Oct 14, 2025
52e5fb1
clean up
mdbirnstiehl Oct 14, 2025
596cf87
fix next steps
mdbirnstiehl Oct 14, 2025
b4a1fc6
clean up
mdbirnstiehl Oct 14, 2025
19afded
add wired vs classic
mdbirnstiehl Oct 14, 2025
c40f976
update advanced
mdbirnstiehl Oct 15, 2025
3fe5f03
Merge branch 'main' into streams-9.2-updates
mdbirnstiehl Oct 15, 2025
ec1c321
update manual pipline config links
mdbirnstiehl Oct 16, 2025
2a3ee4f
Merge branch 'main' into streams-9.2-updates
mdbirnstiehl Oct 16, 2025
0365f2b
make review updates
mdbirnstiehl Oct 17, 2025
f971239
fix link
mdbirnstiehl Oct 17, 2025
21b0db9
update screenshots
mdbirnstiehl Oct 20, 2025
3a68e2a
update wired streams
mdbirnstiehl Oct 20, 2025
7de3e47
fix link
mdbirnstiehl Oct 20, 2025
55f4166
Merge branch 'main' into streams-9.2-updates
mdbirnstiehl Oct 20, 2025
7734b1d
Merge branch 'main' into streams-9.2-updates
mdbirnstiehl Oct 21, 2025
ea250ec
fix screenshot formatting
mdbirnstiehl Oct 21, 2025
6ad9553
fix screenshot formatting
mdbirnstiehl Oct 21, 2025
200474f
add API info
mdbirnstiehl Oct 21, 2025
9da55dc
review updates
mdbirnstiehl Oct 22, 2025
d39e5f4
Merge branch 'main' into streams-9.2-updates
mdbirnstiehl Oct 22, 2025
04f6181
Merge branch 'main' into streams-9.2-updates
mdbirnstiehl Oct 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified solutions/images/logs-streams-failures.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified solutions/images/logs-streams-parsed.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified solutions/images/logs-streams-patterns.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified solutions/images/logs-streams-processor-failures.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
41 changes: 37 additions & 4 deletions solutions/observability/streams/management/advanced.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,44 @@
---
applies_to:
serverless: preview
stack: preview 9.1
serverless: ga
stack: preview 9.1, ga 9.2
navigation_title: Configure advanced settings
---
# Configure advanced settings for streams [streams-advanced-settings]

The **Advanced** tab on the **Manage stream** page shows the underlying configuration details of your stream. While Streams simplifies many configurations, it doesn't support modifying all pipelines and templates. From the **Advanced** tab, you can manually interact with the index or component templates or modify other ingest pipelines that used by the stream.
The **Advanced** tab shows the underlying {{es}} configuration details and advanced configuration options for your stream.

This UI is intended for advanced users.
You can use the **Advanced** tab to add [descriptions](#streams-advanced-description) or [features](#streams-advanced-features) that provide useful information to Stream's AI components. You can also [manually configure](#streams-advanced-index-config) the index or component templates or modify other ingest pipelines used by the stream.

## Stream description [streams-advanced-description]

Describe the data in the stream. AI features like system identification and significant events use this description when generating suggestions.

## Stream feature configuration [streams-advanced-features]

Streams analyzes your data and identifies features. Features are a way to classify some of the data you have in your stream.

Each feature has a natural language description and an optional filter which points to a subset of your data.

For example, in a stream of Kubernetes logs, the feature identification process would be able to identify that you have data from "nginx" which can be found by filtering for `WHERE service.name==nginx`. It would also include a description defining nginx.

Features provide useful information for AI processes, such as significant events, and are used as the foundation for them.

## Index configuration [streams-advanced-index-config]

:::{note}
Processing and schema changes should typically be done through the Streams interface, and none of these configuration processes are required. This feature mainly exists to help advanced users maintain familiar workflows.
:::

For classic streams, you can manually configure:

- [Index templates](../../../../manage-data/data-store/templates.md#index-templates)
- [Component templates](../../../../manage-data/data-store/templates.md#component-templates)
- [Pipelines](../../../../manage-data/ingest/transform-enrich.md)
- [Data streams](../../../../manage-data/data-store/data-streams.md).

For both wired ({applies_to}`stack: preview 9.2`) and classic streams, you can manually configure:

- **Shards:** Control how the index is split across nodes. More shards can improve parallelism but may increase overhead.
- **Replicas:** Define how many copies of the data exist. More replicas improve resilience and read performance but increase storage usage.
- **Refresh interval:** Control how frequently new data becomes visible for search. A longer interval reduces resource usage; a short one makes data searchable sooner.
31 changes: 31 additions & 0 deletions solutions/observability/streams/management/data-quality.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
applies_to:
serverless: preview
stack: preview 9.1, ga 9.2
---

# Manage data quality [streams-data-retention]

After selecting a stream, use the **Data quality** tab to find failed and degraded documents in your stream. Use the following components to monitor the health of your data and identify and fix issues:

- **Degraded documents:** Documents with the `ignored` property usually because of malformed fields or exceeding the limit of total fields when `ignore_above:false`. This component shows the total number of degraded documents, the percentage, and status (**Good**, **Degraded**, **Poor**).
- **Failed documents:**: Documents that were rejected during ingestion because of mapping conflicts or pipeline failures.
- **Quality score:** Streams calculates the overall quality score (**Good**, **Degraded**, **Poor**) based on the percentage of degraded and failed documents.
- **Trends over time:** A time-series chart so you can track how degraded and failed documents are accumulating over time. Use the date picker to zoom into a specific range and understand when problems are spiking.
- **Issues:** {applies_to}`stack: preview 9.2`Find issues with specific fields, how often they've occurred, and when they've occurred.

## Failure store

A [failure store](../../../../manage-data/data-store/data-streams/failure-store.md) is a secondary set of indices inside a data stream, dedicated to storing failed documents. Instead of losing documents that are rejected during ingestion, a failure store retains it in a `::failures` index, so you can review failed documents to understand what went wrong and how to fix it.

### Required permissions
To view and modify failure store in {{stack}}, you need the following data stream level privileges:
- `read_failure_store`
- `manage_failure_store`

For more information, refer to [Granting privileges for data streams and aliases](../../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md).

### Turn on failure stores
In Streams, you need to turn on failure stores to get failed documents. To do this, select **Enable failure store** in the **Failed documents** component. From here you can set your failure store retention period.

For more information on data quality, refer to the [data set quality](../../data-set-quality-monitoring.md) documentation.
Loading