Skip to content

Commit 7d80eff

Browse files
mdbirnstiehlrhr323
authored andcommitted
add Streams 9.2 updates (elastic#3357)
The PR closes elastic/docs-content-internal#282 and updates the docs for classic streams to go GA in 9.2 and wired streams to go tech preview in 9.2.
1 parent d820737 commit 7d80eff

21 files changed

+681
-155
lines changed
1.81 KB
Loading
-6.53 KB
Loading
-44.6 KB
Loading
-113 KB
Loading
Lines changed: 37 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,44 @@
11
---
22
applies_to:
3-
serverless: preview
4-
stack: preview 9.1
3+
serverless: ga
4+
stack: preview 9.1, ga 9.2
55
navigation_title: Configure advanced settings
66
---
77
# Configure advanced settings for streams [streams-advanced-settings]
88

9-
The **Advanced** tab on the **Manage stream** page shows the underlying configuration details of your stream. While Streams simplifies many configurations, it doesn't support modifying all pipelines and templates. From the **Advanced** tab, you can manually interact with the index or component templates or modify other ingest pipelines that used by the stream.
9+
The **Advanced** tab shows the underlying {{es}} configuration details and advanced configuration options for your stream.
1010

11-
This UI is intended for advanced users.
11+
You can use the **Advanced** tab to add [descriptions](#streams-advanced-description) or [features](#streams-advanced-features) that provide useful information to Stream's AI components. You can also [manually configure](#streams-advanced-index-config) the index or component templates or modify other ingest pipelines used by the stream.
12+
13+
## Stream description [streams-advanced-description]
14+
15+
Describe the data in the stream. AI features like system identification and significant events use this description when generating suggestions.
16+
17+
## Stream feature configuration [streams-advanced-features]
18+
19+
Streams analyzes your data and identifies features. Features are a way to classify some of the data you have in your stream.
20+
21+
Each feature has a natural language description and an optional filter which points to a subset of your data.
22+
23+
For example, in a stream of Kubernetes logs, the feature identification process would be able to identify that you have data from "nginx" which can be found by filtering for `WHERE service.name==nginx`. It would also include a description defining nginx.
24+
25+
Features provide useful information for AI processes, such as significant events, and are used as the foundation for them.
26+
27+
## Index configuration [streams-advanced-index-config]
28+
29+
:::{note}
30+
Processing and schema changes should typically be done through the Streams interface, and none of these configuration processes are required. This feature mainly exists to help advanced users maintain familiar workflows.
31+
:::
32+
33+
For classic streams, you can manually configure:
34+
35+
- [Index templates](../../../../manage-data/data-store/templates.md#index-templates)
36+
- [Component templates](../../../../manage-data/data-store/templates.md#component-templates)
37+
- [Pipelines](../../../../manage-data/ingest/transform-enrich.md)
38+
- [Data streams](../../../../manage-data/data-store/data-streams.md).
39+
40+
For both wired ({applies_to}`stack: preview 9.2`) and classic streams, you can manually configure:
41+
42+
- **Shards:** Control how the index is split across nodes. More shards can improve parallelism but may increase overhead.
43+
- **Replicas:** Define how many copies of the data exist. More replicas improve resilience and read performance but increase storage usage.
44+
- **Refresh interval:** Control how frequently new data becomes visible for search. A longer interval reduces resource usage; a short one makes data searchable sooner.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
applies_to:
3+
serverless: preview
4+
stack: preview 9.1, ga 9.2
5+
---
6+
7+
# Manage data quality [streams-data-retention]
8+
9+
After selecting a stream, use the **Data quality** tab to find failed and degraded documents in your stream. Use the following components to monitor the health of your data and identify and fix issues:
10+
11+
- **Degraded documents:** Documents with the `ignored` property usually because of malformed fields or exceeding the limit of total fields when `ignore_above:false`. This component shows the total number of degraded documents, the percentage, and status (**Good**, **Degraded**, **Poor**).
12+
- **Failed documents:**: Documents that were rejected during ingestion because of mapping conflicts or pipeline failures.
13+
- **Quality score:** Streams calculates the overall quality score (**Good**, **Degraded**, **Poor**) based on the percentage of degraded and failed documents.
14+
- **Trends over time:** A time-series chart so you can track how degraded and failed documents are accumulating over time. Use the date picker to zoom into a specific range and understand when problems are spiking.
15+
- **Issues:** {applies_to}`stack: preview 9.2`Find issues with specific fields, how often they've occurred, and when they've occurred.
16+
17+
## Failure store
18+
19+
A [failure store](../../../../manage-data/data-store/data-streams/failure-store.md) is a secondary set of indices inside a data stream, dedicated to storing failed documents. Instead of losing documents that are rejected during ingestion, a failure store retains it in a `::failures` index, so you can review failed documents to understand what went wrong and how to fix it.
20+
21+
### Required permissions
22+
To view and modify failure store in {{stack}}, you need the following data stream level privileges:
23+
- `read_failure_store`
24+
- `manage_failure_store`
25+
26+
For more information, refer to [Granting privileges for data streams and aliases](../../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md).
27+
28+
### Turn on failure stores
29+
In Streams, you need to turn on failure stores to get failed documents. To do this, select **Enable failure store** in the **Failed documents** component. From here you can set your failure store retention period.
30+
31+
For more information on data quality, refer to the [data set quality](../../data-set-quality-monitoring.md) documentation.

0 commit comments

Comments
 (0)