Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions redirects.yml
Original file line number Diff line number Diff line change
Expand Up @@ -364,3 +364,11 @@ redirects:
'deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md': 'elasticsearch://reference/elasticsearch/roles.md'
'deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md': 'elasticsearch://reference/elasticsearch/security-privileges.md'

# Related to https://github.com/elastic/docs-content/pull/3101
'solutions/observability/logs/streams/streams.md': 'solutions/observability/streams/streams.md'
'solutions/observability/logs/streams/management/extract.md': 'solutions/observability/streams/management/extract.md'
'solutions/observability/logs/streams/management/extract/date.md': 'solutions/observability/streams/management/extract/date.md'
'solutions/observability/logs/streams/management/extract/dissect.md': 'solutions/observability/streams/management/extract/dissect.md'
'solutions/observability/logs/streams/management/extract/grok.md': 'solutions/observability/streams/management/extract/grok.md'
'solutions/observability/logs/streams/management/retention.md': 'solutions/observability/streams/management/retention.md'
'solutions/observability/logs/streams/management/advanced.md': 'solutions/observability/streams/management/advanced.md'
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ If you edit the stream again, note the following:
- Changing existing processors or re-ordering them may cause unexpected results. Because the pipeline may have already processed the documents used for sampling, the UI cannot accurately simulate changes to existing data.
- Adding a new processor and moving it before an existing processor may cause unexpected results. The UI only simulates the new processor, not the existing ones, so the simulation may not accurately reflect changes to existing data.

![Screenshot of the Grok processor UI](<../../../../images/logs-streams-grok.png>)
![Screenshot of the Grok processor UI](<../../../images/logs-streams-grok.png>)

### Ignore failures [streams-ignore-failures]

Expand All @@ -87,31 +87,31 @@ Documents fail processing for different reasons. Streams helps you to easily fin

In the following screenshot, the **Failed** percentage shows that not all messages matched the provided Grok pattern:

![Screenshot showing some failed documents](<../../../../images/logs-streams-parsed.png>)
![Screenshot showing some failed documents](<../../../images/logs-streams-parsed.png>)

You can filter your documents by selecting **Parsed** or **Failed** at the top of the table. Select **Failed** to see the documents that weren't parsed correctly:

![Screenshot showing the documents UI with Failed selected](<../../../../images/logs-streams-failures.png>)
![Screenshot showing the documents UI with Failed selected](<../../../images/logs-streams-failures.png>)

Failures are displayed at the bottom of the process editor:

![Screenshot showing failure notifications](<../../../../images/logs-streams-processor-failures.png>)
![Screenshot showing failure notifications](<../../../images/logs-streams-processor-failures.png>)

These failures may require action, but in some cases, they serve more as warnings.

### Mapping conflicts

As part of processing, Streams also checks for mapping conflicts by simulating the change end to end. If a mapping conflict is detected, Streams marks the processor as failed and displays a failure message like the following:

![Screenshot showing mapping conflict notifications](<../../../../images/logs-streams-mapping-conflicts.png>)
![Screenshot showing mapping conflict notifications](<../../../images/logs-streams-mapping-conflicts.png>)

You can then use the information in the failure message to find and troubleshoot mapping issues going forward.

## Processor statistics and detected fields [streams-stats-and-detected-fields]

Once saved, the processor provides a quick look at the processor's success rate and the fields that it added.

![Screenshot showing field stats](<../../../../images/logs-streams-field-stats.png>)
![Screenshot showing field stats](<../../../images/logs-streams-field-stats.png>)

## Advanced: How and where do these changes get applied to the underlying datastream? [streams-applied-changes]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Instead of writing the Grok patterns by hand, you can use the **Generate Pattern

% TODO Elastic LLM?

![generated patterns](<../../../../../images/logs-streams-patterns.png>)
![generated patterns](<../../../../images/logs-streams-patterns.png>)

Click the plus icon next to the pattern to accept it and add it to the list of patterns used by the Grok processor.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ applies_to:

Use the **Data retention** tab on the **Manage stream** page to set how long your stream retains data and to get insight into your stream's data ingestion and storage size.

![Screenshot of the data retention UI](<../../../../images/logs-streams-retention.png>)
![Screenshot of the data retention UI](<../../../images/logs-streams-retention.png>)

The **Data retention** page is made up of the following components that can help you determine how long you want your stream to retain data:

Expand All @@ -30,15 +30,15 @@ The **Retention period** is the minimum number of days after which the data is d
1. Select **Edit data retention** → **Set specific retention days**.
1. From here, set the period of time you want to retain data for this stream.

To define a global default retention policy, refer to [project settings](../../../../../deploy-manage/deploy/elastic-cloud/project-settings.md).
To define a global default retention policy, refer to [project settings](../../../../deploy-manage/deploy/elastic-cloud/project-settings.md).

### Follow an ILM policy
```{applies_to}
stack: ga 9.1
```
[ILM policies](../../../../../manage-data/lifecycle/index-lifecycle-management.md) let you automate and standardize data retention across streams and other data streams. To have your streams follow an existing policy:
[ILM policies](../../../../manage-data/lifecycle/index-lifecycle-management.md) let you automate and standardize data retention across streams and other data streams. To have your streams follow an existing policy:

1. Select **Edit data retention** → **Use a lifecycle policy**.
1. Select a pre-defined ILM policy from the list.

You can also create a new ILM policy. Refer to [Configure a lifecycle policy](../../../../../manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) for more information.
You can also create a new ILM policy. Refer to [Configure a lifecycle policy](../../../../manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) for more information.
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ applies_to:
---

:::{warning}
Streams is currently in Technical Preview and only available on Elastic Cloud Serverless deployments. This feature may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
Streams is currently in Technical Preview. This feature may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
:::

# Streams
Expand All @@ -29,12 +29,12 @@ In {{obs-serverless}}, Streams is automatically available.

In {{stack}} version 9.1 and later, you can enable Streams in the {{observability}} Advanced Settings. To do this:

1. Go to **Management** > **Stack Management** > **Advanced Settings**, or search for "Advanced Settings" in the [global search field](../../../../explore-analyze/find-and-organize/find-apps-and-objects.md).
1. Go to **Management** > **Stack Management** > **Advanced Settings**, or search for "Advanced Settings" in the [global search field](../../../explore-analyze/find-and-organize/find-apps-and-objects.md).
1. Enable **Streams UI** under **Observability**.

In {{serverless-short}} or after enabling Streams in {{stack}}, access the UI in one of the following ways:

- Select **Streams** from the navigation menu or use the [global search field](../../../../explore-analyze/find-and-organize/find-apps-and-objects.md).
- Select **Streams** from the navigation menu or use the [global search field](../../../explore-analyze/find-and-organize/find-apps-and-objects.md).

- From **Discover**, expand a document's details flyout and select **Stream** or an action associated with the document's data stream. Streams will open filtered to only the selected stream. This only works for documents stored in a data stream.

Expand Down
10 changes: 10 additions & 0 deletions solutions/observability/streams/wired-streams.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
applies_to:
stack: all
serverless: all
products:
- id: observability
- id: cloud-serverless
---

# Wired streams
15 changes: 8 additions & 7 deletions solutions/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -452,15 +452,15 @@ toc:
- file: observability/logs/logs-index-template-reference.md
children:
- file: observability/logs/logs-index-template-defaults.md
- file: observability/logs/streams/streams.md
- file: observability/streams/streams.md
children:
- file: observability/logs/streams/management/extract.md
- file: observability/streams/management/extract.md
children:
- file: observability/logs/streams/management/extract/date.md
- file: observability/logs/streams/management/extract/dissect.md
- file: observability/logs/streams/management/extract/grok.md
- file: observability/logs/streams/management/retention.md
- file: observability/logs/streams/management/advanced.md
- file: observability/streams/management/extract/date.md
- file: observability/streams/management/extract/dissect.md
- file: observability/streams/management/extract/grok.md
- file: observability/streams/management/retention.md
- file: observability/streams/management/advanced.md
- file: observability/incident-management.md
children:
- file: observability/incident-management/alerting.md
Expand Down Expand Up @@ -503,6 +503,7 @@ toc:
children:
- file: observability/connect-to-own-local-llm.md
- file: observability/llm-performance-matrix.md
- hidden: observability/streams/wired-streams.md
- file: observability/observability-serverless-feature-tiers.md
- file: security.md
children:
Expand Down
Loading