Skip to content

Commit e6ac0db

Browse files
authored
Reorganize 'data streams' section (#490)
As I proposed [here](elastic/docs-projects#324 (comment)), this moves the "Data streams" section out from underneath "Index types" (since I don't think they belong there), and puts it on the same level as "Mapping", "Text analysis", "Templates", etc. This also removes the "Vector DB" page since that feature isn't yet ready to document. **Still to do:** - Remove the "Index types" page. - Ensure that all parts of the old "Manage indexes" content is still covered somewhere. - Add info about Logs DB and TSDB to the main "Data streams" page and/or the "Index basics" page. See: [docs preview](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/490/manage-data/data-store/data-streams) Rel: elastic/docs-projects#324 --- <img width="568" alt="screen2" src="https://github.com/user-attachments/assets/b0bf800d-d632-4638-a87c-55bd38046c98" />
1 parent 78e3f97 commit e6ac0db

File tree

67 files changed

+142
-151
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+142
-151
lines changed

deploy-manage/deploy/self-managed/configure.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -513,7 +513,7 @@ $$$settings-explore-data-in-chart$$$ `xpack.discoverEnhanced.actions.exploreData
513513
::::{admonition} Deprecated in 8.11.0.
514514
:class: warning
515515

516-
Rollups are deprecated and will be removed in a future version. Use [downsampling](../../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead.
516+
Rollups are deprecated and will be removed in a future version. Use [downsampling](../../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead.
517517
::::
518518

519519

deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,7 @@ node.roles: [ data_content ]
198198

199199
Hot data nodes are part of the hot tier. The hot tier is the {{es}} entry point for time series data and holds your most-recent, most-frequently-searched time series data. Nodes in the hot tier need to be fast for both reads and writes, which requires more hardware resources and faster storage (SSDs). For resiliency, indices in the hot tier should be configured to use one or more replicas.
200200

201-
The hot tier is required. New indices that are part of a [data stream](../../../manage-data/data-store/index-types/data-streams.md) are automatically allocated to the hot tier.
201+
The hot tier is required. New indices that are part of a [data stream](../../../manage-data/data-store/data-streams.md) are automatically allocated to the hot tier.
202202

203203
To create a dedicated hot node, set:
204204

deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,5 +26,5 @@ You can also use the {{es}} API:
2626
* Edit the component template.
2727
* Store the updated component template using the [update component template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template).
2828

29-
After changing the component template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index.
29+
After changing the component template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index.
3030

deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,5 +41,5 @@ You can also use the {{es}} API:
4141
::::
4242

4343

44-
After changing the index template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index.
44+
After changing the index template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index.
4545

deploy-manage/production-guidance/optimize-performance/disk-usage.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,4 +107,4 @@ Due to the fact that multiple documents are compressed together into blocks, it
107107

108108
## Roll up historical data [roll-up-historical-data]
109109

110-
Keeping older data can be useful for later analysis but is often avoided due to storage costs. You can use downsampling to summarize and store historical data at a fraction of the raw data’s storage cost. See [Downsampling a time series data stream](../../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md).
110+
Keeping older data can be useful for later analysis but is often avoided due to storage costs. You can use downsampling to summarize and store historical data at a fraction of the raw data’s storage cost. See [Downsampling a time series data stream](../../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md).

deploy-manage/production-guidance/optimize-performance/size-shards.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ mapped_pages:
55

66
# Size your shards [size-your-shards]
77

8-
Each index in {{es}} is divided into one or more shards, each of which may be replicated across multiple nodes to protect against hardware failures. If you are using [Data streams](../../../manage-data/data-store/index-types/data-streams.md) then each data stream is backed by a sequence of indices. There is a limit to the amount of data you can store on a single node so you can increase the capacity of your cluster by adding nodes and increasing the number of indices and shards to match. However, each index and shard has some overhead and if you divide your data across too many shards then the overhead can become overwhelming. A cluster with too many indices or shards is said to suffer from *oversharding*. An oversharded cluster will be less efficient at responding to searches and in extreme cases it may even become unstable.
8+
Each index in {{es}} is divided into one or more shards, each of which may be replicated across multiple nodes to protect against hardware failures. If you are using [Data streams](../../../manage-data/data-store/data-streams.md) then each data stream is backed by a sequence of indices. There is a limit to the amount of data you can store on a single node so you can increase the capacity of your cluster by adding nodes and increasing the number of indices and shards to match. However, each index and shard has some overhead and if you divide your data across too many shards then the overhead can become overwhelming. A cluster with too many indices or shards is said to suffer from *oversharding*. An oversharded cluster will be less efficient at responding to searches and in extreme cases it may even become unstable.
99

1010

1111
## Create a sharding strategy [create-a-sharding-strategy]
@@ -61,13 +61,13 @@ When possible, delete entire indices instead. {{es}} can immediately remove dele
6161

6262
### Use data streams and {{ilm-init}} for time series data [use-ds-ilm-for-time-series]
6363

64-
[Data streams](../../../manage-data/data-store/index-types/data-streams.md) let you store time series data across multiple, time-based backing indices. You can use [{{ilm}} ({{ilm-init}})](../../../manage-data/lifecycle/index-lifecycle-management.md) to automatically manage these backing indices.
64+
[Data streams](../../../manage-data/data-store/data-streams.md) let you store time series data across multiple, time-based backing indices. You can use [{{ilm}} ({{ilm-init}})](../../../manage-data/lifecycle/index-lifecycle-management.md) to automatically manage these backing indices.
6565

6666
One advantage of this setup is [automatic rollover](../../../manage-data/lifecycle/index-lifecycle-management.md), which creates a new write index when the current one meets a defined `max_primary_shard_size`, `max_age`, `max_docs`, or `max_size` threshold. When an index is no longer needed, you can use {{ilm-init}} to automatically delete it and free up resources.
6767

6868
{{ilm-init}} also makes it easy to change your sharding strategy over time:
6969

70-
* **Want to decrease the shard count for new indices?**<br> Change the [`index.number_of_shards`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/index-types/modify-data-stream.md#data-streams-change-mappings-and-settings).
70+
* **Want to decrease the shard count for new indices?**<br> Change the [`index.number_of_shards`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings).
7171
* **Want larger shards or fewer backing indices?**<br> Increase your {{ilm-init}} policy’s [rollover threshold](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html).
7272
* **Need indices that span shorter intervals?**<br> Offset the increased shard count by deleting older indices sooner. You can do this by lowering the `min_age` threshold for your policy’s [delete phase](../../../manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md).
7373

deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ mapped_pages:
99
# Bi-directional disaster recovery [ccr-disaster-recovery-bi-directional-tutorial]
1010

1111

12-
Learn how to set up disaster recovery between two clusters based on bi-directional {{ccr}}. The following tutorial is designed for data streams which support [update by query](../../../manage-data/data-store/index-types/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/index-types/use-data-stream.md#delete-docs-in-a-data-stream-by-query). You can only perform these actions on the leader index.
12+
Learn how to set up disaster recovery between two clusters based on bi-directional {{ccr}}. The following tutorial is designed for data streams which support [update by query](../../../manage-data/data-store/data-streams/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/data-streams/use-data-stream.md#delete-docs-in-a-data-stream-by-query). You can only perform these actions on the leader index.
1313

1414
This tutorial works with {{ls}} as the source of ingestion. It takes advantage of a {{ls}} feature where [the {{ls}} output to {{es}}](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) can be load balanced across an array of hosts specified. {{beats}} and {{agents}} currently do not support multiple outputs. It should also be possible to set up a proxy (load balancer) to redirect traffic without {{ls}} in this tutorial.
1515

deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ mapped_pages:
55

66
# Granting privileges for data streams and aliases [securing-aliases]
77

8-
{{es}} {{security-features}} allow you to secure operations executed against [data streams](../../../manage-data/data-store/index-types/data-streams.md) and [aliases](../../../manage-data/data-store/aliases.md).
8+
{{es}} {{security-features}} allow you to secure operations executed against [data streams](../../../manage-data/data-store/data-streams.md) and [aliases](../../../manage-data/data-store/aliases.md).
99

1010
## Data stream privileges [data-stream-privileges]
1111

@@ -28,7 +28,7 @@ Because the user is automatically granted the same privileges to the stream’s
2828
GET .ds-my-data-stream-2099.03.08-000002/_doc/2
2929
```
3030

31-
Later `my-data-stream` [rolls over](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream). This creates a new backing index: `.ds-my-data-stream-2099.03.09-000003`. Because the user still has the `read` privilege for `my-data-stream`, the user can retrieve documents directly from `.ds-my-data-stream-2099.03.09-000003`:
31+
Later `my-data-stream` [rolls over](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream). This creates a new backing index: `.ds-my-data-stream-2099.03.09-000003`. Because the user still has the `read` privilege for `my-data-stream`, the user can retrieve documents directly from `.ds-my-data-stream-2099.03.09-000003`:
3232

3333
```console
3434
GET .ds-my-data-stream-2099.03.09-000003/_doc/2

explore-analyze/discover/discover-get-started.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ Select the data you want to explore, and then specify the time range in which to
2828
1. Find **Discover** in the navigation menu or by using the [global search field](../../explore-analyze/find-and-organize/find-apps-and-objects.md).
2929
2. Select the data view that contains the data you want to explore.
3030
::::{tip}
31-
By default, {{kib}} requires a [{{data-source}}](../find-and-organize/data-views.md) to access your Elasticsearch data. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/index-types/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). When adding data to {{es}} using one of the many integrations available, sometimes data views are created automatically, but you can also create your own.
31+
By default, {{kib}} requires a [{{data-source}}](../find-and-organize/data-views.md) to access your Elasticsearch data. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). When adding data to {{es}} using one of the many integrations available, sometimes data views are created automatically, but you can also create your own.
3232

3333
You can also [try {{esql}}](try-esql.md), that let's you query any data you have in {{es}} without specifying a {{data-source}} first.
3434
::::

explore-analyze/find-and-organize/data-views.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ $$$management-cross-cluster-search$$$
3131
$$$data-views-read-only-access$$$
3232

3333

34-
By default, analytics features such as Discover require a {{data-source}} to access the {{es}} data that you want to explore. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/index-types/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). For example, a {{data-source}} can point to your log data from yesterday, or all indices that contain your data.
34+
By default, analytics features such as Discover require a {{data-source}} to access the {{es}} data that you want to explore. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). For example, a {{data-source}} can point to your log data from yesterday, or all indices that contain your data.
3535

3636
::::{note}
3737
In certain apps, you can also query your {{es}} data using [{{esql}}](../query-filter/languages/esql.md). With {{esql}}, data views aren't required.
@@ -109,7 +109,7 @@ Temporary {{data-sources}} are not available in the **Management** menu.
109109
::::{admonition}
110110
:class: warning
111111

112-
Rollups are deprecated. Use [downsampling](../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead.
112+
Rollups are deprecated. Use [downsampling](../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead.
113113
::::
114114

115115

0 commit comments

Comments
 (0)