Skip to content
2 changes: 1 addition & 1 deletion deploy-manage/deploy/self-managed/configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,7 @@ $$$settings-explore-data-in-chart$$$ `xpack.discoverEnhanced.actions.exploreData
::::{admonition} Deprecated in 8.11.0.
:class: warning

Rollups are deprecated and will be removed in a future version. Use [downsampling](../../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead.
Rollups are deprecated and will be removed in a future version. Use [downsampling](../../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead.
::::


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ node.roles: [ data_content ]

Hot data nodes are part of the hot tier. The hot tier is the {{es}} entry point for time series data and holds your most-recent, most-frequently-searched time series data. Nodes in the hot tier need to be fast for both reads and writes, which requires more hardware resources and faster storage (SSDs). For resiliency, indices in the hot tier should be configured to use one or more replicas.

The hot tier is required. New indices that are part of a [data stream](../../../manage-data/data-store/index-types/data-streams.md) are automatically allocated to the hot tier.
The hot tier is required. New indices that are part of a [data stream](../../../manage-data/data-store/data-streams.md) are automatically allocated to the hot tier.

To create a dedicated hot node, set:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,5 @@ You can also use the {{es}} API:
* Edit the component template.
* Store the updated component template using the [update component template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template).

After changing the component template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index.
After changing the component template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index.

Original file line number Diff line number Diff line change
Expand Up @@ -41,5 +41,5 @@ You can also use the {{es}} API:
::::


After changing the index template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index.
After changing the index template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index.

Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,4 @@ Due to the fact that multiple documents are compressed together into blocks, it

## Roll up historical data [roll-up-historical-data]

Keeping older data can be useful for later analysis but is often avoided due to storage costs. You can use downsampling to summarize and store historical data at a fraction of the raw data’s storage cost. See [Downsampling a time series data stream](../../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md).
Keeping older data can be useful for later analysis but is often avoided due to storage costs. You can use downsampling to summarize and store historical data at a fraction of the raw data’s storage cost. See [Downsampling a time series data stream](../../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md).
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ mapped_pages:

# Size your shards [size-your-shards]

Each index in {{es}} is divided into one or more shards, each of which may be replicated across multiple nodes to protect against hardware failures. If you are using [Data streams](../../../manage-data/data-store/index-types/data-streams.md) then each data stream is backed by a sequence of indices. There is a limit to the amount of data you can store on a single node so you can increase the capacity of your cluster by adding nodes and increasing the number of indices and shards to match. However, each index and shard has some overhead and if you divide your data across too many shards then the overhead can become overwhelming. A cluster with too many indices or shards is said to suffer from *oversharding*. An oversharded cluster will be less efficient at responding to searches and in extreme cases it may even become unstable.
Each index in {{es}} is divided into one or more shards, each of which may be replicated across multiple nodes to protect against hardware failures. If you are using [Data streams](../../../manage-data/data-store/data-streams.md) then each data stream is backed by a sequence of indices. There is a limit to the amount of data you can store on a single node so you can increase the capacity of your cluster by adding nodes and increasing the number of indices and shards to match. However, each index and shard has some overhead and if you divide your data across too many shards then the overhead can become overwhelming. A cluster with too many indices or shards is said to suffer from *oversharding*. An oversharded cluster will be less efficient at responding to searches and in extreme cases it may even become unstable.


## Create a sharding strategy [create-a-sharding-strategy]
Expand Down Expand Up @@ -61,13 +61,13 @@ When possible, delete entire indices instead. {{es}} can immediately remove dele

### Use data streams and {{ilm-init}} for time series data [use-ds-ilm-for-time-series]

[Data streams](../../../manage-data/data-store/index-types/data-streams.md) let you store time series data across multiple, time-based backing indices. You can use [{{ilm}} ({{ilm-init}})](../../../manage-data/lifecycle/index-lifecycle-management.md) to automatically manage these backing indices.
[Data streams](../../../manage-data/data-store/data-streams.md) let you store time series data across multiple, time-based backing indices. You can use [{{ilm}} ({{ilm-init}})](../../../manage-data/lifecycle/index-lifecycle-management.md) to automatically manage these backing indices.

One advantage of this setup is [automatic rollover](../../../manage-data/lifecycle/index-lifecycle-management.md), which creates a new write index when the current one meets a defined `max_primary_shard_size`, `max_age`, `max_docs`, or `max_size` threshold. When an index is no longer needed, you can use {{ilm-init}} to automatically delete it and free up resources.

{{ilm-init}} also makes it easy to change your sharding strategy over time:

* **Want to decrease the shard count for new indices?**<br> Change the [`index.number_of_shards`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/index-types/modify-data-stream.md#data-streams-change-mappings-and-settings).
* **Want to decrease the shard count for new indices?**<br> Change the [`index.number_of_shards`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings).
* **Want larger shards or fewer backing indices?**<br> Increase your {{ilm-init}} policy’s [rollover threshold](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html).
* **Need indices that span shorter intervals?**<br> Offset the increased shard count by deleting older indices sooner. You can do this by lowering the `min_age` threshold for your policy’s [delete phase](../../../manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ mapped_pages:
# Bi-directional disaster recovery [ccr-disaster-recovery-bi-directional-tutorial]


Learn how to set up disaster recovery between two clusters based on bi-directional {{ccr}}. The following tutorial is designed for data streams which support [update by query](../../../manage-data/data-store/index-types/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/index-types/use-data-stream.md#delete-docs-in-a-data-stream-by-query). You can only perform these actions on the leader index.
Learn how to set up disaster recovery between two clusters based on bi-directional {{ccr}}. The following tutorial is designed for data streams which support [update by query](../../../manage-data/data-store/data-streams/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/data-streams/use-data-stream.md#delete-docs-in-a-data-stream-by-query). You can only perform these actions on the leader index.

This tutorial works with {{ls}} as the source of ingestion. It takes advantage of a {{ls}} feature where [the {{ls}} output to {{es}}](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) can be load balanced across an array of hosts specified. {{beats}} and {{agents}} currently do not support multiple outputs. It should also be possible to set up a proxy (load balancer) to redirect traffic without {{ls}} in this tutorial.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ mapped_pages:

# Granting privileges for data streams and aliases [securing-aliases]

{{es}} {{security-features}} allow you to secure operations executed against [data streams](../../../manage-data/data-store/index-types/data-streams.md) and [aliases](../../../manage-data/data-store/aliases.md).
{{es}} {{security-features}} allow you to secure operations executed against [data streams](../../../manage-data/data-store/data-streams.md) and [aliases](../../../manage-data/data-store/aliases.md).

## Data stream privileges [data-stream-privileges]

Expand All @@ -28,7 +28,7 @@ Because the user is automatically granted the same privileges to the stream’s
GET .ds-my-data-stream-2099.03.08-000002/_doc/2
```

Later `my-data-stream` [rolls over](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream). This creates a new backing index: `.ds-my-data-stream-2099.03.09-000003`. Because the user still has the `read` privilege for `my-data-stream`, the user can retrieve documents directly from `.ds-my-data-stream-2099.03.09-000003`:
Later `my-data-stream` [rolls over](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream). This creates a new backing index: `.ds-my-data-stream-2099.03.09-000003`. Because the user still has the `read` privilege for `my-data-stream`, the user can retrieve documents directly from `.ds-my-data-stream-2099.03.09-000003`:

```console
GET .ds-my-data-stream-2099.03.09-000003/_doc/2
Expand Down
2 changes: 1 addition & 1 deletion explore-analyze/discover/discover-get-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Select the data you want to explore, and then specify the time range in which to
1. Find **Discover** in the navigation menu or by using the [global search field](../../explore-analyze/find-and-organize/find-apps-and-objects.md).
2. Select the data view that contains the data you want to explore.
::::{tip}
By default, {{kib}} requires a [{{data-source}}](../find-and-organize/data-views.md) to access your Elasticsearch data. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/index-types/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). When adding data to {{es}} using one of the many integrations available, sometimes data views are created automatically, but you can also create your own.
By default, {{kib}} requires a [{{data-source}}](../find-and-organize/data-views.md) to access your Elasticsearch data. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). When adding data to {{es}} using one of the many integrations available, sometimes data views are created automatically, but you can also create your own.

You can also [try {{esql}}](try-esql.md), that let's you query any data you have in {{es}} without specifying a {{data-source}} first.
::::
Expand Down
4 changes: 2 additions & 2 deletions explore-analyze/find-and-organize/data-views.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ $$$management-cross-cluster-search$$$
$$$data-views-read-only-access$$$


By default, analytics features such as Discover require a {{data-source}} to access the {{es}} data that you want to explore. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/index-types/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). For example, a {{data-source}} can point to your log data from yesterday, or all indices that contain your data.
By default, analytics features such as Discover require a {{data-source}} to access the {{es}} data that you want to explore. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). For example, a {{data-source}} can point to your log data from yesterday, or all indices that contain your data.

::::{note}
In certain apps, you can also query your {{es}} data using [{{esql}}](../query-filter/languages/esql.md). With {{esql}}, data views aren't required.
Expand Down Expand Up @@ -109,7 +109,7 @@ Temporary {{data-sources}} are not available in the **Management** menu.
::::{admonition}
:class: warning

Rollups are deprecated. Use [downsampling](../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead.
Rollups are deprecated. Use [downsampling](../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead.
::::


Expand Down
2 changes: 1 addition & 1 deletion explore-analyze/query-filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ mapped_pages:

You can use {{es}} as a basic document store to retrieve documents and their metadata. However, the real power of {{es}} comes from its advanced search and analytics capabilities.

* **{{es}} makes JSON documents searchable and aggregatable.** The documents are stored in an [index](/manage-data/data-store/index-basics.md) or [data stream](/manage-data/data-store/index-types/data-streams.md), which represent one type of data.
* **{{es}} makes JSON documents searchable and aggregatable.** The documents are stored in an [index](/manage-data/data-store/index-basics.md) or [data stream](/manage-data/data-store/data-streams.md), which represent one type of data.
* ***Searchable* means that you can filter the documents for conditions.** For example, you can filter for data "within the last 7 days" or data that "contains the word {{kib}}". {{kib}} provides many ways for you to construct filters, which are also called queries or search terms.
* ***Aggregatable* means that you can extract summaries from matching documents.** The simplest aggregation is **count**, and it is frequently used in combination with the **date histogram**, to see count over time. The **terms** aggregation shows the most frequent values.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ This tutorial uses a test dataset from [Atomic Red Team](https://github.com/redc

To get started:

1. Create an [index template](../../../manage-data/data-store/templates.md) with [data stream enabled](../../../manage-data/data-store/index-types/set-up-data-stream.md#create-index-template):
1. Create an [index template](../../../manage-data/data-store/templates.md) with [data stream enabled](../../../manage-data/data-store/data-streams/set-up-data-stream.md#create-index-template):

```console
PUT /_index_template/my-data-stream-template
Expand Down
2 changes: 1 addition & 1 deletion explore-analyze/transforms/transform-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ If your data uses the [date nanosecond data type](https://www.elastic.co/guide/e

### Data streams as destination indices are not supported [transform-data-streams-destination]

{{transforms-cap}} update data in the destination index which requires writing into the destination. [Data streams](../../manage-data/data-store/index-types/data-streams.md) are designed to be append-only, which means you cannot send update or delete requests directly to a data stream. For this reason, data streams are not supported as destination indices for {{transforms}}.
{{transforms-cap}} update data in the destination index which requires writing into the destination. [Data streams](../../manage-data/data-store/data-streams.md) are designed to be append-only, which means you cannot send update or delete requests directly to a data stream. For this reason, data streams are not supported as destination indices for {{transforms}}.

### ILM as destination index may cause duplicated documents [transform-ilm-destination]

Expand Down
2 changes: 1 addition & 1 deletion manage-data/data-store.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ mapped_pages:

[{{es}}](https://github.com/elastic/elasticsearch/) is a distributed search and analytics engine, scalable data store, and vector database built on Apache Lucene.

The documentation in this section details how {{es}} works as a _data store_ starting with the fundamental unit of storage in Elasticsearch: the index. An index is a collection of documents uniquely identified by a name or an alias. Read more in [Index basics](/manage-data/data-store/index-basics.md) and [Index types](/manage-data/data-store/index-types.md).
The documentation in this section details how {{es}} works as a _data store_ starting with the fundamental unit of storage in Elasticsearch: the index. An index is a collection of documents uniquely identified by a name or an alias. Read more in [Index basics](/manage-data/data-store/index-basics.md).

Then, learn how these documents and the fields they contain are stored and indexed in [Mapping](/manage-data/data-store/mapping.md), and how unstructured text is converted into a structured format that’s optimized for search in [Text analysis](/manage-data/data-store/text-analysis.md).

Expand Down
Loading
Loading