From ad3493a40884feb79da3b195e4a9fa4ed6809b6e Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 18 Feb 2025 15:25:10 -0500 Subject: [PATCH 1/5] Reorganize 'data streams' section --- .../deploy/self-managed/configure.md | 2 +- .../clusters-nodes-shards/node-roles.md | 2 +- ...g-monitoring-data-streams-elastic-agent.md | 2 +- ...ig-monitoring-data-streams-metricbeat-8.md | 2 +- .../optimize-performance/disk-usage.md | 2 +- .../optimize-performance/size-shards.md | 6 ++-- .../bi-directional-disaster-recovery.md | 2 +- ...ing-privileges-for-data-streams-aliases.md | 4 +-- .../discover/discover-get-started.md | 2 +- .../find-and-organize/data-views.md | 4 +-- explore-analyze/query-filter.md | 2 +- .../example-detect-threats-with-eql.md | 2 +- .../transforms/transform-limitations.md | 2 +- .../{index-types => }/data-streams.md | 24 ++++++++-------- .../downsampling-time-series-data-stream.md | 10 +++---- .../logs-data-stream.md | 4 +-- .../manage-data-stream.md | 2 +- .../modify-data-stream.md | 24 ++++++++-------- .../reindex-tsds.md | 4 +-- .../run-downsampling-manually.md | 10 +++---- ...ownsampling-using-data-stream-lifecycle.md | 6 ++-- .../run-downsampling-with-ilm.md | 10 +++---- .../set-up-data-stream.md | 2 +- .../set-up-tsds.md | 14 +++++----- .../time-series-data-stream-tsds.md | 4 +-- .../use-data-stream.md | 4 +-- manage-data/data-store/index-types.md | 4 +-- .../data-store/index-types/vectordb.md | 7 ----- .../data-store/mapping/explicit-mapping.md | 2 +- .../templates/index-template-management.md | 2 +- .../transform-enrich/example-parse-logs.md | 4 +-- .../transform-enrich/ingest-pipelines.md | 2 +- ...orial-create-data-stream-with-lifecycle.md | 2 +- manage-data/lifecycle/data-tiers.md | 6 ++-- .../configure-lifecycle-policy.md | 4 +-- .../index-management-in-kibana.md | 2 +- .../tutorial-automate-rollover.md | 8 +++--- manage-data/lifecycle/rollup.md | 2 +- .../lifecycle/rollup/getting-started-api.md | 2 +- .../rollup/getting-started-kibana.md | 2 +- .../migrating-from-rollup-to-downsampling.md | 6 ++-- .../rollup/rollup-aggregation-limitations.md | 2 +- .../rollup/rollup-search-limitations.md | 2 +- .../lifecycle/rollup/understanding-groups.md | 2 +- manage-data/toc.yml | 28 +++++++++---------- ...lasticsearch-to-manage-time-series-data.md | 4 +-- .../serverless/index-management.md | 4 +-- .../observability-log-monitoring.md | 2 +- ...servability-synthetics-manage-retention.md | 2 +- .../observability-synthetics-settings.md | 2 +- .../serverless/security-data-views-in-sec.md | 2 +- .../documents-indices.md | 2 +- .../elasticsearch-reference/index-mgmt.md | 2 +- .../snapshots-restore-snapshot.md | 4 +-- .../observability/logs-checklist.md | 2 +- .../synthetics-manage-retention.md | 2 +- .../observability/synthetics-settings.md | 2 +- solutions/security/detect-and-alert.md | 2 +- .../data-views-elastic-security.md | 2 +- .../index-lifecycle-management-errors.md | 2 +- .../elasticsearch/mapping-explosion.md | 2 +- .../elasticsearch/troubleshooting-searches.md | 4 +-- troubleshoot/ingest/fleet/common-problems.md | 2 +- 63 files changed, 138 insertions(+), 147 deletions(-) rename manage-data/data-store/{index-types => }/data-streams.md (70%) rename manage-data/data-store/{index-types => data-streams}/downsampling-time-series-data-stream.md (94%) rename manage-data/data-store/{index-types => data-streams}/logs-data-stream.md (98%) rename manage-data/data-store/{index-types => data-streams}/manage-data-stream.md (93%) rename manage-data/data-store/{index-types => data-streams}/modify-data-stream.md (90%) rename manage-data/data-store/{index-types => data-streams}/reindex-tsds.md (90%) rename manage-data/data-store/{index-types => data-streams}/run-downsampling-manually.md (95%) rename manage-data/data-store/{index-types => data-streams}/run-downsampling-using-data-stream-lifecycle.md (97%) rename manage-data/data-store/{index-types => data-streams}/run-downsampling-with-ilm.md (96%) rename manage-data/data-store/{index-types => data-streams}/set-up-data-stream.md (97%) rename manage-data/data-store/{index-types => data-streams}/set-up-tsds.md (93%) rename manage-data/data-store/{index-types => data-streams}/time-series-data-stream-tsds.md (97%) rename manage-data/data-store/{index-types => data-streams}/use-data-stream.md (96%) delete mode 100644 manage-data/data-store/index-types/vectordb.md diff --git a/deploy-manage/deploy/self-managed/configure.md b/deploy-manage/deploy/self-managed/configure.md index 0aa97dbc90..54ef1b25ed 100644 --- a/deploy-manage/deploy/self-managed/configure.md +++ b/deploy-manage/deploy/self-managed/configure.md @@ -513,7 +513,7 @@ $$$settings-explore-data-in-chart$$$ `xpack.discoverEnhanced.actions.exploreData ::::{admonition} Deprecated in 8.11.0. :class: warning - Rollups are deprecated and will be removed in a future version. Use [downsampling](../../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead. + Rollups are deprecated and will be removed in a future version. Use [downsampling](../../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: diff --git a/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md b/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md index c50a91a650..9b2722ad2d 100644 --- a/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md +++ b/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md @@ -198,7 +198,7 @@ node.roles: [ data_content ] Hot data nodes are part of the hot tier. The hot tier is the {{es}} entry point for time series data and holds your most-recent, most-frequently-searched time series data. Nodes in the hot tier need to be fast for both reads and writes, which requires more hardware resources and faster storage (SSDs). For resiliency, indices in the hot tier should be configured to use one or more replicas. -The hot tier is required. New indices that are part of a [data stream](../../../manage-data/data-store/index-types/data-streams.md) are automatically allocated to the hot tier. +The hot tier is required. New indices that are part of a [data stream](../../../manage-data/data-store/data-streams.md) are automatically allocated to the hot tier. To create a dedicated hot node, set: diff --git a/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md b/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md index b3f7491fd5..e5c4eb87fe 100644 --- a/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md +++ b/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md @@ -26,5 +26,5 @@ You can also use the {{es}} API: * Edit the component template. * Store the updated component template using the [update component template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template). -After changing the component template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index. +After changing the component template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index. diff --git a/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md b/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md index f83dbe3025..9449c8618e 100644 --- a/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md +++ b/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md @@ -41,5 +41,5 @@ You can also use the {{es}} API: :::: -After changing the index template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index. +After changing the index template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index. diff --git a/deploy-manage/production-guidance/optimize-performance/disk-usage.md b/deploy-manage/production-guidance/optimize-performance/disk-usage.md index 72d33f680e..e27cfd7080 100644 --- a/deploy-manage/production-guidance/optimize-performance/disk-usage.md +++ b/deploy-manage/production-guidance/optimize-performance/disk-usage.md @@ -107,4 +107,4 @@ Due to the fact that multiple documents are compressed together into blocks, it ## Roll up historical data [roll-up-historical-data] -Keeping older data can be useful for later analysis but is often avoided due to storage costs. You can use downsampling to summarize and store historical data at a fraction of the raw data’s storage cost. See [Downsampling a time series data stream](../../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md). +Keeping older data can be useful for later analysis but is often avoided due to storage costs. You can use downsampling to summarize and store historical data at a fraction of the raw data’s storage cost. See [Downsampling a time series data stream](../../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md). diff --git a/deploy-manage/production-guidance/optimize-performance/size-shards.md b/deploy-manage/production-guidance/optimize-performance/size-shards.md index b89030422a..b7bebf2a72 100644 --- a/deploy-manage/production-guidance/optimize-performance/size-shards.md +++ b/deploy-manage/production-guidance/optimize-performance/size-shards.md @@ -5,7 +5,7 @@ mapped_pages: # Size your shards [size-your-shards] -Each index in {{es}} is divided into one or more shards, each of which may be replicated across multiple nodes to protect against hardware failures. If you are using [Data streams](../../../manage-data/data-store/index-types/data-streams.md) then each data stream is backed by a sequence of indices. There is a limit to the amount of data you can store on a single node so you can increase the capacity of your cluster by adding nodes and increasing the number of indices and shards to match. However, each index and shard has some overhead and if you divide your data across too many shards then the overhead can become overwhelming. A cluster with too many indices or shards is said to suffer from *oversharding*. An oversharded cluster will be less efficient at responding to searches and in extreme cases it may even become unstable. +Each index in {{es}} is divided into one or more shards, each of which may be replicated across multiple nodes to protect against hardware failures. If you are using [Data streams](../../../manage-data/data-store/data-streams.md) then each data stream is backed by a sequence of indices. There is a limit to the amount of data you can store on a single node so you can increase the capacity of your cluster by adding nodes and increasing the number of indices and shards to match. However, each index and shard has some overhead and if you divide your data across too many shards then the overhead can become overwhelming. A cluster with too many indices or shards is said to suffer from *oversharding*. An oversharded cluster will be less efficient at responding to searches and in extreme cases it may even become unstable. ## Create a sharding strategy [create-a-sharding-strategy] @@ -61,13 +61,13 @@ When possible, delete entire indices instead. {{es}} can immediately remove dele ### Use data streams and {{ilm-init}} for time series data [use-ds-ilm-for-time-series] -[Data streams](../../../manage-data/data-store/index-types/data-streams.md) let you store time series data across multiple, time-based backing indices. You can use [{{ilm}} ({{ilm-init}})](../../../manage-data/lifecycle/index-lifecycle-management.md) to automatically manage these backing indices. +[Data streams](../../../manage-data/data-store/data-streams.md) let you store time series data across multiple, time-based backing indices. You can use [{{ilm}} ({{ilm-init}})](../../../manage-data/lifecycle/index-lifecycle-management.md) to automatically manage these backing indices. One advantage of this setup is [automatic rollover](../../../manage-data/lifecycle/index-lifecycle-management.md), which creates a new write index when the current one meets a defined `max_primary_shard_size`, `max_age`, `max_docs`, or `max_size` threshold. When an index is no longer needed, you can use {{ilm-init}} to automatically delete it and free up resources. {{ilm-init}} also makes it easy to change your sharding strategy over time: -* **Want to decrease the shard count for new indices?**
Change the [`index.number_of_shards`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/index-types/modify-data-stream.md#data-streams-change-mappings-and-settings). +* **Want to decrease the shard count for new indices?**
Change the [`index.number_of_shards`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings). * **Want larger shards or fewer backing indices?**
Increase your {{ilm-init}} policy’s [rollover threshold](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html). * **Need indices that span shorter intervals?**
Offset the increased shard count by deleting older indices sooner. You can do this by lowering the `min_age` threshold for your policy’s [delete phase](../../../manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md). diff --git a/deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md b/deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md index ca37398a23..5e2f1f8fe6 100644 --- a/deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md +++ b/deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md @@ -9,7 +9,7 @@ mapped_pages: # Bi-directional disaster recovery [ccr-disaster-recovery-bi-directional-tutorial] -Learn how to set up disaster recovery between two clusters based on bi-directional {{ccr}}. The following tutorial is designed for data streams which support [update by query](../../../manage-data/data-store/index-types/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/index-types/use-data-stream.md#delete-docs-in-a-data-stream-by-query). You can only perform these actions on the leader index. +Learn how to set up disaster recovery between two clusters based on bi-directional {{ccr}}. The following tutorial is designed for data streams which support [update by query](../../../manage-data/data-store/data-streams/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/data-streams/use-data-stream.md#delete-docs-in-a-data-stream-by-query). You can only perform these actions on the leader index. This tutorial works with {{ls}} as the source of ingestion. It takes advantage of a {{ls}} feature where [the {{ls}} output to {{es}}](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) can be load balanced across an array of hosts specified. {{beats}} and {{agents}} currently do not support multiple outputs. It should also be possible to set up a proxy (load balancer) to redirect traffic without {{ls}} in this tutorial. diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md b/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md index ab18fad749..b070116187 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md @@ -5,7 +5,7 @@ mapped_pages: # Granting privileges for data streams and aliases [securing-aliases] -{{es}} {{security-features}} allow you to secure operations executed against [data streams](../../../manage-data/data-store/index-types/data-streams.md) and [aliases](../../../manage-data/data-store/aliases.md). +{{es}} {{security-features}} allow you to secure operations executed against [data streams](../../../manage-data/data-store/data-streams.md) and [aliases](../../../manage-data/data-store/aliases.md). ## Data stream privileges [data-stream-privileges] @@ -28,7 +28,7 @@ Because the user is automatically granted the same privileges to the stream’s GET .ds-my-data-stream-2099.03.08-000002/_doc/2 ``` -Later `my-data-stream` [rolls over](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream). This creates a new backing index: `.ds-my-data-stream-2099.03.09-000003`. Because the user still has the `read` privilege for `my-data-stream`, the user can retrieve documents directly from `.ds-my-data-stream-2099.03.09-000003`: +Later `my-data-stream` [rolls over](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream). This creates a new backing index: `.ds-my-data-stream-2099.03.09-000003`. Because the user still has the `read` privilege for `my-data-stream`, the user can retrieve documents directly from `.ds-my-data-stream-2099.03.09-000003`: ```console GET .ds-my-data-stream-2099.03.09-000003/_doc/2 diff --git a/explore-analyze/discover/discover-get-started.md b/explore-analyze/discover/discover-get-started.md index ee541c227b..ebc0d803a1 100644 --- a/explore-analyze/discover/discover-get-started.md +++ b/explore-analyze/discover/discover-get-started.md @@ -28,7 +28,7 @@ Select the data you want to explore, and then specify the time range in which to 1. Find **Discover** in the navigation menu or by using the [global search field](../../explore-analyze/find-and-organize/find-apps-and-objects.md). 2. Select the data view that contains the data you want to explore. ::::{tip} - By default, {{kib}} requires a [{{data-source}}](../find-and-organize/data-views.md) to access your Elasticsearch data. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/index-types/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). When adding data to {{es}} using one of the many integrations available, sometimes data views are created automatically, but you can also create your own. + By default, {{kib}} requires a [{{data-source}}](../find-and-organize/data-views.md) to access your Elasticsearch data. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). When adding data to {{es}} using one of the many integrations available, sometimes data views are created automatically, but you can also create your own. You can also [try {{esql}}](try-esql.md), that let's you query any data you have in {{es}} without specifying a {{data-source}} first. :::: diff --git a/explore-analyze/find-and-organize/data-views.md b/explore-analyze/find-and-organize/data-views.md index 5f6cb7c6bd..d951b9b320 100644 --- a/explore-analyze/find-and-organize/data-views.md +++ b/explore-analyze/find-and-organize/data-views.md @@ -31,7 +31,7 @@ $$$management-cross-cluster-search$$$ $$$data-views-read-only-access$$$ -By default, analytics features such as Discover require a {{data-source}} to access the {{es}} data that you want to explore. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/index-types/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). For example, a {{data-source}} can point to your log data from yesterday, or all indices that contain your data. +By default, analytics features such as Discover require a {{data-source}} to access the {{es}} data that you want to explore. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). For example, a {{data-source}} can point to your log data from yesterday, or all indices that contain your data. ::::{note} In certain apps, you can also query your {{es}} data using [{{esql}}](../query-filter/languages/esql.md). With {{esql}}, data views aren't required. @@ -109,7 +109,7 @@ Temporary {{data-sources}} are not available in the **Management** menu. ::::{admonition} :class: warning -Rollups are deprecated. Use [downsampling](../../manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead. +Rollups are deprecated. Use [downsampling](../../manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: diff --git a/explore-analyze/query-filter.md b/explore-analyze/query-filter.md index 062fe89859..3b07c36568 100644 --- a/explore-analyze/query-filter.md +++ b/explore-analyze/query-filter.md @@ -10,7 +10,7 @@ mapped_pages: You can use {{es}} as a basic document store to retrieve documents and their metadata. However, the real power of {{es}} comes from its advanced search and analytics capabilities. -* **{{es}} makes JSON documents searchable and aggregatable.** The documents are stored in an [index](/manage-data/data-store/index-basics.md) or [data stream](/manage-data/data-store/index-types/data-streams.md), which represent one type of data. +* **{{es}} makes JSON documents searchable and aggregatable.** The documents are stored in an [index](/manage-data/data-store/index-basics.md) or [data stream](/manage-data/data-store/data-streams.md), which represent one type of data. * ***Searchable* means that you can filter the documents for conditions.** For example, you can filter for data "within the last 7 days" or data that "contains the word {{kib}}". {{kib}} provides many ways for you to construct filters, which are also called queries or search terms. * ***Aggregatable* means that you can extract summaries from matching documents.** The simplest aggregation is **count**, and it is frequently used in combination with the **date histogram**, to see count over time. The **terms** aggregation shows the most frequent values. diff --git a/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md b/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md index 95d8cc3f2a..3cb6149265 100644 --- a/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md +++ b/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md @@ -25,7 +25,7 @@ This tutorial uses a test dataset from [Atomic Red Team](https://github.com/redc To get started: -1. Create an [index template](../../../manage-data/data-store/templates.md) with [data stream enabled](../../../manage-data/data-store/index-types/set-up-data-stream.md#create-index-template): +1. Create an [index template](../../../manage-data/data-store/templates.md) with [data stream enabled](../../../manage-data/data-store/data-streams/set-up-data-stream.md#create-index-template): ```console PUT /_index_template/my-data-stream-template diff --git a/explore-analyze/transforms/transform-limitations.md b/explore-analyze/transforms/transform-limitations.md index a36e2042b2..9cea4ef14b 100644 --- a/explore-analyze/transforms/transform-limitations.md +++ b/explore-analyze/transforms/transform-limitations.md @@ -113,7 +113,7 @@ If your data uses the [date nanosecond data type](https://www.elastic.co/guide/e ### Data streams as destination indices are not supported [transform-data-streams-destination] -{{transforms-cap}} update data in the destination index which requires writing into the destination. [Data streams](../../manage-data/data-store/index-types/data-streams.md) are designed to be append-only, which means you cannot send update or delete requests directly to a data stream. For this reason, data streams are not supported as destination indices for {{transforms}}. +{{transforms-cap}} update data in the destination index which requires writing into the destination. [Data streams](../../manage-data/data-store/data-streams.md) are designed to be append-only, which means you cannot send update or delete requests directly to a data stream. For this reason, data streams are not supported as destination indices for {{transforms}}. ### ILM as destination index may cause duplicated documents [transform-ilm-destination] diff --git a/manage-data/data-store/index-types/data-streams.md b/manage-data/data-store/data-streams.md similarity index 70% rename from manage-data/data-store/index-types/data-streams.md rename to manage-data/data-store/data-streams.md index 2e94dd59c4..36d9a75933 100644 --- a/manage-data/data-store/index-types/data-streams.md +++ b/manage-data/data-store/data-streams.md @@ -14,7 +14,7 @@ applies: A data stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Data streams are well-suited for logs, events, metrics, and other continuously generated data. -You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream’s data. You can use [{{ilm}} ({{ilm-init}})](../../../manage-data/lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows. +You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream’s data. You can use [{{ilm}} ({{ilm-init}})](../lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows. ## Should you use a data stream? [should-you-use-a-data-stream] @@ -25,20 +25,20 @@ To determine whether you should use a data stream for your data, you should cons * You mostly perform indexing requests, with occasional updates and deletes. * You index documents without an `_id`, or when indexing documents with an explicit `_id` you expect first-write-wins behavior. -For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn’t fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See documentation for [managing time series data without a data stream](../../../manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams) for more information. +For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn’t fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See documentation for [managing time series data without a data stream](../lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams) for more information. -Keep in mind that some features such as [Time Series Data Streams (TSDS)](../../../manage-data/data-store/index-types/time-series-data-stream-tsds.md) and [data stream lifecycles](../../../manage-data/lifecycle/data-stream.md) require a data stream. +Keep in mind that some features such as [Time Series Data Streams (TSDS)](../data-store/data-streams/time-series-data-stream-tsds.md) and [data stream lifecycles](../lifecycle/data-stream.md) require a data stream. ## Backing indices [backing-indices] A data stream consists of one or more [hidden](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-hidden), auto-generated backing indices. -:::{image} ../../../images/elasticsearch-reference-data-streams-diagram.svg +:::{image} ../../images/elasticsearch-reference-data-streams-diagram.svg :alt: data streams diagram ::: -A data stream requires a matching [index template](../../../manage-data/data-store/templates.md). The template contains the mappings and settings used to configure the stream’s backing indices. +A data stream requires a matching [index template](templates.md). The template contains the mappings and settings used to configure the stream’s backing indices. Every document indexed to a data stream must contain a `@timestamp` field, mapped as a [`date`](https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html) or [`date_nanos`](https://www.elastic.co/guide/en/elasticsearch/reference/current/date_nanos.html) field type. If the index template doesn’t specify a mapping for the `@timestamp` field, {{es}} maps `@timestamp` as a `date` field with default options. @@ -51,7 +51,7 @@ The name pattern for the backing indices is an implementation detail and no inte When you submit a read request to a data stream, the stream routes the request to all its backing indices. -:::{image} ../../../images/elasticsearch-reference-data-streams-search-request.svg +:::{image} ../../images/elasticsearch-reference-data-streams-search-request.svg :alt: data streams search request ::: @@ -60,7 +60,7 @@ When you submit a read request to a data stream, the stream routes the request t The most recently created backing index is the data stream’s write index. The stream adds new documents to this index only. -:::{image} ../../../images/elasticsearch-reference-data-streams-index-request.svg +:::{image} ../../images/elasticsearch-reference-data-streams-index-request.svg :alt: data streams index request ::: @@ -78,7 +78,7 @@ You also cannot perform operations on a write index that may hinder indexing, su A [rollover](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) creates a new backing index that becomes the stream’s new write index. -We recommend using [{{ilm-init}}](../../../manage-data/lifecycle/index-lifecycle-management.md) to automatically roll over data streams when the write index reaches a specified age or size. If needed, you can also [manually roll over](../../../manage-data/data-store/index-types/use-data-stream.md#manually-roll-over-a-data-stream) a data stream. +We recommend using [{{ilm-init}}](../lifecycle/index-lifecycle-management.md) to automatically roll over data streams when the write index reaches a specified age or size. If needed, you can also [manually roll over](data-streams/use-data-stream.md#manually-roll-over-a-data-stream) a data stream. ## Generation [data-streams-generation] @@ -93,19 +93,19 @@ When a backing index is created, the index is named using the following conventi `` is the backing index’s creation date. Backing indices with a higher generation contain more recent data. For example, the `web-server-logs` data stream has a generation of `34`. The stream’s most recent backing index, created on 7 March 2099, is named `.ds-web-server-logs-2099.03.07-000034`. -Some operations, such as a [shrink](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-shrink) or [restore](../../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md), can change a backing index’s name. These name changes do not remove a backing index from its data stream. +Some operations, such as a [shrink](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-shrink) or [restore](../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md), can change a backing index’s name. These name changes do not remove a backing index from its data stream. The generation of the data stream can change without a new index being added to the data stream (e.g. when an existing backing index is shrunk). This means the backing indices for some generations will never exist. You should not derive any intelligence from the backing indices names. ## Append-only (mostly) [data-streams-append-only] -Data streams are designed for use cases where existing data is rarely updated. You cannot send update or deletion requests for existing documents directly to a data stream. However, you can still [update or delete documents](../../../manage-data/data-store/index-types/use-data-stream.md#update-delete-docs-in-a-backing-index) in a data stream by submitting requests directly to the document’s backing index. +Data streams are designed for use cases where existing data is rarely updated. You cannot send update or deletion requests for existing documents directly to a data stream. However, you can still [update or delete documents](data-streams/use-data-stream.md#update-delete-docs-in-a-backing-index) in a data stream by submitting requests directly to the document’s backing index. -If you need to update a larger number of documents in a data stream, you can use the [update by query](../../../manage-data/data-store/index-types/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/index-types/use-data-stream.md#delete-docs-in-a-data-stream-by-query) APIs. +If you need to update a larger number of documents in a data stream, you can use the [update by query](data-streams/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](data-streams/use-data-stream.md#delete-docs-in-a-data-stream-by-query) APIs. ::::{tip} -If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See [Manage time series data without data streams](../../../manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams). +If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See [Manage time series data without data streams](../lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams). :::: diff --git a/manage-data/data-store/index-types/downsampling-time-series-data-stream.md b/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md similarity index 94% rename from manage-data/data-store/index-types/downsampling-time-series-data-stream.md rename to manage-data/data-store/data-streams/downsampling-time-series-data-stream.md index c57804c15a..d6e6eedc64 100644 --- a/manage-data/data-store/index-types/downsampling-time-series-data-stream.md +++ b/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md @@ -9,7 +9,7 @@ Downsampling provides a method to reduce the footprint of your [time series data Metrics solutions collect large amounts of time series data that grow over time. As that data ages, it becomes less relevant to the current state of the system. The downsampling process rolls up documents within a fixed time interval into a single summary document. Each summary document includes statistical representations of the original data: the `min`, `max`, `sum` and `value_count` for each metric. Data stream [time series dimensions](time-series-data-stream-tsds.md#time-series-dimension) are stored unchanged. -Downsampling, in effect, lets you to trade data resolution and precision for storage size. You can include it in an [{{ilm}} ({{ilm-init}})](../../lifecycle/index-lifecycle-management.md) policy to automatically manage the volume and associated cost of your metrics data at it ages. +Downsampling, in effect, lets you to trade data resolution and precision for storage size. You can include it in an [{{ilm}} ({{ilm-init}})](../index-types/lifecycle/index-lifecycle-management.md) policy to automatically manage the volume and associated cost of your metrics data at it ages. Check the following sections to learn more: @@ -131,17 +131,17 @@ The following restrictions and limitations apply for downsampling: * Only indices in a [time series data stream](time-series-data-stream-tsds.md) are supported. * Data is downsampled based on the time dimension only. All other dimensions are copied to the new index without any modification. * Within a data stream, a downsampled index replaces the original index and the original index is deleted. Only one index can exist for a given time period. -* A source index must be in read-only mode for the downsampling process to succeed. Check the [Run downsampling manually](run-downsampling-manually.md) example for details. +* A source index must be in read-only mode for the downsampling process to succeed. Check the [Run downsampling manually](../index-types/run-downsampling-manually.md) example for details. * Downsampling data for the same period many times (downsampling of a downsampled index) is supported. The downsampling interval must be a multiple of the interval of the downsampled index. * Downsampling is provided as an ILM action. See [Downsample](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-downsample.html). * The new, downsampled index is created on the data tier of the original index and it inherits its settings (for example, the number of shards and replicas). * The numeric `gauge` and `counter` [metric types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-field-meta.html) are supported. -* The downsampling configuration is extracted from the time series data stream [index mapping](set-up-tsds.md#create-tsds-index-template). The only additional required setting is the downsampling `fixed_interval`. +* The downsampling configuration is extracted from the time series data stream [index mapping](../index-types/set-up-tsds.md#create-tsds-index-template). The only additional required setting is the downsampling `fixed_interval`. ## Try it out [try-out-downsampling] -To take downsampling for a test run, try our example of [running downsampling manually](run-downsampling-manually.md). +To take downsampling for a test run, try our example of [running downsampling manually](../index-types/run-downsampling-manually.md). -Downsampling can easily be added to your ILM policy. To learn how, try our [Run downsampling with ILM](run-downsampling-with-ilm.md) example. +Downsampling can easily be added to your ILM policy. To learn how, try our [Run downsampling with ILM](../index-types/run-downsampling-with-ilm.md) example. diff --git a/manage-data/data-store/index-types/logs-data-stream.md b/manage-data/data-store/data-streams/logs-data-stream.md similarity index 98% rename from manage-data/data-store/index-types/logs-data-stream.md rename to manage-data/data-store/data-streams/logs-data-stream.md index 9d565c7b72..37c9934310 100644 --- a/manage-data/data-store/index-types/logs-data-stream.md +++ b/manage-data/data-store/data-streams/logs-data-stream.md @@ -67,7 +67,7 @@ In `logsdb` index mode, indices are sorted by the fields `host.name` and `@times You can override the default sort settings by manually configuring `index.sort.field` and `index.sort.order`. For more details, see [*Index Sorting*](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-index-sorting.html). -To modify the sort configuration of an existing data stream, update the data stream’s component templates, and then perform or wait for a [rollover](data-streams.md#data-streams-rollover). +To modify the sort configuration of an existing data stream, update the data stream’s component templates, and then perform or wait for a [rollover](../data-streams.md#data-streams-rollover). ::::{note} If you apply custom sort settings, the `@timestamp` field is injected into the mappings but is not automatically added to the list of sort fields. For best results, include it manually as the last sort field, with `desc` ordering. @@ -86,7 +86,7 @@ To avoid mapping conflicts, consider these options: * **Switch to a different [index mode](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-mode-setting)**: If resolving `host.name` mapping conflicts is not feasible, you can choose not to use `logsdb` mode. ::::{important} -On existing data streams, `logsdb` mode is applied on [rollover](data-streams.md#data-streams-rollover) (automatic or manual). +On existing data streams, `logsdb` mode is applied on [rollover](../data-streams.md#data-streams-rollover) (automatic or manual). :::: diff --git a/manage-data/data-store/index-types/manage-data-stream.md b/manage-data/data-store/data-streams/manage-data-stream.md similarity index 93% rename from manage-data/data-store/index-types/manage-data-stream.md rename to manage-data/data-store/data-streams/manage-data-stream.md index 309d14f64f..2b37234b27 100644 --- a/manage-data/data-store/index-types/manage-data-stream.md +++ b/manage-data/data-store/data-streams/manage-data-stream.md @@ -6,7 +6,7 @@ The value in the **Indices** column indicates the number of backing indices. Cli A value in the data retention column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. -In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](../../../manage-data/data-store/index-types/logs-data-stream.md). +In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](../index-types/logs-data-stream.md). :::{image} ../../../images/serverless-management-data-stream.png :alt: Data stream details diff --git a/manage-data/data-store/index-types/modify-data-stream.md b/manage-data/data-store/data-streams/modify-data-stream.md similarity index 90% rename from manage-data/data-store/index-types/modify-data-stream.md rename to manage-data/data-store/data-streams/modify-data-stream.md index 40a7194685..6385d1d85e 100644 --- a/manage-data/data-store/index-types/modify-data-stream.md +++ b/manage-data/data-store/data-streams/modify-data-stream.md @@ -8,19 +8,19 @@ mapped_pages: ## Change mappings and settings for a data stream [data-streams-change-mappings-and-settings] -Each [data stream](data-streams.md) has a [matching index template](set-up-data-stream.md#create-index-template). Mappings and index settings from this template are applied to new backing indices created for the stream. This includes the stream’s first backing index, which is auto-generated when the stream is created. +Each [data stream](../data-streams.md) has a [matching index template](../data-streams/set-up-data-stream.md#create-index-template). Mappings and index settings from this template are applied to new backing indices created for the stream. This includes the stream’s first backing index, which is auto-generated when the stream is created. Before creating a data stream, we recommend you carefully consider which mappings and settings to include in this template. If you later need to change the mappings or settings for a data stream, you have a few options: -* [Add a new field mapping to a data stream](modify-data-stream.md#add-new-field-mapping-to-a-data-stream) -* [Change an existing field mapping in a data stream](modify-data-stream.md#change-existing-field-mapping-in-a-data-stream) -* [Change a dynamic index setting for a data stream](modify-data-stream.md#change-dynamic-index-setting-for-a-data-stream) -* [Change a static index setting for a data stream](modify-data-stream.md#change-static-index-setting-for-a-data-stream) +* [Add a new field mapping to a data stream](../data-streams/modify-data-stream.md#add-new-field-mapping-to-a-data-stream) +* [Change an existing field mapping in a data stream](../data-streams/modify-data-stream.md#change-existing-field-mapping-in-a-data-stream) +* [Change a dynamic index setting for a data stream](../data-streams/modify-data-stream.md#change-dynamic-index-setting-for-a-data-stream) +* [Change a static index setting for a data stream](../data-streams/modify-data-stream.md#change-static-index-setting-for-a-data-stream) ::::{tip} -If your changes include modifications to existing field mappings or [static index settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-modules-settings), a reindex is often required to apply the changes to a data stream’s backing indices. If you are already performing a reindex, you can use the same process to add new field mappings and change [dynamic index settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-modules-settings). See [Use reindex to change mappings or settings](modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). +If your changes include modifications to existing field mappings or [static index settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-modules-settings), a reindex is often required to apply the changes to a data stream’s backing indices. If you are already performing a reindex, you can use the same process to add new field mappings and change [dynamic index settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-modules-settings). See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). :::: @@ -165,7 +165,7 @@ The documentation for each [mapping parameter](https://www.elastic.co/guide/en/e Except for supported mapping parameters, we don’t recommend you change the mapping or field data type of existing fields, even in a data stream’s matching index template or its backing indices. Changing the mapping of an existing field could invalidate any data that’s already indexed. -If you need to change the mapping of an existing field, create a new data stream and reindex your data into it. See [Use reindex to change mappings or settings](modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). +If you need to change the mapping of an existing field, create a new data stream and reindex your data into it. See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). ### Change a dynamic index setting for a data stream [change-dynamic-index-setting-for-a-data-stream] @@ -243,9 +243,9 @@ PUT /_index_template/my-data-stream-template 2. Adds the `sort.order` index setting. -If wanted, you can [roll over the data stream](use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the setting to the data stream’s write index. This affects any new data added to the stream after the rollover. However, it does not affect the data stream’s existing backing indices or existing data. +If wanted, you can [roll over the data stream](../data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the setting to the data stream’s write index. This affects any new data added to the stream after the rollover. However, it does not affect the data stream’s existing backing indices or existing data. -To apply static setting changes to existing backing indices, you must create a new data stream and reindex your data into it. See [Use reindex to change mappings or settings](modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). +To apply static setting changes to existing backing indices, you must create a new data stream and reindex your data into it. See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). ### Use reindex to change mappings or settings [data-streams-use-reindex-to-change-mappings-settings] @@ -278,7 +278,7 @@ Follow these steps: 2. Create or update an index template. This template should contain the mappings and settings you’d like to apply to the new data stream’s backing indices. - This index template must meet the [requirements for a data stream template](set-up-data-stream.md#create-index-template). It should also contain your previously chosen name or index pattern in the `index_patterns` property. + This index template must meet the [requirements for a data stream template](../data-streams/set-up-data-stream.md#create-index-template). It should also contain your previously chosen name or index pattern in the `index_patterns` property. ::::{tip} If you are only adding or changing a few things, we recommend you create a new template by copying an existing one and modifying it as needed. @@ -321,7 +321,7 @@ Follow these steps: 3. Use the [create data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream) to manually create the new data stream. The name of the data stream must match the index pattern defined in the new template’s `index_patterns` property. - We do not recommend [indexing new data to create this data stream](set-up-data-stream.md#create-data-stream). Later, you will reindex older data from an existing data stream into this new stream. This could result in one or more backing indices that contains a mix of new and old data. + We do not recommend [indexing new data to create this data stream](../data-streams/set-up-data-stream.md#create-data-stream). Later, you will reindex older data from an existing data stream into this new stream. This could result in one or more backing indices that contains a mix of new and old data. ::::{important} $$$data-stream-mix-new-old-data$$$ @@ -338,7 +338,7 @@ Follow these steps: PUT /_data_stream/new-data-stream ``` -4. If you do not want to mix new and old data in your new data stream, pause the indexing of new documents. While mixing old and new data is safe, it could interfere with data retention. See [Mixing new and old data in a data stream](modify-data-stream.md#data-stream-mix-new-old-data). +4. If you do not want to mix new and old data in your new data stream, pause the indexing of new documents. While mixing old and new data is safe, it could interfere with data retention. See [Mixing new and old data in a data stream](../data-streams/modify-data-stream.md#data-stream-mix-new-old-data). 5. If you use {{ilm-init}} to [automate rollover](../../lifecycle/index-lifecycle-management/tutorial-automate-rollover.md), reduce the {{ilm-init}} poll interval. This ensures the current write index doesn’t grow too large while waiting for the rollover check. By default, {{ilm-init}} checks rollover conditions every 10 minutes. The following [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) request lowers the `indices.lifecycle.poll_interval` setting to `1m` (one minute). diff --git a/manage-data/data-store/index-types/reindex-tsds.md b/manage-data/data-store/data-streams/reindex-tsds.md similarity index 90% rename from manage-data/data-store/index-types/reindex-tsds.md rename to manage-data/data-store/data-streams/reindex-tsds.md index 1dd3bf68f7..2ac7009bb4 100644 --- a/manage-data/data-store/index-types/reindex-tsds.md +++ b/manage-data/data-store/data-streams/reindex-tsds.md @@ -12,7 +12,7 @@ mapped_pages: ## Introduction [tsds-reindex-intro] -With reindexing, you can copy documents from an old [time-series data stream (TSDS)](time-series-data-stream-tsds.md) to a new one. Data streams support reindexing in general, with a few [restrictions](use-data-stream.md#reindex-with-a-data-stream). Still, time-series data streams introduce additional challenges due to tight control on the accepted timestamp range for each backing index they contain. Direct use of the reindex API would likely error out due to attempting to insert documents with timestamps that are outside the current acceptance window. +With reindexing, you can copy documents from an old [time-series data stream (TSDS)](../index-types/TSDS)](../data-streams/time-series-data-stream-tsds.md) to a new one. Data streams support reindexing in general, with a few [restrictions](use-data-stream.md#reindex-with-a-data-stream). Still, time-series data streams introduce additional challenges due to tight control on the accepted timestamp range for each backing index they contain. Direct use of the reindex API would likely error out due to attempting to insert documents with timestamps that are outside the current acceptance window. To avoid these limitations, use the process that is outlined below: @@ -28,7 +28,7 @@ To avoid these limitations, use the process that is outlined below: 5. Invoke the `rollover` api to create a new backing index that can receive new documents. ::::{note} -This process only applies to time-series data streams without [downsampling](downsampling-time-series-data-stream.md) configuration. Data streams with downsampling can only be re-indexed by re-indexing their backing indexes individually and adding them to an empty destination data stream. +This process only applies to time-series data streams without [downsampling](../index-types/downsampling-time-series-data-stream.md) configuration. Data streams with downsampling can only be re-indexed by re-indexing their backing indexes individually and adding them to an empty destination data stream. :::: diff --git a/manage-data/data-store/index-types/run-downsampling-manually.md b/manage-data/data-store/data-streams/run-downsampling-manually.md similarity index 95% rename from manage-data/data-store/index-types/run-downsampling-manually.md rename to manage-data/data-store/data-streams/run-downsampling-manually.md index 78368aab73..f138d21cab 100644 --- a/manage-data/data-store/index-types/run-downsampling-manually.md +++ b/manage-data/data-store/data-streams/run-downsampling-manually.md @@ -9,7 +9,7 @@ mapped_pages: # Run downsampling manually [downsampling-manual] -The recommended way to [downsample](downsampling-time-series-data-stream.md) a [time-series data stream (TSDS)](time-series-data-stream-tsds.md) is [through index lifecycle management (ILM)](run-downsampling-with-ilm.md). However, if you’re not using ILM, you can downsample a TSDS manually. This guide shows you how, using typical Kubernetes cluster monitoring data. +The recommended way to [downsample](../index-types/downsampling-time-series-data-stream.md) a [time-series data stream (TSDS)](../index-types/TSDS)](../data-streams/time-series-data-stream-tsds.md) is [through index lifecycle management (ILM)](../index-types/ILM)](run-downsampling-with-ilm.md). However, if you’re not using ILM, you can downsample a TSDS manually. This guide shows you how, using typical Kubernetes cluster monitoring data. To test out manual downsampling, follow these steps: @@ -22,8 +22,8 @@ To test out manual downsampling, follow these steps: ## Prerequisites [downsampling-manual-prereqs] -* Refer to the [TSDS prerequisites](set-up-tsds.md#tsds-prereqs). -* It is not possible to downsample a [data stream](data-streams.md) directly, nor multiple indices at once. It’s only possible to downsample one time series index (TSDS backing index). +* Refer to the [TSDS prerequisites](../index-types/set-up-tsds.md#tsds-prereqs). +* It is not possible to downsample a [data stream](../data-streams.md) directly, nor multiple indices at once. It’s only possible to downsample one time series index (TSDS backing index). * In order to downsample an index, it needs to be read-only. For a TSDS write index, this means it needs to be rolled over and made read-only first. * Downsampling uses UTC timestamps. * Downsampling needs at least one metric field to exist in the time series index. @@ -399,7 +399,7 @@ You can now delete the old backing index. But be aware this will delete the orig ## View the results [downsampling-manual-view-results] -Re-run the earlier search query (note that when querying downsampled indices there are [a few nuances to be aware of](downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)): +Re-run the earlier search query (note that when querying downsampled indices there are [a few nuances to be aware of](../index-types/downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)): ```console GET /my-data-stream/_search @@ -558,5 +558,5 @@ GET /my-data-stream/_search This example demonstrates how downsampling can dramatically reduce the number of documents stored for time series data, within whatever time boundaries you choose. It’s also possible to perform downsampling on already downsampled data, to further reduce storage and associated costs, as the time series data ages and the data resolution becomes less critical. -The recommended way to downsample a TSDS is with ILM. To learn more, try the [Run downsampling with ILM](run-downsampling-with-ilm.md) example. +The recommended way to downsample a TSDS is with ILM. To learn more, try the [Run downsampling with ILM](../index-types/run-downsampling-with-ilm.md) example. diff --git a/manage-data/data-store/index-types/run-downsampling-using-data-stream-lifecycle.md b/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md similarity index 97% rename from manage-data/data-store/index-types/run-downsampling-using-data-stream-lifecycle.md rename to manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md index bdf9a4865f..7aa7b680e4 100644 --- a/manage-data/data-store/index-types/run-downsampling-using-data-stream-lifecycle.md +++ b/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md @@ -9,7 +9,7 @@ mapped_pages: # Run downsampling using data stream lifecycle [downsampling-dsl] -This is a simplified example that allows you to see quickly how [downsampling](downsampling-time-series-data-stream.md) works as part of a datastream lifecycle to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with data stream lifecycle, follow these steps: +This is a simplified example that allows you to see quickly how [downsampling](../index-types/downsampling-time-series-data-stream.md) works as part of a datastream lifecycle to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with data stream lifecycle, follow these steps: 1. Check the [prerequisites](#downsampling-dsl-prereqs). 2. [Create an index template with data stream lifecycle](#downsampling-dsl-create-index-template). @@ -21,7 +21,7 @@ This is a simplified example that allows you to see quickly how [downsampling](d ## Prerequisites [downsampling-dsl-prereqs] -Refer to [time series data stream prerequisites](set-up-tsds.md#tsds-prereqs). +Refer to [time series data stream prerequisites](../index-types/set-up-tsds.md#tsds-prereqs). ## Create an index template with data stream lifecycle [downsampling-dsl-create-index-template] @@ -345,7 +345,7 @@ After the data stream lifecycle action was executed, original `.ds-datastream-20 ... ``` -Run a search query on the datastream (note that when querying downsampled indices there are [a few nuances to be aware of](downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)). +Run a search query on the datastream (note that when querying downsampled indices there are [a few nuances to be aware of](../index-types/downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)). ```console GET datastream/_search diff --git a/manage-data/data-store/index-types/run-downsampling-with-ilm.md b/manage-data/data-store/data-streams/run-downsampling-with-ilm.md similarity index 96% rename from manage-data/data-store/index-types/run-downsampling-with-ilm.md rename to manage-data/data-store/data-streams/run-downsampling-with-ilm.md index fff6ba0263..2dd8ff4d7f 100644 --- a/manage-data/data-store/index-types/run-downsampling-with-ilm.md +++ b/manage-data/data-store/data-streams/run-downsampling-with-ilm.md @@ -9,7 +9,7 @@ mapped_pages: # Run downsampling with ILM [downsampling-ilm] -This is a simplified example that allows you to see quickly how [downsampling](downsampling-time-series-data-stream.md) works as part of an ILM policy to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with ILM, follow these steps: +This is a simplified example that allows you to see quickly how [downsampling](../index-types/downsampling-time-series-data-stream.md) works as part of an ILM policy to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with ILM, follow these steps: 1. Check the [prerequisites](#downsampling-ilm-prereqs). 2. [Create an index lifecycle policy](#downsampling-ilm-policy). @@ -20,9 +20,9 @@ This is a simplified example that allows you to see quickly how [downsampling](d ## Prerequisites [downsampling-ilm-prereqs] -Refer to [time series data stream prerequisites](set-up-tsds.md#tsds-prereqs). +Refer to [time series data stream prerequisites](../index-types/set-up-tsds.md#tsds-prereqs). -Before running this example you may want to try the [Run downsampling manually](run-downsampling-manually.md) example. +Before running this example you may want to try the [Run downsampling manually](../index-types/run-downsampling-manually.md) example. ## Create an index lifecycle policy [downsampling-ilm-policy] @@ -346,7 +346,7 @@ After the ILM policy has taken effect, the original `.ds-datastream-2022.08.26-0 ... ``` -Run a search query on the datastream (note that when querying downsampled indices there are [a few nuances to be aware of](downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)). +Run a search query on the datastream (note that when querying downsampled indices there are [a few nuances to be aware of](../index-types/downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)). ```console GET datastream/_search @@ -464,4 +464,4 @@ GET /_data_stream/datastream/_stats?human=true This example demonstrates how downsampling works as part of an ILM policy to reduce the storage size of metrics data as it becomes less current and less frequently queried. -You can also try our [Run downsampling manually](run-downsampling-manually.md) example to learn how downsampling can work outside of an ILM policy. +You can also try our [Run downsampling manually](../index-types/run-downsampling-manually.md) example to learn how downsampling can work outside of an ILM policy. diff --git a/manage-data/data-store/index-types/set-up-data-stream.md b/manage-data/data-store/data-streams/set-up-data-stream.md similarity index 97% rename from manage-data/data-store/index-types/set-up-data-stream.md rename to manage-data/data-store/data-streams/set-up-data-stream.md index 5c5e23e4cb..db7836e9a7 100644 --- a/manage-data/data-store/index-types/set-up-data-stream.md +++ b/manage-data/data-store/data-streams/set-up-data-stream.md @@ -173,7 +173,7 @@ PUT _index_template/my-index-template ## Create the data stream [create-data-stream] -[Indexing requests](use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field. +[Indexing requests](../data-streams/use-data-stream.md#add-documents-to-a-data-stream) [Indexing requests](./use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field. To automatically create your data stream, submit an indexing request that targets the stream’s name. This name must match one of your index template’s index patterns. diff --git a/manage-data/data-store/index-types/set-up-tsds.md b/manage-data/data-store/data-streams/set-up-tsds.md similarity index 93% rename from manage-data/data-store/index-types/set-up-tsds.md rename to manage-data/data-store/data-streams/set-up-tsds.md index aba61d16d4..198175160f 100644 --- a/manage-data/data-store/index-types/set-up-tsds.md +++ b/manage-data/data-store/data-streams/set-up-tsds.md @@ -9,18 +9,18 @@ mapped_pages: # Set up a TSDS [set-up-tsds] -To set up a [time series data stream (TSDS)](time-series-data-stream-tsds.md), follow these steps: +To set up a [time series data stream (TSDS)](../index-types/TSDS)](../data-streams/time-series-data-stream-tsds.md), follow these steps: -1. Check the [prerequisites](set-up-tsds.md#tsds-prereqs). -2. [Create an index lifecycle policy](set-up-tsds.md#tsds-ilm-policy). -3. [Create an index template](set-up-tsds.md#create-tsds-index-template). -4. [Create the TSDS](set-up-tsds.md#create-tsds). -5. [Secure the TSDS](set-up-tsds.md#secure-tsds). +1. Check the [prerequisites](../index-types/set-up-tsds.md#tsds-prereqs). +2. [Create an index lifecycle policy](../index-types/set-up-tsds.md#tsds-ilm-policy). +3. [Create an index template](../index-types/set-up-tsds.md#create-tsds-index-template). +4. [Create the TSDS](../index-types/set-up-tsds.md#create-tsds). +5. [Secure the TSDS](../index-types/set-up-tsds.md#secure-tsds). ## Prerequisites [tsds-prereqs] -* Before you create a TSDS, you should be familiar with [data streams](data-streams.md) and [TSDS concepts](time-series-data-stream-tsds.md). +* Before you create a TSDS, you should be familiar with [data streams](../data-streams.md) and [TSDS concepts](time-series-data-stream-tsds.md). * To follow this tutorial, you must have the following permissions: * [Cluster privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `manage_ilm` and `manage_index_templates`. diff --git a/manage-data/data-store/index-types/time-series-data-stream-tsds.md b/manage-data/data-store/data-streams/time-series-data-stream-tsds.md similarity index 97% rename from manage-data/data-store/index-types/time-series-data-stream-tsds.md rename to manage-data/data-store/data-streams/time-series-data-stream-tsds.md index 1dec793060..a4723873a4 100644 --- a/manage-data/data-store/index-types/time-series-data-stream-tsds.md +++ b/manage-data/data-store/data-streams/time-series-data-stream-tsds.md @@ -12,7 +12,7 @@ You can use a TSDS to store metrics data more efficiently. In our benchmarks, me ## When to use a TSDS [when-to-use-tsds] -Both a [regular data stream](data-streams.md) and a TSDS can store timestamped metrics data. Only use a TSDS if you typically add metrics data to {{es}} in near real-time and `@timestamp` order. +Both a [regular data stream](../data-streams.md) and a TSDS can store timestamped metrics data. Only use a TSDS if you typically add metrics data to {{es}} in near real-time and `@timestamp` order. A TSDS is only intended for metrics data. For other timestamped data, such as logs or traces, use a [logs data stream](logs-data-stream.md) or regular data stream. @@ -215,4 +215,4 @@ Internally, each TSDS backing index uses [index sorting](https://www.elastic.co/ ## What’s next? [tsds-whats-next] -Now that you know the basics, you’re ready to [create a TSDS](time-series-data-stream-tsds.md) or [convert an existing data stream to a TSDS](time-series-data-stream-tsds.md). +Now that you know the basics, you’re ready to [create a TSDS](../data-streams/time-series-data-stream-tsds.md) or [convert an existing data stream to a TSDS](../data-streams/time-series-data-stream-tsds.md). diff --git a/manage-data/data-store/index-types/use-data-stream.md b/manage-data/data-store/data-streams/use-data-stream.md similarity index 96% rename from manage-data/data-store/index-types/use-data-stream.md rename to manage-data/data-store/data-streams/use-data-stream.md index afac70f62b..14ccb8cff3 100644 --- a/manage-data/data-store/index-types/use-data-stream.md +++ b/manage-data/data-store/data-streams/use-data-stream.md @@ -70,7 +70,7 @@ GET /_data_stream/my-data-stream/_stats?human=true ## Manually roll over a data stream [manually-roll-over-a-data-stream] -Use the [rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) to manually [roll over](data-streams.md#data-streams-rollover) a data stream. You have two options when manually rolling over: +Use the [rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) to manually [roll over](../data-streams.md#data-streams-rollover) a data stream. You have two options when manually rolling over: 1. To immediately trigger a rollover: @@ -107,7 +107,7 @@ POST /my-data-stream/_open/ ## Reindex with a data stream [reindex-with-a-data-stream] -Use the [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) to copy documents from an existing index, alias, or data stream to a data stream. Because data streams are [append-only](data-streams.md#data-streams-append-only), a reindex into a data stream must use an `op_type` of `create`. A reindex cannot update existing documents in a data stream. +Use the [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) to copy documents from an existing index, alias, or data stream to a data stream. Because data streams are [append-only](../data-streams.md#data-streams-append-only), a reindex into a data stream must use an `op_type` of `create`. A reindex cannot update existing documents in a data stream. ```console POST /_reindex diff --git a/manage-data/data-store/index-types.md b/manage-data/data-store/index-types.md index 08cdd1de48..5b2a3a4051 100644 --- a/manage-data/data-store/index-types.md +++ b/manage-data/data-store/index-types.md @@ -30,7 +30,7 @@ The **{{index-manage-app}}** page contains an overview of your indices. Investigate your data streams and address lifecycle management needs in the **Data Streams** view. -In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](index-types/logs-data-stream.md). +In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](data-streams/logs-data-stream.md). The value in the **Indices** column indicates the number of backing indices. Click this number to drill down into details. @@ -58,7 +58,7 @@ Create, edit, clone, and delete your index templates in the **Index Templates** :class: screenshot ::: -The default **logs** template uses the logsDB index mode to create a [logs data stream](index-types/logs-data-stream.md). +The default **logs** template uses the logsDB index mode to create a [logs data stream](data-streams/logs-data-stream.md). If you don’t have any templates, you can create one using the **Create template** wizard. diff --git a/manage-data/data-store/index-types/vectordb.md b/manage-data/data-store/index-types/vectordb.md deleted file mode 100644 index 2674581b30..0000000000 --- a/manage-data/data-store/index-types/vectordb.md +++ /dev/null @@ -1,7 +0,0 @@ -# vectordb - -% What needs to be done: Write from scratch - -% GitHub issue: docs-projects#325 - -% Scope notes: Confirm with PMs that this is going to be available in 9.0 and ask for any available resources. \ No newline at end of file diff --git a/manage-data/data-store/mapping/explicit-mapping.md b/manage-data/data-store/mapping/explicit-mapping.md index ec72d66322..2b915ff2f0 100644 --- a/manage-data/data-store/mapping/explicit-mapping.md +++ b/manage-data/data-store/mapping/explicit-mapping.md @@ -56,7 +56,7 @@ PUT /my-index-000001/_mapping Except for supported [mapping parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-params.html), you can’t change the mapping or field type of an existing field. Changing an existing field could invalidate data that’s already indexed. -If you need to change the mapping of a field in a data stream’s backing indices, see [Change mappings and settings for a data stream](../index-types/modify-data-stream.md#data-streams-change-mappings-and-settings). +If you need to change the mapping of a field in a data stream’s backing indices, see [Change mappings and settings for a data stream](../data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings). If you need to change the mapping of a field in other indices, create a new index with the correct mapping and [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) your data into that index. diff --git a/manage-data/data-store/templates/index-template-management.md b/manage-data/data-store/templates/index-template-management.md index 8efc7faf98..413a3df5f2 100644 --- a/manage-data/data-store/templates/index-template-management.md +++ b/manage-data/data-store/templates/index-template-management.md @@ -17,7 +17,7 @@ Create, edit, clone, and delete your index templates in the **Index Templates** :class: screenshot ::: -In {{serverless-full}}, the default **logs** template uses the logsDB index mode to create a [logs data stream](../index-types/logs-data-stream.md). +In {{serverless-full}}, the default **logs** template uses the logsDB index mode to create a [logs data stream](../data-streams/logs-data-stream.md). If you don’t have any templates, you can create one using the **Create template** wizard. diff --git a/manage-data/ingest/transform-enrich/example-parse-logs.md b/manage-data/ingest/transform-enrich/example-parse-logs.md index 071d535498..6b2f4d08c6 100644 --- a/manage-data/ingest/transform-enrich/example-parse-logs.md +++ b/manage-data/ingest/transform-enrich/example-parse-logs.md @@ -110,9 +110,9 @@ These logs contain a timestamp, IP address, and user agent. You want to give the 8. Click **Run the pipeline** and verify the pipeline worked as expected. 9. If everything looks correct, close the panel, and then click **Create pipeline**. - You’re now ready to index the logs data to a [data stream](../../data-store/index-types/data-streams.md). + You’re now ready to index the logs data to a [data stream](../../data-store/data-streams.md). -10. Create an [index template](../../data-store/templates.md) with [data stream enabled](../../data-store/index-types/set-up-data-stream.md#create-index-template). +10. Create an [index template](../../data-store/templates.md) with [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). ```console PUT _index_template/my-data-stream-template diff --git a/manage-data/ingest/transform-enrich/ingest-pipelines.md b/manage-data/ingest/transform-enrich/ingest-pipelines.md index 2e30115196..0afcb763bf 100644 --- a/manage-data/ingest/transform-enrich/ingest-pipelines.md +++ b/manage-data/ingest/transform-enrich/ingest-pipelines.md @@ -267,7 +267,7 @@ $$$pipeline-custom-logs-index-template$$$ } ``` -2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-default-pipeline) or [`index.final_pipeline`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/index-types/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs--*`. +2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-default-pipeline) or [`index.final_pipeline`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs--*`. You can create this template using {{kib}}'s [**Index Management**](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). diff --git a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md index 9dfdf8b223..157e0f248b 100644 --- a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md @@ -50,7 +50,7 @@ You can create a data stream in two ways: PUT _data_stream/my-data-stream ``` -2. By [indexing requests](../../data-store/index-types/use-data-stream.md#add-documents-to-a-data-stream) that target the stream’s name. This name must match one of your index template’s index patterns. +2. By [indexing requests](../../data-store/data-streams/use-data-stream.md#add-documents-to-a-data-stream) that target the stream’s name. This name must match one of your index template’s index patterns. ```console PUT my-data-stream/_bulk diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index 75c23810dc..0bec5e43b1 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -37,7 +37,7 @@ The way data tiers are used often depends on the data’s category: * Content data remains on the [content tier](/manage-data/lifecycle/data-tiers.md#content-tier) for its entire data lifecycle. * Time series data may progress through the descending temperature data tiers (hot, warm, cold, and frozen) according to your performance, resiliency, and data retention requirements. - You can automate these lifecycle transitions using the [data stream lifecycle](/manage-data/data-store/index-types/data-streams.md), or custom [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md). + You can automate these lifecycle transitions using the [data stream lifecycle](/manage-data/data-store/data-streams.md), or custom [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md). Learn more about each data tier, including when and how it should be used. @@ -54,7 +54,7 @@ The content tier is required and is often deployed within the same node grouping The hot tier is the {{es}} entry point for time series data and holds your most-recent, most-frequently-searched time series data. Nodes in the hot tier need to be fast for both reads and writes, which requires more hardware resources and faster storage (SSDs). For resiliency, indices in the hot tier should be configured to use one or more replicas. -The hot tier is required. New indices that are part of a [data stream](/manage-data/data-store/index-types/data-streams.md) are automatically allocated to the hot tier. +The hot tier is required. New indices that are part of a [data stream](/manage-data/data-store/data-streams.md) are automatically allocated to the hot tier. ### Warm tier [warm-tier] @@ -437,7 +437,7 @@ The [`index.routing.allocation.include._tier_preference`](https://www.elastic.co When you create an index, by default {{es}} sets the `_tier_preference` to `data_content` to automatically allocate the index shards to the content tier. -When {{es}} creates an index as part of a [data stream](/manage-data/data-store/index-types/data-streams.md), by default {{es}} sets the `_tier_preference` to `data_hot` to automatically allocate the index shards to the hot tier. +When {{es}} creates an index as part of a [data stream](/manage-data/data-store/data-streams.md), by default {{es}} sets the `_tier_preference` to `data_hot` to automatically allocate the index shards to the hot tier. At the time of index creation, you can override the default setting by explicitly setting the preferred value in one of two ways: diff --git a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md index 773867a7a1..53ec02ee16 100644 --- a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md +++ b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md @@ -71,7 +71,7 @@ The rollover action implicitly always rolls over a data stream or alias if one o To use a policy that triggers the rollover action, you need to configure the policy in the index template used to create each new index. You specify the name of the policy and the alias used to reference the rolling indices. ::::{tip} -An `index.lifecycle.rollover_alias` setting is only required if using {{ilm}} with an alias. It is unnecessary when using [Data Streams](../../data-store/index-types/data-streams.md). +An `index.lifecycle.rollover_alias` setting is only required if using {{ilm}} with an alias. It is unnecessary when using [Data Streams](../../data-store/data-streams.md). :::: @@ -108,7 +108,7 @@ PUT _index_template/my_template ### Create an initial managed index [create-initial-index] -When you set up policies for your own rolling indices, if you are not using the recommended [data streams](../../data-store/index-types/data-streams.md), you need to manually create the first index managed by a policy and designate it as the write index. +When you set up policies for your own rolling indices, if you are not using the recommended [data streams](../../data-store/data-streams.md), you need to manually create the first index managed by a policy and designate it as the write index. ::::{important} When you enable {{ilm}} for {{beats}} or the {{ls}} {{es}} output plugin, the necessary policies and configuration changes are applied automatically. You can modify the default policies, but you do not need to explicitly configure a policy or bootstrap an initial index. diff --git a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md index 53fa18e9c4..02a4da5dd4 100644 --- a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md +++ b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md @@ -5,7 +5,7 @@ mapped_pages: # Index management in Kibana [index-mgmt] -{{kib}}'s **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](../../data-store/index-types/data-streams.md), [index templates](../../data-store/templates.md), and [enrich policies](../../ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. +{{kib}}'s **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](../../data-store/data-streams.md), [index templates](../../data-store/templates.md), and [enrich policies](../../ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. To use these features, go to **Stack Management** > **Index Management**. diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md index e5c62b7ed0..cc72a65818 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md @@ -5,10 +5,10 @@ mapped_pages: # Tutorial: Automate rollover [getting-started-index-lifecycle-management] -When you continuously index timestamped documents into {{es}}, you typically use a [data stream](../../data-store/index-types/data-streams.md) so you can periodically [roll over](rollover.md) to a new index. This enables you to implement a [hot-warm-cold architecture](../data-tiers.md) to meet your performance requirements for your newest data, control costs over time, enforce retention policies, and still get the most out of your data. +When you continuously index timestamped documents into {{es}}, you typically use a [data stream](../../data-store/data-streams.md) so you can periodically [roll over](rollover.md) to a new index. This enables you to implement a [hot-warm-cold architecture](../data-tiers.md) to meet your performance requirements for your newest data, control costs over time, enforce retention policies, and still get the most out of your data. ::::{tip} -[Data streams](../../data-store/index-types/data-streams.md) are best suited for [append-only](../../data-store/index-types/data-streams.md#data-streams-append-only) use cases. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. You can still use [ILM](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md) to manage and [roll over](rollover.md) the alias’s indices. Skip to [Manage time series data without data streams](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams). +[Data streams](../../data-store/data-streams.md) are best suited for [append-only](../../data-store/data-streams.md#data-streams-append-only) use cases. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. You can still use [ILM](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md) to manage and [roll over](rollover.md) the alias’s indices. Skip to [Manage time series data without data streams](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#manage-time-series-data-without-data-streams). :::: ## Manage time series data with data streams [manage-time-series-data-with-data-streams] @@ -126,7 +126,7 @@ PUT _index_template/timeseries_template ### Create the data stream [ilm-gs-create-the-data-stream] -To get things started, index a document into the name or wildcard pattern defined in the `index_patterns` of the [index template](../../data-store/templates.md). As long as an existing data stream, index, or index alias does not already use the name, the index request automatically creates a corresponding data stream with a single backing index. {{es}} automatically indexes the request’s documents into this backing index, which also acts as the stream’s [write index](../../data-store/index-types/data-streams.md#data-stream-write-index). +To get things started, index a document into the name or wildcard pattern defined in the `index_patterns` of the [index template](../../data-store/templates.md). As long as an existing data stream, index, or index alias does not already use the name, the index request automatically creates a corresponding data stream with a single backing index. {{es}} automatically indexes the request’s documents into this backing index, which also acts as the stream’s [write index](../../data-store/data-streams.md#data-stream-write-index). For example, the following request creates the `timeseries` data stream and the first generation backing index called `.ds-timeseries-2099.03.08-000001`. @@ -208,7 +208,7 @@ The following response shows the data stream’s first generation backing index ## Manage time series data without data streams [manage-time-series-data-without-data-streams] -Even though [data streams](../../data-store/index-types/data-streams.md) are a convenient way to scale and manage time series data, they are designed to be append-only. We recognise there might be use-cases where data needs to be updated or deleted in place and the data streams don’t support delete and update requests directly, so the index APIs would need to be used directly on the data stream’s backing indices. In these cases we still recommend using a data stream. +Even though [data streams](../../data-store/data-streams.md) are a convenient way to scale and manage time series data, they are designed to be append-only. We recognise there might be use-cases where data needs to be updated or deleted in place and the data streams don’t support delete and update requests directly, so the index APIs would need to be used directly on the data stream’s backing indices. In these cases we still recommend using a data stream. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you can use an index alias instead of a data stream to manage indices containing the time series data and periodically roll over to a new index. diff --git a/manage-data/lifecycle/rollup.md b/manage-data/lifecycle/rollup.md index bec79054f6..536135368f 100644 --- a/manage-data/lifecycle/rollup.md +++ b/manage-data/lifecycle/rollup.md @@ -9,7 +9,7 @@ mapped_urls: ::::{admonition} Deprecated in 8.11.0. :class: warning -Rollups will be removed in a future version. Please [migrate](/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md) to [downsampling](/manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead. +Rollups will be removed in a future version. Please [migrate](/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md) to [downsampling](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: Keeping historical data around for analysis is extremely useful but often avoided due to the financial cost of archiving massive amounts of data. For example, your system may be generating 500 documents every second. That will generate 43 million documents per day, and nearly 16 billion documents a year. Retention periods are thus driven by financial realities rather than by the usefulness of extensive historical data. diff --git a/manage-data/lifecycle/rollup/getting-started-api.md b/manage-data/lifecycle/rollup/getting-started-api.md index 2680b3e903..10d2736f5a 100644 --- a/manage-data/lifecycle/rollup/getting-started-api.md +++ b/manage-data/lifecycle/rollup/getting-started-api.md @@ -9,7 +9,7 @@ mapped_pages: ::::{admonition} Deprecated in 8.11.0. :class: warning -Rollups will be removed in a future version. Please [migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](/manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead. +Rollups will be removed in a future version. Please [migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: ::::{warning} diff --git a/manage-data/lifecycle/rollup/getting-started-kibana.md b/manage-data/lifecycle/rollup/getting-started-kibana.md index 34c27309d6..ef78a43e24 100644 --- a/manage-data/lifecycle/rollup/getting-started-kibana.md +++ b/manage-data/lifecycle/rollup/getting-started-kibana.md @@ -9,7 +9,7 @@ mapped_pages: ::::{admonition} Deprecated in 8.11.0. :class: warning -Rollups are deprecated and will be removed in a future version. Use [downsampling](/manage-data/data-store/index-types/downsampling-time-series-data-stream.md) instead. +Rollups are deprecated and will be removed in a future version. Use [downsampling](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: A rollup job is a periodic task that aggregates data from indices specified by an index pattern, and then rolls it into a new index. Rollup indices are a good way to compactly store months or years of historical data for use in visualizations and reports. diff --git a/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md b/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md index 3965d09a48..3af5515f64 100644 --- a/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md +++ b/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md @@ -17,7 +17,7 @@ The following aspects of downsampling are easier or more robust: * No separate search API. Downsampled indices can be accessed via the search api and es|ql. * No separate rollup configuration. Downsampling uses the time series dimension and metric configuration from the mapping. -It isn’t possible to migrate all rollup usages to downsampling. The main requirement is that the data should be stored in Elasticsearch as [time series data stream (TSDS)](../../data-store/index-types/time-series-data-stream-tsds.md). Rollup usages that basically roll the data up by time and all dimensions can migrate to downsampling. +It isn’t possible to migrate all rollup usages to downsampling. The main requirement is that the data should be stored in Elasticsearch as [time series data stream (TSDS)](../../data-store/data-streams/time-series-data-stream-tsds.md). Rollup usages that basically roll the data up by time and all dimensions can migrate to downsampling. An example rollup usage that can be migrated to downsampling: @@ -50,7 +50,7 @@ PUT _rollup/job/sensor } ``` -The equivalent [time series data stream (TSDS)](../../data-store/index-types/time-series-data-stream-tsds.md) setup that uses downsampling via DSL: +The equivalent [time series data stream (TSDS)](../../data-store/data-streams/time-series-data-stream-tsds.md) setup that uses downsampling via DSL: ```console PUT _index_template/sensor-template @@ -92,7 +92,7 @@ PUT _index_template/sensor-template } ``` -The downsample configuration is included in the above template for a [time series data stream (TSDS)](../../data-store/index-types/time-series-data-stream-tsds.md). Only the `downsampling` part is necessary to enable downsampling, which indicates when to downsample to what fixed interval. +The downsample configuration is included in the above template for a [time series data stream (TSDS)](../../data-store/data-streams/time-series-data-stream-tsds.md). Only the `downsampling` part is necessary to enable downsampling, which indicates when to downsample to what fixed interval. 1. In the rollup job, the `cron` field determines when the rollup documents. In the index template, the `after` field determines when downsampling will rollup documents (note that this the time after a rollover has been performed). 2. In the rollup job, the `groups` field determines all dimensions of the group documents are rolled up to. In the index template, the fields with `time_series_dimension` set `true` and the `@timestamp` field determine the group. diff --git a/manage-data/lifecycle/rollup/rollup-aggregation-limitations.md b/manage-data/lifecycle/rollup/rollup-aggregation-limitations.md index 12b1d18508..fb7db4c4c4 100644 --- a/manage-data/lifecycle/rollup/rollup-aggregation-limitations.md +++ b/manage-data/lifecycle/rollup/rollup-aggregation-limitations.md @@ -8,7 +8,7 @@ mapped_pages: ::::{admonition} Deprecated in 8.11.0. :class: warning -Rollups will be removed in a future version. Please [migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/index-types/downsampling-time-series-data-stream.md) instead. +Rollups will be removed in a future version. Please [migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: diff --git a/manage-data/lifecycle/rollup/rollup-search-limitations.md b/manage-data/lifecycle/rollup/rollup-search-limitations.md index e73248fa2a..9edb089335 100644 --- a/manage-data/lifecycle/rollup/rollup-search-limitations.md +++ b/manage-data/lifecycle/rollup/rollup-search-limitations.md @@ -8,7 +8,7 @@ mapped_pages: ::::{admonition} Deprecated in 8.11.0. :class: warning -Rollups will be removed in a future version. Please [migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/index-types/downsampling-time-series-data-stream.md) instead. +Rollups will be removed in a future version. Please [migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: diff --git a/manage-data/lifecycle/rollup/understanding-groups.md b/manage-data/lifecycle/rollup/understanding-groups.md index 8338296274..1b0ee94a18 100644 --- a/manage-data/lifecycle/rollup/understanding-groups.md +++ b/manage-data/lifecycle/rollup/understanding-groups.md @@ -8,7 +8,7 @@ mapped_pages: ::::{admonition} Deprecated in 8.11.0. :class: warning -Rollups will be removed in a future version. Please [migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/index-types/downsampling-time-series-data-stream.md) instead. +Rollups will be removed in a future version. Please [migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: diff --git a/manage-data/toc.yml b/manage-data/toc.yml index 8e1efcb511..f25d1871be 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -5,23 +5,21 @@ toc: children: - file: data-store/index-basics.md - file: data-store/index-types.md + - file: data-store/data-streams.md children: - - file: data-store/index-types/data-streams.md + - file: data-store/data-streams/set-up-data-stream.md + - file: data-store/data-streams/use-data-stream.md + - file: data-store/data-streams/modify-data-stream.md + - file: data-store/data-streams/manage-data-stream.md + - file: data-store/data-streams/time-series-data-stream-tsds.md children: - - file: data-store/index-types/set-up-data-stream.md - - file: data-store/index-types/use-data-stream.md - - file: data-store/index-types/modify-data-stream.md - - file: data-store/index-types/manage-data-stream.md - - file: data-store/index-types/time-series-data-stream-tsds.md - children: - - file: data-store/index-types/set-up-tsds.md - - file: data-store/index-types/downsampling-time-series-data-stream.md - - file: data-store/index-types/run-downsampling-with-ilm.md - - file: data-store/index-types/run-downsampling-manually.md - - file: data-store/index-types/run-downsampling-using-data-stream-lifecycle.md - - file: data-store/index-types/reindex-tsds.md - - file: data-store/index-types/logs-data-stream.md - - file: data-store/index-types/vectordb.md + - file: data-store/data-streams/set-up-tsds.md + - file: data-store/data-streams/downsampling-time-series-data-stream.md + - file: data-store/data-streams/run-downsampling-with-ilm.md + - file: data-store/data-streams/run-downsampling-manually.md + - file: data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md + - file: data-store/data-streams/reindex-tsds.md + - file: data-store/data-streams/logs-data-stream.md - file: data-store/mapping.md children: - file: data-store/mapping/dynamic-mapping.md diff --git a/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md b/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md index e441b3e68a..4479095eaf 100644 --- a/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md +++ b/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md @@ -105,7 +105,7 @@ You can also use alternative implementations of these repository types, for inst ## Create or edit an index lifecycle policy [create-edit-index-lifecycle-policy] -A [data stream](data-store/index-types/data-streams.md) stores your data across multiple backing indices. {{ilm-init}} uses an [index lifecycle policy](lifecycle/index-lifecycle-management/index-lifecycle.md) to automatically move these indices through your data tiers. +A [data stream](data-store/data-streams.md) stores your data across multiple backing indices. {{ilm-init}} uses an [index lifecycle policy](lifecycle/index-lifecycle-management/index-lifecycle.md) to automatically move these indices through your data tiers. If you use {{fleet}} or {{agent}}, edit one of {{es}}'s built-in lifecycle policies. If you use a custom application, create your own policy. In either case, ensure your policy: @@ -333,7 +333,7 @@ PUT _index_template/my-index-template ## Add data to a data stream [add-data-to-data-stream] -[Indexing requests](data-store/index-types/use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field. +[Indexing requests](data-store/data-streams/use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field. To automatically create your data stream, submit an indexing request that targets the stream’s name. This name must match one of your index template’s index patterns. diff --git a/raw-migrated-files/docs-content/serverless/index-management.md b/raw-migrated-files/docs-content/serverless/index-management.md index 78e34f259b..1e850ce92f 100644 --- a/raw-migrated-files/docs-content/serverless/index-management.md +++ b/raw-migrated-files/docs-content/serverless/index-management.md @@ -25,7 +25,7 @@ The **{{index-manage-app}}** page contains an overview of your indices. Investigate your data streams and address lifecycle management needs in the **Data Streams** view. -In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](../../../manage-data/data-store/index-types/logs-data-stream.md). +In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](../../../manage-data/data-store/data-streams/logs-data-stream.md). The value in the **Indices** column indicates the number of backing indices. Click this number to drill down into details. @@ -53,7 +53,7 @@ Create, edit, clone, and delete your index templates in the **Index Templates** :class: screenshot ::: -The default **logs** template uses the logsDB index mode to create a [logs data stream](../../../manage-data/data-store/index-types/logs-data-stream.md). +The default **logs** template uses the logsDB index mode to create a [logs data stream](../../../manage-data/data-store/data-streams/logs-data-stream.md). If you don’t have any templates, you can create one using the **Create template** wizard. diff --git a/raw-migrated-files/docs-content/serverless/observability-log-monitoring.md b/raw-migrated-files/docs-content/serverless/observability-log-monitoring.md index 9c0e415d36..1b799507db 100644 --- a/raw-migrated-files/docs-content/serverless/observability-log-monitoring.md +++ b/raw-migrated-files/docs-content/serverless/observability-log-monitoring.md @@ -65,7 +65,7 @@ See [install {{agent}} in containers](https://www.elastic.co/guide/en/fleet/curr The following resources provide information on configuring your logs: -* [Data streams](../../../manage-data/data-store/index-types/data-streams.md): Efficiently store append-only time series data in multiple backing indices partitioned by time and size. +* [Data streams](../../../manage-data/data-store/data-streams.md): Efficiently store append-only time series data in multiple backing indices partitioned by time and size. * [Data views](../../../explore-analyze/find-and-organize/data-views.md): Query log entries from the data streams of specific datasets or namespaces. * [Index lifecycle management](../../../manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md): Configure the built-in logs policy based on your application’s performance, resilience, and retention requirements. * [Ingest pipeline](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md): Parse and transform log entries into a suitable format before indexing. diff --git a/raw-migrated-files/docs-content/serverless/observability-synthetics-manage-retention.md b/raw-migrated-files/docs-content/serverless/observability-synthetics-manage-retention.md index 82cf393a39..ab8a428de0 100644 --- a/raw-migrated-files/docs-content/serverless/observability-synthetics-manage-retention.md +++ b/raw-migrated-files/docs-content/serverless/observability-synthetics-manage-retention.md @@ -1,6 +1,6 @@ # Manage data retention [observability-synthetics-manage-retention] -When you set up a synthetic monitor, data from the monitor is saved in [{{es}} data streams](../../../manage-data/data-store/index-types/data-streams.md), an append-only structure in {{es}}. +When you set up a synthetic monitor, data from the monitor is saved in [{{es}} data streams](../../../manage-data/data-store/data-streams.md), an append-only structure in {{es}}. There are six data streams recorded by synthetic monitors: `http`, `tcp`, `icmp`, `browser`, `browser.network`, `browser.screenshot`. Elastic will retain data from each data stream for some time period, and the default time period varies by data stream. If you want to reduce the amount of storage required or store data for longer, you can customize how long to retain data for each data stream. diff --git a/raw-migrated-files/docs-content/serverless/observability-synthetics-settings.md b/raw-migrated-files/docs-content/serverless/observability-synthetics-settings.md index b227dcf4d5..84496e9de9 100644 --- a/raw-migrated-files/docs-content/serverless/observability-synthetics-settings.md +++ b/raw-migrated-files/docs-content/serverless/observability-synthetics-settings.md @@ -74,7 +74,7 @@ In the **Global parameters** tab, you can define variables and parameters. This ## Data retention [synthetics-settings-data-retention] -When you set up a synthetic monitor, data from the monitor is saved in [Elasticsearch data streams](../../../manage-data/data-store/index-types/data-streams.md), an append-only structure in Elasticsearch. You can customize how long synthetics data is stored by creating your own index lifecycle policy and attaching it to the relevant custom Component Template in Stack Management. +When you set up a synthetic monitor, data from the monitor is saved in [Elasticsearch data streams](../../../manage-data/data-store/data-streams.md), an append-only structure in Elasticsearch. You can customize how long synthetics data is stored by creating your own index lifecycle policy and attaching it to the relevant custom Component Template in Stack Management. In the **Data retention** tab, use the links to jump to the relevant policy for each data stream. Learn more about the data included in each data stream in [Manage data retention](../../../solutions/observability/apps/manage-data-retention.md). diff --git a/raw-migrated-files/docs-content/serverless/security-data-views-in-sec.md b/raw-migrated-files/docs-content/serverless/security-data-views-in-sec.md index 92f20df1db..62ca1b118b 100644 --- a/raw-migrated-files/docs-content/serverless/security-data-views-in-sec.md +++ b/raw-migrated-files/docs-content/serverless/security-data-views-in-sec.md @@ -1,6 +1,6 @@ # {{data-sources-cap}} in Elastic Security [security-data-views-in-sec] -{{data-sources-cap}} determine what data displays on {{elastic-sec}} pages with event or alert data. {{data-sources-cap}} are defined by the index patterns they include. Only data from {{es}} [indices](../../../manage-data/data-store/index-basics.md), [data streams](../../../manage-data/data-store/index-types/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html) specified in the active {{data-source}} will appear. +{{data-sources-cap}} determine what data displays on {{elastic-sec}} pages with event or alert data. {{data-sources-cap}} are defined by the index patterns they include. Only data from {{es}} [indices](../../../manage-data/data-store/index-basics.md), [data streams](../../../manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html) specified in the active {{data-source}} will appear. ::::{important} Custom indices are not included in the [default {{data-source}}](../../../solutions/security/get-started/data-views-elastic-security.md#default-data-view-security). Modify it or create a custom {{data-source}} to include custom indices. diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/documents-indices.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/documents-indices.md index 56d987eb57..2a8c01ee25 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/documents-indices.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/documents-indices.md @@ -10,7 +10,7 @@ The index is the fundamental unit of storage in {{es}}, a logical namespace for An index is a collection of documents uniquely identified by a name or an [alias](../../../manage-data/data-store/aliases.md). This unique name is important because it’s used to target the index in search queries and other operations. ::::{tip} -A closely related concept is a [data stream](../../../manage-data/data-store/index-types/data-streams.md). This index abstraction is optimized for append-only timestamped data, and is made up of hidden, auto-generated backing indices. If you’re working with timestamped data, we recommend the [Elastic Observability](https://www.elastic.co/guide/en/observability/current) solution for additional tools and optimized content. +A closely related concept is a [data stream](../../../manage-data/data-store/data-streams.md). This index abstraction is optimized for append-only timestamped data, and is made up of hidden, auto-generated backing indices. If you’re working with timestamped data, we recommend the [Elastic Observability](https://www.elastic.co/guide/en/observability/current) solution for additional tools and optimized content. :::: diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md index 2e3cac401b..b2d02a8ab6 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md @@ -1,6 +1,6 @@ # Index management in {{kib}} [index-mgmt] -{{kib}}'s **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](../../../manage-data/data-store/index-types/data-streams.md), [index templates](../../../manage-data/data-store/templates.md), and [enrich policies](../../../manage-data/ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. +{{kib}}'s **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](../../../manage-data/data-store/data-streams.md), [index templates](../../../manage-data/data-store/templates.md), and [enrich policies](../../../manage-data/ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. To use these features, go to **Stack Management** > **Index Management**. diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md index 7a19064687..30c79f4d16 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md @@ -24,13 +24,13 @@ This guide also provides tips for [restoring to another cluster](../../../deploy * You can only restore a snapshot to a running cluster with an elected [master node](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#master-node-role). The snapshot’s repository must be [registered](../../../deploy-manage/tools/snapshot-and-restore/self-managed.md) and available to the cluster. * The snapshot and cluster versions must be compatible. See [Snapshot compatibility](../../../deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility). * To restore a snapshot, the cluster’s global metadata must be writable. Ensure there aren’t any [cluster blocks](https://www.elastic.co/guide/en/elasticsearch/reference/current/misc-cluster-settings.html#cluster-read-only) that prevent writes. The restore operation ignores [index blocks](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-blocks.html). -* Before you restore a data stream, ensure the cluster contains a [matching index template](../../../manage-data/data-store/index-types/set-up-data-stream.md#create-index-template) with data stream enabled. To check, use {{kib}}'s [**Index Management**](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [get index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-index-template): +* Before you restore a data stream, ensure the cluster contains a [matching index template](../../../manage-data/data-store/data-streams/set-up-data-stream.md#create-index-template) with data stream enabled. To check, use {{kib}}'s [**Index Management**](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [get index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-index-template): ```console GET _index_template/*?filter_path=index_templates.name,index_templates.index_template.index_patterns,index_templates.index_template.data_stream ``` - If no such template exists, you can [create one](../../../manage-data/data-store/index-types/set-up-data-stream.md#create-index-template) or [restore a cluster state](../../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-entire-cluster) that contains one. Without a matching index template, a data stream can’t roll over or create backing indices. + If no such template exists, you can [create one](../../../manage-data/data-store/data-streams/set-up-data-stream.md#create-index-template) or [restore a cluster state](../../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-entire-cluster) that contains one. Without a matching index template, a data stream can’t roll over or create backing indices. * If your snapshot contains data from App Search or Workplace Search, ensure you’ve restored the [Enterprise Search encryption key](https://www.elastic.co/guide/en/enterprise-search/current/encryption-keys.html) before restoring the snapshot. :::{important} diff --git a/raw-migrated-files/observability-docs/observability/logs-checklist.md b/raw-migrated-files/observability-docs/observability/logs-checklist.md index c73d1c652e..ce56e4a496 100644 --- a/raw-migrated-files/observability-docs/observability/logs-checklist.md +++ b/raw-migrated-files/observability-docs/observability/logs-checklist.md @@ -76,7 +76,7 @@ To get started parsing and organizing your logs, refer to [Parse and organize lo The following resources provide information on important concepts related to parsing and organizing your logs: -* [Data streams](../../../manage-data/data-store/index-types/data-streams.md): Efficiently store append-only time series data in multiple backing indices partitioned by time and size. +* [Data streams](../../../manage-data/data-store/data-streams.md): Efficiently store append-only time series data in multiple backing indices partitioned by time and size. * [Data views](../../../explore-analyze/find-and-organize/data-views.md): Query log entries from the data streams of specific datasets or namespaces. * [Index lifecycle management](../../../manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md): Configure the built-in logs policy based on your application’s performance, resilience, and retention requirements. * [Ingest pipeline](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md): Parse and transform log entries into a suitable format before indexing. diff --git a/raw-migrated-files/observability-docs/observability/synthetics-manage-retention.md b/raw-migrated-files/observability-docs/observability/synthetics-manage-retention.md index 143f1e149a..b31a7703ac 100644 --- a/raw-migrated-files/observability-docs/observability/synthetics-manage-retention.md +++ b/raw-migrated-files/observability-docs/observability/synthetics-manage-retention.md @@ -1,6 +1,6 @@ # Manage data retention [synthetics-manage-retention] -When you set up a synthetic monitor, data from the monitor is saved in [{{es}} data streams](../../../manage-data/data-store/index-types/data-streams.md), an append-only structure in {{es}}. +When you set up a synthetic monitor, data from the monitor is saved in [{{es}} data streams](../../../manage-data/data-store/data-streams.md), an append-only structure in {{es}}. There are six data streams recorded by synthetic monitors: `http`, `tcp`, `icmp`, `browser`, `browser.network`, `browser.screenshot`. Elastic will retain data from each data stream for some time period, and the default time period varies by data stream. If you want to reduce the amount of storage required or store data for longer, you can customize how long to retain data for each data stream. diff --git a/raw-migrated-files/observability-docs/observability/synthetics-settings.md b/raw-migrated-files/observability-docs/observability/synthetics-settings.md index 337d4d3f4a..5cd0a8e801 100644 --- a/raw-migrated-files/observability-docs/observability/synthetics-settings.md +++ b/raw-migrated-files/observability-docs/observability/synthetics-settings.md @@ -74,7 +74,7 @@ In the **Global parameters** tab, you can define variables and parameters. This ## Data retention [synthetics-settings-data-retention] -When you set up a synthetic monitor, data from the monitor is saved in [Elasticsearch data streams](../../../manage-data/data-store/index-types/data-streams.md), an append-only structure in Elasticsearch. You can customize how long synthetics data is stored by creating your own index lifecycle policy and attaching it to the relevant custom Component Template in Stack Management. +When you set up a synthetic monitor, data from the monitor is saved in [Elasticsearch data streams](../../../manage-data/data-store/data-streams.md), an append-only structure in Elasticsearch. You can customize how long synthetics data is stored by creating your own index lifecycle policy and attaching it to the relevant custom Component Template in Stack Management. In the **Data retention** tab, use the links to jump to the relevant policy for each data stream. Learn more about the data included in each data stream in [Manage data retention](../../../solutions/observability/apps/manage-data-retention.md). diff --git a/solutions/security/detect-and-alert.md b/solutions/security/detect-and-alert.md index b4dca2c01e..24df97bc72 100644 --- a/solutions/security/detect-and-alert.md +++ b/solutions/security/detect-and-alert.md @@ -141,5 +141,5 @@ Depending on your privileges and whether detection system indices have already b ## Using logsdb index mode [detections-logsdb-index-mode] -To learn how your rules and alerts are affected by using the [logsdb index mode](/manage-data/data-store/index-types/logs-data-stream.md), refer to [*Using logsdb index mode with {{elastic-sec}}*](/solutions/security/detect-and-alert/using-logsdb-index-mode-with-elastic-security.md). +To learn how your rules and alerts are affected by using the [logsdb index mode](/manage-data/data-store/data-streams/logs-data-stream.md), refer to [*Using logsdb index mode with {{elastic-sec}}*](/solutions/security/detect-and-alert/using-logsdb-index-mode-with-elastic-security.md). diff --git a/solutions/security/get-started/data-views-elastic-security.md b/solutions/security/get-started/data-views-elastic-security.md index 0fe353a501..9055be5cd3 100644 --- a/solutions/security/get-started/data-views-elastic-security.md +++ b/solutions/security/get-started/data-views-elastic-security.md @@ -17,7 +17,7 @@ mapped_urls: $$$default-data-view-security$$$ -{{data-sources-cap}} determine what data displays on {{elastic-sec}} pages with event or alert data. {{data-sources-cap}} are defined by the index patterns they include. Only data from {{es}} [indices](/manage-data/data-store/index-basics.md), [data streams](/manage-data/data-store/index-types/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html) specified in the active {{data-source}} will appear. +{{data-sources-cap}} determine what data displays on {{elastic-sec}} pages with event or alert data. {{data-sources-cap}} are defined by the index patterns they include. Only data from {{es}} [indices](/manage-data/data-store/index-basics.md), [data streams](/manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html) specified in the active {{data-source}} will appear. ::::{important} Custom indices are not included in the [default {{data-source}}](/solutions/security/get-started/data-views-elastic-security.md#default-data-view-security). Modify it or create a custom {{data-source}} to include custom indices. diff --git a/troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md b/troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md index 116e917279..cae953341f 100644 --- a/troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md +++ b/troubleshoot/elasticsearch/elasticsearch-reference/index-lifecycle-management-errors.md @@ -154,7 +154,7 @@ You can override how `min_age` is calculated using the `index.lifecycle.originat Here’s how to resolve the most common errors reported in the `ERROR` step. ::::{tip} -Problems with rollover aliases are a common cause of errors. Consider using [data streams](../../../manage-data/data-store/index-types/data-streams.md) instead of managing rollover with aliases. +Problems with rollover aliases are a common cause of errors. Consider using [data streams](../../../manage-data/data-store/data-streams.md) instead of managing rollover with aliases. :::: diff --git a/troubleshoot/elasticsearch/mapping-explosion.md b/troubleshoot/elasticsearch/mapping-explosion.md index 7f583ce48a..b24814e031 100644 --- a/troubleshoot/elasticsearch/mapping-explosion.md +++ b/troubleshoot/elasticsearch/mapping-explosion.md @@ -52,7 +52,7 @@ You can use [analyze index disk usage](https://www.elastic.co/docs/api/doc/elast Mapping explosions also covers when an individual index field totals are within limits but combined indices fields totals are very high. It’s very common for symptoms to first be noticed on a [data view](../../explore-analyze/find-and-organize/data-views.md) and be traced back to an individual index or a subset of indices via the [resolve index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-resolve-index). -However, though less common, it is possible to only experience mapping explosions on the combination of backing indices. For example, if a [data stream](../../manage-data/data-store/index-types/data-streams.md)'s backing indices are all at field total limit but each contain unique fields from one another. +However, though less common, it is possible to only experience mapping explosions on the combination of backing indices. For example, if a [data stream](../../manage-data/data-store/data-streams.md)'s backing indices are all at field total limit but each contain unique fields from one another. This situation most easily surfaces by adding a [data view](../../explore-analyze/find-and-organize/data-views.md) and checking its **Fields** tab for its total fields count. This statistic does tells you overall fields and not only where [`index:true`](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index.html), but serves as a good baseline. diff --git a/troubleshoot/elasticsearch/troubleshooting-searches.md b/troubleshoot/elasticsearch/troubleshooting-searches.md index f6515436c4..a3b721f77b 100644 --- a/troubleshoot/elasticsearch/troubleshooting-searches.md +++ b/troubleshoot/elasticsearch/troubleshooting-searches.md @@ -217,9 +217,9 @@ To troubleshoot queries in {{kib}}, select **Inspect** in the toolbar. Next, sel GET /my-index-000001/_settings ``` -You can update dynamic index settings with the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). [Changing dynamic index settings for a data stream](../../manage-data/data-store/index-types/modify-data-stream.md#change-dynamic-index-setting-for-a-data-stream) requires changing the index template used by the data stream. +You can update dynamic index settings with the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). [Changing dynamic index settings for a data stream](../../manage-data/data-store/data-streams/modify-data-stream.md#change-dynamic-index-setting-for-a-data-stream) requires changing the index template used by the data stream. -For static settings, you need to create a new index with the correct settings. Next, you can reindex the data into that index. For data streams, refer to [Change a static index setting for a data stream](../../manage-data/data-store/index-types/modify-data-stream.md#change-static-index-setting-for-a-data-stream). +For static settings, you need to create a new index with the correct settings. Next, you can reindex the data into that index. For data streams, refer to [Change a static index setting for a data stream](../../manage-data/data-store/data-streams/modify-data-stream.md#change-static-index-setting-for-a-data-stream). ## Find slow queries [troubleshooting-slow-searches] diff --git a/troubleshoot/ingest/fleet/common-problems.md b/troubleshoot/ingest/fleet/common-problems.md index 29b47a7db3..4f53ca963e 100644 --- a/troubleshoot/ingest/fleet/common-problems.md +++ b/troubleshoot/ingest/fleet/common-problems.md @@ -803,7 +803,7 @@ If the installation is correct and all resources are deployed, but data is not f * Missing cluster-level metrics (provided by `kube-state-metrics`): - As described in [Run {{agent}} Standalone on Kubernetes](https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-standalone.html), the {{agent}} Pod acting as `leader` is responsible for retrieving cluster-level metrics from `kube-state-metrics` and delivering them to [data streams](../../../manage-data/data-store/index-types/data-streams.md) prefixed as `metrics-kubernetes.state_`. In order to troubleshoot a situation where these metrics are not appearing: + As described in [Run {{agent}} Standalone on Kubernetes](https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-standalone.html), the {{agent}} Pod acting as `leader` is responsible for retrieving cluster-level metrics from `kube-state-metrics` and delivering them to [data streams](../../../manage-data/data-store/data-streams.md) prefixed as `metrics-kubernetes.state_`. In order to troubleshoot a situation where these metrics are not appearing: 1. Determine which Pod owns the [leadership](https://www.elastic.co/guide/en/fleet/current/kubernetes_leaderelection-provider.html) `lease` in the cluster, with: From 3db4f1196b1e8a1ca77520bd39af8e599c9f297a Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 18 Feb 2025 15:49:36 -0500 Subject: [PATCH 2/5] Fix links --- .../downsampling-time-series-data-stream.md | 10 +++++----- .../data-store/data-streams/manage-data-stream.md | 2 +- manage-data/data-store/data-streams/reindex-tsds.md | 4 ++-- .../data-streams/run-downsampling-manually.md | 8 ++++---- .../run-downsampling-using-data-stream-lifecycle.md | 6 +++--- .../data-streams/run-downsampling-with-ilm.md | 10 +++++----- manage-data/data-store/data-streams/set-up-tsds.md | 12 ++++++------ 7 files changed, 26 insertions(+), 26 deletions(-) diff --git a/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md b/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md index d6e6eedc64..795c2e6a37 100644 --- a/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md +++ b/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md @@ -9,7 +9,7 @@ Downsampling provides a method to reduce the footprint of your [time series data Metrics solutions collect large amounts of time series data that grow over time. As that data ages, it becomes less relevant to the current state of the system. The downsampling process rolls up documents within a fixed time interval into a single summary document. Each summary document includes statistical representations of the original data: the `min`, `max`, `sum` and `value_count` for each metric. Data stream [time series dimensions](time-series-data-stream-tsds.md#time-series-dimension) are stored unchanged. -Downsampling, in effect, lets you to trade data resolution and precision for storage size. You can include it in an [{{ilm}} ({{ilm-init}})](../index-types/lifecycle/index-lifecycle-management.md) policy to automatically manage the volume and associated cost of your metrics data at it ages. +Downsampling, in effect, lets you to trade data resolution and precision for storage size. You can include it in an [{{ilm}} ({{ilm-init}})](../../lifecycle/index-lifecycle-management.md) policy to automatically manage the volume and associated cost of your metrics data at it ages. Check the following sections to learn more: @@ -131,17 +131,17 @@ The following restrictions and limitations apply for downsampling: * Only indices in a [time series data stream](time-series-data-stream-tsds.md) are supported. * Data is downsampled based on the time dimension only. All other dimensions are copied to the new index without any modification. * Within a data stream, a downsampled index replaces the original index and the original index is deleted. Only one index can exist for a given time period. -* A source index must be in read-only mode for the downsampling process to succeed. Check the [Run downsampling manually](../index-types/run-downsampling-manually.md) example for details. +* A source index must be in read-only mode for the downsampling process to succeed. Check the [Run downsampling manually](./run-downsampling-manually.md) example for details. * Downsampling data for the same period many times (downsampling of a downsampled index) is supported. The downsampling interval must be a multiple of the interval of the downsampled index. * Downsampling is provided as an ILM action. See [Downsample](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-downsample.html). * The new, downsampled index is created on the data tier of the original index and it inherits its settings (for example, the number of shards and replicas). * The numeric `gauge` and `counter` [metric types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-field-meta.html) are supported. -* The downsampling configuration is extracted from the time series data stream [index mapping](../index-types/set-up-tsds.md#create-tsds-index-template). The only additional required setting is the downsampling `fixed_interval`. +* The downsampling configuration is extracted from the time series data stream [index mapping](./set-up-tsds.md#create-tsds-index-template). The only additional required setting is the downsampling `fixed_interval`. ## Try it out [try-out-downsampling] -To take downsampling for a test run, try our example of [running downsampling manually](../index-types/run-downsampling-manually.md). +To take downsampling for a test run, try our example of [running downsampling manually](./run-downsampling-manually.md). -Downsampling can easily be added to your ILM policy. To learn how, try our [Run downsampling with ILM](../index-types/run-downsampling-with-ilm.md) example. +Downsampling can easily be added to your ILM policy. To learn how, try our [Run downsampling with ILM](./run-downsampling-with-ilm.md) example. diff --git a/manage-data/data-store/data-streams/manage-data-stream.md b/manage-data/data-store/data-streams/manage-data-stream.md index 2b37234b27..975031c5af 100644 --- a/manage-data/data-store/data-streams/manage-data-stream.md +++ b/manage-data/data-store/data-streams/manage-data-stream.md @@ -6,7 +6,7 @@ The value in the **Indices** column indicates the number of backing indices. Cli A value in the data retention column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. -In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](../index-types/logs-data-stream.md). +In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](./logs-data-stream.md). :::{image} ../../../images/serverless-management-data-stream.png :alt: Data stream details diff --git a/manage-data/data-store/data-streams/reindex-tsds.md b/manage-data/data-store/data-streams/reindex-tsds.md index 2ac7009bb4..af2a9c5a8a 100644 --- a/manage-data/data-store/data-streams/reindex-tsds.md +++ b/manage-data/data-store/data-streams/reindex-tsds.md @@ -12,7 +12,7 @@ mapped_pages: ## Introduction [tsds-reindex-intro] -With reindexing, you can copy documents from an old [time-series data stream (TSDS)](../index-types/TSDS)](../data-streams/time-series-data-stream-tsds.md) to a new one. Data streams support reindexing in general, with a few [restrictions](use-data-stream.md#reindex-with-a-data-stream). Still, time-series data streams introduce additional challenges due to tight control on the accepted timestamp range for each backing index they contain. Direct use of the reindex API would likely error out due to attempting to insert documents with timestamps that are outside the current acceptance window. +With reindexing, you can copy documents from an old [time-series data stream (TSDS)](../data-streams/time-series-data-stream-tsds.md) to a new one. Data streams support reindexing in general, with a few [restrictions](use-data-stream.md#reindex-with-a-data-stream). Still, time-series data streams introduce additional challenges due to tight control on the accepted timestamp range for each backing index they contain. Direct use of the reindex API would likely error out due to attempting to insert documents with timestamps that are outside the current acceptance window. To avoid these limitations, use the process that is outlined below: @@ -28,7 +28,7 @@ To avoid these limitations, use the process that is outlined below: 5. Invoke the `rollover` api to create a new backing index that can receive new documents. ::::{note} -This process only applies to time-series data streams without [downsampling](../index-types/downsampling-time-series-data-stream.md) configuration. Data streams with downsampling can only be re-indexed by re-indexing their backing indexes individually and adding them to an empty destination data stream. +This process only applies to time-series data streams without [downsampling](./downsampling-time-series-data-stream.md) configuration. Data streams with downsampling can only be re-indexed by re-indexing their backing indexes individually and adding them to an empty destination data stream. :::: diff --git a/manage-data/data-store/data-streams/run-downsampling-manually.md b/manage-data/data-store/data-streams/run-downsampling-manually.md index f138d21cab..9cefc127a8 100644 --- a/manage-data/data-store/data-streams/run-downsampling-manually.md +++ b/manage-data/data-store/data-streams/run-downsampling-manually.md @@ -9,7 +9,7 @@ mapped_pages: # Run downsampling manually [downsampling-manual] -The recommended way to [downsample](../index-types/downsampling-time-series-data-stream.md) a [time-series data stream (TSDS)](../index-types/TSDS)](../data-streams/time-series-data-stream-tsds.md) is [through index lifecycle management (ILM)](../index-types/ILM)](run-downsampling-with-ilm.md). However, if you’re not using ILM, you can downsample a TSDS manually. This guide shows you how, using typical Kubernetes cluster monitoring data. +The recommended way to [downsample](./downsampling-time-series-data-stream.md) a [time-series data stream (TSDS)](../data-streams/time-series-data-stream-tsds.md) is [through index lifecycle management (ILM)](run-downsampling-with-ilm.md). However, if you’re not using ILM, you can downsample a TSDS manually. This guide shows you how, using typical Kubernetes cluster monitoring data. To test out manual downsampling, follow these steps: @@ -22,7 +22,7 @@ To test out manual downsampling, follow these steps: ## Prerequisites [downsampling-manual-prereqs] -* Refer to the [TSDS prerequisites](../index-types/set-up-tsds.md#tsds-prereqs). +* Refer to the [TSDS prerequisites](./set-up-tsds.md#tsds-prereqs). * It is not possible to downsample a [data stream](../data-streams.md) directly, nor multiple indices at once. It’s only possible to downsample one time series index (TSDS backing index). * In order to downsample an index, it needs to be read-only. For a TSDS write index, this means it needs to be rolled over and made read-only first. * Downsampling uses UTC timestamps. @@ -399,7 +399,7 @@ You can now delete the old backing index. But be aware this will delete the orig ## View the results [downsampling-manual-view-results] -Re-run the earlier search query (note that when querying downsampled indices there are [a few nuances to be aware of](../index-types/downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)): +Re-run the earlier search query (note that when querying downsampled indices there are [a few nuances to be aware of](.downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)): ```console GET /my-data-stream/_search @@ -558,5 +558,5 @@ GET /my-data-stream/_search This example demonstrates how downsampling can dramatically reduce the number of documents stored for time series data, within whatever time boundaries you choose. It’s also possible to perform downsampling on already downsampled data, to further reduce storage and associated costs, as the time series data ages and the data resolution becomes less critical. -The recommended way to downsample a TSDS is with ILM. To learn more, try the [Run downsampling with ILM](../index-types/run-downsampling-with-ilm.md) example. +The recommended way to downsample a TSDS is with ILM. To learn more, try the [Run downsampling with ILM](./run-downsampling-with-ilm.md) example. diff --git a/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md b/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md index 7aa7b680e4..eccbb9ec57 100644 --- a/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md +++ b/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md @@ -9,7 +9,7 @@ mapped_pages: # Run downsampling using data stream lifecycle [downsampling-dsl] -This is a simplified example that allows you to see quickly how [downsampling](../index-types/downsampling-time-series-data-stream.md) works as part of a datastream lifecycle to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with data stream lifecycle, follow these steps: +This is a simplified example that allows you to see quickly how [downsampling](./downsampling-time-series-data-stream.md) works as part of a datastream lifecycle to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with data stream lifecycle, follow these steps: 1. Check the [prerequisites](#downsampling-dsl-prereqs). 2. [Create an index template with data stream lifecycle](#downsampling-dsl-create-index-template). @@ -21,7 +21,7 @@ This is a simplified example that allows you to see quickly how [downsampling](. ## Prerequisites [downsampling-dsl-prereqs] -Refer to [time series data stream prerequisites](../index-types/set-up-tsds.md#tsds-prereqs). +Refer to [time series data stream prerequisites](./set-up-tsds.md#tsds-prereqs). ## Create an index template with data stream lifecycle [downsampling-dsl-create-index-template] @@ -345,7 +345,7 @@ After the data stream lifecycle action was executed, original `.ds-datastream-20 ... ``` -Run a search query on the datastream (note that when querying downsampled indices there are [a few nuances to be aware of](../index-types/downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)). +Run a search query on the datastream (note that when querying downsampled indices there are [a few nuances to be aware of](./downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)). ```console GET datastream/_search diff --git a/manage-data/data-store/data-streams/run-downsampling-with-ilm.md b/manage-data/data-store/data-streams/run-downsampling-with-ilm.md index 2dd8ff4d7f..8161b333f3 100644 --- a/manage-data/data-store/data-streams/run-downsampling-with-ilm.md +++ b/manage-data/data-store/data-streams/run-downsampling-with-ilm.md @@ -9,7 +9,7 @@ mapped_pages: # Run downsampling with ILM [downsampling-ilm] -This is a simplified example that allows you to see quickly how [downsampling](../index-types/downsampling-time-series-data-stream.md) works as part of an ILM policy to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with ILM, follow these steps: +This is a simplified example that allows you to see quickly how [downsampling](./downsampling-time-series-data-stream.md) works as part of an ILM policy to reduce the storage size of a sampled set of metrics. The example uses typical Kubernetes cluster monitoring data. To test out downsampling with ILM, follow these steps: 1. Check the [prerequisites](#downsampling-ilm-prereqs). 2. [Create an index lifecycle policy](#downsampling-ilm-policy). @@ -20,9 +20,9 @@ This is a simplified example that allows you to see quickly how [downsampling](. ## Prerequisites [downsampling-ilm-prereqs] -Refer to [time series data stream prerequisites](../index-types/set-up-tsds.md#tsds-prereqs). +Refer to [time series data stream prerequisites](./set-up-tsds.md#tsds-prereqs). -Before running this example you may want to try the [Run downsampling manually](../index-types/run-downsampling-manually.md) example. +Before running this example you may want to try the [Run downsampling manually](./run-downsampling-manually.md) example. ## Create an index lifecycle policy [downsampling-ilm-policy] @@ -346,7 +346,7 @@ After the ILM policy has taken effect, the original `.ds-datastream-2022.08.26-0 ... ``` -Run a search query on the datastream (note that when querying downsampled indices there are [a few nuances to be aware of](../index-types/downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)). +Run a search query on the datastream (note that when querying downsampled indices there are [a few nuances to be aware of](./downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)). ```console GET datastream/_search @@ -464,4 +464,4 @@ GET /_data_stream/datastream/_stats?human=true This example demonstrates how downsampling works as part of an ILM policy to reduce the storage size of metrics data as it becomes less current and less frequently queried. -You can also try our [Run downsampling manually](../index-types/run-downsampling-manually.md) example to learn how downsampling can work outside of an ILM policy. +You can also try our [Run downsampling manually](./run-downsampling-manually.md) example to learn how downsampling can work outside of an ILM policy. diff --git a/manage-data/data-store/data-streams/set-up-tsds.md b/manage-data/data-store/data-streams/set-up-tsds.md index 198175160f..7af4637186 100644 --- a/manage-data/data-store/data-streams/set-up-tsds.md +++ b/manage-data/data-store/data-streams/set-up-tsds.md @@ -9,13 +9,13 @@ mapped_pages: # Set up a TSDS [set-up-tsds] -To set up a [time series data stream (TSDS)](../index-types/TSDS)](../data-streams/time-series-data-stream-tsds.md), follow these steps: +To set up a [time series data stream (TSDS)](../data-streams/time-series-data-stream-tsds.md), follow these steps: -1. Check the [prerequisites](../index-types/set-up-tsds.md#tsds-prereqs). -2. [Create an index lifecycle policy](../index-types/set-up-tsds.md#tsds-ilm-policy). -3. [Create an index template](../index-types/set-up-tsds.md#create-tsds-index-template). -4. [Create the TSDS](../index-types/set-up-tsds.md#create-tsds). -5. [Secure the TSDS](../index-types/set-up-tsds.md#secure-tsds). +1. Check the [prerequisites](#tsds-prereqs). +2. [Create an index lifecycle policy](#tsds-ilm-policy). +3. [Create an index template](#create-tsds-index-template). +4. [Create the TSDS](#create-tsds). +5. [Secure the TSDS](#secure-tsds). ## Prerequisites [tsds-prereqs] From 3cdb7e72ee058c5a494e9972bad071acaa766f92 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 18 Feb 2025 15:52:14 -0500 Subject: [PATCH 3/5] link fix --- .../data-store/data-streams/run-downsampling-manually.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manage-data/data-store/data-streams/run-downsampling-manually.md b/manage-data/data-store/data-streams/run-downsampling-manually.md index 9cefc127a8..621f739928 100644 --- a/manage-data/data-store/data-streams/run-downsampling-manually.md +++ b/manage-data/data-store/data-streams/run-downsampling-manually.md @@ -399,7 +399,7 @@ You can now delete the old backing index. But be aware this will delete the orig ## View the results [downsampling-manual-view-results] -Re-run the earlier search query (note that when querying downsampled indices there are [a few nuances to be aware of](.downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)): +Re-run the earlier search query (note that when querying downsampled indices there are [a few nuances to be aware of](./downsampling-time-series-data-stream.md#querying-downsampled-indices-notes)): ```console GET /my-data-stream/_search From 8de3cc06b166ccd8cf5789a9aea700aaf7824127 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 18 Feb 2025 15:56:49 -0500 Subject: [PATCH 4/5] Remove links to index types --- manage-data/data-store.md | 2 +- .../docs-content/serverless/elasticsearch-differences.md | 2 +- .../docs-content/serverless/project-setting-data.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/manage-data/data-store.md b/manage-data/data-store.md index 6d6ae89516..dafaed4e5b 100644 --- a/manage-data/data-store.md +++ b/manage-data/data-store.md @@ -7,7 +7,7 @@ mapped_pages: [{{es}}](https://github.com/elastic/elasticsearch/) is a distributed search and analytics engine, scalable data store, and vector database built on Apache Lucene. -The documentation in this section details how {{es}} works as a _data store_ starting with the fundamental unit of storage in Elasticsearch: the index. An index is a collection of documents uniquely identified by a name or an alias. Read more in [Index basics](/manage-data/data-store/index-basics.md) and [Index types](/manage-data/data-store/index-types.md). +The documentation in this section details how {{es}} works as a _data store_ starting with the fundamental unit of storage in Elasticsearch: the index. An index is a collection of documents uniquely identified by a name or an alias. Read more in [Index basics](/manage-data/data-store/index-basics.md). Then, learn how these documents and the fields they contain are stored and indexed in [Mapping](/manage-data/data-store/mapping.md), and how unstructured text is converted into a structured format that’s optimized for search in [Text analysis](/manage-data/data-store/text-analysis.md). diff --git a/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md b/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md index de0d09c079..0df47cae96 100644 --- a/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md +++ b/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md @@ -118,7 +118,7 @@ Some features that are available in Elastic Cloud Hosted and self-managed offeri These features have been replaced by a new feature and are therefore not available on {{es-serverless}}: -* **Index lifecycle management ({{ilm-init}})** is not available, in favor of [**data stream lifecycle**](../../../manage-data/data-store/index-types.md). +* **Index lifecycle management ({{ilm-init}})** is not available, in favor of **data stream lifecycle**. In an Elastic Cloud Hosted or self-managed environment, {{ilm-init}} lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {{es-serverless}} eliminates this complexity by optimizing your cluster performance for you. diff --git a/raw-migrated-files/docs-content/serverless/project-setting-data.md b/raw-migrated-files/docs-content/serverless/project-setting-data.md index 7d4ba0e1b3..6faceb8a4b 100644 --- a/raw-migrated-files/docs-content/serverless/project-setting-data.md +++ b/raw-migrated-files/docs-content/serverless/project-setting-data.md @@ -14,7 +14,7 @@ Access to individual features is governed by Elastic user roles. Consult your ad | [Integrations](https://www.elastic.co/guide/en/serverless/current/integrations.html) | Connect your data to your project. | [![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) | | [Fleet and Elastic Agent](https://www.elastic.co/guide/en/serverless/current/fleet-and-elastic-agent.html) | Add monitoring for logs, metrics, and other types of data to a host. | [![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) | | [{{data-sources-cap}}](../../../explore-analyze/find-and-organize/data-views.md) | Manage the fields in the data views that retrieve your data from {{es-serverless}}. | [![Elasticsearch](../../../images/serverless-es-badge.svg "")](../../../solutions/search.md)[![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) | -| [Index management](../../../manage-data/data-store/index-types.md) | View index settings, mappings, and statistics and perform operations on indices. | [![Elasticsearch](../../../images/serverless-es-badge.svg "")](../../../solutions/search.md)[![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) | +| [Index management](../../../manage-data/data-store/index-basics.md) | View index settings, mappings, and statistics and perform operations on indices. | [![Elasticsearch](../../../images/serverless-es-badge.svg "")](../../../solutions/search.md)[![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) | | [{{ingest-pipelines-cap}}](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md) | Create and manage ingest pipelines that parse, transform, and enrich your data. | [![Elasticsearch](../../../images/serverless-es-badge.svg "")](../../../solutions/search.md)[![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) | | [{{ls-pipelines}}](../../../manage-data/ingest/transform-enrich/logstash-pipelines.md) | Create and manage {{ls}} pipelines that parse, transform, and enrich your data. | [![Elasticsearch](../../../images/serverless-es-badge.svg "")](../../../solutions/search.md)[![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) | | [{{ml-cap}}](../../../explore-analyze/machine-learning.md) | View, export, and import your {{anomaly-detect}} and {{dfanalytics}} jobs and trained models. | [![Elasticsearch](../../../images/serverless-es-badge.svg "")](../../../solutions/search.md)[![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) | From 39941401b0d39e05c92059b1001a64ea8e850d81 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 18 Feb 2025 16:13:43 -0500 Subject: [PATCH 5/5] Fix links in newly merged 'index-basics.md' --- manage-data/data-store/index-basics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/manage-data/data-store/index-basics.md b/manage-data/data-store/index-basics.md index 537c8c8df8..ddadfdc602 100644 --- a/manage-data/data-store/index-basics.md +++ b/manage-data/data-store/index-basics.md @@ -12,7 +12,7 @@ This content applies to: [![Elasticsearch](/images/serverless-es-badge.svg "")]( An index is a fundamental unit of storage in {{es}}. It is a collection of documents uniquely identified by a name or an [alias](/manage-data/data-store/aliases.md). This unique name is important because it’s used to target the index in search queries and other operations. ::::{tip} -A closely related concept is a [data stream](/manage-data/data-store/index-types/data-streams.md). This index abstraction is optimized for append-only timestamped data, and is made up of hidden, auto-generated backing indices. If you’re working with timestamped data, we recommend the [Elastic Observability](https://www.elastic.co/guide/en/observability/current) solution for additional tools and optimized content. +A closely related concept is a [data stream](/manage-data/data-store/data-streams.md). This index abstraction is optimized for append-only timestamped data, and is made up of hidden, auto-generated backing indices. If you’re working with timestamped data, we recommend the [Elastic Observability](https://www.elastic.co/guide/en/observability/current) solution for additional tools and optimized content. :::: ## Index components @@ -84,7 +84,7 @@ Investigate your indices and perform operations from the **Indices** view. ### Manage data streams -A [data stream](/manage-data/data-store/index-types/data-streams.md) lets you store append-only time series data across multiple indices while giving you a single named resource for requests. +A [data stream](/manage-data/data-store/data-streams.md) lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Investigate your data streams and address lifecycle management needs in the **Data Streams** view.