Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions manage-data/_snippets/ilm-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ To restart {{ilm-init}} and resume executing policies, use the [{{ilm-init}} sta
POST _ilm/start
```

The response will look like this:
The response looks like this:

```console-result
{
Expand All @@ -20,7 +20,7 @@ Verify that {{ilm}} is now running:
GET _ilm/status
```

The response will look like this:
The response looks like this:

```console-result
{
Expand Down
2 changes: 1 addition & 1 deletion manage-data/_snippets/ilm-status.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
To see the current status of the {{ilm-init}} service, use the [{{ilm-init}} status API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-get-status):
To view the current status of the {{ilm-init}} service, use the [{{ilm-init}} status API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-get-status):

```console
GET _ilm/status
Expand Down
6 changes: 3 additions & 3 deletions manage-data/_snippets/ilm-stop.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ By default, the {{ilm}} service is in the `RUNNING` state and manages all indice
You can stop {{ilm-init}} to suspend management operations for all indices. For example, you might stop {{ilm}} when performing scheduled maintenance or making changes to the cluster that could impact the execution of {{ilm-init}} actions.

::::{important}
When you stop {{ilm-init}}, [{{slm-init}}](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm) operations are also suspended. No snapshots will be taken as scheduled until you restart {{ilm-init}}. In-progress snapshots are not affected.
When you stop {{ilm-init}}, [{{slm-init}}](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm) operations are also suspended. {{slm-init}} will not take snapshots as scheduled until you restart {{ilm-init}}. In-progress snapshots are not affected.
::::

To stop the {{ilm-init}} service and pause execution of all lifecycle policies, use the [{{ilm-init}} stop API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ilm-stop):
Expand All @@ -12,7 +12,7 @@ To stop the {{ilm-init}} service and pause execution of all lifecycle policies,
POST _ilm/stop
```

The response will look like this:
The response looks like this:

```console-result
{
Expand All @@ -28,7 +28,7 @@ While the {{ilm-init}} service is shutting down, run the status API to verify th
GET _ilm/status
```

The response will look like this:
The response looks like this:

```console-result
{
Expand Down
4 changes: 2 additions & 2 deletions manage-data/data-store.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,6 @@ The documentation in this section details how {{es}} works as a _data store_ sta

Then, learn how these documents and the fields they contain are stored and indexed in [Mapping](/manage-data/data-store/mapping.md), and how unstructured text is converted into a structured format that’s optimized for search in [Text analysis](/manage-data/data-store/text-analysis.md).

You can also read more about working with {{es}} as a data store including how to use [index templates](/manage-data/data-store/templates.md) to tell {{es}} how to configure an index when it is created, how to use [aliases](/manage-data/data-store/aliases.md) to point to multiple indices, and how to use the [command line to manage data](/manage-data/data-store/manage-data-from-the-command-line.md) stored in {{es}}.
You can also read more about working with {{es}} as a data store including how to use [index templates](/manage-data/data-store/templates.md) to tell {{es}} how to configure an index when you create it, how to use [aliases](/manage-data/data-store/aliases.md) to point to multiple indices, and how to use the [command line to manage data](/manage-data/data-store/manage-data-from-the-command-line.md) stored in {{es}}.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is possibly a confusing change - the reason behind index templates is so the index is created properly when it is created by some automated means. "you create it" implies manual effort.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ agreed. thanks for calling this out, @shainaraskas.

@natasha-moore-elastic can you please fix this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #3941.


If your use case involves working with continuous streams of time series data, you may consider using a [data stream](./data-store/data-streams.md). These are optimally suited for storing append-only data. The data can be accessed through a single, named resource, while it is stored in a series of hidden, auto-generated backing indices.
If your use case involves working with continuous streams of time series data, you can consider using a [data stream](./data-store/data-streams.md). These are optimally suited for storing append-only data. You can access the data through a single, named resource, while {{es}} stores it in a series of hidden, auto-generated backing indices.
10 changes: 5 additions & 5 deletions manage-data/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,31 +9,31 @@ description: Learn how to ingest, store, and manage data in Elasticsearch. Under

# Manage data

Whether you're looking to build a fast and relevant search solution, monitor business-critical applications and infrastructure, monitor endpoint security data, or one of the [many other use cases Elastic supports](/get-started/introduction.md), you'll need to understand how to ingest and manage data stored in {{es}}.
Whether you're looking to build a fast and relevant search solution, monitor business-critical applications and infrastructure, monitor endpoint security data, or one of the [many other use cases Elastic supports](/get-started/introduction.md), you need to understand how to ingest and manage data stored in {{es}}.

## Learn how data is stored

% Topic: Learning about Elastic data store primitives

The fundamental unit of storage in {{es}}, the index, is a collection of documents uniquely identified by a name or an alias. These documents go through a process called mapping, which defines how a document and the fields it contains are stored and indexed, and a process called text analysis in which unstructured text is converted into a structured format thats optimized for search.
The fundamental unit of storage in {{es}}, the index, is a collection of documents uniquely identified by a name or an alias. These documents go through a process called mapping, which defines how {{es}} stores and indexes a document and the fields it contains, and a process called text analysis in which {{es}} converts unstructured text into a structured format that's optimized for search.

**Learn more in [The Elasticsearch data store](/manage-data/data-store.md)**.

## Get data into {{es}}

% Topic: Evaluating and implementing ingestion and data enrichment technologies

Before you can start searching, visualizing, and pulling actionable insights from Elastic, you have to get your data into {{es}}. Elastic offers a wide range of tools and methods for getting data into {{es}}. The best approach will depend on the kind of data you're ingesting and your specific use case.
Before you can start searching, visualizing, and pulling actionable insights from Elastic, you have to get your data into {{es}}. Elastic offers a wide range of tools and methods for getting data into {{es}}. The best approach depends on the kind of data you're ingesting and your specific use case.

**Learn more in [Ingestion](/manage-data/ingest.md).**

## Manage data over time

% Topic: Managing your data volume (lifecycle)

After you've added data to {{es}}, you'll need to manage it over time. For example, you may specify that data be deleted after a retention period or store data in multiple tiers with different performance characteristics.
After you've added data to {{es}}, you need to manage it over time. For example, you can specify that data be deleted after a retention period or store data in multiple tiers with different performance characteristics.

Strategies for managing data depend on the type of data and how it's being used. For example, with a collection of items you want to search, like a catalog of products, the value of the content remains relatively constant over time so you want to be able to retrieve items quickly regardless of how old they are. Whereas with a stream of continuously-generated timestamped data, such as log entries, the data keeps accumulating over time, so you need strategies for balancing the value of the data against the cost of storing it.
Strategies for managing data depend on the type of data and how you're using it. For example, with a collection of items you want to search, like a catalog of products, the value of the content remains relatively constant over time so you want to be able to retrieve items quickly regardless of how old they are. Whereas with a stream of continuously-generated timestamped data, such as log entries, the data keeps accumulating over time, so you need strategies for balancing the value of the data against the cost of storing it.

**Learn more in [Data lifecycle](/manage-data/lifecycle.md).**

Expand Down
6 changes: 3 additions & 3 deletions manage-data/ingest.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ products:

Whether you call it *adding*, *indexing*, or *ingesting* data, you have to get the data into {{es}} before you can search it, visualize it, and use it for insights.

Our ingest tools are flexible, and support a wide range of scenarios. We can help you with everything from popular and straightforward use cases, all the way to advanced use cases that require additional processing in order to modify or reshape your data before it goes to {{es}}.
Our ingest tools are flexible, and support a wide range of scenarios. We can help you with everything from popular and straightforward use cases, all the way to advanced use cases that require additional processing to modify or reshape your data before it goes to {{es}}.

You can ingest:

Expand All @@ -42,10 +42,10 @@ If you would like to try things out before you add your own data, try using our

## Ingesting time series data [ingest-time-series]

::::{admonition} Whats the best approach for ingesting time series data?
::::{admonition} What's the best approach for ingesting time series data?
The best approach for ingesting data is the *simplest option* that *meets your needs* and *satisfies your use case*.

In most cases, the *simplest option* for ingesting time series data is using {{agent}} paired with an Elastic integration.
Usually, the *simplest option* for ingesting time series data is using {{agent}} paired with an Elastic integration.

* Install [Elastic Agent](/reference/fleet/index.md) on the computer(s) from which you want to collect data.
* Add the [Elastic integration](https://docs.elastic.co/en/integrations) for the data source to your deployment.
Expand Down
14 changes: 7 additions & 7 deletions manage-data/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The data you store in {{es}} generally falls into one of two categories:
To help you manage your data, {{es}} offers you the following options: {{ilm-cap}}, Data stream lifecycle, and Elastic Curator.

::::{note}
[Data rollup](/manage-data/lifecycle/rollup.md) is a deprecated {{es}} feature that allows you to manage the amount of data that is stored in your cluster, similar to the downsampling functionality of {{ilm-init}} and data stream lifecycle. This feature should not be used for new deployments.
[Data rollup](/manage-data/lifecycle/rollup.md) is a deprecated {{es}} feature that allows you to manage the amount of data that your cluster stores, similar to the downsampling functionality of{{ilm-init}} and data stream lifecycle. Do not use this feature for new deployments.
::::

## {{ilm-init}} [ilm]
Expand All @@ -40,9 +40,9 @@ In an {{ecloud}} or self-managed environment, ILM lets you automatically transit
:::
::::

**{{ilm-init}}** can be used to manage both indices and data streams. It allows you to do the following:
**{{ilm-init}}** can manage both indices and data streams. It allows you to do the following:

* Define the retention period of your data. The retention period is the minimum time your data will be stored in {{es}}. Data older than this period can be deleted by {{es}}.
* Define the retention period of your data. The retention period is the minimum time {{es}} stores your data. {{es}} can delete data older than this period.
* Define [multiple tiers](/manage-data/lifecycle/data-tiers.md) of data nodes with different performance characteristics.
* Automatically transition indices through the data tiers according to your performance needs and retention policies.
* Leverage [searchable snapshots](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md) stored in a remote repository to provide resiliency for your older indices while reducing operating costs and maintaining search performance.
Expand All @@ -52,10 +52,10 @@ In an {{ecloud}} or self-managed environment, ILM lets you automatically transit

## Data stream lifecycle [data-stream-lifecycle]

**Data stream lifecycle** is less feature rich but is focused on simplicity. It allows you to do the following:
**Data stream lifecycle** is less feature rich but focuses on simplicity. It allows you to do the following:

* Define the retention period of your data. The retention period is the minimum time your data will be stored in {{es}}. Data older than this period can be deleted by {{es}} at a later time.
* Improve the performance of your data stream by performing background operations that will optimize the way your data stream is stored.
* Define the retention period of your data. The retention period is the minimum time {{es}} stores your data. {{es}} can delete data older than this period at a later time.
* Improve the performance of your data stream by performing background operations that optimize how {{es}} stores your data stream.

**[Read more in Data stream lifecycle ->](/manage-data/lifecycle/data-stream.md)**

Expand All @@ -64,6 +64,6 @@ In an {{ecloud}} or self-managed environment, ILM lets you automatically transit
serverless: unavailable
```

**Elastic Curator** is a tool that allows you to manage your indices and snapshots using user-defined filters and predefined actions. If ILM provides the functionality to manage your index lifecycle, and you have at least a Basic license, consider using ILM in place of Curator. Many stack components make use of ILM by default.
**Elastic Curator** is a tool that allows you to manage your indices and snapshots using user-defined filters and predefined actions. If ILM provides the functionality to manage your index lifecycle, and you have at least a Basic license, consider using ILM in place of Curator. Many stack components use ILM by default.

**[Read more in Elastic Curator ->](/manage-data/lifecycle/curator.md)**
Loading
Loading