-
Notifications
You must be signed in to change notification settings - Fork 156
[Docs] Add Index basics page #464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 7 commits
ec71eed
edbde83
0744cc9
2272419
f0c9d1a
9950050
241f671
1a34c3c
0ec85ef
aa5c924
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -15,10 +15,152 @@ mapped_urls: | |||||
|
||||||
% Use migrated content from existing pages that map to this page: | ||||||
|
||||||
% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/documents-indices.md | ||||||
% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md | ||||||
% - [ ] ./raw-migrated-files/docs-content/serverless/index-management.md | ||||||
% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/documents-indices.md | ||||||
% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md | ||||||
% - [x] ./raw-migrated-files/docs-content/serverless/index-management.md | ||||||
|
||||||
% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): | ||||||
|
||||||
$$$elasticsearch-intro-documents-fields-mappings$$$ | ||||||
$$$elasticsearch-intro-documents-fields-mappings$$$ | ||||||
|
||||||
This content applies to: [](/solutions/search.md) [](/solutions/observability.md) [](/solutions/security/elastic-security-serverless.md) | ||||||
|
||||||
An index is a fundamental unit of storage in {{es}}. It is a collection of documents uniquely identified by a name or an [alias](/manage-data/data-store/aliases.md). This unique name is important because it’s used to target the index in search queries and other operations. | ||||||
|
||||||
::::{tip} | ||||||
A closely related concept is a [data stream](/manage-data/data-store/index-types/data-streams.md). This index abstraction is optimized for append-only timestamped data, and is made up of hidden, auto-generated backing indices. If you’re working with timestamped data, we recommend the [Elastic Observability](https://www.elastic.co/guide/en/observability/current) solution for additional tools and optimized content. | ||||||
:::: | ||||||
|
||||||
## Index components | ||||||
|
||||||
An index is made up of the following components. | ||||||
|
||||||
### Documents [elasticsearch-intro-documents-fields] | ||||||
|
||||||
{{es}} serializes and stores data in the form of JSON documents. A document is a set of fields, which are key-value pairs that contain your data. Each document has a unique ID, which you can create or have {{es}} auto-generate. | ||||||
|
||||||
A simple {{es}} document might look like this: | ||||||
|
||||||
```js | ||||||
{ | ||||||
"_index": "my-first-elasticsearch-index", | ||||||
"_id": "DyFpo5EBxE8fzbb95DOa", | ||||||
"_version": 1, | ||||||
"_seq_no": 0, | ||||||
"_primary_term": 1, | ||||||
"found": true, | ||||||
"_source": { | ||||||
"email": "[email protected]", | ||||||
"first_name": "John", | ||||||
"last_name": "Smith", | ||||||
"info": { | ||||||
"bio": "Eco-warrior and defender of the weak", | ||||||
"age": 25, | ||||||
"interests": [ | ||||||
"dolphins", | ||||||
"whales" | ||||||
] | ||||||
}, | ||||||
"join_date": "2024/05/01" | ||||||
} | ||||||
} | ||||||
``` | ||||||
|
||||||
### Metadata fields [elasticsearch-intro-documents-fields-data-metadata] | ||||||
|
||||||
An indexed document contains data and metadata. [Metadata fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html) are system fields that store information about the documents. In {{es}}, metadata fields are prefixed with an underscore. For example, the following fields are metadata fields: | ||||||
|
||||||
* `_index`: The name of the index where the document is stored. | ||||||
* `_id`: The document’s ID. IDs must be unique per index. | ||||||
|
||||||
|
||||||
### Mappings and data types [elasticsearch-intro-documents-fields-mappings] | ||||||
|
||||||
Each index has a [mapping](/manage-data/data-store/mapping.md) or schema for how the fields in your documents are indexed. A mapping defines the [data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) for each field, how the field should be indexed, and how it should be stored. | ||||||
|
||||||
## Index management | ||||||
|
||||||
Elastic's index management features are an easy, convenient way to manage your cluster's indices, data streams, index templates, and enrich policies. Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. | ||||||
|
||||||
Go to **{{project-settings}} → {{manage-app}} → {{index-manage-app}}**. | ||||||
|
||||||
### Manage indices | ||||||
|
||||||
Investigate your indices and perform operations from the **Indices** view. | ||||||
|
||||||
:::{image} /images/serverless-index-management-indices.png | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The images in this PR aren't showing up for me in the preview... I'm not sure, but this might help.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure what the issue was but when I was adding the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah I had to hard refresh to get the page built with the latest docs-builder update (including the redesign). After that I can see the images. 👍 |
||||||
:alt: Index Management indices | ||||||
:class: screenshot | ||||||
::: | ||||||
|
||||||
* To show details and perform operations, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices.html). | ||||||
* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-put-follow.html), a [rollup index](https://www.elastic.co/guide/en/elasticsearch/reference/current/rollup-get-rollup-index-caps.html), or [frozen](https://www.elastic.co/guide/en/elasticsearch/reference/current/unfreeze-index-api.html). | ||||||
* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-modules-settings), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. | ||||||
* To create new indices, use the **Create index** wizard. | ||||||
|
||||||
### Manage data streams | ||||||
|
||||||
A [data stream](/manage-data/data-store/index-types/data-streams.md) lets you store append-only time series data across multiple indices while giving you a single named resource for requests. | ||||||
|
||||||
Investigate your data streams and address lifecycle management needs in the **Data Streams** view. | ||||||
|
||||||
:::{image} /images/serverless-management-data-stream.png | ||||||
:alt: Data stream details | ||||||
:class: screenshot | ||||||
::: | ||||||
|
||||||
In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](https://www.elastic.co/guide/en/elasticsearch/reference/master/logs-data-stream.html). | ||||||
|
||||||
* To view information about the stream's backing indices, click the number in the **Indices** column. | ||||||
* A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. | ||||||
* To modify the data retention value, select an index, open the **Manage** menu, and click **Edit data retention**. | ||||||
* To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream's name. From this view, you can navigate to **Discover** to further explore data within the data stream. | ||||||
|
||||||
### Manage index templates | ||||||
|
||||||
An [index template](/manage-data/data-store/templates.md) is a way to tell {{es}} how to configure an index when it is created. | ||||||
|
||||||
Create, edit, clone, and delete your index templates in the **Index Templates** view. Changes made to an index template do not affect existing indices. | ||||||
|
||||||
:::{image} /images/serverless-index-management-index-templates.png | ||||||
:alt: Index templates | ||||||
:class: screenshot | ||||||
::: | ||||||
|
||||||
* To show details and perform operations, click the template name. | ||||||
* To view more information about the component templates within an index template, click the value in the **Component templates** column. | ||||||
* Values in the **Content** column indicate whether a template contains index mappings, settings, and aliases. | ||||||
* To create new index templates, use the **Create template** wizard. | ||||||
|
||||||
### Manage component templates | ||||||
|
||||||
[Component templates](/manage-data/data-store/templates.md) are reusable building blocks that configure mappings, settings, and aliases. | ||||||
|
||||||
Create, edit, clone, and delete your component templates in the **Component Templates** view. | ||||||
|
||||||
:::{image} /images/serverless-management-component-templates.png | ||||||
:alt: Component templates | ||||||
:class: screenshot | ||||||
::: | ||||||
|
||||||
* To show details and perform operations, click the template name. | ||||||
* To create new component templates, use the **Create component template** wizard. | ||||||
|
||||||
### Manage enrich policies | ||||||
|
||||||
An [enrich policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-enriching-data.html#enrich-policy) is a set of configuration options used to add the right enrich data to the right incoming documents. | ||||||
|
||||||
Add data from your existing indices to incoming documents using the **Enrich Policies** view. | ||||||
|
||||||
:::{image} /images/serverless-management-enrich-policies.png | ||||||
:alt: Enrich policies | ||||||
:class: screenshot | ||||||
::: | ||||||
|
||||||
* To show details click the policy name. | ||||||
* To perform operations, click the policy name or use the buttons in the **Actions** column. | ||||||
* To create new policies, use the **Create enrich policy** wizard. | ||||||
|
||||||
You must execute a new enrich policy before you can use it with an enrich processor. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. | ||||||
|
||||||
|
Uh oh!
There was an error while loading. Please reload this page.