-
Notifications
You must be signed in to change notification settings - Fork 156
[Docs] Add Index basics page #464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
ec71eed
[Docs] Add index basics
wajihaparvez edbde83
Merge branch 'main' into 369/index-basics
wajihaparvez 0744cc9
Fix image
wajihaparvez 2272419
merging the latestMerge branch '369/index-basics' of github.com:wajih…
wajihaparvez f0c9d1a
Attempting to fix iamges
wajihaparvez 9950050
Applying fix to all images
wajihaparvez 241f671
Fixing link
wajihaparvez 1a34c3c
Applying PR feedback
wajihaparvez 0ec85ef
Applying PR feedback
wajihaparvez aa5c924
Merge branch 'main' into 369/index-basics
wajihaparvez File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,18 +7,144 @@ mapped_urls: | |
|
||
# Index basics | ||
|
||
% What needs to be done: Refine | ||
This content applies to: [](/solutions/search.md) [](/solutions/observability.md) [](/solutions/security/elastic-security-serverless.md) | ||
|
||
% GitHub issue: docs-projects#369 | ||
An index is a fundamental unit of storage in {{es}}. It is a collection of documents uniquely identified by a name or an [alias](/manage-data/data-store/aliases.md). This unique name is important because it’s used to target the index in search queries and other operations. | ||
|
||
% Scope notes: Combine content from linked sources. | ||
::::{tip} | ||
A closely related concept is a [data stream](/manage-data/data-store/index-types/data-streams.md). This index abstraction is optimized for append-only timestamped data, and is made up of hidden, auto-generated backing indices. If you’re working with timestamped data, we recommend the [Elastic Observability](https://www.elastic.co/guide/en/observability/current) solution for additional tools and optimized content. | ||
:::: | ||
|
||
% Use migrated content from existing pages that map to this page: | ||
## Index components | ||
|
||
% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/documents-indices.md | ||
% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md | ||
% - [ ] ./raw-migrated-files/docs-content/serverless/index-management.md | ||
An index is made up of the following components. | ||
|
||
### Documents [elasticsearch-intro-documents-fields] | ||
|
||
{{es}} serializes and stores data in the form of JSON documents. A document is a set of fields, which are key-value pairs that contain your data. Each document has a unique ID, which you can create or have {{es}} auto-generate. | ||
|
||
A simple {{es}} document might look like this: | ||
|
||
```js | ||
{ | ||
"_index": "my-first-elasticsearch-index", | ||
"_id": "DyFpo5EBxE8fzbb95DOa", | ||
"_version": 1, | ||
"_seq_no": 0, | ||
"_primary_term": 1, | ||
"found": true, | ||
"_source": { | ||
"email": "[email protected]", | ||
"first_name": "John", | ||
"last_name": "Smith", | ||
"info": { | ||
"bio": "Eco-warrior and defender of the weak", | ||
"age": 25, | ||
"interests": [ | ||
"dolphins", | ||
"whales" | ||
] | ||
}, | ||
"join_date": "2024/05/01" | ||
} | ||
} | ||
``` | ||
|
||
### Metadata fields [elasticsearch-intro-documents-fields-data-metadata] | ||
|
||
An indexed document contains data and metadata. [Metadata fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html) are system fields that store information about the documents. In {{es}}, metadata fields are prefixed with an underscore. For example, the following fields are metadata fields: | ||
|
||
* `_index`: The name of the index where the document is stored. | ||
* `_id`: The document’s ID. IDs must be unique per index. | ||
|
||
|
||
### Mappings and data types [elasticsearch-intro-documents-fields-mappings] | ||
|
||
Each index has a [mapping](/manage-data/data-store/mapping.md) or schema for how the fields in your documents are indexed. A mapping defines the [data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) for each field, how the field should be indexed, and how it should be stored. | ||
|
||
## Index management | ||
|
||
Elastic's index management features are an easy, convenient way to manage your cluster's indices, data streams, index templates, and enrich policies. Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. | ||
|
||
Go to **{{project-settings}} → {{manage-app}} → {{index-manage-app}}**. | ||
|
||
### Manage indices | ||
|
||
Investigate your indices and perform operations from the **Indices** view. | ||
|
||
:::{image} /images/serverless-index-management-indices.png | ||
:alt: Index Management indices | ||
:class: screenshot | ||
::: | ||
|
||
* To show details and perform operations, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices.html). | ||
* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-put-follow.html), a [rollup index](https://www.elastic.co/guide/en/elasticsearch/reference/current/rollup-get-rollup-index-caps.html), or [frozen](https://www.elastic.co/guide/en/elasticsearch/reference/current/unfreeze-index-api.html). | ||
* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-modules-settings), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. | ||
* To create new indices, use the **Create index** wizard. | ||
|
||
### Manage data streams | ||
|
||
A [data stream](/manage-data/data-store/index-types/data-streams.md) lets you store append-only time series data across multiple indices while giving you a single named resource for requests. | ||
|
||
Investigate your data streams and address lifecycle management needs in the **Data Streams** view. | ||
|
||
:::{image} /images/serverless-management-data-stream.png | ||
:alt: Data stream details | ||
:class: screenshot | ||
::: | ||
|
||
In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](https://www.elastic.co/guide/en/elasticsearch/reference/master/logs-data-stream.html). | ||
|
||
* To view information about the stream's backing indices, click the number in the **Indices** column. | ||
* A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. | ||
* To modify the data retention value, select an index, open the **Manage** menu, and click **Edit data retention**. | ||
* To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream's name. From this view, you can navigate to **Discover** to further explore data within the data stream. | ||
|
||
### Manage index templates | ||
|
||
An [index template](/manage-data/data-store/templates.md) is a way to tell {{es}} how to configure an index when it is created. | ||
|
||
Create, edit, clone, and delete your index templates in the **Index Templates** view. Changes made to an index template do not affect existing indices. | ||
|
||
:::{image} /images/serverless-index-management-index-templates.png | ||
:alt: Index templates | ||
:class: screenshot | ||
::: | ||
|
||
* To show details and perform operations, click the template name. | ||
* To view more information about the component templates within an index template, click the value in the **Component templates** column. | ||
* Values in the **Content** column indicate whether a template contains index mappings, settings, and aliases. | ||
* To create new index templates, use the **Create template** wizard. | ||
|
||
### Manage component templates | ||
|
||
[Component templates](/manage-data/data-store/templates.md) are reusable building blocks that configure mappings, settings, and aliases. | ||
|
||
Create, edit, clone, and delete your component templates in the **Component Templates** view. | ||
|
||
:::{image} /images/serverless-management-component-templates.png | ||
:alt: Component templates | ||
:class: screenshot | ||
::: | ||
|
||
* To show details and perform operations, click the template name. | ||
* To create new component templates, use the **Create component template** wizard. | ||
|
||
### Manage enrich policies | ||
|
||
An [enrich policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-enriching-data.html#enrich-policy) is a set of configuration options used to add the right enrich data to the right incoming documents. | ||
|
||
Add data from your existing indices to incoming documents using the **Enrich Policies** view. | ||
|
||
:::{image} /images/serverless-management-enrich-policies.png | ||
:alt: Enrich policies | ||
:class: screenshot | ||
::: | ||
|
||
* To show details click the policy name. | ||
* To perform operations, click the policy name or use the buttons in the **Actions** column. | ||
* To create new policies, use the **Create enrich policy** wizard. | ||
|
||
You must execute a new enrich policy before you can use it with an enrich processor. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. | ||
|
||
% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): | ||
|
||
$$$elasticsearch-intro-documents-fields-mappings$$$ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The images in this PR aren't showing up for me in the preview... I'm not sure, but this might help.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what the issue was but when I was adding the
../../
on Friday, the deployment was failing. I can see the images in the preview now, could you confirm if that's true for you as well?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I had to hard refresh to get the page built with the latest docs-builder update (including the redesign). After that I can see the images. 👍