-
Notifications
You must be signed in to change notification settings - Fork 25.6k
[DOCS][9.x] Update ESQL metadata fields page #129939
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,24 +6,41 @@ mapped_pages: | |
|
|
||
| # {{esql}} metadata fields [esql-metadata-fields] | ||
|
|
||
| {{esql}} can access [metadata fields](/reference/elasticsearch/mapping-reference/document-metadata-fields.md). | ||
|
|
||
| {{esql}} can access [metadata fields](/reference/elasticsearch/mapping-reference/document-metadata-fields.md). The currently supported ones are: | ||
|
|
||
| * [`_index`](/reference/elasticsearch/mapping-reference/mapping-index-field.md): the index to which the document belongs. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md). | ||
| * [`_id`](/reference/elasticsearch/mapping-reference/mapping-id-field.md): the source document’s ID. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md). | ||
| * `_version`: the source document’s version. The field is of the type [long](/reference/elasticsearch/mapping-reference/number.md). | ||
| * [`_ignored`](/reference/elasticsearch/mapping-reference/mapping-ignored-field.md): the ignored source document fields. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md). | ||
| * `_score`: when enabled, the final score assigned to each row matching an ES|QL query. Scoring will be updated when using [full text search functions](/reference/query-languages/esql/functions-operators/search-functions.md). | ||
|
|
||
| To enable the access to these fields, the [`FROM`](/reference/query-languages/esql/commands/source-commands.md#esql-from) source command needs to be provided with a dedicated directive: | ||
| To access these fields, use the `METADATA` directive with the [`FROM`](/reference/query-languages/esql/commands/source-commands.md#esql-from) source command. For example: | ||
|
|
||
| ```esql | ||
| FROM index METADATA _index, _id | ||
| ``` | ||
|
|
||
| Metadata fields are only available if the source of the data is an index. Consequently, `FROM` is the only source commands that supports the `METADATA` directive. | ||
| ## Available metadata fields | ||
|
|
||
| The following metadata fields are available in {{esql}}: | ||
|
|
||
| | Metadata field | Type | Description | | ||
| |---------------|------|-------------| | ||
| | [`_id`](/reference/elasticsearch/mapping-reference/mapping-id-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Unique document ID. | | ||
| | [`_ignored`](/reference/elasticsearch/mapping-reference/mapping-ignored-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Names every field in a document that was ignored when the document was indexed. | | ||
| | [`_index`](/reference/elasticsearch/mapping-reference/mapping-index-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Index name. | | ||
| | `_index_mode` | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | [Index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting). For example: `standard`, `lookup`, or `logsdb`. | | ||
| | `_score` | [`float`](/reference/elasticsearch/mapping-reference/number.md) | Query relevance score (when enabled). Scores are updated when using [full text search functions](/reference/query-languages/esql/functions-operators/search-functions.md). | | ||
| | [`_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md) | Special `_source` type | Original JSON document body passed at index time (or a reconstructed version if [synthetic `_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) is enabled). This field is not supported by functions. | | ||
leemthompo marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| | `_version` | [`long`](/reference/elasticsearch/mapping-reference/number.md) | Document version number | | ||
|
|
||
| ## Usage and limitations | ||
|
|
||
| - Metadata fields are only available when the data source is an index | ||
| - The `_source` field is not supported by functions | ||
leemthompo marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - Only the `FROM` command supports the `METADATA` directive | ||
| - Once enabled, metadata fields work like regular index fields | ||
| - Fields are dropped after aggregations unless used for grouping ([example](#metadata-fields-and-aggregations)) | ||
|
||
|
|
||
| ## Examples | ||
|
|
||
| Once enabled, these fields will be available to subsequent processing commands, just like other index fields: | ||
| ### Basic metadata usage | ||
|
|
||
| Once enabled, metadata fields are available to subsequent processing commands, just like other index fields: | ||
|
|
||
| ```esql | ||
| FROM ul_logs, apps METADATA _index, _version | ||
|
|
@@ -40,6 +57,8 @@ FROM ul_logs, apps METADATA _index, _version | |
| | 14 | apps | 1 | apps_14 | | ||
| | 14 | ul_logs | 1 | ul_logs_14 | | ||
|
|
||
| ### Metadata fields and aggregations | ||
|
|
||
| Similar to index fields, once an aggregation is performed, a metadata field will no longer be accessible to subsequent commands, unless used as a grouping field: | ||
|
|
||
| ```esql | ||
|
|
@@ -51,3 +70,15 @@ FROM employees METADATA _index, _id | |
| | --- | --- | | ||
| | 10100 | employees | | ||
|
|
||
| ### Sort results by search score | ||
|
|
||
| ```esql | ||
| FROM products METADATA _score | ||
| | WHERE MATCH(description, "wireless headphones") | ||
| | SORT _score DESC | ||
| | KEEP name, description, _score | ||
| ``` | ||
|
|
||
| :::{tip} | ||
| Refer to [{{esql}} for search](docs-content://solutions/search/esql-for-search.md#esql-for-search-scoring) for more information on relevance scoring and how to use `_score` in your queries. | ||
| ::: | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍