Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 42 additions & 11 deletions docs/reference/query-languages/esql/esql-metadata-fields.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,41 @@ mapped_pages:

# {{esql}} metadata fields [esql-metadata-fields]

{{esql}} can access [metadata fields](/reference/elasticsearch/mapping-reference/document-metadata-fields.md).

{{esql}} can access [metadata fields](/reference/elasticsearch/mapping-reference/document-metadata-fields.md). The currently supported ones are:

* [`_index`](/reference/elasticsearch/mapping-reference/mapping-index-field.md): the index to which the document belongs. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md).
* [`_id`](/reference/elasticsearch/mapping-reference/mapping-id-field.md): the source document’s ID. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md).
* `_version`: the source document’s version. The field is of the type [long](/reference/elasticsearch/mapping-reference/number.md).
* [`_ignored`](/reference/elasticsearch/mapping-reference/mapping-ignored-field.md): the ignored source document fields. The field is of the type [keyword](/reference/elasticsearch/mapping-reference/keyword.md).
* `_score`: when enabled, the final score assigned to each row matching an ES|QL query. Scoring will be updated when using [full text search functions](/reference/query-languages/esql/functions-operators/search-functions.md).

To enable the access to these fields, the [`FROM`](/reference/query-languages/esql/commands/source-commands.md#esql-from) source command needs to be provided with a dedicated directive:
To access these fields, use the `METADATA` directive with the [`FROM`](/reference/query-languages/esql/commands/source-commands.md#esql-from) source command. For example:

```esql
FROM index METADATA _index, _id
```

Metadata fields are only available if the source of the data is an index. Consequently, `FROM` is the only source commands that supports the `METADATA` directive.
## Available metadata fields

The following metadata fields are available in {{esql}}:

| Metadata field | Type | Description |
|---------------|------|-------------|
| [`_id`](/reference/elasticsearch/mapping-reference/mapping-id-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Unique document ID. |
| [`_ignored`](/reference/elasticsearch/mapping-reference/mapping-ignored-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Names every field in a document that was ignored when the document was indexed. |
| [`_index`](/reference/elasticsearch/mapping-reference/mapping-index-field.md) | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | Index name. |
| `_index_mode` | [keyword](/reference/elasticsearch/mapping-reference/keyword.md) | [Index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting). For example: `standard`, `lookup`, or `logsdb`. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

| `_score` | [`float`](/reference/elasticsearch/mapping-reference/number.md) | Query relevance score (when enabled). Scores are updated when using [full text search functions](/reference/query-languages/esql/functions-operators/search-functions.md). |
| [`_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md) | Special `_source` type | Original JSON document body passed at index time (or a reconstructed version if [synthetic `_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) is enabled). This field is not supported by functions. |
| `_version` | [`long`](/reference/elasticsearch/mapping-reference/number.md) | Document version number |

## Usage and limitations

- Metadata fields are only available when the data source is an index
- The `_source` field is not supported by functions
- Only the `FROM` command supports the `METADATA` directive
- Once enabled, metadata fields work like regular index fields
- Fields are dropped after aggregations unless used for grouping ([example](#metadata-fields-and-aggregations))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this is true, but this is not metadata fields-specific.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove 👍


## Examples

Once enabled, these fields will be available to subsequent processing commands, just like other index fields:
### Basic metadata usage

Once enabled, metadata fields are available to subsequent processing commands, just like other index fields:

```esql
FROM ul_logs, apps METADATA _index, _version
Expand All @@ -40,6 +57,8 @@ FROM ul_logs, apps METADATA _index, _version
| 14 | apps | 1 | apps_14 |
| 14 | ul_logs | 1 | ul_logs_14 |

### Metadata fields and aggregations

Similar to index fields, once an aggregation is performed, a metadata field will no longer be accessible to subsequent commands, unless used as a grouping field:

```esql
Expand All @@ -51,3 +70,15 @@ FROM employees METADATA _index, _id
| --- | --- |
| 10100 | employees |

### Sort results by search score

```esql
FROM products METADATA _score
| WHERE MATCH(description, "wireless headphones")
| SORT _score DESC
| KEEP name, description, _score
```

:::{tip}
Refer to [{{esql}} for search](docs-content://solutions/search/esql-for-search.md#esql-for-search-scoring) for more information on relevance scoring and how to use `_score` in your queries.
:::
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,27 @@ mapped_pages:

# {{esql}} Search functions [esql-search-functions]

:::{tip}
Get started with {{esql}} for search use cases with
our [hands-on tutorial](docs-content://solutions/search/esql-search-tutorial.md).

For a high-level overview of search functionalities in {{esql}}, and to learn about relevance scoring, refer to [{{esql}} for search](docs-content://solutions/search/esql-for-search.md#esql-for-search-scoring).
:::

{{esql}} provides a set of functions for performing searching on text fields.

Use these functions
for [full-text search](docs-content://solutions/search/full-text.md)
and [semantic search](docs-content://solutions/search/semantic-search/semantic-search-semantic-text.md).

Get started with {{esql}} for search use cases with
our [hands-on tutorial](docs-content://solutions/search/esql-search-tutorial.md).

Full text functions can be used to
match [multivalued fields](/reference/query-languages/esql/esql-multivalued-fields.md).
A multivalued field that contains a value that matches a full text query is
considered to match the query.

Full text functions are significantly more performant for text search use cases
on large data sets than using pattern matching or regular expressions with
`LIKE` or `RLIKE`
`LIKE` or `RLIKE`.

See [full text search limitations](/reference/query-languages/esql/limitations.md#esql-limitations-full-text-search)
for information on the limitations of full text search.
Expand Down
Loading