diff --git a/explore-analyze/alerts-cases/watcher/actions-index.md b/explore-analyze/alerts-cases/watcher/actions-index.md index 2af2b29d06..e85250ef12 100644 --- a/explore-analyze/alerts-cases/watcher/actions-index.md +++ b/explore-analyze/alerts-cases/watcher/actions-index.md @@ -40,7 +40,7 @@ The following snippet shows a simple `index` action definition: | --- | --- | --- | --- | | `index` | yes* | - | The index, alias, or data stream to index into. Date math expressions like `` are also supported.

*If you dynamically set an `_index` value, this parameter isn’t required. See [Multi-document support](#anatomy-actions-index-multi-doc-support).
| | `doc_id` | no | - | The optional `_id` of the document. | -| `op_type` | no | `index` | The [op_type](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#docs-index-api-op_type) for the index operation. Must be one of either `index` or `create`. Must be `create` if `index` is a data stream. | +| `op_type` | no | `index` | The [op_type](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) for the index operation. Must be one of either `index` or `create`. Must be `create` if `index` is a data stream. | | `execution_time_field` | no | - | The field that will store/index the watch execution time. | | `timeout` | no | 60s | The timeout for waiting for the index api call to return. If no response is returned within this time, the index action times out and fails. This setting overrides the default timeouts. | | `refresh` | no | - | Optional setting of the [refresh policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html) for the write request | diff --git a/explore-analyze/alerts-cases/watcher/actions.md b/explore-analyze/alerts-cases/watcher/actions.md index 9bd1ec3469..99b4ebc92e 100644 --- a/explore-analyze/alerts-cases/watcher/actions.md +++ b/explore-analyze/alerts-cases/watcher/actions.md @@ -146,9 +146,9 @@ If you do not define a throttle period at the action or watch level, the global xpack.watcher.execution.default_throttle_period: 15m ``` -{{watcher}} also supports acknowledgement-based throttling. You can acknowledge a watch using the [ack watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-ack-watch.html) to prevent the watch actions from being executed again while the watch condition remains `true`. This essentially tells {{watcher}} "I received the notification and I’m handling it, please do not notify me about this error again". An acknowledged watch action remains in the `acked` state until the watch’s condition evaluates to `false`. When that happens, the action’s state changes to `awaits_successful_execution`. +{{watcher}} also supports acknowledgement-based throttling. You can acknowledge a watch using the [ack watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-ack-watch) to prevent the watch actions from being executed again while the watch condition remains `true`. This essentially tells {{watcher}} "I received the notification and I’m handling it, please do not notify me about this error again". An acknowledged watch action remains in the `acked` state until the watch’s condition evaluates to `false`. When that happens, the action’s state changes to `awaits_successful_execution`. -To acknowledge an action, you use the [ack watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-ack-watch.html): +To acknowledge an action, you use the [ack watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-ack-watch): ```console POST _watcher/watch//_ack/ diff --git a/explore-analyze/alerts-cases/watcher/how-watcher-works.md b/explore-analyze/alerts-cases/watcher/how-watcher-works.md index 17270242d8..a6659706d2 100644 --- a/explore-analyze/alerts-cases/watcher/how-watcher-works.md +++ b/explore-analyze/alerts-cases/watcher/how-watcher-works.md @@ -35,7 +35,7 @@ A watch consists of a *trigger*, *input*, *condition*, and *actions*. The action : Specify what happens when the watch condition is met. $$$watch-definition-example$$$ -For example, the following snippet shows a [create or update watch](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-put-watch.html) request that defines a watch that looks for log error events: +For example, the following snippet shows a [create or update watch](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-put-watch) request that defines a watch that looks for log error events: ```console PUT _watcher/watch/log_errors @@ -167,10 +167,10 @@ By default, when you add a watch it is immediately set to the *active* state, re You can also set a watch to the *inactive* state. Inactive watches are not registered with a trigger engine and can never be triggered. -To set a watch to the inactive state when you create it, set the [`active`](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-put-watch.html) parameter to *inactive*. To deactivate an existing watch, use the [deactivate watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-deactivate-watch.html). To reactivate an inactive watch, use the [activate watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-activate-watch.html). +To set a watch to the inactive state when you create it, set the [`active`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-put-watch) parameter to *inactive*. To deactivate an existing watch, use the [deactivate watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-deactivate-watch). To reactivate an inactive watch, use the [activate watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-activate-watch). ::::{note} -You can use the [execute watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-execute-watch.html) to force the execution of a watch even when it is inactive. +You can use the [execute watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch) to force the execution of a watch even when it is inactive. :::: diff --git a/explore-analyze/alerts-cases/watcher/input-http.md b/explore-analyze/alerts-cases/watcher/input-http.md index 58bfdfefa0..d247530b15 100644 --- a/explore-analyze/alerts-cases/watcher/input-http.md +++ b/explore-analyze/alerts-cases/watcher/input-http.md @@ -14,7 +14,7 @@ Use the `http` input to submit a request to an HTTP endpoint and load the respon With the `http` input, you can: * Query external Elasticsearch clusters. The `http` input provides a way to submit search requests to clusters other than the one {{watcher}} is running on. This is useful when you’re running a dedicated {{watcher}} cluster or if you need to search clusters that are running different Elasticsearch versions. -* Query Elasticsearch APIs other than the search API. For example, you might want to load data from the [nodes stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html), [cluster health](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html) or [cluster state](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-state.html) APIs. +* Query Elasticsearch APIs other than the search API. For example, you might want to load data from the [nodes stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-stats), [cluster health](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health) or [cluster state](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-state) APIs. * Query external web services. The `http` input enables you to load data from any service that exposes an HTTP endpoint. This provides a bridge between Elasticsearch clusters and other systems. ## Querying external Elasticsearch clusters [_querying_external_elasticsearch_clusters] @@ -50,7 +50,7 @@ You can use the full Elasticsearch [Query DSL](../../query-filter/languages/quer ## Calling Elasticsearch APIs [_calling_elasticsearch_apis] -To load the data from other Elasticsearch APIs, specify the API endpoint as the `path` attribute. Use the `params` attribute to specify query string parameters. For example, the following `http` input calls the [cluster stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html) API and enables the `human` attribute: +To load the data from other Elasticsearch APIs, specify the API endpoint as the `path` attribute. Use the `params` attribute to specify query string parameters. For example, the following `http` input calls the [cluster stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-stats) API and enables the `human` attribute: ```js "input" : { diff --git a/explore-analyze/alerts-cases/watcher/input-search.md b/explore-analyze/alerts-cases/watcher/input-search.md index 2f22219b0c..2728985051 100644 --- a/explore-analyze/alerts-cases/watcher/input-search.md +++ b/explore-analyze/alerts-cases/watcher/input-search.md @@ -14,7 +14,7 @@ Use the `search` input to load the results of an Elasticsearch search request in In the search input’s `request` object, you specify: * The indices you want to search -* The [search type](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-type) +* The [search type](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) * The search request body The search request body supports the full Elasticsearch Query DSL—​it’s the same as the body of an Elasticsearch `_search` request. @@ -137,9 +137,9 @@ The total number of hits in the search response is returned as an object in the | Name | Required | Default | Description | | --- | --- | --- | --- | -| `request.search_type` | no | `query_then_fetch` | The [type](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-type) of search request to perform. Valid values are: `dfs_query_then_fetch` and `query_then_fetch`. The Elasticsearch default is `query_then_fetch`. | +| `request.search_type` | no | `query_then_fetch` | The [type](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) of search request to perform. Valid values are: `dfs_query_then_fetch` and `query_then_fetch`. The Elasticsearch default is `query_then_fetch`. | | `request.indices` | no | - | The indices to search. If omitted, all indices are searched, which is the default behaviour in Elasticsearch. | -| `request.body` | no | - | The body of the request. The [request body](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` [templates](how-watcher-works.md#templates). | +| `request.body` | no | - | The body of the request. The [request body](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` [templates](how-watcher-works.md#templates). | | `request.template` | no | - | The body of the search template. See [configure templates](how-watcher-works.md#templates) for more information. | | `request.indices_options.expand_wildcards` | no | `open` | How to expand wildcards. Valid values are: `all`, `open`, `closed`, and `none` See [`expand_wildcards`](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index) for more information. | | `request.indices_options.ignore_unavailable` | no | `true` | Whether the search should ignore unavailable indices. See [`ignore_unavailable`](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index) for more information. | diff --git a/explore-analyze/alerts-cases/watcher/managing-watches.md b/explore-analyze/alerts-cases/watcher/managing-watches.md index 4a7be33ebb..8c0e216e3f 100644 --- a/explore-analyze/alerts-cases/watcher/managing-watches.md +++ b/explore-analyze/alerts-cases/watcher/managing-watches.md @@ -10,12 +10,12 @@ mapped_pages: {{watcher}} provides as set of APIs you can use to manage your watches: -* Use the [create or update watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-put-watch.html) to add or update watches -* Use the [get watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-get-watch.html) to retrieve watches -* Use the [delete watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-delete-watch.html) to delete watches -* Use the [activate watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-activate-watch.html) to activate watches -* Use the [deactivate watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-deactivate-watch.html) to deactivate watches -* Use the [ack watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-ack-watch.html) to acknowledge watches +* Use the [create or update watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-put-watch) to add or update watches +* Use the [get watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-get-watch) to retrieve watches +* Use the [delete watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-delete-watch) to delete watches +* Use the [activate watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-activate-watch) to activate watches +* Use the [deactivate watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-deactivate-watch) to deactivate watches +* Use the [ack watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-ack-watch) to acknowledge watches ## Listing watches [listing-watches] diff --git a/explore-analyze/alerts-cases/watcher/transform-search.md b/explore-analyze/alerts-cases/watcher/transform-search.md index a55e9f9eda..c67fa35fdf 100644 --- a/explore-analyze/alerts-cases/watcher/transform-search.md +++ b/explore-analyze/alerts-cases/watcher/transform-search.md @@ -49,9 +49,9 @@ The following table lists all available settings for the search {{watcher-transf | Name | Required | Default | Description | | --- | --- | --- | --- | -| `request.search_type` | no | query_then_fetch | The search [type](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-type). | +| `request.search_type` | no | query_then_fetch | The search [type](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). | | `request.indices` | no | all indices | One or more indices to search on. | -| `request.body` | no | `match_all` query | The body of the request. The [request body](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` [templates](how-watcher-works.md#templates). | +| `request.body` | no | `match_all` query | The body of the request. The [request body](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` [templates](how-watcher-works.md#templates). | | `request.indices_options.expand_wildcards` | no | `open` | Determines how to expand indices wildcards. An array consisting of a combination of `open`, `closed`, and `hidden`. Alternatively a value of `none` or `all`. (see [multi-target syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index)) | | `request.indices_options.ignore_unavailable` | no | `true` | A boolean value that determines whether the search should leniently ignore unavailable indices (see [multi-target syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index)) | | `request.indices_options.allow_no_indices` | no | `true` | A boolean value that determines whether the search should leniently return no results when no indices are resolved (see [multi-target syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index)) | diff --git a/explore-analyze/alerts-cases/watcher/watch-cluster-status.md b/explore-analyze/alerts-cases/watcher/watch-cluster-status.md index 0851a1034d..9a08061f64 100644 --- a/explore-analyze/alerts-cases/watcher/watch-cluster-status.md +++ b/explore-analyze/alerts-cases/watcher/watch-cluster-status.md @@ -31,7 +31,7 @@ PUT _watcher/watch/cluster_health_watch 1. Schedules are typically configured to run less frequently. This example sets the interval to 10 seconds to you can easily see the watches being triggered. Since this watch runs so frequently, don’t forget to [delete the watch](#health-delete) when you’re done experimenting. -To get the status of your cluster, you can call the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html): +To get the status of your cluster, you can call the [cluster health API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health): ```console GET _cluster/health?pretty @@ -224,7 +224,7 @@ GET .watcher-history*/_search?pretty Since the `cluster_health_watch` is configured to run every 10 seconds, make sure you delete it when you’re done experimenting. Otherwise, you’ll spam yourself indefinitely. -To remove the watch, use the [delete watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-delete-watch.html): +To remove the watch, use the [delete watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-delete-watch): ```console DELETE _watcher/watch/cluster_health_watch diff --git a/explore-analyze/alerts-cases/watcher/watcher-getting-started.md b/explore-analyze/alerts-cases/watcher/watcher-getting-started.md index 20be3483bb..64f27eca82 100644 --- a/explore-analyze/alerts-cases/watcher/watcher-getting-started.md +++ b/explore-analyze/alerts-cases/watcher/watcher-getting-started.md @@ -154,7 +154,7 @@ PUT _watcher/watch/log_error_watch Since the `log_error_watch` is configured to run every 10 seconds, make sure you delete it when you’re done experimenting. Otherwise, the noise from this sample watch will make it hard to see what else is going on in your watch history and log file. -To remove the watch, use the [delete watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-delete-watch.html): +To remove the watch, use the [delete watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-delete-watch): ```console DELETE _watcher/watch/log_error_watch diff --git a/explore-analyze/alerts-cases/watcher/watcher-ui.md b/explore-analyze/alerts-cases/watcher/watcher-ui.md index aad72aac3d..a66d440114 100644 --- a/explore-analyze/alerts-cases/watcher/watcher-ui.md +++ b/explore-analyze/alerts-cases/watcher/watcher-ui.md @@ -132,11 +132,11 @@ The **Simulate** tab allows you to override parts of the watch, and then run a s * Trigger overrides use [date math](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#date-math). * Input overrides accepts a JSON blob. * Condition overrides indicates if you want to force the condition to always be `true`. -* Action overrides support [multiple options](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-execute-watch.html#watcher-api-execute-watch-action-mode). +* Action overrides support [multiple options](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch). Some watches, such as those using basic auth credentials, contain passwords. For security reasons, these passwords will be stripped from any watches that are loaded by the Watcher UI. You will need to manually re-enter these passwords to simulate the watch. -After starting the simulation, you’ll see a results screen. For more information on the fields in the response, see the [Execute watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-execute-watch.html). +After starting the simulation, you’ll see a results screen. For more information on the fields in the response, see the [Execute watch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch). ![Create advanced watch](../../../images/kibana-advanced-watch-simulate.png "") diff --git a/explore-analyze/geospatial-analysis.md b/explore-analyze/geospatial-analysis.md index 3b4c30ebfc..8c4bfbc745 100644 --- a/explore-analyze/geospatial-analysis.md +++ b/explore-analyze/geospatial-analysis.md @@ -68,7 +68,7 @@ Combine aggregations to perform complex geospatial analysis. For example, to cal ## Integrate [geospatial-integrate] -Use [vector tile search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html) to consume {{es}} geo data within existing GIS infrastructure. +Use [vector tile search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search-mvt) to consume {{es}} geo data within existing GIS infrastructure. ## Visualize [geospatial-visualize] diff --git a/explore-analyze/machine-learning/anomaly-detection/anomaly-detection-scale.md b/explore-analyze/machine-learning/anomaly-detection/anomaly-detection-scale.md index a5ccbb3f2e..1dcdbda6dc 100644 --- a/explore-analyze/machine-learning/anomaly-detection/anomaly-detection-scale.md +++ b/explore-analyze/machine-learning/anomaly-detection/anomaly-detection-scale.md @@ -40,7 +40,7 @@ For large jobs, use a dedicated results index. This ensures that results from a ## 3. Disable model plot [model-plot] -By default, model plot is enabled when you create jobs in {{kib}}. If you have a large job, however, consider disabling it. You can disable model plot for existing jobs by using the [Update {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-update-job.html). +By default, model plot is enabled when you create jobs in {{kib}}. If you have a large job, however, consider disabling it. You can disable model plot for existing jobs by using the [Update {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-update-job). Model plot calculates and stores the model bounds for each analyzed entity, including both anomalous and non-anomalous entities. These bounds are used to display the shaded area in the Single Metric Viewer charts. Model plot creates one result document per bucket per split field value. If you have high cardinality fields and/or a short bucket span, disabling model plot reduces processing workload and results stored. @@ -77,7 +77,7 @@ The `model_memory_limit` job configuration option sets the approximate maximum a If you change the resources available on your {{ml}} nodes or make significant changes to the characteristics or cardinality of your data, the model memory requirements might also change. You can update the model memory limit for a job while it is closed. If you want to decrease the limit below the current model memory usage, however, you must clone and re-run the job. ::::{tip} -You can view the current model size statistics with the [get {{anomaly-job}} stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-job-stats.html) and [get model snapshots](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-snapshot.html) APIs. You can also obtain a model memory limit estimate at any time by running the [estimate {{anomaly-jobs}} model memory API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-estimate-model-memory.html). However, you must provide your own cardinality estimates. +You can view the current model size statistics with the [get {{anomaly-job}} stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-job-stats) and [get model snapshots](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-model-snapshots) APIs. You can also obtain a model memory limit estimate at any time by running the [estimate {{anomaly-jobs}} model memory API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-estimate-model-memory). However, you must provide your own cardinality estimates. :::: As a job approaches its model memory limit, the memory status is `soft_limit` and older models are more aggressively pruned to free up space. If you have categorization jobs, no further examples are stored. When a job exceeds its limit, the memory status is `hard_limit` and the job no longer models new entities. It is therefore important to have appropriate memory model limits for each job. If you reach the hard limit and are concerned about the missing data, ensure that you have adequate resources then clone and re-run the job with a larger model memory limit. diff --git a/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md b/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md index 6ae4e92aac..8553bf12be 100644 --- a/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md +++ b/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md @@ -41,7 +41,7 @@ There are a few limitations to consider before you create this type of job: If those limitations are acceptable, try creating an {{anomaly-job}} that uses the [`lat_long` function](https://www.elastic.co/guide/en/machine-learning/current/ml-geo-functions.html#ml-lat-long) to analyze your own data or the sample data sets. -To create an {{anomaly-job}} that uses the `lat_long` function, in {{kib}} you must click **Create job** on the **{{ml-cap}} > {{anomaly-detect-cap}} > Jobs** page and select the advanced job wizard. Alternatively, use the [create {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html). +To create an {{anomaly-job}} that uses the `lat_long` function, in {{kib}} you must click **Create job** on the **{{ml-cap}} > {{anomaly-detect-cap}} > Jobs** page and select the advanced job wizard. Alternatively, use the [create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job). For example, create a job that analyzes the sample eCommerce orders data set to find orders with unusual coordinates (`geoip.location` values) relative to the past behavior of each customer (`user` ID): diff --git a/explore-analyze/machine-learning/anomaly-detection/mapping-anomalies.md b/explore-analyze/machine-learning/anomaly-detection/mapping-anomalies.md index d62ec2b58d..d9710a00ef 100644 --- a/explore-analyze/machine-learning/anomaly-detection/mapping-anomalies.md +++ b/explore-analyze/machine-learning/anomaly-detection/mapping-anomalies.md @@ -27,7 +27,7 @@ If you have fields that contain valid vector layers, you can use the **{{data-vi ## Create an {{anomaly-job}} [mapping-anomalies-jobs] -To create an {{anomaly-job}} in {{kib}}, click **Create job** on the **{{ml-cap}} > {{anomaly-detect-cap}}** page and select an appropriate job wizard. Alternatively, use the [create {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html). +To create an {{anomaly-job}} in {{kib}}, click **Create job** on the **{{ml-cap}} > {{anomaly-detect-cap}}** page and select an appropriate job wizard. Alternatively, use the [create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job). For example, use the multi-metric job wizard to create a job that analyzes the sample web logs data set to detect anomalous behavior in the sum of the data transferred (`bytes` values) for each destination country (`geo.dest` values): diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-ad-explain.md b/explore-analyze/machine-learning/anomaly-detection/ml-ad-explain.md index 7d7d482877..cb87b497bc 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-ad-explain.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-ad-explain.md @@ -58,4 +58,4 @@ Real-world anomalies often show the impacts of several factors. The **Anomaly ex :class: screenshot ::: -You can also find this information in the `anomaly_score_explanation` field of the [get record API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-record.html). +You can also find this information in the `anomaly_score_explanation` field of the [get record API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-records). diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-ad-forecast.md b/explore-analyze/machine-learning/anomaly-detection/ml-ad-forecast.md index b04ee2d01a..cd2520dac3 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-ad-forecast.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-ad-forecast.md @@ -15,7 +15,7 @@ You can use a forecast to estimate a time series value at a specific future date You can also use it to estimate the probability of a time series value occurring at a future date. For example, you might want to determine how likely it is that your disk utilization will reach 100% before the end of next week. -Each forecast has a unique ID, which you can use to distinguish between forecasts that you created at different times. You can create a forecast by using the [forecast {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-forecast.html) or by using {{kib}}. For example: +Each forecast has a unique ID, which you can use to distinguish between forecasts that you created at different times. You can create a forecast by using the [forecast {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-forecast) or by using {{kib}}. For example: :::{image} ../../../images/machine-learning-overview-forecast.jpg :alt: Example screenshot from the Machine Learning Single Metric Viewer in Kibana @@ -26,4 +26,4 @@ The yellow line in the chart represents the predicted data values. The shaded ye When you create a forecast, you specify its *duration*, which indicates how far the forecast extends beyond the last record that was processed. By default, the duration is 1 day. Typically the farther into the future that you forecast, the lower the confidence levels become (that is to say, the bounds increase). Eventually if the confidence levels are too low, the forecast stops. For more information about limitations that affect your ability to create a forecast, see [Unsupported forecast configurations](ml-limitations.md#ml-forecast-config-limitations). -You can also optionally specify when the forecast expires. By default, it expires in 14 days and is deleted automatically thereafter. You can specify a different expiration period by using the `expires_in` parameter in the [forecast {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-forecast.html). +You can also optionally specify when the forecast expires. By default, it expires in 14 days and is deleted automatically thereafter. You can specify a different expiration period by using the `expires_in` parameter in the [forecast {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-forecast). diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md b/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md index c976179d4e..b4dd2e507a 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md @@ -22,7 +22,7 @@ If your data is located outside of {{es}}, you cannot use {{kib}} to create your ## Create an {{anomaly-job}} [ml-ad-create-job] -You can create {{anomaly-jobs}} by using the [create {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html). {{kib}} also provides wizards to simplify the process, which vary depending on whether you are using the {{ml-app}} app, {{security-app}} or {{observability}} apps. To open **Anomaly Detection**, find **{{ml-app}}** in the main menu, or use the [global search field](../../find-and-organize/find-apps-and-objects.md). +You can create {{anomaly-jobs}} by using the [create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job). {{kib}} also provides wizards to simplify the process, which vary depending on whether you are using the {{ml-app}} app, {{security-app}} or {{observability}} apps. To open **Anomaly Detection**, find **{{ml-app}}** in the main menu, or use the [global search field](../../find-and-organize/find-apps-and-objects.md). :::{image} ../../../images/machine-learning-ml-create-job.png :alt: Create New Job @@ -37,7 +37,7 @@ You can create {{anomaly-jobs}} by using the [create {{anomaly-jobs}} API](https {{kib}} can also recognize certain types of data and provide specialized wizards for that context. For example, there are {{anomaly-jobs}} for the sample eCommerce orders and sample web logs data sets, as well as for data generated by the {{elastic-sec}} and {{observability}} solutions, {{beats}}, and {{agent}} {{integrations}}. For a list of all the preconfigured jobs, see [Supplied configurations](ootb-ml-jobs.md). -When you create an {{anomaly-job}} in {{kib}}, the job creation wizards can provide advice based on the characteristics of your data. By heeding these suggestions, you can create jobs that are more likely to produce insightful {{ml}} results. The most important concepts are covered here; for a description of all the job properties, see the [create {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html). +When you create an {{anomaly-job}} in {{kib}}, the job creation wizards can provide advice based on the characteristics of your data. By heeding these suggestions, you can create jobs that are more likely to produce insightful {{ml}} results. The most important concepts are covered here; for a description of all the job properties, see the [create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job). ### Bucket span [ml-ad-bucket-span] @@ -112,7 +112,7 @@ If you are using {{ece}} or the hosted Elasticsearch Service on Elastic Cloud, ` ### Dedicated indices [ml-ad-dedicated-indices] -For each {{anomaly-job}}, you can optionally specify a dedicated index to store the {{anomaly-detect}} results. As {{anomaly-jobs}} may produce a large amount of results (for example, jobs with many time series, small bucket span, or with long running period), it is recommended to use a dedicated results index by choosing the **Use dedicated index** option in {{kib}} or specifying the `results_index_name` via the [Create {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html). +For each {{anomaly-job}}, you can optionally specify a dedicated index to store the {{anomaly-detect}} results. As {{anomaly-jobs}} may produce a large amount of results (for example, jobs with many time series, small bucket span, or with long running period), it is recommended to use a dedicated results index by choosing the **Use dedicated index** option in {{kib}} or specifying the `results_index_name` via the [Create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job). ### {{dfeeds-cap}} [ml-ad-datafeeds] @@ -125,11 +125,11 @@ You can associate only one {{dfeed}} with each {{anomaly-job}}. The {{dfeed}} co ::::{important} When the {{es}} {{security-features}} are enabled, a {{dfeed}} stores the roles of the user who created or updated the {{dfeed}} at that time. This means that if those roles are updated, the {{dfeed}} subsequently runs with the new permissions that are associated with the roles. However, if the user’s roles are adjusted after creating or updating the {{dfeed}}, the {{dfeed}} continues to run with the permissions that were associated with the original roles. -One way to update the roles that are stored within the {{dfeed}} without changing any other settings is to submit an empty JSON document ({}) to the [update {{dfeed}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-update-datafeed.html). +One way to update the roles that are stored within the {{dfeed}} without changing any other settings is to submit an empty JSON document ({}) to the [update {{dfeed}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-update-datafeed). :::: -If the data that you want to analyze is not stored in {{es}}, you cannot use {{dfeeds}}. You can however send batches of data directly to the job by using the [post data to jobs API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-post-data.html). [7.11.0] +If the data that you want to analyze is not stored in {{es}}, you cannot use {{dfeeds}}. You can however send batches of data directly to the job by using the [post data to jobs API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-post-data). [7.11.0] ## Open the job [ml-ad-open-job] @@ -137,7 +137,7 @@ An {{anomaly-job}} must be opened in order for it to be ready to receive and ana After you start the job, you can start the {{dfeed}}, which retrieves data from your cluster. A {{dfeed}} can be started and stopped multiple times throughout its lifecycle. When you start it, you can optionally specify start and end times. If you do not specify an end time, the {{dfeed}} runs continuously. -You can perform both these tasks in {{kib}} or use the [open {{anomaly-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-open-job.html) and [start {{dfeeds}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-start-datafeed.html) APIs. +You can perform both these tasks in {{kib}} or use the [open {{anomaly-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-open-job) and [start {{dfeeds}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-datafeed) APIs. ## Tune the job [ml-ad-tune] @@ -147,13 +147,13 @@ Typically after you open a job, the next step is to [view the results](ml-ad-vie Sometimes there are periods when you expect unusual activity to take place, such as bank holidays, "Black Friday", or planned system outages. If you identify these events in advance, no anomalies are generated during that period. The {{ml}} model is not ill-affected and you do not receive spurious results. -You can create calendars and scheduled events in the **Settings** pane on the **Machine Learning** page in {{kib}} or by using [{{ml-cap}} {{anomaly-detect}} APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-ad-apis.html). +You can create calendars and scheduled events in the **Settings** pane on the **Machine Learning** page in {{kib}} or by using [{{ml-cap}} {{anomaly-detect}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-anomaly). A scheduled event must have a start time, end time, and description. In general, scheduled events are short in duration (typically lasting from a few hours to a day) and occur infrequently. If you have regularly occurring events, such as weekly maintenance periods, you do not need to create scheduled events for these circumstances; they are already handled by the {{ml}} analytics. You can identify zero or more scheduled events in a calendar. {{anomaly-jobs-cap}} can then subscribe to calendars and the {{ml}} analytics handle all subsequent scheduled events appropriately. -If you want to add multiple scheduled events at once, you can import an iCalendar (`.ics`) file in {{kib}} or a JSON file in the [add events to calendar API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-post-calendar-event.html). +If you want to add multiple scheduled events at once, you can import an iCalendar (`.ics`) file in {{kib}} or a JSON file in the [add events to calendar API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-post-calendar-events). ::::{note} @@ -184,9 +184,9 @@ For more information, see [Customizing detectors with custom rules](ml-configuri To ensure resilience in the event of a system failure, snapshots of the {{ml}} model for each {{anomaly-job}} are saved to an internal index within the {{es}} cluster. The amount of time necessary to save these snapshots is proportional to the size of the model in memory. By default, snapshots are captured approximately every 3 to 4 hours. You can change this interval (`background_persist_interval`) when you create or update a job. -To reduce the number of snapshots consuming space on your cluster, at the end of each day, old snapshots are automatically deleted. The age of each snapshot is calculated relative to the timestamp of the most recent snapshot. By default, if there are snapshots over one day older than the newest snapshot, they are deleted except for the first snapshot each day. As well, all snapshots over ten days older than the newest snapshot are deleted. You can change these retention settings (`daily_model_snapshot_retention_after_days` and `model_snapshot_retention_days`) when you create or update a job. If you want to exempt a specific snapshot from this clean up, use {{kib}} or the [update model snapshots API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-update-snapshot.html) to set `retain` to `true`. +To reduce the number of snapshots consuming space on your cluster, at the end of each day, old snapshots are automatically deleted. The age of each snapshot is calculated relative to the timestamp of the most recent snapshot. By default, if there are snapshots over one day older than the newest snapshot, they are deleted except for the first snapshot each day. As well, all snapshots over ten days older than the newest snapshot are deleted. You can change these retention settings (`daily_model_snapshot_retention_after_days` and `model_snapshot_retention_days`) when you create or update a job. If you want to exempt a specific snapshot from this clean up, use {{kib}} or the [update model snapshots API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-update-model-snapshot) to set `retain` to `true`. -You can see the list of model snapshots for each job with the [get model snapshots API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-snapshot.html) or in the **Model snapshots** tab on the **Job Management** page in {{kib}}: +You can see the list of model snapshots for each job with the [get model snapshots API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-model-snapshots) or in the **Model snapshots** tab on the **Job Management** page in {{kib}}: :::{image} ../../../images/machine-learning-ml-model-snapshots.png :alt: Example screenshot with a list of model snapshots @@ -194,7 +194,7 @@ You can see the list of model snapshots for each job with the [get model snapsho ::: ::::{tip} -There are situations other than system failures where you might want to [revert](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-revert-snapshot.html) to using a specific model snapshot. The {{ml-features}} react quickly to anomalous input and new behaviors in data. Highly anomalous input increases the variance in the models and {{ml}} analytics must determine whether it is a new step-change in behavior or a one-off event. In the case where you know this anomalous input is a one-off, it might be appropriate to reset the model state to a time before this event. For example, after a Black Friday sales day you might consider reverting to a saved snapshot. If you know about such events in advance, however, you can use [calendars and scheduled events](https://www.elastic.co/guide/en/machine-learning/current/ml-ad-run-jobs.html#ml-ad-calendars) to avoid impacting your model. +There are situations other than system failures where you might want to [revert](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-revert-model-snapshot) to using a specific model snapshot. The {{ml-features}} react quickly to anomalous input and new behaviors in data. Highly anomalous input increases the variance in the models and {{ml}} analytics must determine whether it is a new step-change in behavior or a one-off event. In the case where you know this anomalous input is a one-off, it might be appropriate to reset the model state to a time before this event. For example, after a Black Friday sales day you might consider reverting to a saved snapshot. If you know about such events in advance, however, you can use [calendars and scheduled events](https://www.elastic.co/guide/en/machine-learning/current/ml-ad-run-jobs.html#ml-ad-calendars) to avoid impacting your model. :::: ## Close the job [ml-ad-close-job] @@ -214,7 +214,7 @@ This process ensures that jobs are in a consistent state in case you want to sub ### Stopping {{dfeeds}} [stopping-ml-datafeeds] -When you stop a {{dfeed}}, it ceases to retrieve data from {{es}}. You can stop a {{dfeed}} by using {{kib}} or the [stop {{dfeeds}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-stop-datafeed.html). For example, the following request stops the `feed1` {{dfeed}}: +When you stop a {{dfeed}}, it ceases to retrieve data from {{es}}. You can stop a {{dfeed}} by using {{kib}} or the [stop {{dfeeds}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-stop-datafeed). For example, the following request stops the `feed1` {{dfeed}}: ```console POST _ml/datafeeds/feed1/_stop @@ -236,7 +236,7 @@ POST _ml/datafeeds/_all/_stop ### Closing {{anomaly-jobs}} [closing-ml-jobs] -When you close an {{anomaly-job}}, it cannot receive data or perform analysis operations. You can close a job by using the [close {{anomaly-job}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-close-job.html). For example, the following request closes the `job1` job: +When you close an {{anomaly-job}}, it cannot receive data or perform analysis operations. You can close a job by using the [close {{anomaly-job}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-close-job). For example, the following request closes the `job1` job: ```console POST _ml/anomaly_detectors/job1/_close @@ -246,7 +246,7 @@ POST _ml/anomaly_detectors/job1/_close You must have `manage_ml`, or `manage` cluster privileges to stop {{anomaly-jobs}}. For more information, see [Security privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md). :::: -If you submit a request to close an {{anomaly-job}} and its {{dfeed}} is running, the request first tries to stop the {{dfeed}}. This behavior is equivalent to calling the [stop {{dfeeds}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-stop-datafeed.html) with the same `timeout` and `force` parameters as the close job request. +If you submit a request to close an {{anomaly-job}} and its {{dfeed}} is running, the request first tries to stop the {{dfeed}}. This behavior is equivalent to calling the [stop {{dfeeds}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-stop-datafeed) with the same `timeout` and `force` parameters as the close job request. {{anomaly-jobs-cap}} can be opened and closed multiple times throughout their lifecycle. diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-ad-troubleshooting.md b/explore-analyze/machine-learning/anomaly-detection/ml-ad-troubleshooting.md index d44ebb494b..68b85088a8 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-ad-troubleshooting.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-ad-troubleshooting.md @@ -17,7 +17,7 @@ If an {{anomaly-job}} fails, try to restart the job by following the procedure d If an {{anomaly-job}} has failed, do the following to recover from `failed` state: -1. *Force* stop the corresponding {{dfeed}} by using the [Stop {{dfeed}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-stop-datafeed.html) with the `force` parameter being `true`. For example, the following request force stops the `my_datafeed` {{dfeed}}. +1. *Force* stop the corresponding {{dfeed}} by using the [Stop {{dfeed}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-stop-datafeed) with the `force` parameter being `true`. For example, the following request force stops the `my_datafeed` {{dfeed}}. ```console POST _ml/datafeeds/my_datafeed/_stop { @@ -25,7 +25,7 @@ If an {{anomaly-job}} has failed, do the following to recover from `failed` stat } ``` -2. *Force* close the {{anomaly-job}} by using the [Close {{anomaly-job}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-close-job.html) with the `force` parameter being `true`. For example, the following request force closes the `my_job` {{anomaly-job}}: +2. *Force* close the {{anomaly-job}} by using the [Close {{anomaly-job}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-close-job) with the `force` parameter being `true`. For example, the following request force closes the `my_job` {{anomaly-job}}: ```console POST _ml/anomaly_detectors/my_job/_close?force=true ``` diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-ad-view-results.md b/explore-analyze/machine-learning/anomaly-detection/ml-ad-view-results.md index 7261eabacb..28329242ae 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-ad-view-results.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-ad-view-results.md @@ -39,7 +39,7 @@ If you have [{{anomaly-detect-cap}} alert rules](https://www.elastic.co/guide/en :class: screenshot ::: -If you have more than one {{anomaly-job}}, you can also obtain *overall bucket* results, which combine and correlate anomalies from multiple jobs into an overall score. When you view the results for job groups in {{kib}}, it provides the overall bucket scores. For more information, see [Get overall buckets API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-overall-buckets.html). +If you have more than one {{anomaly-job}}, you can also obtain *overall bucket* results, which combine and correlate anomalies from multiple jobs into an overall score. When you view the results for job groups in {{kib}}, it provides the overall bucket scores. For more information, see [Get overall buckets API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-overall-buckets). Bucket results provide the top level, overall view of the {{anomaly-job}} and are ideal for alerts. For example, the bucket results might indicate that at 16:05 the system was unusual. This information is a summary of all the anomalies, pinpointing when they occurred. When you identify an anomalous bucket, you can investigate further by examining the pertinent records. @@ -47,7 +47,7 @@ Bucket results provide the top level, overall view of the {{anomaly-job}} and ar The influencer results show which entities were anomalous and when. One influencer result is written per bucket for each influencer that affects the anomalousness of the bucket. The {{ml}} analytics determine the impact of an influencer by performing a series of experiments that remove all data points with a specific influencer value and check whether the bucket is still anomalous. That means that only influencers with statistically significant impact on the anomaly are reported in the results. For jobs with more than one detector, influencer scores provide a powerful view of the most anomalous entities. -For example, the `high_sum_total_sales` {{anomaly-job}} for the eCommerce orders sample data uses `customer_full_name.keyword` and `category.keyword` as influencers. You can examine the influencer results with the [get influencers API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-influencer.html). Alternatively, you can use the **Anomaly Explorer** in {{kib}}: +For example, the `high_sum_total_sales` {{anomaly-job}} for the eCommerce orders sample data uses `customer_full_name.keyword` and `category.keyword` as influencers. You can examine the influencer results with the [get influencers API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-influencers). Alternatively, you can use the **Anomaly Explorer** in {{kib}}: :::{image} ../../../images/machine-learning-influencers.jpg :alt: Influencers in the {{kib}} Anomaly Explorer diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-anomaly-detection-job-types.md b/explore-analyze/machine-learning/anomaly-detection/ml-anomaly-detection-job-types.md index 4cd82fd3fd..1386d84843 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-anomaly-detection-job-types.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-anomaly-detection-job-types.md @@ -43,7 +43,7 @@ Refer to [Performing population analysis](https://www.elastic.co/guide/en/machin ## Advanced jobs [advanced-jobs] -Advanced jobs give you all the flexibility that’s possible in the [create {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html). At the extreme, you can switch to directly edit the JSON that will be sent to this endpoint. All the other types of jobs described in this page *can* be created as advanced jobs, but the more specialized wizards make it easier to create jobs for common situations. You can create an advanced job if you are familiar with all the functionality that {{ml}} {{anomaly-detect}} provides and want to do something that the more specialized wizards do not allow you to do. +Advanced jobs give you all the flexibility that’s possible in the [create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job). At the extreme, you can switch to directly edit the JSON that will be sent to this endpoint. All the other types of jobs described in this page *can* be created as advanced jobs, but the more specialized wizards make it easier to create jobs for common situations. You can create an advanced job if you are familiar with all the functionality that {{ml}} {{anomaly-detect}} provides and want to do something that the more specialized wizards do not allow you to do. ## Categorization jobs [categorization-jobs] diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-api-quickref.md b/explore-analyze/machine-learning/anomaly-detection/ml-api-quickref.md index 8905de7a16..a0c7e6964c 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-api-quickref.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-api-quickref.md @@ -16,11 +16,11 @@ All {{ml}} {anomaly-detect} endpoints have the following base: The main resources can be accessed with a variety of endpoints: -* [`/anomaly_detectors/`](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-ad-apis.html#ml-api-anomaly-job-endpoint): Create and manage {anomaly-jobs} -* [`/calendars/`](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-ad-apis.html#ml-api-calendar-endpoint): Create and manage calendars and scheduled events -* [`/datafeeds/`](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-ad-apis.html#ml-api-datafeed-endpoint): Select data from {{es}} to be analyzed -* [`/filters/`](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-ad-apis.html#ml-api-filter-endpoint): Create and manage filters for custom rules -* [`/results/`](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-ad-apis.html#ml-api-result-endpoint): Access the results of an {anomaly-job} -* [`/model_snapshots/`](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-ad-apis.html#ml-api-snapshot-endpoint): Manage model snapshots +* [`/anomaly_detectors/`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-anomaly): Create and manage {anomaly-jobs} +* [`/calendars/`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-anomaly): Create and manage calendars and scheduled events +* [`/datafeeds/`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-anomaly): Select data from {{es}} to be analyzed +* [`/filters/`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-anomaly): Create and manage filters for custom rules +* [`/results/`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-anomaly): Access the results of an {anomaly-job} +* [`/model_snapshots/`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-anomaly): Manage model snapshots -For a full list, see [{{ml-cap}} {anomaly-detect} APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-ad-apis.html). +For a full list, see [{{ml-cap}} {anomaly-detect} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-anomaly). diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md index 95d7cf0fdc..49e266acc9 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md @@ -78,7 +78,7 @@ For this type of job, the results contain extra information for each anomaly: th ### Advanced configuration options [advanced-categorization-options] -If you use the advanced {{anomaly-job}} wizard in {{kib}} or the [create {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html), there are additional configuration options. For example, the optional `categorization_examples_limit` property specifies the maximum number of examples that are stored in memory and in the results data store for each category. The default value is `4`. Note that this setting does not affect the categorization; it just affects the list of visible examples. If you increase this value, more examples are available, but you must have more storage available. If you set this value to `0`, no examples are stored. +If you use the advanced {{anomaly-job}} wizard in {{kib}} or the [create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job), there are additional configuration options. For example, the optional `categorization_examples_limit` property specifies the maximum number of examples that are stored in memory and in the results data store for each category. The default value is `4`. Note that this setting does not affect the categorization; it just affects the list of visible examples. If you increase this value, more examples are available, but you must have more storage available. If you set this value to `0`, no examples are stored. Another advanced option is the `categorization_filters` property, which can contain an array of regular expressions. If a categorization field value matches the regular expression, the portion of the field that is matched is not taken into consideration when defining categories. The categorization filters are applied in the order they are listed in the job configuration, which enables you to disregard multiple sections of the categorization field value. In this example, you might create a filter like `[ "\\[statement:.*\\]"]` to remove the SQL statement from the categorization algorithm. @@ -101,7 +101,7 @@ If you use the categorization wizard in {{kib}}, you can see which categorizatio :class: screenshot ::: -The categorization analyzer can refer to a built-in {{es}} analyzer or a combination of zero or more character filters, a tokenizer, and zero or more token filters. In this example, adding a [`pattern_replace` character filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-replace-charfilter.html) achieves the same behavior as the `categorization_filters` job configuration option described earlier. For more details about these properties, refer to the [`categorization_analyzer` API object](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html#ml-put-job-request-body). +The categorization analyzer can refer to a built-in {{es}} analyzer or a combination of zero or more character filters, a tokenizer, and zero or more token filters. In this example, adding a [`pattern_replace` character filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-replace-charfilter.html) achieves the same behavior as the `categorization_filters` job configuration option described earlier. For more details about these properties, refer to the [`categorization_analyzer` API object](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job#ml-put-job-request-body). If you use the default categorization analyzer in {{kib}} or omit the `categorization_analyzer` property from the API, the following default values are used: diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-detector-custom-rules.md b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-detector-custom-rules.md index 51124e08ea..a0ae5db977 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-detector-custom-rules.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-detector-custom-rules.md @@ -10,4 +10,4 @@ mapped_pages: [Custom rules](ml-ad-run-jobs.md#ml-ad-rules) – or *job rules* as {{kib}} refers to them – enable you to change the behavior of anomaly detectors based on domain-specific knowledge. -Custom rules describe *when* a detector should take a certain *action* instead of following its default behavior. To specify the *when* a rule uses a `scope` and `conditions`. You can think of `scope` as the categorical specification of a rule, while `conditions` are the numerical part. A rule can have a scope, one or more conditions, or a combination of scope and conditions. For the full list of specification details, see the [`custom_rules` object](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-job.html#put-customrules) in the create {{anomaly-jobs}} API. +Custom rules describe *when* a detector should take a certain *action* instead of following its default behavior. To specify the *when* a rule uses a `scope` and `conditions`. You can think of `scope` as the categorical specification of a rule, while `conditions` are the numerical part. A rule can have a scope, one or more conditions, or a combination of scope and conditions. For the full list of specification details, see the [`custom_rules` object](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job) in the create {{anomaly-jobs}} API. diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-delayed-data-detection.md b/explore-analyze/machine-learning/anomaly-detection/ml-delayed-data-detection.md index 6cf91de2c0..4774265a80 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-delayed-data-detection.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-delayed-data-detection.md @@ -10,7 +10,7 @@ mapped_pages: Delayed data are documents that are indexed late. That is to say, it is data related to a time that your {{dfeed}} has already processed and it is therefore never analyzed by your {{anomaly-job}}. -When you create a {{dfeed}}, you can specify a [`query_delay`](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-datafeed.html#ml-put-datafeed-request-body) setting. This setting enables the {{dfeed}} to wait for some time past real-time, which means any "late" data in this period is fully indexed before the {{dfeed}} tries to gather it. However, if the setting is set too low, the {{dfeed}} may query for data before it has been indexed and consequently miss that document. Conversely, if it is set too high, analysis drifts farther away from real-time. The balance that is struck depends upon each use case and the environmental factors of the cluster. +When you create a {{dfeed}}, you can specify a [`query_delay`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-datafeed) setting. This setting enables the {{dfeed}} to wait for some time past real-time, which means any "late" data in this period is fully indexed before the {{dfeed}} tries to gather it. However, if the setting is set too low, the {{dfeed}} may query for data before it has been indexed and consequently miss that document. Conversely, if it is set too high, analysis drifts farther away from real-time. The balance that is struck depends upon each use case and the environmental factors of the cluster. ::::{important} If you get an error that says `Datafeed missed XXXX documents due to ingest latency`, consider increasing the value of query_delay. If it doesn’t help, investigate the ingest latency and its cause. You can do this by comparing event and ingest timestamps. High latency is often caused by bursts of ingested documents, misconfiguration of the ingest pipeline, or misalignment of system clocks. diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md b/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md index 973640c56c..d3891fe7d2 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md @@ -293,7 +293,7 @@ To create a forecast in {{kib}}: As the job processes more data, you can click the **Forecast** button again and choose to see one of your forecasts overlaid on the actual data. The chart then contains the actual data values, the bounds for the expected values, the anomalies, the forecast data values, and the bounds for the forecast. This combination of actual and forecast data gives you an indication of how well the {{ml-features}} can extrapolate the future behavior of the data. -If you want to see this type of comparison for the {{kib}} sample data, which has a finite number of documents, you can reset the job and analyze only a subset of the data before you create a forecast. For example, reset one of your {{anomaly-jobs}} from the **Job Management** page in {{kib}} or use the [reset {{anomaly-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-reset-job.html). When you restart the {{dfeed}} for this job, choose a date part way through your sample data as the search end date. By default, the {{dfeed}} stops and the {{anomaly-job}} closes when it reaches that date. Create the forecast. You can then restart the {{dfeed}} to process the remaining data and generate the type of results shown here. +If you want to see this type of comparison for the {{kib}} sample data, which has a finite number of documents, you can reset the job and analyze only a subset of the data before you create a forecast. For example, reset one of your {{anomaly-jobs}} from the **Job Management** page in {{kib}} or use the [reset {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-reset-job). When you restart the {{dfeed}} for this job, choose a date part way through your sample data as the search end date. By default, the {{dfeed}} stops and the {{anomaly-job}} closes when it reaches that date. Create the forecast. You can then restart the {{dfeed}} to process the remaining data and generate the type of results shown here. ::::{tip} The {{kib}} sample data sets have timestamps that are relative to when you added the data sets. However, some of these dates are in the future. Therefore, for the purposes of this tutorial, when you restart your {{dfeed}} do not use the **No end time (Real-time search)** option. Specify the appropriate end dates so that it processes all of the data immediately. diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md b/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md index 1d75e4a402..3cade2a565 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md @@ -94,7 +94,7 @@ One of the counts associated with a {{ml}} job is `missing_field_count`, which i Since jobs analyze JSON data, the `missing_field_count` might be misleading. Missing fields might be expected due to the structure of the data and therefore do not generate poor results. -For more information about `missing_field_count`, see the [get {{anomaly-job}} statistics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-job-stats.html). +For more information about `missing_field_count`, see the [get {{anomaly-job}} statistics API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-job-stats). ### Security integration [_security_integration] @@ -102,7 +102,7 @@ When the {{es}} {{security-features}} are enabled, a {{dfeed}} stores the roles ### Job and {{dfeed}} APIs have a maximum search size [ml-result-size-limitations] -In 6.6 and later releases, the [get jobs API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-job.html) and the [get job statistics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-job-stats.html) return a maximum of 10,000 jobs. Likewise, the [get {{dfeeds}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-datafeed.html) and the [get {{dfeed}} statistics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-datafeed-stats.html) return a maximum of 10,000 {{dfeeds}}. +In 6.6 and later releases, the [get jobs API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-jobs) and the [get job statistics API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-job-stats) return a maximum of 10,000 jobs. Likewise, the [get {{dfeeds}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-datafeeds) and the [get {{dfeed}} statistics API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-datafeed-stats) return a maximum of 10,000 {{dfeeds}}. ### Forecast operational limitations [ml-forecast-limitations] diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-reverting-model-snapshot.md b/explore-analyze/machine-learning/anomaly-detection/ml-reverting-model-snapshot.md index 897d60d060..fc1f24583c 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-reverting-model-snapshot.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-reverting-model-snapshot.md @@ -33,7 +33,7 @@ mapped_pages: You can use [custom rules](ml-ad-run-jobs.md#ml-ad-rules) to avoid a model being updated in case of a known event you want to exclude from the analysis. Using custom rules might help you to avoid situations where you need to revert to a snapshot. :::: -Alternatively, you can use the [revert model snapshots](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-revert-snapshot.html) API. In this case, you need to manually close the corresponding job before reverting to the saved snapshot. +Alternatively, you can use the [revert model snapshots](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-revert-model-snapshot) API. In this case, you need to manually close the corresponding job before reverting to the saved snapshot. ::::{note} diff --git a/explore-analyze/machine-learning/anomaly-detection/move-jobs.md b/explore-analyze/machine-learning/anomaly-detection/move-jobs.md index 7014d25f81..adf57b9962 100644 --- a/explore-analyze/machine-learning/anomaly-detection/move-jobs.md +++ b/explore-analyze/machine-learning/anomaly-detection/move-jobs.md @@ -16,5 +16,5 @@ The exported file contains configuration details; it does not contain the {{ml}} There are some additional actions that you must take before you can successfully import and run your jobs: 1. The {{kib}} [{{data-sources}}](https://www.elastic.co/guide/en/kibana/current/data-views.html) that are used by {{anomaly-detect}} {dfeeds} and {{dfanalytics}} source indices must exist; otherwise, the import fails. -2. If your {{anomaly-jobs}} use [custom rules](ml-configuring-detector-custom-rules.md) with filter lists, the filter lists must exist; otherwise, the import fails. To create filter lists, use {{kib}} or the [create filters API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-filter.html). -3. If your {{anomaly-jobs}} were associated with [calendars](https://www.elastic.co/guide/en/machine-learning/current/ml-ad-run-jobs.html#ml-ad-calendars), you must create the calendar in the new environment and add your imported jobs to the calendar. Use {{kib}} or the [create calendars](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-calendar.html), [add events to calendar](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-post-calendar-event.html), and [add jobs to calendar](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-put-calendar-job.html) APIs. +2. If your {{anomaly-jobs}} use [custom rules](ml-configuring-detector-custom-rules.md) with filter lists, the filter lists must exist; otherwise, the import fails. To create filter lists, use {{kib}} or the [create filters API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-filter). +3. If your {{anomaly-jobs}} were associated with [calendars](https://www.elastic.co/guide/en/machine-learning/current/ml-ad-run-jobs.html#ml-ad-calendars), you must create the calendar in the new environment and add your imported jobs to the calendar. Use {{kib}} or the [create calendars](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-calendar), [add events to calendar](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-post-calendar-events), and [add jobs to calendar](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-calendar-job) APIs. diff --git a/explore-analyze/machine-learning/data-frame-analytics/hyperparameters.md b/explore-analyze/machine-learning/data-frame-analytics/hyperparameters.md index 21453600d2..f0c0be3814 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/hyperparameters.md +++ b/explore-analyze/machine-learning/data-frame-analytics/hyperparameters.md @@ -12,9 +12,9 @@ When you create a {{dfanalytics-job}} for {{classification}} or {{reganalysis}}, Hyperparameter optimization involves multiple rounds of analysis. Each round involves a different combination of hyperparameter values, which are determined through a combination of random search and Bayesian optimization techniques. If you explicitly set a hyperparameter, that value is not optimized and remains the same in each round. To determine which round produces the best results, stratified K-fold cross-validation methods are used to split the data set, train a model, and calculate its performance on validation data. -You can view the hyperparameter values that were ultimately chosen by expanding the job details in {{kib}} or by using the [get trained models API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html). You can also see the specific type of validation loss (such as mean squared error or binomial cross entropy) that was used to compare each round of optimization using the [get {{dfanalytics-job}} stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-dfanalytics-stats.html). +You can view the hyperparameter values that were ultimately chosen by expanding the job details in {{kib}} or by using the [get trained models API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models). You can also see the specific type of validation loss (such as mean squared error or binomial cross entropy) that was used to compare each round of optimization using the [get {{dfanalytics-job}} stats API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-data-frame-analytics-stats). -Different hyperparameters may affect the model performance to a different degree. To estimate the importance of the optimized hyperparameters, analysis of variance decomposition is used. The resulting `absolute importance` shows how much the variation of a hyperparameter impacts the variation in the validation loss. Additionally, `relative importance` is also computed which gives the importance of the hyperparameter compared to the rest of the tuneable hyperparameters. The sum of all relative importances is 1. You can check these results in the response of the [get {{dfanalytics-job}} stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-dfanalytics-stats.html). +Different hyperparameters may affect the model performance to a different degree. To estimate the importance of the optimized hyperparameters, analysis of variance decomposition is used. The resulting `absolute importance` shows how much the variation of a hyperparameter impacts the variation in the validation loss. Additionally, `relative importance` is also computed which gives the importance of the hyperparameter compared to the rest of the tuneable hyperparameters. The sum of all relative importances is 1. You can check these results in the response of the [get {{dfanalytics-job}} stats API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-data-frame-analytics-stats). ::::{tip} Unless you fully understand the purpose of a hyperparameter, it is highly recommended that you leave it unset and allow hyperparameter optimization to occur. diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-classification.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-classification.md index 895793dc1b..30753ac18a 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-classification.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-classification.md @@ -16,7 +16,7 @@ mapped_pages: In reality, {{classification}} problems are more complex, such as classifying malicious and benign domains to detect DGA activities for security reasons or predicting customer churn based on customer calling data. {{classification-cap}} is for predicting discrete, categorical values. -When you create a {{classification}} job, you must specify which field contains the classes that you want to predict. This field is known as the *{{depvar}}*. It can contain maximum 100 classes. By default, all other [supported fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html#dfa-supported-fields) are included in the analysis and are known as *{{feature-vars}}*. You can optionally include or exclude fields. For more information about field selection, refer to the [explain data frame analytics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/explain-dfanalytics.html). +When you create a {{classification}} job, you must specify which field contains the classes that you want to predict. This field is known as the *{{depvar}}*. It can contain maximum 100 classes. By default, all other [supported fields](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics) are included in the analysis and are known as *{{feature-vars}}*. You can optionally include or exclude fields. For more information about field selection, refer to the [explain data frame analytics API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-explain-data-frame-analytics). ## {{classification-cap}} algorithms [dfa-classification-algorithm] @@ -42,7 +42,7 @@ To learn more about how to prepare your data, refer to [the relevant section](ml ## 4. Create a job [dfa-classification-create-job] -{{dfanalytics-jobs-cap}} contain the configuration information and metadata necessary to perform an analytics task. You can create {{dfanalytics-jobs}} via {{kib}} or using the [create {{dfanalytics-jobs}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html). +{{dfanalytics-jobs-cap}} contain the configuration information and metadata necessary to perform an analytics task. You can create {{dfanalytics-jobs}} via {{kib}} or using the [create {{dfanalytics-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics). Select {{classification}} as the analytics type, then select the field that you want to predict (the {{depvar}}). You can also include and exclude fields. @@ -54,7 +54,7 @@ To improve performance, consider using a small `training_percent` value to train ## 5. Start the job [dfa-classification-start] -You can start the job via {{kib}} or using the [start {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-dfanalytics.html) API. A {{classification}} job has the following phases: +You can start the job via {{kib}} or using the [start {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-data-frame-analytics) API. A {{classification}} job has the following phases: * `reindexing`: Documents are copied from the source index to the destination index. * `loading_data`: The job fetches the necessary data from the destination index. @@ -73,11 +73,11 @@ When you create a {{dfanalytics-job}}, the inference step of the process might f ## 6. Evaluate and interpret the result [ml-dfanalytics-classification-evaluation] -Using the {{dfanalytics}} features to gain insights from a data set is an iterative process. After you defined the problem you want to solve, and chose the analytics type that can help you to do so, you need to produce a high-quality data set and create the appropriate {{dfanalytics-job}}. You might need to experiment with different configurations, parameters, and ways to transform data before you arrive at a result that satisfies your use case. A valuable companion to this process is the [{{evaluatedf-api}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/evaluate-dfanalytics.html), which enables you to evaluate the {{dfanalytics}} performance. It helps you understand error distributions and identifies the points where the {{dfanalytics}} model performs well or less trustworthily. +Using the {{dfanalytics}} features to gain insights from a data set is an iterative process. After you defined the problem you want to solve, and chose the analytics type that can help you to do so, you need to produce a high-quality data set and create the appropriate {{dfanalytics-job}}. You might need to experiment with different configurations, parameters, and ways to transform data before you arrive at a result that satisfies your use case. A valuable companion to this process is the [{{evaluatedf-api}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-evaluate-data-frame), which enables you to evaluate the {{dfanalytics}} performance. It helps you understand error distributions and identifies the points where the {{dfanalytics}} model performs well or less trustworthily. To evaluate the analysis with this API, you need to annotate your index that contains the results of the analysis with a field that marks each document with the ground truth. The {{evaluatedf-api}} evaluates the performance of the {{dfanalytics}} against this manually provided ground truth. -You can measure how well the model has performed on your training data set by using the `classification` evaluation type of the [evaluate {{dfanalytics}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/evaluate-dfanalytics.html) or by viewing the job results in {{kib}}. The {{classification}} evaluation offers the following metrics to evaluate the model performance: +You can measure how well the model has performed on your training data set by using the `classification` evaluation type of the [evaluate {{dfanalytics}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-evaluate-data-frame) or by viewing the job results in {{kib}}. The {{classification}} evaluation offers the following metrics to evaluate the model performance: * Multiclass confusion matrix * Area under the curve of receiver operating characteristic (AUC ROC) @@ -193,16 +193,16 @@ For instance, suppose you have an online service and you would like to predict w {{infer-cap}} can be used as a processor specified in an [ingest pipeline](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md). It uses a trained model to infer against the data that is being ingested in the pipeline. The model is used on the ingest node. {{infer-cap}} pre-processes the data by using the model and provides a prediction. After the process, the pipeline continues executing (if there is any other processor in the pipeline), finally the new data together with the results are indexed into the destination index. -Check the [{{infer}} processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-processor.html) and [the {{ml}} {dfanalytics} API documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-df-analytics-apis.html) to learn more. +Check the [{{infer}} processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-processor.html) and [the {{ml}} {dfanalytics} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. #### {{infer-cap}} aggregation [ml-inference-aggregation-class] {{infer-cap}} can also be used as a pipeline aggregation. You can reference a trained model in the aggregation to infer on the result field of the parent bucket aggregation. The {{infer}} aggregation uses the model on the results to provide a prediction. This aggregation enables you to run {{classification}} or {{reganalysis}} at search time. If you want to perform the analysis on a small set of data, this aggregation enables you to generate predictions without the need to set up a processor in the ingest pipeline. -Check the [{{infer}} bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-inference-bucket-aggregation.html) and [the {{ml}} {dfanalytics} API documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-df-analytics-apis.html) to learn more. +Check the [{{infer}} bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-inference-bucket-aggregation.html) and [the {{ml}} {dfanalytics} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. ::::{note} -If you use trained model aliases to reference your trained model in an {{infer}} processor or {{infer}} aggregation, you can replace your trained model with a new one without the need of updating the processor or the aggregation. Reassign the alias you used to a new trained model ID by using the [Create or update trained model aliases API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-trained-models-aliases.html). The new trained model needs to use the same type of {{dfanalytics}} as the old one. +If you use trained model aliases to reference your trained model in an {{infer}} processor or {{infer}} aggregation, you can replace your trained model with a new one without the need of updating the processor or the aggregation. Reassign the alias you used to a new trained model ID by using the [Create or update trained model aliases API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model-alias). The new trained model needs to use the same type of {{dfanalytics}} as the old one. :::: ## Performing {{classanalysis}} in the sample flight data set [performing-classification] @@ -279,7 +279,7 @@ The sample flight data set is used in this example because it is easily accessib To predict whether a specific flight is delayed: 1. Create a {{dfanalytics-job}}. - You can use the wizard on the **{{ml-app}}** > **Data Frame Analytics** tab in {{kib}} or the [create {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html) API. + You can use the wizard on the **{{ml-app}}** > **Data Frame Analytics** tab in {{kib}} or the [create {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics) API. :::{image} ../../../images/machine-learning-flights-classification-job-1.jpg :alt: Creating a {{dfanalytics-job}} in {kib} :class: screenshot @@ -341,7 +341,7 @@ PUT _ml/data_frame/analytics/model-flight-delays-classification After you configured your job, the configuration details are automatically validated. If the checks are successful, you can start the job. A warning message is shown if the configuration is invalid. The message contains a suggestion to improve the configuration to be validated. -2. Start the job in {{kib}} or use the [start {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-dfanalytics.html) API. +2. Start the job in {{kib}} or use the [start {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-data-frame-analytics) API. The job takes a few minutes to run. Runtime depends on the local hardware and also on the number of documents and fields that are analyzed. The more fields and documents, the longer the job runs. It stops automatically when the analysis is complete. @@ -353,7 +353,7 @@ POST _ml/data_frame/analytics/model-flight-delays-classification/_start :::: -3. Check the job stats to follow the progress in {{kib}} or use the [get {{dfanalytics-jobs}} statistics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-dfanalytics-stats.html). +3. Check the job stats to follow the progress in {{kib}} or use the [get {{dfanalytics-jobs}} statistics API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-data-frame-analytics-stats). :::{image} ../../../images/machine-learning-flights-classification-details.jpg :alt: Statistics for a {{dfanalytics-job}} in {kib} @@ -530,7 +530,7 @@ You can also see the {{feat-imp}} values for each individual prediction in the f In {{kib}}, the decision path shows the relative impact of each feature on the probability of the prediction. The features with the most significant positive or negative impact appear at the top of the decision plot. Thus in this example, the features related to flight time and distance had the most significant influence on the probability value for this prediction. This type of information can help you to understand how models arrive at their predictions. It can also indicate which aspects of your data set are most influential or least useful when you are training and tuning your model. -If you do not use {{kib}}, you can see the summarized {{feat-imp}} values by using the [get trained model API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html) and the individual values by searching the destination index. +If you do not use {{kib}}, you can see the summarized {{feat-imp}} values by using the [get trained model API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) and the individual values by searching the destination index. ::::{dropdown} API example @@ -691,7 +691,7 @@ Likewise if you select other quadrants in the matrix, it shows the number of doc :class: screenshot ::: -You can also generate these metrics with the [{{dfanalytics}} evaluate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/evaluate-dfanalytics.html). For more information about interpreting the evaluation metrics, see [6. Evaluate and interpret the result](#ml-dfanalytics-classification-evaluation). +You can also generate these metrics with the [{{dfanalytics}} evaluate API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-evaluate-data-frame). For more information about interpreting the evaluation metrics, see [6. Evaluate and interpret the result](#ml-dfanalytics-classification-evaluation). ::::{dropdown} API example First, we want to know the training error that represents how well the model performed on the training data set. @@ -799,7 +799,7 @@ The returned confusion matrix shows us how many data points were classified corr :::: -If you don’t want to keep the {{dfanalytics-job}}, you can delete it in {{kib}} or by using the [delete {{dfanalytics-job}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-dfanalytics.html). When you delete {{dfanalytics-jobs}} in {{kib}}, you have the option to also remove the destination indices and {{data-sources}}. +If you don’t want to keep the {{dfanalytics-job}}, you can delete it in {{kib}} or by using the [delete {{dfanalytics-job}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-delete-data-frame-analytics). When you delete {{dfanalytics-jobs}} in {{kib}}, you have the option to also remove the destination indices and {{data-sources}}. ### Further readings [dfa-classification-readings] diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-finding-outliers.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-finding-outliers.md index cb7ae7b9e9..e0f0483391 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-finding-outliers.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-finding-outliers.md @@ -49,7 +49,7 @@ You can find an example of how to transform your data into an entity-centric ind ## 4. Create a job [dfa-outlier-detection-create-job] -{{dfanalytics-cap}} jobs contain the configuration information and metadata necessary to perform an analytics task. You can create {{dfanalytics}} jobs via {{kib}} or using the [create {{dfanalytics}} jobs API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html). Select {{oldetection}} as the analytics type that the {{dfanalytics}} job performs. You can also decide to include and exclude fields to/from the analysis when you create the job. +{{dfanalytics-cap}} jobs contain the configuration information and metadata necessary to perform an analytics task. You can create {{dfanalytics}} jobs via {{kib}} or using the [create {{dfanalytics}} jobs API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics). Select {{oldetection}} as the analytics type that the {{dfanalytics}} job performs. You can also decide to include and exclude fields to/from the analysis when you create the job. ::::{tip} You can view the statistics of the selectable fields in the {{dfanalytics}} wizard. The field statistics displayed in a flyout provide more meaningful context to help you select relevant fields. @@ -57,7 +57,7 @@ You can view the statistics of the selectable fields in the {{dfanalytics}} wiza ## 5. Start the job [dfa-outlier-detection-start] -You can start the job via {{kib}} or using the [start {{dfanalytics}} job](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-dfanalytics.html) API. An {{oldetection}} job has four phases: +You can start the job via {{kib}} or using the [start {{dfanalytics}} job](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-data-frame-analytics) API. An {{oldetection}} job has four phases: * `reindexing`: documents are copied from the source index to the destination index. * `loading_data`: the job fetches the necessary data from the destination index. @@ -70,7 +70,7 @@ After the last phase is finished, the job stops and the results are ready for ev ## 6. Evaluate the results [ml-outlier-detection-evaluate] -Using the {{dfanalytics}} features to gain insights from a data set is an iterative process. After you defined the problem you want to solve, and chose the analytics type that can help you to do so, you need to produce a high-quality data set and create the appropriate {{dfanalytics}} job. You might need to experiment with different configurations, parameters, and ways to transform data before you arrive at a result that satisfies your use case. A valuable companion to this process is the [evaluate {{dfanalytics}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/evaluate-dfanalytics.html), which enables you to evaluate the {{dfanalytics}} performance. It helps you understand error distributions and identifies the points where the {{dfanalytics}} model performs well or less trustworthily. +Using the {{dfanalytics}} features to gain insights from a data set is an iterative process. After you defined the problem you want to solve, and chose the analytics type that can help you to do so, you need to produce a high-quality data set and create the appropriate {{dfanalytics}} job. You might need to experiment with different configurations, parameters, and ways to transform data before you arrive at a result that satisfies your use case. A valuable companion to this process is the [evaluate {{dfanalytics}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-evaluate-data-frame), which enables you to evaluate the {{dfanalytics}} performance. It helps you understand error distributions and identifies the points where the {{dfanalytics}} model performs well or less trustworthily. To evaluate the analysis with this API, you need to annotate your index that contains the results of the analysis with a field that marks each document with the ground truth. The evaluate {{dfanalytics}} API evaluates the performance of the {{dfanalytics}} against this manually provided ground truth. @@ -124,7 +124,7 @@ The goal of {{oldetection}} is to find the most unusual documents in an index. L :alt: Creating a {{transform}} in {kib} :class: screenshot ::: - Alternatively, you can use the [preview {{transform}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html) and the [create {{transform}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html). + Alternatively, you can use the [preview {{transform}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-preview-transform) and the [create {{transform}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform). ::::{dropdown} API example @@ -224,7 +224,7 @@ POST _transform/_preview Even though resource utilization is automatically adjusted based on the cluster load, a {{transform}} increases search and indexing load on your cluster while it runs. If you’re experiencing an excessive load, however, you can stop it. :::: - You can start, stop, and manage {{transforms}} in {{kib}}. Alternatively, you can use the [start {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-transform.html) API. + You can start, stop, and manage {{transforms}} in {{kib}}. Alternatively, you can use the [start {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-start-transform) API. ::::{dropdown} API example @@ -250,7 +250,7 @@ POST _transform/logs-by-clientip/_start If you want these charts to represent data from a larger sample size or from a randomized selection of documents, you can change the default behavior. However, a larger sample size might slow down the performance of the matrix and a randomized selection might put more load on the cluster due to the more intensive query. - Alternatively, you can use the [create {{dfanalytics}} jobs API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html). + Alternatively, you can use the [create {{dfanalytics}} jobs API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics). ::::{dropdown} API example @@ -278,7 +278,7 @@ PUT _ml/data_frame/analytics/weblog-outliers After you configured your job, the configuration details are automatically validated. If the checks are successful, you can proceed and start the job. A warning message is shown if the configuration is invalid. The message contains a suggestion to improve the configuration to be validated. 5. Start the {{dfanalytics}} job. - You can start, stop, and manage {{dfanalytics-jobs}} on the **Machine Learning** > **Data Frame Analytics** page. Alternatively, you can use the [start {{dfanalytics}} jobs](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-dfanalytics.html) and [stop {{dfanalytics}} jobs](https://www.elastic.co/guide/en/elasticsearch/reference/current/stop-dfanalytics.html) APIs. + You can start, stop, and manage {{dfanalytics-jobs}} on the **Machine Learning** > **Data Frame Analytics** page. Alternatively, you can use the [start {{dfanalytics}} jobs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-data-frame-analytics) and [stop {{dfanalytics}} jobs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-stop-data-frame-analytics) APIs. ::::{dropdown} API example @@ -350,7 +350,7 @@ GET weblog-outliers/_search?q="111.237.144.54" Now that you’ve found unusual behavior in the sample data set, consider how you might apply these steps to other data sets. If you have data that is already marked up with true outliers, you can determine how well the {{oldetection}} algorithms perform by using the evaluate {{dfanalytics}} API. See [6. Evaluate the results](#ml-outlier-detection-evaluate). ::::{tip} -If you do not want to keep the {{transform}} and the {{dfanalytics}} job, you can delete them in {{kib}} or use the [delete {{transform}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-transform.html) and [delete {{dfanalytics}} job API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-dfanalytics.html). When you delete {{transforms}} and {{dfanalytics}} jobs in {{kib}}, you have the option to also remove the destination indices and {{data-sources}}. +If you do not want to keep the {{transform}} and the {{dfanalytics}} job, you can delete them in {{kib}} or use the [delete {{transform}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-delete-transform) and [delete {{dfanalytics}} job API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-delete-data-frame-analytics). When you delete {{transforms}} and {{dfanalytics}} jobs in {{kib}}, you have the option to also remove the destination indices and {{data-sources}}. :::: ## Further reading [outlier-detection-reading] diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-limitations.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-limitations.md index f586688f3f..d5975cdec1 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-limitations.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-limitations.md @@ -53,7 +53,7 @@ Trained models created in version 7.8.0 are not backwards compatible with older ### Deleting a {{dfanalytics-job}} does not delete the destination index [dfa-deletion-limitations] -The [delete {{dfanalytics-job}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-dfanalytics.html) does not delete the destination index that contains the annotated data of the {{dfanalytics}}. That index must be deleted separately. +The [delete {{dfanalytics-job}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-delete-data-frame-analytics) does not delete the destination index that contains the annotated data of the {{dfanalytics}}. That index must be deleted separately. ### {{dfanalytics-jobs-cap}} runtime may vary [dfa-time-limitations] diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-phases.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-phases.md index eef424f58b..3e867d49d2 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-phases.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-phases.md @@ -25,7 +25,7 @@ Let’s take a look at the phases one-by-one. During the reindexing phase the documents from the source index or indices are copied to the destination index. If you want to define settings or mappings, create the index before you start the job. Otherwise, the job creates it using default settings. -Once the destination index is built, the {{dfanalytics-job}} task calls the {{es}} [Reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) to launch the reindexing task. +Once the destination index is built, the {{dfanalytics-job}} task calls the {{es}} [Reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) to launch the reindexing task. ## Loading data [ml-dfa-phases-load] diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-regression.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-regression.md index 8f1f8d50f3..bb7b77fa1f 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-regression.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-regression.md @@ -36,7 +36,7 @@ To learn more about how to prepare your data, refer to [the relevant section](ml ## 4. Create a job [dfa-regression-create-job] -{{dfanalytics-cap}} jobs contain the configuration information and metadata necessary to perform an analytics task. You can create {{dfanalytics}} jobs via {{kib}} or using the [create {{dfanalytics}} jobs API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html). +{{dfanalytics-cap}} jobs contain the configuration information and metadata necessary to perform an analytics task. You can create {{dfanalytics}} jobs via {{kib}} or using the [create {{dfanalytics}} jobs API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics). Select {{regression}} as the analytics type for the job, then select the field that you want to predict (the {{depvar}}). You can also include and exclude fields to/from the analysis. @@ -46,7 +46,7 @@ You can view the statistics of the selectable fields in the {{dfanalytics}} wiza ## 5. Start the job [dfa-regression-start] -You can start the job via {{kib}} or using the [start {{dfanalytics}} jobs](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-dfanalytics.html) API. A {{regression}} job has the following phases: +You can start the job via {{kib}} or using the [start {{dfanalytics}} jobs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-data-frame-analytics) API. A {{regression}} job has the following phases: * `reindexing`: Documents are copied from the source index to the destination index. * `loading_data`: The job fetches the necessary data from the destination index. @@ -65,11 +65,11 @@ When you create a {{dfanalytics-job}}, the inference step of the process might f ## 6. Evaluate the result [ml-dfanalytics-regression-evaluation] -Using the {{dfanalytics}} features to gain insights from a data set is an iterative process. After you defined the problem you want to solve, and chose the analytics type that can help you to do so, you need to produce a high-quality data set and create the appropriate {{dfanalytics}} job. You might need to experiment with different configurations, parameters, and ways to transform data before you arrive at a result that satisfies your use case. A valuable companion to this process is the [evaluate {{dfanalytics}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/evaluate-dfanalytics.html), which enables you to evaluate the {{dfanalytics}} performance. It helps you understand error distributions and identifies the points where the {{dfanalytics}} model performs well or less trustworthily. +Using the {{dfanalytics}} features to gain insights from a data set is an iterative process. After you defined the problem you want to solve, and chose the analytics type that can help you to do so, you need to produce a high-quality data set and create the appropriate {{dfanalytics}} job. You might need to experiment with different configurations, parameters, and ways to transform data before you arrive at a result that satisfies your use case. A valuable companion to this process is the [evaluate {{dfanalytics}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-evaluate-data-frame), which enables you to evaluate the {{dfanalytics}} performance. It helps you understand error distributions and identifies the points where the {{dfanalytics}} model performs well or less trustworthily. To evaluate the analysis with this API, you need to annotate your index that contains the results of the analysis with a field that marks each document with the ground truth. The {{evaluatedf-api}} evaluates the performance of the {{dfanalytics}} against this manually provided ground truth. -You can measure how well the model has performed on your training data by using the `regression` evaluation type of the [evaluate {{dfanalytics}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/evaluate-dfanalytics.html). The [mean squared error (MSE)](#ml-dfanalytics-mse) value that the evaluation provides you on the training data set is the *training error*. Training and evaluating the model iteratively means finding the combination of model parameters that produces the lowest possible training error. +You can measure how well the model has performed on your training data by using the `regression` evaluation type of the [evaluate {{dfanalytics}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-evaluate-data-frame). The [mean squared error (MSE)](#ml-dfanalytics-mse) value that the evaluation provides you on the training data set is the *training error*. Training and evaluating the model iteratively means finding the combination of model parameters that produces the lowest possible training error. Another crucial measurement is how well your model performs on unseen data. To assess how well the trained model will perform on data it has never seen before, you must set aside a proportion of the training set for testing (testing data). Once the model is trained, you can let it predict the value of the data points it has never seen before and compare the prediction to the actual value. This test provides an estimate of a quantity known as the *model generalization error*. @@ -139,16 +139,16 @@ For instance, suppose you have an online service and you would like to predict w {{infer-cap}} can be used as a processor specified in an [ingest pipeline](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md). It uses a trained model to infer against the data that is being ingested in the pipeline. The model is used on the ingest node. {{infer-cap}} pre-processes the data by using the model and provides a prediction. After the process, the pipeline continues executing (if there is any other processor in the pipeline), finally the new data together with the results are indexed into the destination index. -Check the [{{infer}} processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-processor.html) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-df-analytics-apis.html) to learn more. +Check the [{{infer}} processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-processor.html) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. #### {{infer-cap}} aggregation [ml-inference-aggregation-reg] {{infer-cap}} can also be used as a pipeline aggregation. You can reference a trained model in the aggregation to infer on the result field of the parent bucket aggregation. The {{infer}} aggregation uses the model on the results to provide a prediction. This aggregation enables you to run {{classification}} or {{reganalysis}} at search time. If you want to perform the analysis on a small set of data, this aggregation enables you to generate predictions without the need to set up a processor in the ingest pipeline. -Check the [{{infer}} bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-inference-bucket-aggregation.html) and [the {{ml}} {dfanalytics} API documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-df-analytics-apis.html) to learn more. +Check the [{{infer}} bucket aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-inference-bucket-aggregation.html) and [the {{ml}} {dfanalytics} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. ::::{note} -If you use trained model aliases to reference your trained model in an {{infer}} processor or {{infer}} aggregation, you can replace your trained model with a new one without the need of updating the processor or the aggregation. Reassign the alias you used to a new trained model ID by using the [Create or update trained model aliases API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-trained-models-aliases.html). The new trained model needs to use the same type of {{dfanalytics}} as the old one. +If you use trained model aliases to reference your trained model in an {{infer}} processor or {{infer}} aggregation, you can replace your trained model with a new one without the need of updating the processor or the aggregation. Reassign the alias you used to a new trained model ID by using the [Create or update trained model aliases API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model-alias). The new trained model needs to use the same type of {{dfanalytics}} as the old one. :::: ## Performing {{reganalysis}} in the sample flight data set [performing-regression] @@ -222,7 +222,7 @@ To predict the number of minutes delayed for each flight: 1. Verify that your environment is set up properly to use {{ml-features}}. The {{stack}} {security-features} require a user that has authority to create and manage {{dfanalytics-jobs}}. See [Setup and security](../setting-up-machine-learning.md). 2. Create a {{dfanalytics-job}}. - You can use the wizard on the **{{ml-app}}** > **Data Frame Analytics** tab in {{kib}} or the [create {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html) API. + You can use the wizard on the **{{ml-app}}** > **Data Frame Analytics** tab in {{kib}} or the [create {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics) API. :::{image} ../../../images/machine-learning-flights-regression-job-1.jpg :alt: Creating a {{dfanalytics-job}} in {kib} :class: screenshot @@ -289,7 +289,7 @@ PUT _ml/data_frame/analytics/model-flight-delays-regression After you configured your job, the configuration details are automatically validated. If the checks are successful, you can proceed and start the job. A warning message is shown if the configuration is invalid. The message contains a suggestion to improve the configuration to be validated. -3. Start the job in {{kib}} or use the [start {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-dfanalytics.html) API. +3. Start the job in {{kib}} or use the [start {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-data-frame-analytics) API. The job takes a few minutes to run. Runtime depends on the local hardware and also on the number of documents and fields that are analyzed. The more fields and documents, the longer the job runs. It stops automatically when the analysis is complete. @@ -301,7 +301,7 @@ POST _ml/data_frame/analytics/model-flight-delays-regression/_start :::: -4. Check the job stats to follow the progress in {{kib}} or use the [get {{dfanalytics-jobs}} statistics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-dfanalytics-stats.html). +4. Check the job stats to follow the progress in {{kib}} or use the [get {{dfanalytics-jobs}} statistics API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-data-frame-analytics-stats). :::{image} ../../../images/machine-learning-flights-regression-details.jpg :alt: Statistics for a {{dfanalytics-job}} in {kib} @@ -434,7 +434,7 @@ You can also see the {{feat-imp}} values for each individual prediction in the f The decision path starts at a baseline, which is the average of the predictions for all the data points in the training data set. From there, the feature importance values are added to the decision path until it arrives at its final prediction. The features with the most significant positive or negative impact appear at the top. Thus in this example, the features related to the flight distance had the most significant influence on this particular predicted flight delay. This type of information can help you to understand how models arrive at their predictions. It can also indicate which aspects of your data set are most influential or least useful when you are training and tuning your model. -If you do not use {{kib}}, you can see summarized {{feat-imp}} values by using the [get trained model API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html) and the individual values by searching the destination index. +If you do not use {{kib}}, you can see summarized {{feat-imp}} values by using the [get trained model API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) and the individual values by searching the destination index. ::::{dropdown} API example @@ -542,7 +542,7 @@ A mean squared error (MSE) of zero means that the models predicts the {{depvar}} For more information about the interpreting the evaluation metrics, see [6. Evaluate the result](#ml-dfanalytics-regression-evaluation). -You can alternatively generate these metrics with the [{{dfanalytics}} evaluate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/evaluate-dfanalytics.html). +You can alternatively generate these metrics with the [{{dfanalytics}} evaluate API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-evaluate-data-frame). ::::{dropdown} API example @@ -628,7 +628,7 @@ POST _ml/data_frame/_evaluate When you have trained a satisfactory model, you can [deploy it](#dfa-regression-deploy) to make predictions about new data. -If you don’t want to keep the {{dfanalytics-job}}, you can delete it. For example, use {{kib}} or the [delete {{dfanalytics-job}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-dfanalytics.html). When you delete {{dfanalytics-jobs}} in {{kib}}, you have the option to also remove the destination indices and {{data-sources}}. +If you don’t want to keep the {{dfanalytics-job}}, you can delete it. For example, use {{kib}} or the [delete {{dfanalytics-job}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-delete-data-frame-analytics). When you delete {{dfanalytics-jobs}} in {{kib}}, you have the option to also remove the destination indices and {{data-sources}}. ## Further reading [dfa-regression-reading] diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfanalytics-apis.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfanalytics-apis.md index 5d3b55b176..0db93dbc12 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfanalytics-apis.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfanalytics-apis.md @@ -20,15 +20,15 @@ The evaluation API endpoint has the following base: /_ml/data_frame/_evaluate ``` -* [Create {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html) -* [Delete {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-dfanalytics.html) -* [Evaluate {{dfanalytics}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/evaluate-dfanalytics.html) -* [Explain {{dfanalytics}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/explain-dfanalytics.html) -* [Get {{dfanalytics-jobs}} info](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-dfanalytics.html) -* [Get {{dfanalytics-jobs}} statistics](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-dfanalytics-stats.html) -* [Preview {{dfanalytics}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-dfanalytics.html) -* [Start {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-dfanalytics.html) -* [Stop {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/stop-dfanalytics.html) -* [Update {{dfanalytics-jobs}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/update-dfanalytics.html) +* [Create {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics) +* [Delete {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-delete-data-frame-analytics) +* [Evaluate {{dfanalytics}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-evaluate-data-frame) +* [Explain {{dfanalytics}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-explain-data-frame-analytics) +* [Get {{dfanalytics-jobs}} info](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-data-frame-analytics) +* [Get {{dfanalytics-jobs}} statistics](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-data-frame-analytics-stats) +* [Preview {{dfanalytics}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-preview-data-frame-analytics) +* [Start {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-data-frame-analytics) +* [Stop {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-stop-data-frame-analytics) +* [Update {{dfanalytics-jobs}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-update-data-frame-analytics) For information about the APIs related to trained models, refer to [*API quick reference*](../nlp/ml-nlp-apis.md). diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-feature-importance.md b/explore-analyze/machine-learning/data-frame-analytics/ml-feature-importance.md index e4fe924316..532367faa8 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-feature-importance.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-feature-importance.md @@ -12,7 +12,7 @@ mapped_pages: The purpose of {{feat-imp}} is to help you determine whether the predictions are sensible. Is the relationship between the dependent variable and the important features supported by your domain knowledge? The lessons you learn about the importance of specific features might also affect your decision to include them in future iterations of your trained model. -You can see the average magnitude of the {{feat-imp}} values for each field across all the training data in {{kib}} or by using the [get trained model API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html). For example, {{kib}} shows the total feature importance for each field in {{regression}} or binary {{classanalysis}} results as follows: +You can see the average magnitude of the {{feat-imp}} values for each field across all the training data in {{kib}} or by using the [get trained model API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models). For example, {{kib}} shows the total feature importance for each field in {{regression}} or binary {{classanalysis}} results as follows: :::{image} ../../../images/machine-learning-flights-regression-total-importance.jpg :alt: Total {{feat-imp}} values for a {{regression}} {dfanalytics-job} in {kib} diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-feature-processors.md b/explore-analyze/machine-learning/data-frame-analytics/ml-feature-processors.md index 6a3c7d14f9..30b9ea4839 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-feature-processors.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-feature-processors.md @@ -13,7 +13,7 @@ mapped_pages: The feature processors that you defined are the part of the analytics process, when data comes through the aggregation or pipeline, the processors run against the new data. The resulting features are ephemeral; they are not stored in the index. This provides a mechanism to create features that can be used at search and ingest time and don’t take up space in the index. -Refer to the `feature_processors` property of the [Create {{dfanalytics-job}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html) to learn more. +Refer to the `feature_processors` property of the [Create {{dfanalytics-job}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-data-frame-analytics) to learn more. Available feature processors: diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md b/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md index 647ad9f73f..99b248c66f 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md @@ -12,7 +12,7 @@ When you use a {{dfanalytics-job}} to perform {{classification}} or {{reganalysi In {{kib}}, you can view and manage your trained models in **{{stack-manage-app}}** > **Alerts and Insights** > **{{ml-app}}** and **{{ml-app}}** > **Model Management**. -Alternatively, you can use APIs like [get trained models](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html) and [delete trained models](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-trained-models.html). +Alternatively, you can use APIs like [get trained models](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) and [delete trained models](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-delete-trained-model). ## Deploying trained models [deploy-dfa-trained-models] @@ -49,13 +49,13 @@ The model is deployed and ready to use through the {{infer}} pipeline. ### Models trained by other methods [_models_trained_by_other_methods] -You can also supply trained models that are not created by {{dfanalytics-job}} but adhere to the appropriate [JSON schema](https://github.com/elastic/ml-json-schemas). Likewise, you can use third-party models to perform natural language processing (NLP) tasks. If you want to use these trained models in the {{stack}}, you must store them in {{es}} documents by using the [create trained models API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-trained-models.html). For more information about NLP models, refer to [*Deploy trained models*](../nlp/ml-nlp-deploy-models.md). +You can also supply trained models that are not created by {{dfanalytics-job}} but adhere to the appropriate [JSON schema](https://github.com/elastic/ml-json-schemas). Likewise, you can use third-party models to perform natural language processing (NLP) tasks. If you want to use these trained models in the {{stack}}, you must store them in {{es}} documents by using the [create trained models API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model). For more information about NLP models, refer to [*Deploy trained models*](../nlp/ml-nlp-deploy-models.md). ## Exporting and importing models [export-import] Models trained in Elasticsearch are portable and can be transferred between clusters. This is particularly useful when models are trained in isolation from the cluster where they are used for inference. The following instructions show how to use [`curl`](https://curl.se/) and [`jq`](https://stedolan.github.io/jq/) to export a model as JSON and import it to another cluster. -1. Given a model *name*, find the model *ID*. You can use `curl` to call the [get trained model API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html) to list all models with their IDs. +1. Given a model *name*, find the model *ID*. You can use `curl` to call the [get trained model API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) to list all models with their IDs. ```bash curl -s -u username:password \ @@ -80,7 +80,7 @@ Models trained in Elasticsearch are portable and can be transferred between clus In this example, you are exporting the model with ID `flights1-1607953694065`. -2. Using `curl` from the command line, again use the [get trained models API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html) to export the entire model definition and save it to a JSON file. +2. Using `curl` from the command line, again use the [get trained models API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) to export the entire model definition and save it to a JSON file. ```bash curl -u username:password \ @@ -95,7 +95,7 @@ A few observations: * Note the query parameters that are used during export. These parameters are necessary to export the model in a way that it can later be imported again and used for inference. * You must unnest the JSON object by one level to extract just the model definition. You must also remove the existing model ID in order to not have ID collisions when you import again. You can do these steps using `jq` inline or alternatively it can be done to the resulting JSON file after downloading using `jq` or other tools. -3. Import the saved model using `curl` to upload the JSON file to the [created trained model API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-trained-models.html). When you specify the URL, you can also set the model ID to something new using the last path part of the URL. +3. Import the saved model using `curl` to upload the JSON file to the [created trained model API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model). When you specify the URL, you can also set the model ID to something new using the last path part of the URL. ```bash curl -u username:password \ @@ -106,7 +106,7 @@ A few observations: ::::{note} -* Models exported from the [get trained models API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html) are limited in size by the [http.max_content_length](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html) global configuration value in {{es}}. The default value is `100mb` and may need to be increased depending on the size of model being exported. +* Models exported from the [get trained models API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) are limited in size by the [http.max_content_length](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html) global configuration value in {{es}}. The default value is `100mb` and may need to be increased depending on the size of model being exported. * Connection timeouts can occur, for example, when model sizes are very large or your cluster is under load. If needed, you can increase [timeout configurations](https://ec.haxx.se/usingcurl/usingcurl-timeouts) for `curl` (for example, `curl --max-time 600`) or your client of choice. :::: diff --git a/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md b/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md index 912558fc14..97e0ac8807 100644 --- a/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md +++ b/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md @@ -104,7 +104,7 @@ Here, you’ll be able to: Inference processors added to your index-specific ML {{infer}} pipelines are normal Elasticsearch pipelines. Once created, each processor will have options to **View in Stack Management** and **Delete Pipeline**. Deleting an {{infer}} processor from within the **Content** UI deletes the pipeline and also removes its reference from your index-specific ML {{infer}} pipeline. -These pipelines can also be viewed, edited, and deleted in Kibana via **Stack Management → Ingest Pipelines**, just like all other Elasticsearch ingest pipelines. You may also use the [Ingest pipeline APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-apis.html). If you delete any of these pipelines outside of the **Content** UI in Kibana, make sure to edit the ML {{infer}} pipelines that reference them. +These pipelines can also be viewed, edited, and deleted in Kibana via **Stack Management → Ingest Pipelines**, just like all other Elasticsearch ingest pipelines. You may also use the [Ingest pipeline APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ingest). If you delete any of these pipelines outside of the **Content** UI in Kibana, make sure to edit the ML {{infer}} pipelines that reference them. ## Test your ML {{infer}} pipeline [ingest-pipeline-search-inference-test-inference-pipeline] diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-apis.md b/explore-analyze/machine-learning/nlp/ml-nlp-apis.md index 992e807f37..1a3eb19b2c 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-apis.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-apis.md @@ -14,16 +14,16 @@ All the trained models endpoints have the following base: /_ml/trained_models/ ``` -* [Create trained model aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-trained-models-aliases.html) -* [Create trained model definition part](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-trained-model-definition-part.html) -* [Create trained models](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-trained-models.html) -* [Delete trained models](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-trained-models.html) -* [Get trained models](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html) -* [Get trained models statistics](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models-stats.html) -* [Infer trained model](https://www.elastic.co/guide/en/elasticsearch/reference/current/infer-trained-model.html) -* [Start trained model deployment](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trained-model-deployment.html) -* [Stop trained model deployment](https://www.elastic.co/guide/en/elasticsearch/reference/current/stop-trained-model-deployment.html) -* [Update trained model aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-trained-models-aliases.html) +* [Create trained model aliases](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model-alias) +* [Create trained model definition part](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model-definition-part) +* [Create trained models](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model) +* [Delete trained models](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-delete-trained-model) +* [Get trained models](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) +* [Get trained models statistics](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models-stats) +* [Infer trained model](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-infer-trained-model) +* [Start trained model deployment](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-trained-model-deployment) +* [Stop trained model deployment](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-stop-trained-model-deployment) +* [Update trained model aliases](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model-alias) You can also integrate NLP models from different providers such as Cohere, HuggingFace, or OpenAI and use them as a service through the {{infer}} API. @@ -33,7 +33,7 @@ The {{infer}} APIs have the following base: /_inference/ ``` -* [Create inference endpoint](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html) -* [Delete inference endpoint](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-inference-api.html) -* [Get inference endpoint](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-inference-api.html) -* [Perform inference](https://www.elastic.co/guide/en/elasticsearch/reference/current/post-inference-api.html) +* [Create inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) +* [Delete inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-delete) +* [Get inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-get) +* [Perform inference](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-inference) diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-auto-scale.md b/explore-analyze/machine-learning/nlp/ml-nlp-auto-scale.md index 2367e39215..ccb5d5ce21 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-auto-scale.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-auto-scale.md @@ -28,9 +28,9 @@ When adaptive allocations are enabled, the number of allocations of the model is You can enable adaptive allocations by using: * the create inference endpoint API for [ELSER](../../../solutions/search/inference-api/elser-inference-integration.md), [E5 and models uploaded through Eland](../../../solutions/search/inference-api/elasticsearch-inference-integration.md) that are used as {{infer}} services. -* the [start trained model deployment](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trained-model-deployment.html) or [update trained model deployment](https://www.elastic.co/guide/en/elasticsearch/reference/current/update-trained-model-deployment.html) APIs for trained models that are deployed on {{ml}} nodes. +* the [start trained model deployment](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-trained-model-deployment) or [update trained model deployment](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-update-trained-model-deployment) APIs for trained models that are deployed on {{ml}} nodes. -If the new allocations fit on the current {{ml}} nodes, they are immediately started. If more resource capacity is needed for creating new model allocations, then your {{ml}} node will be scaled up if {{ml}} autoscaling is enabled to provide enough resources for the new allocation. The number of model allocations can be scaled down to 0. They cannot be scaled up to more than 32 allocations, unless you explicitly set the maximum number of allocations to more. Adaptive allocations must be set up independently for each deployment and [{{infer}} endpoint](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html). +If the new allocations fit on the current {{ml}} nodes, they are immediately started. If more resource capacity is needed for creating new model allocations, then your {{ml}} node will be scaled up if {{ml}} autoscaling is enabled to provide enough resources for the new allocation. The number of model allocations can be scaled down to 0. They cannot be scaled up to more than 32 allocations, unless you explicitly set the maximum number of allocations to more. Adaptive allocations must be set up independently for each deployment and [{{infer}} endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put). ### Optimizing for typical use cases [optimize-use-case] @@ -60,7 +60,7 @@ The used resources for trained model deployments depend on three factors: * the use case you optimize the model deployment for (ingest or search) * whether model autoscaling is enabled with adaptive allocations/resources to have dynamic resources, or disabled for static resources -If you use {{es}} on-premises, vCPUs level ranges are derived from the `total_ml_processors` and `max_single_ml_node_processors` values. Use the [get {{ml}} info API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-ml-info.html) to check these values. The following tables show you the number of allocations, threads, and vCPUs available in Cloud when adaptive resources are enabled or disabled. +If you use {{es}} on-premises, vCPUs level ranges are derived from the `total_ml_processors` and `max_single_ml_node_processors` values. Use the [get {{ml}} info API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-info) to check these values. The following tables show you the number of allocations, threads, and vCPUs available in Cloud when adaptive resources are enabled or disabled. ::::{note} On Serverless, adaptive allocations are automatically enabled for all project types. However, the "Adaptive resources" control is not displayed in {{kib}} for Observability and Security projects. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model.md b/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model.md index 32ccb8a1b2..6fc96f9393 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model.md @@ -8,7 +8,7 @@ mapped_pages: # Deploy the model in your cluster [ml-nlp-deploy-model] -After you import the model and vocabulary, you can use {{kib}} to view and manage their deployment across your cluster under **{{ml-app}}** > **Model Management**. Alternatively, you can use the [start trained model deployment API](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trained-model-deployment.html). +After you import the model and vocabulary, you can use {{kib}} to view and manage their deployment across your cluster under **{{ml-app}}** > **Model Management**. Alternatively, you can use the [start trained model deployment API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-trained-model-deployment). You can deploy a model multiple times by assigning a unique deployment ID when starting the deployment. @@ -35,6 +35,6 @@ For the resource levels when adaptive resources are enabled, refer to <[*Trained ## Request queues and search priority [infer-request-queues] -Each allocation of a model deployment has a dedicated queue to buffer {{infer}} requests. The size of this queue is determined by the `queue_capacity` parameter in the [start trained model deployment API](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trained-model-deployment.html). When the queue reaches its maximum capacity, new requests are declined until some of the queued requests are processed, creating available capacity once again. When multiple ingest pipelines reference the same deployment, the queue can fill up, resulting in rejected requests. Consider using dedicated deployments to prevent this situation. +Each allocation of a model deployment has a dedicated queue to buffer {{infer}} requests. The size of this queue is determined by the `queue_capacity` parameter in the [start trained model deployment API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-trained-model-deployment). When the queue reaches its maximum capacity, new requests are declined until some of the queued requests are processed, creating available capacity once again. When multiple ingest pipelines reference the same deployment, the queue can fill up, resulting in rejected requests. Consider using dedicated deployments to prevent this situation. {{infer-cap}} requests originating from search, such as the [`text_expansion` query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-text-expansion-query.html), have a higher priority compared to non-search requests. The {{infer}} ingest processor generates normal priority requests. If both a search query and an ingest processor use the same deployment, the search requests with higher priority skip ahead in the queue for processing before the lower priority ingest requests. This prioritization accelerates search responses while potentially slowing down ingest where response time is less critical. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-e5.md b/explore-analyze/machine-learning/nlp/ml-nlp-e5.md index e61bc5f4cd..2d99273c86 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-e5.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-e5.md @@ -25,7 +25,7 @@ Enabling trained model autoscaling for your E5 deployment is recommended. Refer ## Download and deploy E5 [download-deploy-e5] -The easiest and recommended way to download and deploy E5 is to use the [{{infer}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-apis.html). +The easiest and recommended way to download and deploy E5 is to use the [{{infer}} API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-inference). 1. In {{kib}}, navigate to the **Dev Console**. 2. Create an {{infer}} endpoint with the `elasticsearch` service by running the following API request: @@ -127,7 +127,7 @@ PUT _ml/trained_models/.multilingual-e5-small The API call automatically initiates the model download if the model is not downloaded yet. -3. Deploy the model by using the [start trained model deployment API](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trained-model-deployment.html) with a delpoyment ID: +3. Deploy the model by using the [start trained model deployment API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-trained-model-deployment) with a delpoyment ID: ```console POST _ml/trained_models/.multilingual-e5-small/deployment/_start?deployment_id=for_search diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-elser.md b/explore-analyze/machine-learning/nlp/ml-nlp-elser.md index 2c1b73a75a..4f1e061004 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-elser.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-elser.md @@ -51,7 +51,7 @@ Additionally, the `elasticearch-labs` GitHub repository contains an interactive ## Download and deploy ELSER [download-deploy-elser] -The easiest and recommended way to download and deploy ELSER is to use the [{{infer}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-apis.html). +The easiest and recommended way to download and deploy ELSER is to use the [{{infer}} API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-inference). 1. In {{kib}}, navigate to the **Dev Console**. 2. Create an {{infer}} endpoint with the ELSER service by running the following API request: @@ -157,7 +157,7 @@ PUT _ml/trained_models/.elser_model_2 The API call automatically initiates the model download if the model is not downloaded yet. -3. Deploy the model by using the [start trained model deployment API](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trained-model-deployment.html) with a delpoyment ID: +3. Deploy the model by using the [start trained model deployment API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-trained-model-deployment) with a delpoyment ID: ```console POST _ml/trained_models/.elser_model_2/deployment/_start?deployment_id=for_search diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-inference.md b/explore-analyze/machine-learning/nlp/ml-nlp-inference.md index 724874a674..c94ffba544 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-inference.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-inference.md @@ -191,7 +191,7 @@ To learn more about ingest pipelines and all of the other processors that you ca If you encounter problems while using your trained model in an ingest pipeline, check the following possible causes: -1. The trained model is not deployed in your cluster. You can view its status in **{{ml-app}}** > **Model Management** or use the [get trained models statistics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models-stats.html). Unless you are using the built-in `lang_ident_model_1` model, you must ensure your model is successfully deployed. Refer to [Deploy the model in your cluster](ml-nlp-deploy-model.md). +1. The trained model is not deployed in your cluster. You can view its status in **{{ml-app}}** > **Model Management** or use the [get trained models statistics API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models-stats). Unless you are using the built-in `lang_ident_model_1` model, you must ensure your model is successfully deployed. Refer to [Deploy the model in your cluster](ml-nlp-deploy-model.md). 2. The default input field name expected by your trained model is not present in your source document. Use the **Field Map** option in your {{infer}} processor to set the appropriate field name. 3. There are too many requests. If you are using bulk ingest, reduce the number of documents in the bulk request. If you are reindexing, use the `size` parameter to decrease the number of documents processed in each batch. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md b/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md index 8b7a4e4a05..36625fd7ba 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md @@ -59,7 +59,7 @@ Deployed models can be evaluated in {{kib}} under **{{ml-app}}** > **Trained Mod ::: ::::{dropdown} **Test the model by using the _infer API** -You can also evaluate your models by using the [_infer API](https://www.elastic.co/guide/en/elasticsearch/reference/current/infer-trained-model.html). In the following request, `text_field` is the field name where the model expects to find the input, as defined in the model configuration. By default, if the model was uploaded via Eland, the input field is `text_field`. +You can also evaluate your models by using the [_infer API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-infer-trained-model). In the following request, `text_field` is the field name where the model expects to find the input, as defined in the model configuration. By default, if the model was uploaded via Eland, the input field is `text_field`. ```js POST _ml/trained_models/elastic__distilbert-base-uncased-finetuned-conll03-english/_infer diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-overview.md b/explore-analyze/machine-learning/nlp/ml-nlp-overview.md index 8e1d4803bf..6660b41ef9 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-overview.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-overview.md @@ -16,7 +16,7 @@ Elastic offers a wide range of possibilities to leverage natural language proces You can **integrate NLP models from different providers** such as Cohere, HuggingFace, or OpenAI and use them as a service through the [semantic_text](../../../solutions/search/semantic-search/semantic-search-semantic-text.md) workflow. You can also use [ELSER](ml-nlp-elser.md) (the retrieval model trained by Elastic) and [E5](ml-nlp-e5.md) in the same way. -The [{{infer}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-apis.html) enables you to use the same services with a more complex workflow, for greater control over your configurations settings. This [tutorial](../../../solutions/search/inference-api.md) walks you through the process of using the various services with the {{infer}} API. +The [{{infer}} API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-inference) enables you to use the same services with a more complex workflow, for greater control over your configurations settings. This [tutorial](../../../solutions/search/inference-api.md) walks you through the process of using the various services with the {{infer}} API. You can **upload and manage NLP models** using the Eland client and the [{{stack}}](ml-nlp-deploy-models.md). Find the [list of recommended and compatible models here](ml-nlp-model-ref.md). Refer to [*Examples*](ml-nlp-examples.md) to learn more about how to use {{ml}} models deployed in your cluster. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-rerank.md b/explore-analyze/machine-learning/nlp/ml-nlp-rerank.md index 62646558b0..85897f7b65 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-rerank.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-rerank.md @@ -81,7 +81,7 @@ You might see a 502 bad gateway error in the response when using the {{kib}} Con :::: -After creating the Elastic Rerank {{infer}} endpoint, it’s ready to use with a [`text_similarity_reranker`](https://www.elastic.co/guide/en/elasticsearch/reference/current/retriever.html#text-similarity-reranker-retriever-example-elastic-rerank) retriever. +After creating the Elastic Rerank {{infer}} endpoint, it’s ready to use with a [`text_similarity_reranker`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#operation-search-body-application-json-retriever) retriever. ## Deploy in an air-gapped environment [ml-nlp-rerank-deploy-verify] diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-test-inference.md b/explore-analyze/machine-learning/nlp/ml-nlp-test-inference.md index 15721b1a43..8976685092 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-test-inference.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-test-inference.md @@ -17,7 +17,7 @@ The simplest method to test your model against new data is to use the **Test mod :class: screenshot ::: -Alternatively, you can use the [infer trained model API](https://www.elastic.co/guide/en/elasticsearch/reference/current/infer-trained-model.html). For example, to try a named entity recognition task, provide some sample text: +Alternatively, you can use the [infer trained model API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-infer-trained-model). For example, to try a named entity recognition task, provide some sample text: ```console POST /_ml/trained_models/elastic__distilbert-base-cased-finetuned-conll03-english/_infer diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md b/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md index 76f32b77b7..bed59cf75c 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md @@ -63,7 +63,7 @@ Deployed models can be evaluated in {{kib}} under **{{ml-app}}** > **Trained Mod ::: ::::{dropdown} **Test the model by using the _infer API** -You can also evaluate your models by using the [_infer API](https://www.elastic.co/guide/en/elasticsearch/reference/current/infer-trained-model.html). In the following request, `text_field` is the field name where the model expects to find the input, as defined in the model configuration. By default, if the model was uploaded via Eland, the input field is `text_field`. +You can also evaluate your models by using the [_infer API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-infer-trained-model). In the following request, `text_field` is the field name where the model expects to find the input, as defined in the model configuration. By default, if the model was uploaded via Eland, the input field is `text_field`. ```js POST /_ml/trained_models/sentence-transformers__msmarco-minilm-l-12-v3/_infer @@ -152,7 +152,7 @@ PUT _ingest/pipeline/text-embeddings The passages are in a field named `text`. The `field_map` maps the text to the field `text_field` that the model expects. The `on_failure` handler is set to index failures into a different index. -Before ingesting the data through the pipeline, create the mappings of the destination index, in particular for the field `text_embedding.predicted_value` where the ingest processor stores the embeddings. The `dense_vector` field must be configured with the same number of dimensions (`dims`) as the text embedding produced by the model. That value can be found in the `embedding_size` option in the model configuration either under the Trained Models page in {{kib}} or in the response body of the [Get trained models API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-trained-models.html) call. The msmarco-MiniLM-L-12-v3 model has embedding_size of 384, so `dims` is set to 384. +Before ingesting the data through the pipeline, create the mappings of the destination index, in particular for the field `text_embedding.predicted_value` where the ingest processor stores the embeddings. The `dense_vector` field must be configured with the same number of dimensions (`dims`) as the text embedding produced by the model. That value can be found in the `embedding_size` option in the model configuration either under the Trained Models page in {{kib}} or in the response body of the [Get trained models API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) call. The msmarco-MiniLM-L-12-v3 model has embedding_size of 384, so `dims` is set to 384. ```js PUT collection-with-embeddings diff --git a/explore-analyze/machine-learning/nlp/nlp-end-to-end-tutorial.md b/explore-analyze/machine-learning/nlp/nlp-end-to-end-tutorial.md index 16741ce5b4..873d904b59 100644 --- a/explore-analyze/machine-learning/nlp/nlp-end-to-end-tutorial.md +++ b/explore-analyze/machine-learning/nlp/nlp-end-to-end-tutorial.md @@ -107,7 +107,7 @@ We want to run our documents through an inference processor that uses the traine * Choose to **Use the API** and give your index a name. It will automatically be prefixed with `search-`. For this demo, we will name the index `search-photo-comments`. * After clicking **Create Index**, you will be redirected to the overview page for your new index. -To configure the ML inference pipeline, we need the index to have an existing field mapping so we can choose which field to analyze. This can be done via the [index mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html) in the Kibana Dev Tools or simply through a cURL command: +To configure the ML inference pipeline, we need the index to have an existing field mapping so we can choose which field to analyze. This can be done via the [index mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping) in the Kibana Dev Tools or simply through a cURL command: ```js PUT search-photo-comments/_mapping diff --git a/explore-analyze/query-filter/aggregations.md b/explore-analyze/query-filter/aggregations.md index 6d56663526..c2905f81ef 100644 --- a/explore-analyze/query-filter/aggregations.md +++ b/explore-analyze/query-filter/aggregations.md @@ -23,7 +23,7 @@ An aggregation summarizes your data as metrics, statistics, or other analytics. ## Run an aggregation [run-an-agg] -You can run aggregations as part of a [search](../../solutions/search/querying-for-search.md) by specifying the [search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html)'s `aggs` parameter. The following search runs a [terms aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html) on `my-field`: +You can run aggregations as part of a [search](../../solutions/search/querying-for-search.md) by specifying the [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search)'s `aggs` parameter. The following search runs a [terms aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html) on `my-field`: ```console GET /my-index-000001/_search @@ -113,7 +113,7 @@ GET /my-index-000001/_search } ``` -## Run multiple aggregations [run-multiple-aggs] +## Run multiple aggregations [run-multiple-aggs] You can specify multiple aggregations in the same request: @@ -185,7 +185,7 @@ The response nests sub-aggregation results under their parent aggregation: 1. Results for the parent aggregation, `my-agg-name`. 2. Results for `my-agg-name`'s sub-aggregation, `my-sub-agg-name`. -## Add custom metadata [add-metadata-to-an-agg] +## Add custom metadata [add-metadata-to-an-agg] Use the `meta` object to associate custom metadata with an aggregation: @@ -260,7 +260,7 @@ Some aggregations return a different aggregation type from the type in the reque 1. The aggregation type, `histogram`, followed by a `#` separator and the aggregation’s name, `my-agg-name`. -## Use scripts in an aggregation [use-scripts-in-an-agg] +## Use scripts in an aggregation [use-scripts-in-an-agg] When a field doesn’t exactly match the aggregation you need, you should aggregate on a [runtime field](../../manage-data/data-store/mapping/runtime-fields.md): diff --git a/explore-analyze/query-filter/languages.md b/explore-analyze/query-filter/languages.md index f1be0ffda6..87cb77ab13 100644 --- a/explore-analyze/query-filter/languages.md +++ b/explore-analyze/query-filter/languages.md @@ -11,10 +11,10 @@ applies: | Name | Description | Use cases | API endpoint | | --- | --- | --- | --- | -| [Query DSL](languages/querydsl.md) | The primary query language for {{es}}. A powerful and flexible JSON-style language that enables complex queries. | Full-text search, semantic search, keyword search, filtering, aggregations, and more. | [`_search`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) | +| [Query DSL](languages/querydsl.md) | The primary query language for {{es}}. A powerful and flexible JSON-style language that enables complex queries. | Full-text search, semantic search, keyword search, filtering, aggregations, and more. | [`_search`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) | | [{{esql}}](languages/esql.md) | Introduced in **8.11**, the Elasticsearch Query Language ({{esql}}) is a piped query language language for filtering, transforming, and analyzing data. | Initially tailored towards working with time series data like logs and metrics.Robust integration with {{kib}} for querying, visualizing, and analyzing data.Does not yet support full-text search. | [`_query`](languages/esql-rest.md) | -| [EQL](languages/eql.md) | Event Query Language (EQL) is a query language for event-based time series data. Data must contain the `@timestamp` field to use EQL. | Designed for the threat hunting security use case. | [`_eql`](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-apis.html) | -| [Elasticsearch SQL](languages/sql.md) | Allows native, real-time SQL-like querying against {{es}} data. JDBC and ODBC drivers are available for integration with business intelligence (BI) tools. | Enables users familiar with SQL to query {{es}} data using familiar syntax for BI and reporting. | [`_sql`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-apis.html) | +| [EQL](languages/eql.md) | Event Query Language (EQL) is a query language for event-based time series data. Data must contain the `@timestamp` field to use EQL. | Designed for the threat hunting security use case. | [`_eql`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-eql) | +| [Elasticsearch SQL](languages/sql.md) | Allows native, real-time SQL-like querying against {{es}} data. JDBC and ODBC drivers are available for integration with business intelligence (BI) tools. | Enables users familiar with SQL to query {{es}} data using familiar syntax for BI and reporting. | [`_sql`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-sql) | | [Kibana Query Language (KQL)](languages/kql.md) | {{kib}} Query Language (KQL) is a text-based query language for filtering data when you access it through the {{kib}} UI. | Use KQL to filter documents where a value for a field exists, matches a given value, or is within a given range. | N/A | > {{esql}} does not yet support all the features of Query DSL. Look forward to new {{esql}} features and functionalities in each release. diff --git a/explore-analyze/query-filter/languages/eql.md b/explore-analyze/query-filter/languages/eql.md index 85b845491a..aac6dd3d94 100644 --- a/explore-analyze/query-filter/languages/eql.md +++ b/explore-analyze/query-filter/languages/eql.md @@ -38,7 +38,7 @@ While no schema is required to use EQL, we recommend using the [ECS](https://www ## Run an EQL search [run-an-eql-search] -Use the [EQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-search-api.html) to run a [basic EQL query](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html#eql-basic-syntax). +Use the [EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) to run a [basic EQL query](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html#eql-basic-syntax). ```console GET /my-data-stream/_eql/search @@ -1134,7 +1134,7 @@ The async search continues to run in the background without blocking other reque } ``` -To check the progress of an async search, use the [get async EQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-async-eql-search-api.html) with the search ID. Specify how long you’d like for complete results in the `wait_for_completion_timeout` parameter. +To check the progress of an async search, use the [get async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-get) with the search ID. Specify how long you’d like for complete results in the `wait_for_completion_timeout` parameter. ```console GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s @@ -1153,7 +1153,7 @@ If the response’s `is_running` value is `false`, the async search has finished } ``` -Another more lightweight way to check the progress of an async search is to use the [get async EQL status API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-async-eql-status-api.html) with the search ID. +Another more lightweight way to check the progress of an async search is to use the [get async EQL status API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-get-status) with the search ID. ```console GET /_eql/search/status/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE= @@ -1185,13 +1185,13 @@ GET /my-data-stream/_eql/search } ``` -You can use the [get async EQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-async-eql-search-api.html)'s `keep_alive` parameter to later change the retention period. The new retention period starts after the get request runs. +You can use the [get async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-get)'s `keep_alive` parameter to later change the retention period. The new retention period starts after the get request runs. ```console GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d ``` -Use the [delete async EQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-async-eql-search-api.html) to manually delete an async EQL search before the `keep_alive` period ends. If the search is still ongoing, {{es}} cancels the search request. +Use the [delete async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-delete) to manually delete an async EQL search before the `keep_alive` period ends. If the search is still ongoing, {{es}} cancels the search request. ```console DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE= @@ -1226,7 +1226,7 @@ The response includes a search ID. `is_partial` and `is_running` are `false`, in } ``` -Use the [get async EQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-async-eql-search-api.html) to get the same results later: +Use the [get async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-get) to get the same results later: ```console GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY= @@ -1234,9 +1234,9 @@ GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUTo Saved synchronous searches are still subject to the `keep_alive` parameter’s retention period. When this period ends, the search and its results are deleted. -You can also check only the status of the saved synchronous search without results by using [get async EQL status API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-async-eql-status-api.html). +You can also check only the status of the saved synchronous search without results by using [get async EQL status API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-get-status). -You can also manually delete saved synchronous searches using the [delete async EQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-async-eql-search-api.html). +You can also manually delete saved synchronous searches using the [delete async EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-delete). ## Run an EQL search across clusters [run-eql-search-across-clusters] @@ -1248,7 +1248,7 @@ This functionality is in technical preview and may be changed or removed in a fu The EQL search API supports [cross-cluster search](../../../solutions/search/cross-cluster-search.md). However, the local and [remote clusters](../../../deploy-manage/remote-clusters.md) must use the same {{es}} version if they have versions prior to 7.17.7 (included) or prior to 8.5.1 (included). -The following [cluster update settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html) request adds two remote clusters: `cluster_one` and `cluster_two`. +The following [cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) request adds two remote clusters: `cluster_one` and `cluster_two`. ```console PUT /_cluster/settings diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index d03736a101..69b1842b0d 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -72,7 +72,7 @@ API key based cross-cluster search (CCS) enables more granular control over allo You will need to: -* Create an API key on the **remote cluster** using the [Create cross-cluster API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-cross-cluster-api-key.html) API or using the [Kibana API keys UI](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). +* Create an API key on the **remote cluster** using the [Create cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) API or using the [Kibana API keys UI](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). * Add the API key to the keystore on the **local cluster**, as part of the steps in [configuring the local cluster](../../../deploy-manage/remote-clusters/remote-clusters-api-key.md#remote-clusters-security-api-key-local-actions). All cross-cluster requests from the local cluster are bound by the API key’s privileges. Using {{esql}} with the API key based security model requires some additional permissions that may not be needed when using the traditional query DSL based search. The following example API call creates a role that can query remote indices using {{esql}} when using the API key based security model. The final privilege, `remote_cluster`, is required to allow remote enrich operations. @@ -109,9 +109,9 @@ POST /_security/role/remote1 1. The `cross_cluster_search` cluster privilege is required for the *local* cluster. 2. Typically, users will have permissions to read both local and remote indices. However, for cases where the role is intended to ONLY search the remote cluster, the `read` permission is still required for the local cluster. To provide read access to the local cluster, but disallow reading any indices in the local cluster, the `names` field may be an empty string. -3. The indices allowed read access to the remote cluster. The configured [cross-cluster API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-cross-cluster-api-key.html) must also allow this index to be read. +3. The indices allowed read access to the remote cluster. The configured [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) must also allow this index to be read. 4. The `read_cross_cluster` privilege is always required when using {{esql}} across clusters with the API key based security model. -5. The remote clusters to which these privileges apply. This remote cluster must be configured with a [cross-cluster API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-cross-cluster-api-key.html) and connected to the remote cluster before the remote index can be queried. Verify connection using the [Remote cluster info](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-remote-info.html) API. +5. The remote clusters to which these privileges apply. This remote cluster must be configured with a [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) and connected to the remote cluster before the remote index can be queried. Verify connection using the [Remote cluster info](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-remote-info) API. 6. Required to allow remote enrichment. Without this, the user cannot read from the `.enrich` indices on the remote cluster. The `remote_cluster` security privilege was introduced in version **8.15.0**. @@ -138,7 +138,7 @@ Cross cluster API keys created in versions prior to 8.15.0 will need to replaced Once the security model is configured, you can add remote clusters. -The following [cluster update settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html) API request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. +The following [cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) API request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. ```console PUT _cluster/settings diff --git a/explore-analyze/query-filter/languages/esql-getting-started.md b/explore-analyze/query-filter/languages/esql-getting-started.md index dcc7425faf..7e48623704 100644 --- a/explore-analyze/query-filter/languages/esql-getting-started.md +++ b/explore-analyze/query-filter/languages/esql-getting-started.md @@ -74,7 +74,7 @@ In {{kib}}, you can use Console or Discover to run {{esql}} queries: ::::::{tab-item} Console To get started with {{esql}} in Console, open the main menu and select **Dev Tools**. -The general structure of an [{{esql}} query API](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-query-api.html) request is: +The general structure of an [{{esql}} query API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql) request is: ```txt POST /_query?format=txt diff --git a/explore-analyze/query-filter/languages/esql-rest.md b/explore-analyze/query-filter/languages/esql-rest.md index e69427a817..f6d74b32ae 100644 --- a/explore-analyze/query-filter/languages/esql-rest.md +++ b/explore-analyze/query-filter/languages/esql-rest.md @@ -15,7 +15,7 @@ mapped_pages: ## Overview [esql-rest-overview] -The [{{esql}} query API](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-query-api.html) accepts an {{esql}} query string in the `query` parameter, runs it, and returns the results. For example: +The [{{esql}} query API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-esql) accepts an {{esql}} query string in the `query` parameter, runs it, and returns the results. For example: ```console POST /_query?format=txt @@ -248,11 +248,11 @@ POST /_query ### Running an async {{esql}} query [esql-rest-async-query] -The [{{esql}} async query API](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-async-query-api.html) lets you asynchronously execute a query request, monitor its progress, and retrieve results when they become available. +The [{{esql}} async query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-async-query) lets you asynchronously execute a query request, monitor its progress, and retrieve results when they become available. Executing an {{esql}} query is commonly quite fast, however queries across large data sets or frozen data can take some time. To avoid long waits, run an async {{esql}} query. -Queries initiated by the async query API may return results or not. The `wait_for_completion_timeout` property determines how long to wait for the results. If the results are not available by this time, a [query id](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-async-query-api.html#esql-async-query-api-response-body-query-id) is returned which can be later used to retrieve the results. For example: +Queries initiated by the async query API may return results or not. The `wait_for_completion_timeout` property determines how long to wait for the results. If the results are not available by this time, a [query id](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-async-query#esql-async-query-api-response-body-query-id) is returned which can be later used to retrieve the results. For example: ```console POST /_query/async @@ -282,7 +282,7 @@ The query continues to run in the background without blocking other requests. } ``` -To check the progress of an async query, use the [{{esql}} async query get API](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-async-query-get-api.html) with the query ID. Specify how long you’d like to wait for complete results in the `wait_for_completion_timeout` parameter. +To check the progress of an async query, use the [{{esql}} async query get API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-async-query-get) with the query ID. Specify how long you’d like to wait for complete results in the `wait_for_completion_timeout` parameter. ```console GET /_query/async/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=30s @@ -298,7 +298,7 @@ If the response’s `is_running` value is `false`, the query has finished and th } ``` -Use the [{{esql}} async query delete API](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-async-query-delete-api.html) to delete an async query before the `keep_alive` period ends. If the query is still running, {{es}} cancels it. +Use the [{{esql}} async query delete API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-esql-async-query-delete) to delete an async query before the `keep_alive` period ends. If the query is still running, {{es}} cancels it. ```console DELETE /_query/async/FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI= diff --git a/explore-analyze/query-filter/languages/esql-task-management.md b/explore-analyze/query-filter/languages/esql-task-management.md index ee3d46157d..aa0da370f3 100644 --- a/explore-analyze/query-filter/languages/esql-task-management.md +++ b/explore-analyze/query-filter/languages/esql-task-management.md @@ -12,7 +12,7 @@ mapped_pages: # Task management [esql-task-management] -You can list running {{esql}} queries with the [task management API](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html): +You can list running {{esql}} queries with the [task management API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks): $$$esql-task-management-get-all$$$ @@ -43,7 +43,7 @@ Which returns a list of statuses like this: 2. Time the query has been running. -You can use this to find long running queries and, if you need to, cancel them with the [task cancellation API](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html#task-cancellation): +You can use this to find long running queries and, if you need to, cancel them with the [task cancellation API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks#task-cancellation): $$$esql-task-management-cancelEsqlQueryRequestTests$$$ diff --git a/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md b/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md index 223465b120..6f299e53c0 100644 --- a/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md +++ b/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md @@ -37,13 +37,13 @@ To get started: ``` 2. Download [`normalized-T1117-AtomicRed-regsvr32.json`](https://raw.githubusercontent.com/elastic/elasticsearch/master/docs/src/yamlRestTest/resources/normalized-T1117-AtomicRed-regsvr32.json). -3. Use the [bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) to index the data to a matching stream: +3. Use the [bulk API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) to index the data to a matching stream: ```sh curl -H "Content-Type: application/json" -XPOST "localhost:9200/my-data-stream/_bulk?pretty&refresh" --data-binary "@normalized-T1117-AtomicRed-regsvr32.json" ``` -4. Use the [cat indices API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html) to verify the data was indexed: +4. Use the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices) to verify the data was indexed: ```console GET /_cat/indices/my-data-stream?v=true&h=health,status,index,docs.count diff --git a/explore-analyze/query-filter/languages/querydsl.md b/explore-analyze/query-filter/languages/querydsl.md index 0c269792ed..6e207eaeb6 100644 --- a/explore-analyze/query-filter/languages/querydsl.md +++ b/explore-analyze/query-filter/languages/querydsl.md @@ -58,7 +58,7 @@ The following aggregation types are available: * [Bucket](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html): Group documents into buckets based on field values, ranges, or other criteria. * [Pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html): Run aggregations on the results of other aggregations. -Run aggregations by specifying the [search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html)'s `aggs` parameter. Learn more in [Run an aggregation](/explore-analyze/query-filter/aggregations.md#run-an-agg). +Run aggregations by specifying the [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search)'s `aggs` parameter. Learn more in [Run an aggregation](/explore-analyze/query-filter/aggregations.md#run-an-agg). ## How does it work? [query-dsl] @@ -103,14 +103,14 @@ The execution of such queries can be prevented by setting the value of the `sear By default, Elasticsearch sorts matching search results by **relevance score**, which measures how well each document matches a query. -The relevance score is a positive floating point number, returned in the `_score` metadata field of the [search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) API. The higher the `_score`, the more relevant the document. While each query type can calculate relevance scores differently, score calculation also depends on whether the query clause is run in a **query** or **filter** context. +The relevance score is a positive floating point number, returned in the `_score` metadata field of the [search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) API. The higher the `_score`, the more relevant the document. While each query type can calculate relevance scores differently, score calculation also depends on whether the query clause is run in a **query** or **filter** context. ### Query context [query-context] In the query context, a query clause answers the question *How well does this document match this query clause?* Besides deciding whether or not the document matches, the query clause also calculates a relevance score in the `_score` metadata field. -Query context is in effect whenever a query clause is passed to a `query` parameter, such as the `query` parameter in the [search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#request-body-search-query) API. +Query context is in effect whenever a query clause is passed to a `query` parameter, such as the `query` parameter in the [search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search#request-body-search-query) API. ### Filter context [filter-context] diff --git a/explore-analyze/query-filter/languages/sql-async.md b/explore-analyze/query-filter/languages/sql-async.md index 0679b7abc2..8d84cc0ef2 100644 --- a/explore-analyze/query-filter/languages/sql-async.md +++ b/explore-analyze/query-filter/languages/sql-async.md @@ -38,7 +38,7 @@ For CSV, TSV, and TXT responses, the API returns these values in the respective } ``` -To check the progress of an async search, use the search ID with the [get async SQL search status API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-async-sql-search-status-api.html). +To check the progress of an async search, use the search ID with the [get async SQL search status API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-get-async-status). ```console GET _sql/async/status/FnR0TDhyWUVmUmVtWXRWZER4MXZiNFEad2F5UDk2ZVdTVHV1S0xDUy00SklUdzozMTU= @@ -56,7 +56,7 @@ If `is_running` and `is_partial` are `false`, the async search has finished with } ``` -To get the results, use the search ID with the [get async SQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-async-sql-search-api.html). If the search is still running, specify how long you’d like to wait using `wait_for_completion_timeout`. You can also specify the response `format`. +To get the results, use the search ID with the [get async SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-get-async). If the search is still running, specify how long you’d like to wait using `wait_for_completion_timeout`. You can also specify the response `format`. ```console GET _sql/async/FnR0TDhyWUVmUmVtWXRWZER4MXZiNFEad2F5UDk2ZVdTVHV1S0xDUy00SklUdzozMTU=?wait_for_completion_timeout=2s&format=json @@ -83,7 +83,7 @@ You can use the get async SQL search API’s `keep_alive` parameter to later cha GET _sql/async/FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI=?keep_alive=5d&wait_for_completion_timeout=2s&format=json ``` -Use the [delete async SQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-async-sql-search-api.html) to delete an async search before the `keep_alive` period ends. If the search is still running, {{es}} cancels it. +Use the [delete async SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-delete-async) to delete an async search before the `keep_alive` period ends. If the search is still running, {{es}} cancels it. ```console DELETE _sql/async/delete/FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI= @@ -117,7 +117,7 @@ If `is_partial` and `is_running` are `false`, the search was synchronous and ret } ``` -You can get the same results later using the search ID with the [get async SQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-async-sql-search-api.html). +You can get the same results later using the search ID with the [get async SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-get-async). -Saved synchronous searches are still subject to the `keep_alive` retention period. When this period ends, {{es}} deletes the search results. You can also delete saved searches using the [delete async SQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-async-sql-search-api.html). +Saved synchronous searches are still subject to the `keep_alive` retention period. When this period ends, {{es}} deletes the search results. You can also delete saved searches using the [delete async SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-delete-async). diff --git a/explore-analyze/query-filter/languages/sql-functions-datetime.md b/explore-analyze/query-filter/languages/sql-functions-datetime.md index a2f8b59a5e..7f8590d224 100644 --- a/explore-analyze/query-filter/languages/sql-functions-datetime.md +++ b/explore-analyze/query-filter/languages/sql-functions-datetime.md @@ -594,7 +594,7 @@ SELECT DATE_PARSE('07/04/2020', 'dd/MM/yyyy') AS "date"; ``` ::::{note} -The resulting `date` will have the time zone specified by the user through the [`time_zone`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html#sql-search-api-time-zone)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters with no conversion applied. +The resulting `date` will have the time zone specified by the user through the [`time_zone`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters with no conversion applied. ```sql { @@ -699,7 +699,7 @@ SELECT DATETIME_PARSE('10:20:30 07/04/2020 Europe/Berlin', 'HH:mm:ss dd/MM/yyyy ``` ::::{note} -If timezone is not specified in the datetime string expression and the parsing pattern, the resulting `datetime` will have the time zone specified by the user through the [`time_zone`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html#sql-search-api-time-zone)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters with no conversion applied. +If timezone is not specified in the datetime string expression and the parsing pattern, the resulting `datetime` will have the time zone specified by the user through the [`time_zone`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters with no conversion applied. ```sql { @@ -756,7 +756,7 @@ SELECT TIME_PARSE('10:20:30-01:00', 'HH:mm:ssXXX') AS "time"; ``` ::::{note} -If timezone is not specified in the time string expression and the parsing pattern, the resulting `time` will have the offset of the time zone specified by the user through the [`time_zone`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html#sql-search-api-time-zone)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters at the Unix epoch date (`1970-01-01`) with no conversion applied. +If timezone is not specified in the time string expression and the parsing pattern, the resulting `time` will have the offset of the time zone specified by the user through the [`time_zone`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters at the Unix epoch date (`1970-01-01`) with no conversion applied. ```sql { diff --git a/explore-analyze/query-filter/languages/sql-getting-started.md b/explore-analyze/query-filter/languages/sql-getting-started.md index fd8cb89529..52d66bd63e 100644 --- a/explore-analyze/query-filter/languages/sql-getting-started.md +++ b/explore-analyze/query-filter/languages/sql-getting-started.md @@ -21,7 +21,7 @@ PUT /library/_bulk?refresh {"name": "Dune", "author": "Frank Herbert", "release_date": "1965-06-01", "page_count": 604} ``` -And now you can execute SQL using the [SQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html): +And now you can execute SQL using the [SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query): ```console POST /_sql?format=txt diff --git a/explore-analyze/query-filter/languages/sql-index-frozen.md b/explore-analyze/query-filter/languages/sql-index-frozen.md index 71fc969d98..4c722ed5dc 100644 --- a/explore-analyze/query-filter/languages/sql-index-frozen.md +++ b/explore-analyze/query-filter/languages/sql-index-frozen.md @@ -8,10 +8,10 @@ mapped_pages: # Frozen Indices [sql-index-frozen] -By default, Elasticsearch SQL doesn’t search [frozen indices](https://www.elastic.co/guide/en/elasticsearch/reference/current/unfreeze-index-api.html). To search frozen indices, use one of the following features: +By default, Elasticsearch SQL doesn’t search [frozen indices](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). To search frozen indices, use one of the following features: dedicated configuration parameter -: Set to `true` properties `index_include_frozen` in the [SQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html) or `index.include.frozen` in the drivers to include frozen indices. +: Set to `true` properties `index_include_frozen` in the [SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query) or `index.include.frozen` in the drivers to include frozen indices. dedicated keyword : Explicitly perform the inclusion through the dedicated `FROZEN` keyword in the `FROM` clause or `INCLUDE FROZEN` in the `SHOW` commands: diff --git a/explore-analyze/query-filter/languages/sql-odbc-setup.md b/explore-analyze/query-filter/languages/sql-odbc-setup.md index b0149fc854..36a855a2de 100644 --- a/explore-analyze/query-filter/languages/sql-odbc-setup.md +++ b/explore-analyze/query-filter/languages/sql-odbc-setup.md @@ -235,7 +235,7 @@ The connection configuration can further be tweaked by the following parameters. * Max page size (rows) - The maximum number of rows that Elasticsearch SQL server should send the driver for one page. This corresponds to the SQL search API’s [`fetch_size`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html#sql-search-api-fetch-size) parameter. A `0` value indicates a server default. + The maximum number of rows that Elasticsearch SQL server should send the driver for one page. This corresponds to the SQL search API’s [`fetch_size`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query) parameter. A `0` value indicates a server default. * Max page length (MB) @@ -287,7 +287,7 @@ The connection configuration can further be tweaked by the following parameters. * Multi value field lenient - This setting controls the behavior of the server in case a multi-value field is queried. In case this is set and the server encounters such a field, it will pick a value in the set - without any guarantees of what that will be, but typically the first in natural ascending order - and return it as the value for the column. If not set, the server will return an error. This corresponds to the SQL search API’s [`field_multi_value_leniency`](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html#sql-search-api-field-multi-value-leniency) parameter. + This setting controls the behavior of the server in case a multi-value field is queried. In case this is set and the server encounters such a field, it will pick a value in the set - without any guarantees of what that will be, but typically the first in natural ascending order - and return it as the value for the column. If not set, the server will return an error. This corresponds to the SQL search API’s [`field_multi_value_leniency`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query) parameter. * Include frozen indices diff --git a/explore-analyze/query-filter/languages/sql-pagination.md b/explore-analyze/query-filter/languages/sql-pagination.md index c976616502..168a6139a8 100644 --- a/explore-analyze/query-filter/languages/sql-pagination.md +++ b/explore-analyze/query-filter/languages/sql-pagination.md @@ -36,7 +36,7 @@ Note that the `columns` object is only part of the first page. You’ve reached the last page when there is no `cursor` returned in the results. Like Elasticsearch’s [scroll](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results), SQL may keep state in Elasticsearch to support the cursor. Unlike scroll, receiving the last page is enough to guarantee that the Elasticsearch state is cleared. -To clear the state earlier, use the [clear cursor API](https://www.elastic.co/guide/en/elasticsearch/reference/current/clear-sql-cursor-api.html): +To clear the state earlier, use the [clear cursor API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-clear-cursor): ```console POST /_sql/close diff --git a/explore-analyze/query-filter/languages/sql-rest-overview.md b/explore-analyze/query-filter/languages/sql-rest-overview.md index 8e028cb270..fdd34f7082 100644 --- a/explore-analyze/query-filter/languages/sql-rest-overview.md +++ b/explore-analyze/query-filter/languages/sql-rest-overview.md @@ -8,7 +8,7 @@ mapped_pages: # Overview [sql-rest-overview] -The [SQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html) accepts SQL in a JSON document, executes it, and returns the results. For example: +The [SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query) accepts SQL in a JSON document, executes it, and returns the results. For example: ```console POST /_sql?format=txt diff --git a/explore-analyze/query-filter/languages/sql-translate.md b/explore-analyze/query-filter/languages/sql-translate.md index c2ac26f107..77860281c6 100644 --- a/explore-analyze/query-filter/languages/sql-translate.md +++ b/explore-analyze/query-filter/languages/sql-translate.md @@ -52,7 +52,7 @@ Which returns: } ``` -Which is the request that SQL will run to provide the results. In this case, SQL will use the [scroll](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results) API. If the result contained an aggregation then SQL would use the normal [search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html). +Which is the request that SQL will run to provide the results. In this case, SQL will use the [scroll](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results) API. If the result contained an aggregation then SQL would use the normal [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). -The request body accepts the same [parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html#sql-search-api-request-body) as the [SQL search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-search-api.html), excluding `cursor`. +The request body accepts the same [parameters](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query) as the [SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query), excluding `cursor`. diff --git a/explore-analyze/report-and-share.md b/explore-analyze/report-and-share.md index 13fe3d1296..304e08fdcb 100644 --- a/explore-analyze/report-and-share.md +++ b/explore-analyze/report-and-share.md @@ -130,7 +130,7 @@ We recommend using CSV reports to export moderate amounts of data only. The feat To work around the limitations, use filters to create multiple smaller reports, or extract the data you need directly with the Elasticsearch APIs. -For more information on using Elasticsearch APIs directly, see [Scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html), [Point in time API](https://www.elastic.co/guide/en/elasticsearch/reference/current/point-in-time-api.html), [ES|QL](/explore-analyze/query-filter/languages/esql-rest.md) or [SQL](/explore-analyze/query-filter/languages/sql-rest-format.md#_csv) with CSV response data format. We recommend that you use an official Elastic language client: details for each programming language library that Elastic provides are in the [{{es}} Client documentation](https://www.elastic.co/guide/en/elasticsearch/client/index.html). +For more information on using Elasticsearch APIs directly, see [Scroll API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-scroll), [Point in time API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time), [ES|QL](/explore-analyze/query-filter/languages/esql-rest.md) or [SQL](/explore-analyze/query-filter/languages/sql-rest-format.md#_csv) with CSV response data format. We recommend that you use an official Elastic language client: details for each programming language library that Elastic provides are in the [{{es}} Client documentation](https://www.elastic.co/guide/en/elasticsearch/client/index.html). [Reporting parameters](https://www.elastic.co/guide/en/kibana/current/reporting-settings-kb.html) can be adjusted to overcome some of these limiting scenarios. Results are dependent on data size, availability, and latency factors and are not guaranteed. diff --git a/explore-analyze/report-and-share/reporting-troubleshooting-csv.md b/explore-analyze/report-and-share/reporting-troubleshooting-csv.md index 7758a748fa..ba8355836e 100644 --- a/explore-analyze/report-and-share/reporting-troubleshooting-csv.md +++ b/explore-analyze/report-and-share/reporting-troubleshooting-csv.md @@ -24,7 +24,7 @@ We recommend using CSV reports to export moderate amounts of data only. The feat To work around the limitations, use filters to create multiple smaller reports, or extract the data you need directly with the Elasticsearch APIs. -For more information on using Elasticsearch APIs directly, see [Scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html), [Point in time API](https://www.elastic.co/guide/en/elasticsearch/reference/current/point-in-time-api.html), [ES|QL](../query-filter/languages/esql-rest.md) or [SQL](../query-filter/languages/sql-rest-format.md#_csv) with CSV response data format. We recommend that you use an official Elastic language client: details for each programming language library that Elastic provides are in the [{{es}} Client documentation](https://www.elastic.co/guide/en/elasticsearch/client/index.html). +For more information on using Elasticsearch APIs directly, see [Scroll API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-scroll), [Point in time API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time), [ES|QL](../query-filter/languages/esql-rest.md) or [SQL](../query-filter/languages/sql-rest-format.md#_csv) with CSV response data format. We recommend that you use an official Elastic language client: details for each programming language library that Elastic provides are in the [{{es}} Client documentation](https://www.elastic.co/guide/en/elasticsearch/client/index.html). [Reporting parameters](https://www.elastic.co/guide/en/kibana/current/reporting-settings-kb.html) can be adjusted to overcome some of these limiting scenarios. Results are dependent on data size, availability, and latency factors and are not guaranteed. @@ -38,7 +38,7 @@ For advice about common problems, refer to [Troubleshooting](reporting-troublesh ## Configuring CSV export to use the scroll API [reporting-troubleshooting-csv-configure-scan-api] -The Kibana CSV export feature collects all of the data from Elasticsearch by using multiple requests to page over all of the documents. Internally, the feature uses the [Point in time API and `search_after` parameters in the queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/point-in-time-api.html) to do so. There are some limitations related to the point in time API: +The Kibana CSV export feature collects all of the data from Elasticsearch by using multiple requests to page over all of the documents. Internally, the feature uses the [Point in time API and `search_after` parameters in the queries](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time) to do so. There are some limitations related to the point in time API: 1. Permissions to read data aliases alone will not work: the permissions are needed on the underlying indices or data streams. 2. In cases where data shards are unavailable or time out, the export will be empty rather than returning partial data. diff --git a/explore-analyze/scripting/dissect.md b/explore-analyze/scripting/dissect.md index ffe36406a1..a047772161 100644 --- a/explore-analyze/scripting/dissect.md +++ b/explore-analyze/scripting/dissect.md @@ -155,7 +155,7 @@ PUT my-index/_mappings } ``` -After mapping the fields you want to retrieve, index a few records from your log data into {{es}}. The following request uses the [bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) to index raw log data into `my-index`: +After mapping the fields you want to retrieve, index a few records from your log data into {{es}}. The following request uses the [bulk API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) to index raw log data into `my-index`: ```console POST /my-index/_bulk?refresh=true diff --git a/explore-analyze/scripting/grok.md b/explore-analyze/scripting/grok.md index fa19c25f86..eef5d16235 100644 --- a/explore-analyze/scripting/grok.md +++ b/explore-analyze/scripting/grok.md @@ -88,7 +88,7 @@ PUT /my-index/ } ``` -Next, use the [bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) to index some log data into `my-index`. +Next, use the [bulk API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) to index some log data into `my-index`. ```console POST /my-index/_bulk?refresh diff --git a/explore-analyze/scripting/modules-scripting-fields.md b/explore-analyze/scripting/modules-scripting-fields.md index 24b3bb46ca..0ac0ed4119 100644 --- a/explore-analyze/scripting/modules-scripting-fields.md +++ b/explore-analyze/scripting/modules-scripting-fields.md @@ -13,7 +13,7 @@ Depending on where a script is used, it will have access to certain special vari ## Update scripts [_update_scripts] -A script used in the [update](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html), [update-by-query](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html), or [reindex](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) API will have access to the `ctx` variable which exposes: +A script used in the [update](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update), [update-by-query](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query), or [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) API will have access to the `ctx` variable which exposes: `ctx._source` : Access to the document [`_source` field](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html). diff --git a/explore-analyze/scripting/modules-scripting-using.md b/explore-analyze/scripting/modules-scripting-using.md index 18860115a7..b56848ea14 100644 --- a/explore-analyze/scripting/modules-scripting-using.md +++ b/explore-analyze/scripting/modules-scripting-using.md @@ -22,7 +22,7 @@ Wherever scripting is supported in the {{es}} APIs, the syntax follows the same : Specifies the language the script is written in. Defaults to `painless`. `source`, `id` -: The script itself, which you specify as `source` for an inline script or `id` for a stored script. Use the [stored script APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/script-apis.html#stored-script-apis) to create and manage stored scripts. +: The script itself, which you specify as `source` for an inline script or `id` for a stored script. Use the [stored script APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-script) to create and manage stored scripts. `params` : Specifies any named parameters that are passed into the script as variables. [Use parameters](#prefer-params) instead of hard-coded values to decrease compile time. @@ -154,7 +154,7 @@ Use this abbreviated syntax anywhere that {{es}} supports scripts, such as when ## Store and retrieve scripts [script-stored-scripts] -You can store and retrieve scripts from the cluster state using the [stored script APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/script-apis.html#stored-script-apis). Stored scripts allow you to reference shared scripts for operations like scoring, aggregating, filtering, and reindexing. Instead of embedding scripts inline in each query, you can reference these shared operations. +You can store and retrieve scripts from the cluster state using the [stored script APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-script). Stored scripts allow you to reference shared scripts for operations like scoring, aggregating, filtering, and reindexing. Instead of embedding scripts inline in each query, you can reference these shared operations. Stored scripts can also reduce request payload size. Depending on script size and request frequency, this can help lower latency and data transfer costs. @@ -163,7 +163,7 @@ Unlike regular scripts, stored scripts require that you specify a script languag :::: -To create a script, use the [create stored script API](https://www.elastic.co/guide/en/elasticsearch/reference/current/create-stored-script-api.html). For example, the following request creates a stored script named `calculate-score`. +To create a script, use the [create stored script API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-put-script). For example, the following request creates a stored script named `calculate-score`. ```console POST _scripts/calculate-score @@ -175,7 +175,7 @@ POST _scripts/calculate-score } ``` -You can retrieve that script by using the [get stored script API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-stored-script-api.html). +You can retrieve that script by using the [get stored script API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-get-script). ```console GET _scripts/calculate-score @@ -207,7 +207,7 @@ GET my-index-000001/_search 1. `id` of the stored script -To delete a stored script, submit a [delete stored script API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-stored-script-api.html) request. +To delete a stored script, submit a [delete stored script API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-delete-script) request. ```console DELETE _scripts/calculate-score @@ -216,7 +216,7 @@ DELETE _scripts/calculate-score ## Update documents with scripts [scripts-update-scripts] -You can use the [update API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html) to update documents with a specified script. The script can update, delete, or skip modifying the document. The update API also supports passing a partial document, which is merged into the existing document. +You can use the [update API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update) to update documents with a specified script. The script can update, delete, or skip modifying the document. The update API also supports passing a partial document, which is merged into the existing document. First, let’s index a simple document: diff --git a/explore-analyze/scripting/painless-lab.md b/explore-analyze/scripting/painless-lab.md index b97e7fbc88..4c34f698d9 100644 --- a/explore-analyze/scripting/painless-lab.md +++ b/explore-analyze/scripting/painless-lab.md @@ -12,7 +12,7 @@ mapped_pages: This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: -The **Painless Lab** is an interactive code editor that lets you test and debug [Painless scripts](modules-scripting-painless.md) in real-time. You can use the Painless scripting language to create [{{kib}} runtime fields](../find-and-organize/data-views.md#runtime-fields), process [reindexed data](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html), define complex [Watcher conditions](../alerts-cases/watcher.md), and work with data in other contexts. +The **Painless Lab** is an interactive code editor that lets you test and debug [Painless scripts](modules-scripting-painless.md) in real-time. You can use the Painless scripting language to create [{{kib}} runtime fields](../find-and-organize/data-views.md#runtime-fields), process [reindexed data](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex), define complex [Watcher conditions](../alerts-cases/watcher.md), and work with data in other contexts. Find **Painless Lab** by navigating to the **Developer tools** page using the navigation menu or the [global search field](../../explore-analyze/find-and-organize/find-apps-and-objects.md). diff --git a/explore-analyze/scripting/scripting-field-extraction.md b/explore-analyze/scripting/scripting-field-extraction.md index e16ca1faa1..9a97dcab68 100644 --- a/explore-analyze/scripting/scripting-field-extraction.md +++ b/explore-analyze/scripting/scripting-field-extraction.md @@ -34,7 +34,7 @@ PUT /my-index/ } ``` -After mapping the fields you want to retrieve, index a few records from your log data into {{es}}. The following request uses the [bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) to index raw log data into `my-index`. Instead of indexing all of your log data, you can use a small sample to experiment with runtime fields. +After mapping the fields you want to retrieve, index a few records from your log data into {{es}}. The following request uses the [bulk API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) to index raw log data into `my-index`. Instead of indexing all of your log data, you can use a small sample to experiment with runtime fields. ```console POST /my-index/_bulk?refresh diff --git a/explore-analyze/scripting/scripts-search-speed.md b/explore-analyze/scripting/scripts-search-speed.md index a48b2c3752..c14c6c5526 100644 --- a/explore-analyze/scripting/scripts-search-speed.md +++ b/explore-analyze/scripting/scripts-search-speed.md @@ -12,7 +12,7 @@ mapped_pages: Cache sizing is important. Your script cache should be large enough to hold all of the scripts that users need to be accessed concurrently. -If you see a large number of script cache evictions and a rising number of compilations in [node stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html), your cache might be too small. +If you see a large number of script cache evictions and a rising number of compilations in [node stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-stats), your cache might be too small. All scripts are cached by default so that they only need to be recompiled when updates occur. By default, scripts do not have a time-based expiration. You can change this behavior by using the `script.cache.expire` setting. Use the `script.cache.max_size` setting to configure the size of the cache. @@ -88,7 +88,7 @@ PUT _ingest/pipeline/my_test_scores_pipeline } ``` -To update existing data, use this pipeline to [reindex](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) any documents from `my_test_scores` to a new index named `my_test_scores_2`. +To update existing data, use this pipeline to [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) any documents from `my_test_scores` to a new index named `my_test_scores_2`. ```console POST /_reindex diff --git a/explore-analyze/transforms/ecommerce-transforms.md b/explore-analyze/transforms/ecommerce-transforms.md index 869f1a29a8..a627ca9150 100644 --- a/explore-analyze/transforms/ecommerce-transforms.md +++ b/explore-analyze/transforms/ecommerce-transforms.md @@ -35,10 +35,10 @@ mapped_pages: ::: ::::{tip} - If you’re interested in a subset of the data, you can optionally include a [query](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#request-body-search-query) element. In this example, we’ve filtered the data so that we’re only looking at orders with a `currency` of `EUR`. Alternatively, we could group the data by that field too. If you want to use more complex queries, you can create your {{dataframe}} from a [saved search](../discover/save-open-search.md). + If you’re interested in a subset of the data, you can optionally include a [query](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) element. In this example, we’ve filtered the data so that we’re only looking at orders with a `currency` of `EUR`. Alternatively, we could group the data by that field too. If you want to use more complex queries, you can create your {{dataframe}} from a [saved search](../discover/save-open-search.md). :::: - If you prefer, you can use the [preview {{transforms}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html). + If you prefer, you can use the [preview {{transforms}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-preview-transform). ::::{dropdown} API example ```console @@ -105,7 +105,7 @@ mapped_pages: :class: screenshot ::: - If you prefer, you can use the [create {{transforms}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html). + If you prefer, you can use the [create {{transforms}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform). ::::{dropdown} API example ```console @@ -173,7 +173,7 @@ mapped_pages: 5. Optional: Create the destination index. If the destination index does not exist, it is created the first time you start your {{transform}}. A pivot transform deduces the mappings for the destination index from the source indices and the transform aggregations. If there are fields in the destination index that are derived from scripts (for example, if you use [`scripted_metrics`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html) or [`bucket_scripts`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html) aggregations), they’re created with [dynamic mappings](../../manage-data/data-store/mapping/dynamic-mapping.md). You can use the preview {{transform}} API to preview the mappings it will use for the destination index. In {{kib}}, if you copied the API request to your clipboard, paste it into the console, then refer to the `generated_dest_index` object in the API response. ::::{note} - {{transforms-cap}} might have more configuration options provided by the APIs than the options available in {{kib}}. For example, you can set an ingest pipeline for `dest` by calling the [Create {{transform}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html). For all the {{transform}} configuration options, refer to the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-apis.html). + {{transforms-cap}} might have more configuration options provided by the APIs than the options available in {{kib}}. For example, you can set an ingest pipeline for `dest` by calling the [Create {{transform}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform). For all the {{transform}} configuration options, refer to the [documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-transform). :::: ::::{dropdown} API example @@ -247,7 +247,7 @@ mapped_pages: :::: - In some instances the deduced mappings might be incompatible with the actual data. For example, numeric overflows might occur or dynamically mapped fields might contain both numbers and strings. To avoid this problem, create your destination index before you start the {{transform}}. For more information, see the [create index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html). + In some instances the deduced mappings might be incompatible with the actual data. For example, numeric overflows might occur or dynamically mapped fields might contain both numbers and strings. To avoid this problem, create your destination index before you start the {{transform}}. For more information, see the [create index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create). ::::{dropdown} API example You can use the information from the {{transform}} preview to create the destination index. For example: @@ -299,7 +299,7 @@ mapped_pages: :class: screenshot ::: - Alternatively, you can use the [start {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-transform.html), [stop {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/stop-transform.html) and [reset {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/reset-transform.html) APIs. + Alternatively, you can use the [start {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-start-transform), [stop {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-stop-transform) and [reset {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-reset-transform) APIs. If you reset a {{transform}}, all checkpoints, states, and the destination index (if it was created by the {{transform}}) are deleted. The {{transform}} is ready to start again as if it had just been created. @@ -355,6 +355,6 @@ mapped_pages: If the destination index does not exist, it is created the first time you start your {{transform}}. Unlike pivot {{transforms}}, however, latest {{transforms}} do not deduce mapping definitions when they create the index. Instead, they use dynamic mappings. To use explicit mappings, create the destination index before you start the {{transform}}. :::: -9. If you do not want to keep a {{transform}}, you can delete it in {{kib}} or use the [delete {{transform}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-transform.html). By default, when you delete a {{transform}}, its destination index and {{kib}} index patterns remain. +9. If you do not want to keep a {{transform}}, you can delete it in {{kib}} or use the [delete {{transform}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-delete-transform). By default, when you delete a {{transform}}, its destination index and {{kib}} index patterns remain. Now that you’ve created simple {{transforms}} for {{kib}} sample data, consider possible use cases for your own data. For more ideas, see [When to use {{transforms}}](transform-usage.md) and [Examples](transform-examples.md). diff --git a/explore-analyze/transforms/transform-api-quickref.md b/explore-analyze/transforms/transform-api-quickref.md index c0ee73cba0..68319e7edf 100644 --- a/explore-analyze/transforms/transform-api-quickref.md +++ b/explore-analyze/transforms/transform-api-quickref.md @@ -14,15 +14,15 @@ All {{transform}} endpoints have the following base: _transform/ ``` -* [Create {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html) -* [Delete {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-transform.html) -* [Get {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-transform.html) -* [Get {{transforms}} statistics](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-transform-stats.html) -* [Preview {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html) -* [Reset {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/reset-transform.html) -* [Schedule now {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/schedule-now-transform.html) -* [Start {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-transform.html) -* [Stop {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/stop-transform.html) -* [Update {{transforms}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/update-transform.html) +* [Create {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform) +* [Delete {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-delete-transform) +* [Get {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-get-transform) +* [Get {{transforms}} statistics](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-get-transform-stats) +* [Preview {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-preview-transform) +* [Reset {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-reset-transform) +* [Schedule now {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-schedule-now-transform) +* [Start {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-start-transform) +* [Stop {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-stop-transform) +* [Update {{transforms}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-update-transform) -For the full list, see [*{{transform-cap}} APIs*](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-apis.html). +For the full list, see [*{{transform-cap}} APIs*](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-transform). diff --git a/explore-analyze/transforms/transform-checkpoints.md b/explore-analyze/transforms/transform-checkpoints.md index 1ff3ab57a7..a63c0323d3 100644 --- a/explore-analyze/transforms/transform-checkpoints.md +++ b/explore-analyze/transforms/transform-checkpoints.md @@ -41,7 +41,7 @@ If the cluster experiences unsuitable performance degradation due to the {{trans In most cases, it is strongly recommended to use the ingest timestamp of the source indices for syncing the {{transform}}. This is the most optimal way for {{transforms}} to be able to identify new changes. If your data source follows the [ECS standard](https://www.elastic.co/guide/en/ecs/{{ecs_version}}/ecs-reference.html), you might already have an [`event.ingested`](https://www.elastic.co/guide/en/ecs/{{ecs_version}}/ecs-event.html#field-event-ingested) field. In this case, use `event.ingested` as the `sync`.`time`.`field` property of your {{transform}}. -If you don’t have a `event.ingested` field or it isn’t populated, you can set it by using an ingest pipeline. Create an ingest pipeline either using the [ingest pipeline API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-pipeline-api.html) (like the example below) or via {{kib}} under **Stack Management > Ingest Pipelines**. Use a [`set` processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html) to set the field and associate it with the value of the ingest timestamp. +If you don’t have a `event.ingested` field or it isn’t populated, you can set it by using an ingest pipeline. Create an ingest pipeline either using the [ingest pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) (like the example below) or via {{kib}} under **Stack Management > Ingest Pipelines**. Use a [`set` processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html) to set the field and associate it with the value of the ingest timestamp. ```console PUT _ingest/pipeline/set_ingest_time @@ -58,7 +58,7 @@ PUT _ingest/pipeline/set_ingest_time } ``` -After you created the ingest pipeline, apply it to the source indices of your {{transform}}. The pipeline adds the field `event.ingested` to every document with the value of the ingest timestamp. Configure the `sync`.`time`.`field` property of your {{transform}} to use the field by using the [create {{transform}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html) for new {{transforms}} or the [update {{transform}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/update-transform.html) for existing {{transforms}}. The `event.ingested` field is used for syncing the {{transform}}. +After you created the ingest pipeline, apply it to the source indices of your {{transform}}. The pipeline adds the field `event.ingested` to every document with the value of the ingest timestamp. Configure the `sync`.`time`.`field` property of your {{transform}} to use the field by using the [create {{transform}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform) for new {{transforms}} or the [update {{transform}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-update-transform) for existing {{transforms}}. The `event.ingested` field is used for syncing the {{transform}}. Refer to [Add a pipeline to an indexing request](../../manage-data/ingest/transform-enrich/ingest-pipelines.md#add-pipeline-to-indexing-request) and [Ingest pipelines](../../manage-data/ingest/transform-enrich/ingest-pipelines.md) to learn more about how to use an ingest pipeline. diff --git a/explore-analyze/transforms/transform-examples.md b/explore-analyze/transforms/transform-examples.md index 7a055cd8f3..143e18155a 100644 --- a/explore-analyze/transforms/transform-examples.md +++ b/explore-analyze/transforms/transform-examples.md @@ -27,7 +27,7 @@ This example uses the eCommerce orders sample data set to find the customers who :class: screenshot ::: -Alternatively, you can use the [preview {{transform}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html) and the [create {{transform}} API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html). +Alternatively, you can use the [preview {{transform}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-preview-transform) and the [create {{transform}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform). :::::{dropdown} API example ```console diff --git a/explore-analyze/transforms/transform-limitations.md b/explore-analyze/transforms/transform-limitations.md index e64ea0d07d..a36e2042b2 100644 --- a/explore-analyze/transforms/transform-limitations.md +++ b/explore-analyze/transforms/transform-limitations.md @@ -51,9 +51,9 @@ A {{ctransform}} periodically checks for changes to source data. The functionali When a pivot {{transform}} is first started, it will deduce the mappings required for the destination index. This process is based on the field types of the source index and the aggregations used. If the fields are derived from [`scripted_metrics`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html) or [`bucket_scripts`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html), [dynamic mappings](../../manage-data/data-store/mapping/dynamic-mapping.md) will be used. In some instances the deduced mappings may be incompatible with the actual data. For example, numeric overflows might occur or dynamically mapped fields might contain both numbers and strings. Please check {{es}} logs if you think this may have occurred. -You can view the deduced mappings by using the [preview transform API](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html). See the `generated_dest_index` object in the API response. +You can view the deduced mappings by using the [preview transform API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-preview-transform). See the `generated_dest_index` object in the API response. -If it’s required, you may define custom mappings prior to starting the {{transform}} by creating a custom destination index using the [create index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html). As deduced mappings cannot be overwritten by an index template, use the create index API to define custom mappings. The index templates only apply to fields derived from scripts that use dynamic mappings. +If it’s required, you may define custom mappings prior to starting the {{transform}} by creating a custom destination index using the [create index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create). As deduced mappings cannot be overwritten by an index template, use the create index API to define custom mappings. The index templates only apply to fields derived from scripts that use dynamic mappings. ### Batch {{transforms}} may not account for changed documents [transform-batch-limitations] @@ -133,4 +133,4 @@ The {{transforms}} management page in {{kib}} lists up to 1000 {{transforms}}. ### {{kib}} might not support every {{transform}} configuration option [transform-ui-support] -There might be configuration options available via the {{transform}} APIs that are not supported in {{kib}}. For an exhaustive list of configuration options, refer to the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-apis.html). +There might be configuration options available via the {{transform}} APIs that are not supported in {{kib}}. For an exhaustive list of configuration options, refer to the [documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-transform). diff --git a/explore-analyze/transforms/transform-overview.md b/explore-analyze/transforms/transform-overview.md index d3d9d7dee4..2094e1e04d 100644 --- a/explore-analyze/transforms/transform-overview.md +++ b/explore-analyze/transforms/transform-overview.md @@ -14,7 +14,7 @@ You can choose either of the following methods to transform your data: [pivot](# ::::{important} * All {{transforms}} leave your source index intact. They create a new index that is dedicated to the transformed data. -* {{transforms-cap}} might have more configuration options provided by the APIs than the options available in {{kib}}. For all the {{transform}} configuration options, refer to the [API documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-apis.html). +* {{transforms-cap}} might have more configuration options provided by the APIs than the options available in {{kib}}. For all the {{transform}} configuration options, refer to the [API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-transform). :::: @@ -30,7 +30,7 @@ A lot of {{es}} indices are organized as a stream of events: each event is an in To define a pivot, first you select one or more fields that you will use to group your data. You can select categorical fields (terms) and numerical fields for grouping. If you use numerical fields, the field values are bucketed using an interval that you specify. -The second step is deciding how you want to aggregate the grouped data. When using aggregations, you practically ask questions about the index. There are different types of aggregations, each with its own purpose and output. To learn more about the supported aggregations and group-by fields, see [Create {{transform}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html). +The second step is deciding how you want to aggregate the grouped data. When using aggregations, you practically ask questions about the index. There are different types of aggregations, each with its own purpose and output. To learn more about the supported aggregations and group-by fields, see [Create {{transform}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform). As an optional step, you can also add a query to further limit the scope of the aggregation. @@ -64,7 +64,7 @@ If your {{transform}} must process a lot of historic data, it has high resource For better performance, make sure that your search aggregations and queries are optimized and that your {{transform}} is processing only necessary data. Consider whether you can apply a source query to the {{transform}} to reduce the scope of data it processes. Also consider whether the cluster has sufficient resources in place to support both the composite aggregation search and the indexing of its results. -If you prefer to spread out the impact on your cluster (at the cost of a slower {{transform}}), you can throttle the rate at which it performs search and index requests. Set the `docs_per_second` limit when you [create](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html) or [update](https://www.elastic.co/guide/en/elasticsearch/reference/current/update-transform.html) your {{transform}}. If you want to calculate the current rate, use the following information from the [get {{transform}} stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-transform-stats.html): +If you prefer to spread out the impact on your cluster (at the cost of a slower {{transform}}), you can throttle the rate at which it performs search and index requests. Set the `docs_per_second` limit when you [create](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform) or [update](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-update-transform) your {{transform}}. If you want to calculate the current rate, use the following information from the [get {{transform}} stats API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-get-transform-stats): ``` documents_processed / search_time_in_ms * 1000 diff --git a/explore-analyze/transforms/transform-painless-examples.md b/explore-analyze/transforms/transform-painless-examples.md index 6c0daebc4a..6ef9b2a155 100644 --- a/explore-analyze/transforms/transform-painless-examples.md +++ b/explore-analyze/transforms/transform-painless-examples.md @@ -96,7 +96,7 @@ You can retrieve the last value in a similar way: ### Getting top hits by using stored scripts [top-hits-stored-scripts] -You can also use the power of [stored scripts](https://www.elastic.co/guide/en/elasticsearch/reference/current/create-stored-script-api.html) to get the latest value. Stored scripts are updatable, enable collaboration, and avoid duplication across queries. +You can also use the power of [stored scripts](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-put-script) to get the latest value. Stored scripts are updatable, enable collaboration, and avoid duplication across queries. 1. Create the stored scripts: diff --git a/explore-analyze/transforms/transform-scale.md b/explore-analyze/transforms/transform-scale.md index 39bb5a0d12..9671fe8aed 100644 --- a/explore-analyze/transforms/transform-scale.md +++ b/explore-analyze/transforms/transform-scale.md @@ -29,7 +29,7 @@ The keywords in parenthesis at the end of each recommendation title indicates th ## Measure {{transforms}} performance [measure-performance] -In order to optimize {{transform}} performance, start by identifying the areas where most work is being done. The **Stats** interface of the **{{transforms-cap}}** page in {{kib}} contains information that covers three main areas: indexing, searching, and processing time (alternatively, you can use the [{{transforms}} stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-transform-stats.html)). If, for example, the results show that the highest proportion of time is spent on search, then prioritize efforts on optimizing the search query of the {{transform}}. {{transforms-cap}} also has [Rally support](https://esrally.readthedocs.io) that makes it possible to run performance checks on {{transforms}} configurations if it is required. If you optimized the crucial factors and you still experience performance issues, you may also want to consider improving your hardware. +In order to optimize {{transform}} performance, start by identifying the areas where most work is being done. The **Stats** interface of the **{{transforms-cap}}** page in {{kib}} contains information that covers three main areas: indexing, searching, and processing time (alternatively, you can use the [{{transforms}} stats API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-get-transform-stats)). If, for example, the results show that the highest proportion of time is spent on search, then prioritize efforts on optimizing the search query of the {{transform}}. {{transforms-cap}} also has [Rally support](https://esrally.readthedocs.io) that makes it possible to run performance checks on {{transforms}} configurations if it is required. If you optimized the crucial factors and you still experience performance issues, you may also want to consider improving your hardware. ## 1. Optimize `frequency` (index) [frequency] @@ -40,7 +40,7 @@ In a {{ctransform}}, the `frequency` configuration option sets the interval betw Depending on the size of the destination index, you may consider increasing its shard count. {{transforms-cap}} use one shard by default when creating the destination index. To override the index settings, create the destination index before starting the {{transform}}. For more information about how the number of shards affects scalability and resilience, refer to [Get ready for production](../../deploy-manage/index.md) ::::{tip} -Use the [Preview {{transform}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html) to check the settings that the {{transform}} would use to create the destination index. You can copy and adjust these in order to create the destination index prior to starting the {{transform}}. +Use the [Preview {{transform}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-preview-transform) to check the settings that the {{transform}} would use to create the destination index. You can copy and adjust these in order to create the destination index prior to starting the {{transform}}. :::: ## 3. Profile and optimize your search queries (search) [search-queries] diff --git a/explore-analyze/visualize/custom-visualizations-with-vega.md b/explore-analyze/visualize/custom-visualizations-with-vega.md index 0fc2a6867f..0cf6b0d15d 100644 --- a/explore-analyze/visualize/custom-visualizations-with-vega.md +++ b/explore-analyze/visualize/custom-visualizations-with-vega.md @@ -1451,7 +1451,7 @@ Use the contextual **Inspect** tool to gain insights into different elements. ##### Inspect {{es}} requests [inspect-elasticsearch-requests] -**Vega** uses the [{{es}} search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to get documents and aggregation results from {{es}}. To troubleshoot these requests, click **Inspect**, which shows the most recent requests. In case your specification has more than one request, you can switch between the views using the **View** dropdown. +**Vega** uses the [{{es}} search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) to get documents and aggregation results from {{es}}. To troubleshoot these requests, click **Inspect**, which shows the most recent requests. In case your specification has more than one request, you can switch between the views using the **View** dropdown. :::{image} ../../images/kibana-vega_tutorial_inspect_requests.png :alt: vega tutorial inspect requests diff --git a/explore-analyze/visualize/graph.md b/explore-analyze/visualize/graph.md index 057d09b1a4..27a6c5e2d6 100644 --- a/explore-analyze/visualize/graph.md +++ b/explore-analyze/visualize/graph.md @@ -12,7 +12,7 @@ The {{graph-features}} enable you to discover how items in an {{es}} index are r For example, graph exploration could help you uncover website vulnerabilities that hackers are targeting so you can harden your website. Or, you might provide graph-based personalized recommendations to your e-commerce customers. -The {{graph-features}} provide a simple, yet powerful [graph exploration API](https://www.elastic.co/guide/en/elasticsearch/reference/current/graph-explore-api.html), and an interactive graph visualization app for {{kib}}. Both work out of the box with existing {{es}} indices—you don’t need to store any additional data to use these features. +The {{graph-features}} provide a simple, yet powerful [graph exploration API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-graph), and an interactive graph visualization app for {{kib}}. Both work out of the box with existing {{es}} indices—you don’t need to store any additional data to use these features. ## How Graph works [how-graph-works] diff --git a/explore-analyze/visualize/maps/maps-troubleshooting.md b/explore-analyze/visualize/maps/maps-troubleshooting.md index 330150a4f6..25a6d0e078 100644 --- a/explore-analyze/visualize/maps/maps-troubleshooting.md +++ b/explore-analyze/visualize/maps/maps-troubleshooting.md @@ -17,7 +17,7 @@ Use the information in this section to inspect Elasticsearch requests and find s ## Inspect Elasticsearch requests [_inspect_elasticsearch_requests] -Maps uses the [{{es}} vector tile search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html) and the [{{es}} search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to get documents and aggregation results from {{es}}. Use **Vector tiles** inspector to view {{es}} vector tile search API requests. Use **Requests** inspector to view {{es}} search API requests. +Maps uses the [{{es}} vector tile search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search-mvt) and the [{{es}} search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) to get documents and aggregation results from {{es}}. Use **Vector tiles** inspector to view {{es}} vector tile search API requests. Use **Requests** inspector to view {{es}} search API requests. :::{image} ../../../images/kibana-vector_tile_inspector.png :alt: vector tile inspector