diff --git a/deploy-manage/deploy/cloud-on-k8s/nodes-orchestration.md b/deploy-manage/deploy/cloud-on-k8s/nodes-orchestration.md index 75250f6cd3..b1bf5835f8 100644 --- a/deploy-manage/deploy/cloud-on-k8s/nodes-orchestration.md +++ b/deploy-manage/deploy/cloud-on-k8s/nodes-orchestration.md @@ -176,7 +176,7 @@ Advanced users may force an upgrade by manually deleting Pods themselves. The de Operations that reduce the number of nodes in the cluster cannot make progress without user intervention, if the Elasticsearch index replica settings are incompatible with the intended downscale. Specifically, if the Elasticsearch index settings demand a higher number of shard copies than data nodes in the cluster after the downscale operation, ECK cannot migrate the data away from the node about to be removed. You can address this in the following ways: * Adjust the Elasticsearch [index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to a number of replicas that allow the desired node removal. -* Use [`auto_expand_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-settings) to automatically adjust the replicas to the number of data nodes in the cluster. +* Use [`auto_expand_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md) to automatically adjust the replicas to the number of data nodes in the cluster. ## Advanced control during rolling upgrades [k8s-advanced-upgrade-control] diff --git a/deploy-manage/deploy/cloud-on-k8s/virtual-memory.md b/deploy-manage/deploy/cloud-on-k8s/virtual-memory.md index 83206c8c02..b8221ccc25 100644 --- a/deploy-manage/deploy/cloud-on-k8s/virtual-memory.md +++ b/deploy-manage/deploy/cloud-on-k8s/virtual-memory.md @@ -13,7 +13,7 @@ The kernel setting `vm.max_map_count=262144` can be set on the host directly, by For more information, check the Elasticsearch documentation on [Virtual memory](/deploy-manage/deploy/self-managed/vm-max-map-count.md). -Optionally, you can select a different type of file system implementation for the storage. For possible options, check the [store module documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-store-settings.md). +Optionally, you can select a different type of file system implementation for the storage. For possible options, check the [store module documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md). ```yaml spec: diff --git a/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md b/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md index f6cd65335a..45755f78aa 100644 --- a/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md +++ b/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md @@ -92,7 +92,7 @@ When attempting to use an unavailable API, you’ll receive a clear error messag ## Settings availability [elasticsearch-differences-serverless-settings-availability] -In {{es-serverless}}, you can only configure [index-level settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. +In {{es-serverless}}, you can only configure [index-level settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. Available settings : **Index-level settings**: Settings that control how {{es}} documents are processed, stored, and searched are available to end users. These include: diff --git a/deploy-manage/deploy/self-managed/bootstrap-checks-max-map-count.md b/deploy-manage/deploy/self-managed/bootstrap-checks-max-map-count.md index 2e0808a412..46c4cac7be 100644 --- a/deploy-manage/deploy/self-managed/bootstrap-checks-max-map-count.md +++ b/deploy-manage/deploy/self-managed/bootstrap-checks-max-map-count.md @@ -7,5 +7,5 @@ mapped_pages: Continuing from the previous [point](max-size-virtual-memory-check.md), to use `mmap` effectively, Elasticsearch also requires the ability to create many memory-mapped areas. The maximum map count check checks that the kernel allows a process to have at least 262,144 memory-mapped areas and is enforced on Linux only. To pass the maximum map count check, you must configure `vm.max_map_count` via `sysctl` to be at least `262144`. -Alternatively, the maximum map count check is only needed if you are using `mmapfs` or `hybridfs` as the [store type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-store-settings.md) for your indices. If you [do not allow](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-store-settings.md#allow-mmap) the use of `mmap` then this bootstrap check will not be enforced. +Alternatively, the maximum map count check is only needed if you are using `mmapfs` or `hybridfs` as the [store type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md) for your indices. If you [do not allow](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md#allow-mmap) the use of `mmap` then this bootstrap check will not be enforced. diff --git a/deploy-manage/deploy/self-managed/important-settings-configuration.md b/deploy-manage/deploy/self-managed/important-settings-configuration.md index e3a877408d..398a0166a6 100644 --- a/deploy-manage/deploy/self-managed/important-settings-configuration.md +++ b/deploy-manage/deploy/self-managed/important-settings-configuration.md @@ -60,7 +60,7 @@ Don’t modify anything within the data directory or run processes that might in :::: -Elasticsearch offers a deprecated setting that allows you to specify multiple paths in `path.data`. To learn about this setting, and how to migrate away from it, refer to [Multiple data paths](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/path-settings.md#multiple-data-paths). +Elasticsearch offers a deprecated setting that allows you to specify multiple paths in `path.data`. To learn about this setting, and how to migrate away from it, refer to [Multiple data paths](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/path.md#multiple-data-paths). ## Cluster name setting [_cluster_name_setting] diff --git a/deploy-manage/deploy/self-managed/vm-max-map-count.md b/deploy-manage/deploy/self-managed/vm-max-map-count.md index bc5e646378..808376a7da 100644 --- a/deploy-manage/deploy/self-managed/vm-max-map-count.md +++ b/deploy-manage/deploy/self-managed/vm-max-map-count.md @@ -5,7 +5,7 @@ mapped_pages: # Virtual memory [vm-max-map-count] -Elasticsearch uses a [`mmapfs`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-store-settings.md#mmapfs) directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. +Elasticsearch uses a [`mmapfs`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md#mmapfs) directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. On Linux, you can increase the limits by running the following command as `root`: diff --git a/deploy-manage/distributed-architecture/reading-and-writing-documents.md b/deploy-manage/distributed-architecture/reading-and-writing-documents.md index fc4a2409cc..559d0290ed 100644 --- a/deploy-manage/distributed-architecture/reading-and-writing-documents.md +++ b/deploy-manage/distributed-architecture/reading-and-writing-documents.md @@ -41,7 +41,7 @@ These indexing stages (coordinating, primary, and replica) are sequential. To en Many things can go wrong during indexing — disks can get corrupted, nodes can be disconnected from each other, or some configuration mistake could cause an operation to fail on a replica despite it being successful on the primary. These are infrequent but the primary has to respond to them. -In the case that the primary itself fails, the node hosting the primary will send a message to the master about it. The indexing operation will wait (up to 1 minute, by [default](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-settings)) for the master to promote one of the replicas to be a new primary. The operation will then be forwarded to the new primary for processing. Note that the master also monitors the health of the nodes and may decide to proactively demote a primary. This typically happens when the node holding the primary is isolated from the cluster by a networking issue. See [here](#demoted-primary) for more details. +In the case that the primary itself fails, the node hosting the primary will send a message to the master about it. The indexing operation will wait (up to 1 minute, by [default](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md)) for the master to promote one of the replicas to be a new primary. The operation will then be forwarded to the new primary for processing. Note that the master also monitors the health of the nodes and may decide to proactively demote a primary. This typically happens when the node holding the primary is isolated from the cluster by a networking issue. See [here](#demoted-primary) for more details. Once the operation has been successfully performed on the primary, the primary has to deal with potential failures when executing it on the replica shards. This may be caused by an actual failure on the replica or due to a network issue preventing the operation from reaching the replica (or preventing the replica from responding). All of these share the same end result: a replica which is part of the in-sync replica set misses an operation that is about to be acknowledged. In order to avoid violating the invariant, the primary sends a message to the master requesting that the problematic shard be removed from the in-sync replica set. Only once removal of the shard has been acknowledged by the master does the primary acknowledge the operation. Note that the master will also instruct another node to start building a new shard copy in order to restore the system to a healthy state. diff --git a/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md b/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md index 0f6c1f8a38..75f4757220 100644 --- a/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md +++ b/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md @@ -26,7 +26,7 @@ If you’re using {{agent}}, do not deploy {{filebeat}} for log collection. Inst 2. Identify which logs you want to monitor. - The {{filebeat}} {{es}} module can handle [audit logs](../logging-configuration/logfile-audit-output.md), [deprecation logs](../logging-configuration/elasticsearch-log4j-configuration-self-managed.md#deprecation-logging), [gc logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#gc-logging), [server logs](../logging-configuration/elasticsearch-log4j-configuration-self-managed.md), and [slow logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log-settings.md). For more information about the location of your {{es}} logs, see the [path.logs](../../deploy/self-managed/important-settings-configuration.md#path-settings) setting. + The {{filebeat}} {{es}} module can handle [audit logs](../logging-configuration/logfile-audit-output.md), [deprecation logs](../logging-configuration/elasticsearch-log4j-configuration-self-managed.md#deprecation-logging), [gc logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#gc-logging), [server logs](../logging-configuration/elasticsearch-log4j-configuration-self-managed.md), and [slow logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md). For more information about the location of your {{es}} logs, see the [path.logs](../../deploy/self-managed/important-settings-configuration.md#path-settings) setting. ::::{important} If there are both structured (`*.json`) and unstructured (plain text) versions of the logs, you must use the structured logs. Otherwise, they might not appear in the appropriate context in {{kib}}. diff --git a/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md b/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md index 1da4335e25..d58578ffe3 100644 --- a/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md +++ b/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md @@ -181,7 +181,7 @@ When shipping logs to a monitoring deployment there are more logging features av #### For {{es}}: [ece-extra-logging-features-elasticsearch] * [Audit logging](../logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log-settings.md) - helps find and debug slow queries and indexing +* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing * Verbose logging - helps debug stack issues by increasing component logs After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../deploy/cloud-enterprise/edit-stack-settings.md) to enable these features. diff --git a/deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md b/deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md index b9bfe783b7..8cbb4ca47c 100644 --- a/deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md +++ b/deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md @@ -11,7 +11,7 @@ In smaller clusters, it is most important to be resilient to single-node failure If your cluster consists of one node, that single node must do everything. To accommodate this, {{es}} assigns nodes every role by default. -A single node cluster is not resilient. If the node fails, the cluster will stop working. Because there are no replicas in a one-node cluster, you cannot store your data redundantly. However, by default at least one replica is required for a [`green` cluster health status](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health). To ensure your cluster can report a `green` status, override the default by setting [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-settings) to `0` on every index. +A single node cluster is not resilient. If the node fails, the cluster will stop working. Because there are no replicas in a one-node cluster, you cannot store your data redundantly. However, by default at least one replica is required for a [`green` cluster health status](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health). To ensure your cluster can report a `green` status, override the default by setting [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md) to `0` on every index. If the node fails, you may need to restore an older copy of any lost indices from a [snapshot](../../tools/snapshot-and-restore.md). @@ -20,7 +20,7 @@ Because they are not resilient to any failures, we do not recommend using one-no ## Two-node clusters [high-availability-cluster-design-two-nodes] -If you have two nodes, we recommend they both be data nodes. You should also ensure every shard is stored redundantly on both nodes by setting [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-settings) to `1` on every index that is not a [searchable snapshot index](../../tools/snapshot-and-restore/searchable-snapshots.md). This is the default behaviour but may be overridden by an [index template](../../../manage-data/data-store/templates.md). [Auto-expand replicas](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-settings) can also achieve the same thing, but it’s not necessary to use this feature in such a small cluster. +If you have two nodes, we recommend they both be data nodes. You should also ensure every shard is stored redundantly on both nodes by setting [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md) to `1` on every index that is not a [searchable snapshot index](../../tools/snapshot-and-restore/searchable-snapshots.md). This is the default behaviour but may be overridden by an [index template](../../../manage-data/data-store/templates.md). [Auto-expand replicas](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md) can also achieve the same thing, but it’s not necessary to use this feature in such a small cluster. We recommend you set only one of your two nodes to be [master-eligible](../../distributed-architecture/clusters-nodes-shards/node-roles.md#master-node-role). This means you can be certain which of your nodes is the elected master of the cluster. The cluster can tolerate the loss of the other master-ineligible node. If you set both nodes to master-eligible, two nodes are required for a master election. Since the election will fail if either node is unavailable, your cluster cannot reliably tolerate the loss of either node. diff --git a/deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md b/deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md index 92bf030aba..3c493cb70f 100644 --- a/deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md +++ b/deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md @@ -78,7 +78,7 @@ Generally, if you are using a quantized index, you should only preload the relev ## Reduce the number of index segments [_reduce_the_number_of_index_segments] -{{es}} shards are composed of segments, which are internal storage elements in the index. For approximate kNN search, {{es}} stores the vector values of each segment as a separate HNSW graph, so kNN search must check each segment. The recent parallelization of kNN search made it much faster to search across multiple segments, but still kNN search can be up to several times faster if there are fewer segments. By default, {{es}} periodically merges smaller segments into larger ones through a background [merge process](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge-scheduler-settings.md). If this isn’t sufficient, you can take explicit steps to reduce the number of index segments. +{{es}} shards are composed of segments, which are internal storage elements in the index. For approximate kNN search, {{es}} stores the vector values of each segment as a separate HNSW graph, so kNN search must check each segment. The recent parallelization of kNN search made it much faster to search across multiple segments, but still kNN search can be up to several times faster if there are fewer segments. By default, {{es}} periodically merges smaller segments into larger ones through a background [merge process](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge.md). If this isn’t sufficient, you can take explicit steps to reduce the number of index segments. ### Increase maximum segment size [_increase_maximum_segment_size] @@ -90,8 +90,8 @@ Generally, if you are using a quantized index, you should only preload the relev A common pattern is to first perform an initial bulk upload, then make an index available for searches. Instead of force merging, you can adjust the index settings to encourage {{es}} to create larger initial segments: -* Ensure there are no searches during the bulk upload and disable [`index.refresh_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-refresh-interval-setting) by setting it to `-1`. This prevents refresh operations and avoids creating extra segments. -* Give {{es}} a large indexing buffer so it can accept more documents before flushing. By default, the [`indices.memory.index_buffer_size`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/indexing-buffer-settings.md) is set to 10% of the heap size. With a substantial heap size like 32GB, this is often enough. To allow the full indexing buffer to be used, you should also increase the limit [`index.translog.flush_threshold_size`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/translog-settings.md). +* Ensure there are no searches during the bulk upload and disable [`index.refresh_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-refresh-interval-setting) by setting it to `-1`. This prevents refresh operations and avoids creating extra segments. +* Give {{es}} a large indexing buffer so it can accept more documents before flushing. By default, the [`indices.memory.index_buffer_size`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/indexing-buffer-settings.md) is set to 10% of the heap size. With a substantial heap size like 32GB, this is often enough. To allow the full indexing buffer to be used, you should also increase the limit [`index.translog.flush_threshold_size`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/translog.md). ## Avoid heavy indexing during searches [_avoid_heavy_indexing_during_searches] @@ -103,7 +103,7 @@ When possible, it’s best to avoid heavy indexing during approximate kNN search ## Avoid page cache thrashing by using modest readahead values on Linux [_avoid_page_cache_thrashing_by_using_modest_readahead_values_on_linux_2] -Search can cause a lot of randomized read I/O. When the underlying block device has a high readahead value, there may be a lot of unnecessary read I/O done, especially when files are accessed using memory mapping (see [storage types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-store-settings.md#file-system)). +Search can cause a lot of randomized read I/O. When the underlying block device has a high readahead value, there may be a lot of unnecessary read I/O done, especially when files are accessed using memory mapping (see [storage types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md#file-system)). Most Linux distributions use a sensible readahead value of `128KiB` for a single plain device, however, when using software raid, LVM or dm-crypt the resulting block device (backing Elasticsearch [path.data](../../deploy/self-managed/important-settings-configuration.md#path-settings)) may end up having a very large readahead value (in the range of several MiB). This usually results in severe page (filesystem) cache thrashing adversely affecting search (or [update](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document)) performance. diff --git a/deploy-manage/production-guidance/optimize-performance/disk-usage.md b/deploy-manage/production-guidance/optimize-performance/disk-usage.md index eab2374dfb..7d29c1cfd7 100644 --- a/deploy-manage/production-guidance/optimize-performance/disk-usage.md +++ b/deploy-manage/production-guidance/optimize-performance/disk-usage.md @@ -68,7 +68,7 @@ The [`_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch ## Use `best_compression` [best-compression] -The `_source` and stored fields can easily take a non negligible amount of disk space. They can be compressed more aggressively by using the `best_compression` [codec](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-codec). +The `_source` and stored fields can easily take a non negligible amount of disk space. They can be compressed more aggressively by using the `best_compression` [codec](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-codec). ## Force merge [_force_merge] @@ -97,7 +97,7 @@ The type that you pick for [numeric data](asciidocalypse://docs/elasticsearch/do When Elasticsearch stores `_source`, it compresses multiple documents at once in order to improve the overall compression ratio. For instance it is very common that documents share the same field names, and quite common that they share some field values, especially on fields that have a low cardinality or a [zipfian](https://en.wikipedia.org/wiki/Zipf%27s_law) distribution. -By default documents are compressed together in the order that they are added to the index. If you enabled [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-sorting-settings.md) then instead they are compressed in sorted order. Sorting documents with similar structure, fields, and values together should improve the compression ratio. +By default documents are compressed together in the order that they are added to the index. If you enabled [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) then instead they are compressed in sorted order. Sorting documents with similar structure, fields, and values together should improve the compression ratio. ## Put fields in the same order in documents [_put_fields_in_the_same_order_in_documents] diff --git a/deploy-manage/production-guidance/optimize-performance/indexing-speed.md b/deploy-manage/production-guidance/optimize-performance/indexing-speed.md index c1c42d653e..36757e4f4c 100644 --- a/deploy-manage/production-guidance/optimize-performance/indexing-speed.md +++ b/deploy-manage/production-guidance/optimize-performance/indexing-speed.md @@ -28,7 +28,7 @@ By default, Elasticsearch periodically refreshes indices every second, but only This is the optimal configuration if you have no or very little search traffic (e.g. less than one search request every 5 minutes) and want to optimize for indexing speed. This behavior aims to automatically optimize bulk indexing in the default case when no searches are performed. In order to opt out of this behavior set the refresh interval explicitly. -On the other hand, if your index experiences regular search requests, this default behavior means that Elasticsearch will refresh your index every 1 second. If you can afford to increase the amount of time between when a document gets indexed and when it becomes visible, increasing the [`index.refresh_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-refresh-interval-setting) to a larger value, e.g. `30s`, might help improve indexing speed. +On the other hand, if your index experiences regular search requests, this default behavior means that Elasticsearch will refresh your index every 1 second. If you can afford to increase the amount of time between when a document gets indexed and when it becomes visible, increasing the [`index.refresh_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-refresh-interval-setting) to a larger value, e.g. `30s`, might help improve indexing speed. ## Disable replicas for initial loads [_disable_replicas_for_initial_loads] diff --git a/deploy-manage/production-guidance/optimize-performance/search-speed.md b/deploy-manage/production-guidance/optimize-performance/search-speed.md index 5d53126967..a2eb961719 100644 --- a/deploy-manage/production-guidance/optimize-performance/search-speed.md +++ b/deploy-manage/production-guidance/optimize-performance/search-speed.md @@ -13,7 +13,7 @@ Elasticsearch heavily relies on the filesystem cache in order to make search fas ## Avoid page cache thrashing by using modest readahead values on Linux [_avoid_page_cache_thrashing_by_using_modest_readahead_values_on_linux] -Search can cause a lot of randomized read I/O. When the underlying block device has a high readahead value, there may be a lot of unnecessary read I/O done, especially when files are accessed using memory mapping (see [storage types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-store-settings.md#file-system)). +Search can cause a lot of randomized read I/O. When the underlying block device has a high readahead value, there may be a lot of unnecessary read I/O done, especially when files are accessed using memory mapping (see [storage types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md#file-system)). Most Linux distributions use a sensible readahead value of `128KiB` for a single plain device, however, when using software raid, LVM or dm-crypt the resulting block device (backing Elasticsearch [path.data](../../deploy/self-managed/important-settings-configuration.md#path-settings)) may end up having a very large readahead value (in the range of several MiB). This usually results in severe page (filesystem) cache thrashing adversely affecting search (or [update](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document)) performance. @@ -303,7 +303,7 @@ Loading data into the filesystem cache eagerly on too many indices or too many f ## Use index sorting to speed up conjunctions [_use_index_sorting_to_speed_up_conjunctions] -[Index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-sorting-settings.md) can be useful in order to make conjunctions faster at the cost of slightly slower indexing. Read more about it in the [index sorting documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules-index-sorting-conjunctions.md). +[Index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) can be useful in order to make conjunctions faster at the cost of slightly slower indexing. Read more about it in the [index sorting documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting-conjunctions.md). ## Use `preference` to optimize cache utilization [preference-cache-optimization] diff --git a/deploy-manage/production-guidance/optimize-performance/size-shards.md b/deploy-manage/production-guidance/optimize-performance/size-shards.md index 4402b41c7a..5d559037b6 100644 --- a/deploy-manage/production-guidance/optimize-performance/size-shards.md +++ b/deploy-manage/production-guidance/optimize-performance/size-shards.md @@ -35,7 +35,7 @@ Most searches hit multiple shards. Each shard runs the search on a single CPU th Every index and every shard requires some memory and CPU resources. In most cases, a small set of large shards uses fewer resources than many small shards. -Segments play a big role in a shard’s resource usage. Most shards contain several segments, which store its index data. {{es}} keeps some segment metadata in heap memory so it can be quickly retrieved for searches. As a shard grows, its segments are [merged](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge-scheduler-settings.md) into fewer, larger segments. This decreases the number of segments, which means less metadata is kept in heap memory. +Segments play a big role in a shard’s resource usage. Most shards contain several segments, which store its index data. {{es}} keeps some segment metadata in heap memory so it can be quickly retrieved for searches. As a shard grows, its segments are [merged](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge.md) into fewer, larger segments. This decreases the number of segments, which means less metadata is kept in heap memory. Every mapped field also carries some overhead in terms of memory usage and disk space. By default {{es}} will automatically create a mapping for every field in every document it indexes, but you can switch off this behaviour to [take control of your mappings](../../../manage-data/data-store/mapping/explicit-mapping.md). @@ -54,7 +54,7 @@ Where applicable, use the following best practices as starting points for your s ### Delete indices, not documents [delete-indices-not-documents] -Deleted documents aren’t immediately removed from {{es}}'s file system. Instead, {{es}} marks the document as deleted on each related shard. The marked document will continue to use resources until it’s removed during a periodic [segment merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge-scheduler-settings.md). +Deleted documents aren’t immediately removed from {{es}}'s file system. Instead, {{es}} marks the document as deleted on each related shard. The marked document will continue to use resources until it’s removed during a periodic [segment merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge.md). When possible, delete entire indices instead. {{es}} can immediately remove deleted indices directly from the file system and free up resources. @@ -67,7 +67,7 @@ One advantage of this setup is [automatic rollover](../../../manage-data/lifecyc {{ilm-init}} also makes it easy to change your sharding strategy over time: -* **Want to decrease the shard count for new indices?**
Change the [`index.number_of_shards`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings). +* **Want to decrease the shard count for new indices?**
Change the [`index.number_of_shards`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings). * **Want larger shards or fewer backing indices?**
Increase your {{ilm-init}} policy’s [rollover threshold](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md). * **Need indices that span shorter intervals?**
Offset the increased shard count by deleting older indices sooner. You can do this by lowering the `min_age` threshold for your policy’s [delete phase](../../../manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md). @@ -273,7 +273,7 @@ DELETE my-index-000001 ### Force merge during off-peak hours [force-merge-during-off-peak-hours] -If you no longer write to an index, you can use the [force merge API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) to [merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge-scheduler-settings.md) smaller segments into larger ones. This can reduce shard overhead and improve search speeds. However, force merges are resource-intensive. If possible, run the force merge during off-peak hours. +If you no longer write to an index, you can use the [force merge API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) to [merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge.md) smaller segments into larger ones. This can reduce shard overhead and improve search speeds. However, force merges are resource-intensive. If possible, run the force merge during off-peak hours. ```console POST my-index-000001/_forcemerge diff --git a/deploy-manage/tools/cross-cluster-replication.md b/deploy-manage/tools/cross-cluster-replication.md index 60eab48d05..ae1b0b3c6b 100644 --- a/deploy-manage/tools/cross-cluster-replication.md +++ b/deploy-manage/tools/cross-cluster-replication.md @@ -191,11 +191,11 @@ Use the [recovery API](https://www.elastic.co/docs/api/doc/elasticsearch/operati ## Replicating a leader requires soft deletes [ccr-leader-requirements] -{{ccr-cap}} works by replaying the history of individual write operations that were performed on the shards of the leader index. {{es}} needs to retain the [history of these operations](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/history-retention-settings.md) on the leader shards so that they can be pulled by the follower shard tasks. The underlying mechanism used to retain these operations is *soft deletes*. +{{ccr-cap}} works by replaying the history of individual write operations that were performed on the shards of the leader index. {{es}} needs to retain the [history of these operations](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/history-retention.md) on the leader shards so that they can be pulled by the follower shard tasks. The underlying mechanism used to retain these operations is *soft deletes*. A soft delete occurs whenever an existing document is deleted or updated. By retaining these soft deletes up to configurable limits, the history of operations can be retained on the leader shards and made available to the follower shard tasks as it replays the history of operations. -The [`index.soft_deletes.retention_lease.period`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#ccr-index-soft-deletes-retention-period) setting defines the maximum time to retain a shard history retention lease before it is considered expired. This setting determines how long the cluster containing your follower index can be offline, which is 12 hours by default. If a shard copy recovers after its retention lease expires, but the missing operations are still available on the leader index, then {{es}} will establish a new lease and copy the missing operations. However {{es}} does not guarantee to retain unleased operations, so it is also possible that some of the missing operations have been discarded by the leader and are now completely unavailable. If this happens then the follower cannot recover automatically so you must [recreate it](cross-cluster-replication/ccr-recreate-follower-index.md). +The [`index.soft_deletes.retention_lease.period`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) setting defines the maximum time to retain a shard history retention lease before it is considered expired. This setting determines how long the cluster containing your follower index can be offline, which is 12 hours by default. If a shard copy recovers after its retention lease expires, but the missing operations are still available on the leader index, then {{es}} will establish a new lease and copy the missing operations. However {{es}} does not guarantee to retain unleased operations, so it is also possible that some of the missing operations have been discarded by the leader and are now completely unavailable. If this happens then the follower cannot recover automatically so you must [recreate it](cross-cluster-replication/ccr-recreate-follower-index.md). Soft deletes must be enabled for indices that you want to use as leader indices. Soft deletes are enabled by default on new indices created on or after {{es}} 7.0.0. diff --git a/deploy-manage/tools/cross-cluster-replication/_failback_when_clustera_comes_back.md b/deploy-manage/tools/cross-cluster-replication/_failback_when_clustera_comes_back.md index 74c6b43d50..13e71bb2a5 100644 --- a/deploy-manage/tools/cross-cluster-replication/_failback_when_clustera_comes_back.md +++ b/deploy-manage/tools/cross-cluster-replication/_failback_when_clustera_comes_back.md @@ -62,7 +62,7 @@ When `clusterA` comes back, `clusterB` becomes the new leader and `clusterA` bec ``` ::::{tip} - If a soft delete is merged away before it can be replicated to a follower the following process will fail due to incomplete history on the leader, see [index.soft_deletes.retention_lease.period](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#ccr-index-soft-deletes-retention-period) for more details. + If a soft delete is merged away before it can be replicated to a follower the following process will fail due to incomplete history on the leader, see [index.soft_deletes.retention_lease.period](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) for more details. :::: diff --git a/deploy-manage/tools/cross-cluster-replication/_perform_update_or_delete_by_query.md b/deploy-manage/tools/cross-cluster-replication/_perform_update_or_delete_by_query.md index ae27e1fe09..f9fe7389a2 100644 --- a/deploy-manage/tools/cross-cluster-replication/_perform_update_or_delete_by_query.md +++ b/deploy-manage/tools/cross-cluster-replication/_perform_update_or_delete_by_query.md @@ -54,7 +54,7 @@ It is possible to update or delete the documents but you can only perform these ``` ::::{tip} - If a soft delete is merged away before it can be replicated to a follower the following process will fail due to incomplete history on the leader, see [index.soft_deletes.retention_lease.period](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#ccr-index-soft-deletes-retention-period) for more details. + If a soft delete is merged away before it can be replicated to a follower the following process will fail due to incomplete history on the leader, see [index.soft_deletes.retention_lease.period](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) for more details. :::: diff --git a/deploy-manage/tools/cross-cluster-replication/ccr-recreate-follower-index.md b/deploy-manage/tools/cross-cluster-replication/ccr-recreate-follower-index.md index 9e4225889d..77cdf96fb5 100644 --- a/deploy-manage/tools/cross-cluster-replication/ccr-recreate-follower-index.md +++ b/deploy-manage/tools/cross-cluster-replication/ccr-recreate-follower-index.md @@ -12,7 +12,7 @@ applies_to: # Recreate a follower index [ccr-recreate-follower-index] -When a document is updated or deleted, the underlying operation is retained in the Lucene index for a period of time defined by the [`index.soft_deletes.retention_lease.period`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#ccr-index-soft-deletes-retention-period) parameter. You configure this setting on the [leader index](../cross-cluster-replication.md#ccr-leader-requirements). +When a document is updated or deleted, the underlying operation is retained in the Lucene index for a period of time defined by the [`index.soft_deletes.retention_lease.period`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) parameter. You configure this setting on the [leader index](../cross-cluster-replication.md#ccr-leader-requirements). When a follower index starts, it acquires a retention lease from the leader index. This lease informs the leader that it should not allow a soft delete to be pruned until either the follower indicates that it has received the operation, or until the lease expires. diff --git a/deploy-manage/tools/cross-cluster-replication/manage-auto-follow-patterns.md b/deploy-manage/tools/cross-cluster-replication/manage-auto-follow-patterns.md index 93486c1bd7..1c8816cb19 100644 --- a/deploy-manage/tools/cross-cluster-replication/manage-auto-follow-patterns.md +++ b/deploy-manage/tools/cross-cluster-replication/manage-auto-follow-patterns.md @@ -15,7 +15,7 @@ applies_to: To replicate time series indices, you configure an auto-follow pattern so that each new index in the series is replicated automatically. Whenever the name of a new index on the remote cluster matches the auto-follow pattern, a corresponding follower index is added to the local cluster. ::::{note} -Auto-follow patterns only match open indices on the remote cluster that have all primary shards started. Auto-follow patterns do not match indices that can’t be used for {{ccr-init}} such as [closed indices](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-open) or [{{search-snaps}}](../snapshot-and-restore/searchable-snapshots.md). Avoid using an auto-follow pattern that matches indices with a [read or write block](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block-settings.md#index-block-settings). These blocks prevent follower indices from replicating such indices. +Auto-follow patterns only match open indices on the remote cluster that have all primary shards started. Auto-follow patterns do not match indices that can’t be used for {{ccr-init}} such as [closed indices](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-open) or [{{search-snaps}}](../snapshot-and-restore/searchable-snapshots.md). Avoid using an auto-follow pattern that matches indices with a [read or write block](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md). These blocks prevent follower indices from replicating such indices. :::: diff --git a/deploy-manage/tools/snapshot-and-restore/create-snapshots.md b/deploy-manage/tools/snapshot-and-restore/create-snapshots.md index 661317ec4e..9696b86e67 100644 --- a/deploy-manage/tools/snapshot-and-restore/create-snapshots.md +++ b/deploy-manage/tools/snapshot-and-restore/create-snapshots.md @@ -31,7 +31,7 @@ The guide also provides tips for creating dedicated cluster state snapshots and * You can only take a snapshot from a running cluster with an elected [master node](../../distributed-architecture/clusters-nodes-shards/node-roles.md#master-node-role). * A snapshot repository must be [registered](self-managed.md) and available to the cluster. -* The cluster’s global metadata must be readable. To include an index in a snapshot, the index and its metadata must also be readable. Ensure there aren’t any [cluster blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-read-only) or [index blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block-settings.md) that prevent read access. +* The cluster’s global metadata must be readable. To include an index in a snapshot, the index and its metadata must also be readable. Ensure there aren’t any [cluster blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-read-only) or [index blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md) that prevent read access. ## Considerations [create-snapshot-considerations] diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md b/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md index d33a53f949..db63722162 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md @@ -12,7 +12,7 @@ The following limitations and known problems apply to the 9.0.0-beta1 release of ## Document size limitations when using `semantic_text` fields [ml-nlp-large-documents-limit-10k-10mb] -When using semantic text to ingest documents, chunking takes place automatically. The number of chunks is limited by the [`index.mapping.nested_objects.limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit-settings.md) cluster setting, which defaults to 10k. Documents that are too large will cause errors during ingestion. To avoid this issue, please split your documents into roughly 1MB parts before ingestion. +When using semantic text to ingest documents, chunking takes place automatically. The number of chunks is limited by the [`index.mapping.nested_objects.limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit.md) cluster setting, which defaults to 10k. Documents that are too large will cause errors during ingestion. To avoid this issue, please split your documents into roughly 1MB parts before ingestion. ## ELSER semantic search is limited to 512 tokens per field that inference is applied to [ml-nlp-elser-v1-limit-512] diff --git a/explore-analyze/transforms/transform-limitations.md b/explore-analyze/transforms/transform-limitations.md index 5c3bfadf14..febf4c0d1a 100644 --- a/explore-analyze/transforms/transform-limitations.md +++ b/explore-analyze/transforms/transform-limitations.md @@ -89,7 +89,7 @@ For each checkpoint, entities are identified that have changed since the last ti The page `size` is defined by `max_page_search_size` which is also used to define the number of buckets returned by the composite aggregation search. The default value is 500, the minimum is 10. -The index setting [`index.max_terms_count`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-settings) defines the maximum number of terms that can be used in a terms query. The default value is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the {{transform}} will fail. +The index setting [`index.max_terms_count`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-settings) defines the maximum number of terms that can be used in a terms query. The default value is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the {{transform}} will fail. Using smaller values for `max_page_search_size` may result in a longer duration for the {{transform}} checkpoint to complete. diff --git a/manage-data/data-store/data-streams.md b/manage-data/data-store/data-streams.md index 540c7a7994..eb09652681 100644 --- a/manage-data/data-store/data-streams.md +++ b/manage-data/data-store/data-streams.md @@ -30,7 +30,7 @@ Keep in mind that some features such as [Time Series Data Streams (TSDS)](../dat ## Backing indices [backing-indices] -A data stream consists of one or more [hidden](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-hidden), auto-generated backing indices. +A data stream consists of one or more [hidden](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-hidden), auto-generated backing indices. :::{image} ../../images/elasticsearch-reference-data-streams-diagram.svg :alt: data streams diagram diff --git a/manage-data/data-store/data-streams/logs-data-stream.md b/manage-data/data-store/data-streams/logs-data-stream.md index 7f3f649d0f..8de970a006 100644 --- a/manage-data/data-store/data-streams/logs-data-stream.md +++ b/manage-data/data-store/data-streams/logs-data-stream.md @@ -68,7 +68,7 @@ In `logsdb` index mode, indices are sorted by the fields `host.name` and `@times * To prioritize the latest data, `host.name` is sorted in ascending order and `@timestamp` is sorted in descending order. -You can override the default sort settings by manually configuring `index.sort.field` and `index.sort.order`. For more details, see [*Index Sorting*](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-sorting-settings.md). +You can override the default sort settings by manually configuring `index.sort.field` and `index.sort.order`. For more details, see [*Index Sorting*](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md). To modify the sort configuration of an existing data stream, update the data stream’s component templates, and then perform or wait for a [rollover](../data-streams.md#data-streams-rollover). @@ -86,7 +86,7 @@ To avoid mapping conflicts, consider these options: * **Adjust mappings:** Check your existing mappings to ensure that `host.name` is mapped as a keyword. * **Change sorting:** If needed, you can remove `host.name` from the sort settings and use a different set of fields. Sorting by `@timestamp` can be a good fallback. -* **Switch to a different [index mode](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-mode-setting)**: If resolving `host.name` mapping conflicts is not feasible, you can choose not to use `logsdb` mode. +* **Switch to a different [index mode](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting)**: If resolving `host.name` mapping conflicts is not feasible, you can choose not to use `logsdb` mode. ::::{important} On existing data streams, `logsdb` mode is applied on [rollover](../data-streams.md#data-streams-rollover) (automatic or manual). @@ -103,7 +103,7 @@ In benchmarks, routing optimizations reduced storage requirements by 20% compare To configure a routing optimization: * Include the index setting `[index.logsdb.route_on_sort_fields:true]` in the data stream configuration. -* [Configure index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-sorting-settings.md) with two or more fields, in addition to `@timestamp`. +* [Configure index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) with two or more fields, in addition to `@timestamp`. * Make sure the [`_id`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-id-field.md) field is not populated in ingested documents. It should be auto-generated instead. A custom sort configuration is required, to improve storage efficiency and to minimize hotspots from logging spikes that may route documents to a single shard. For best results, use a few sort fields that have a relatively low cardinality and don’t co-vary (for example, `host.name` and `host.id` are not optimal). @@ -111,7 +111,7 @@ A custom sort configuration is required, to improve storage efficiency and to mi ## Specialized codecs [logsdb-specialized-codecs] -By default, `logsdb` index mode uses the `best_compression` [codec](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-codec), which applies [ZSTD](https://en.wikipedia.org/wiki/Zstd) compression to stored fields. You can switch to the `default` codec for faster compression with a slightly larger storage footprint. +By default, `logsdb` index mode uses the `best_compression` [codec](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-codec), which applies [ZSTD](https://en.wikipedia.org/wiki/Zstd) compression to stored fields. You can switch to the `default` codec for faster compression with a slightly larger storage footprint. The `logsdb` index mode also automatically applies specialized codecs for numeric doc values, in order to optimize storage usage. Numeric fields are encoded using the following sequence of codecs: diff --git a/manage-data/data-store/data-streams/modify-data-stream.md b/manage-data/data-store/data-streams/modify-data-stream.md index bd637a62b2..5a565e6cde 100644 --- a/manage-data/data-store/data-streams/modify-data-stream.md +++ b/manage-data/data-store/data-streams/modify-data-stream.md @@ -23,7 +23,7 @@ If you later need to change the mappings or settings for a data stream, you have * [Change a static index setting for a data stream](../data-streams/modify-data-stream.md#change-static-index-setting-for-a-data-stream) ::::{tip} -If your changes include modifications to existing field mappings or [static index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings), a reindex is often required to apply the changes to a data stream’s backing indices. If you are already performing a reindex, you can use the same process to add new field mappings and change [dynamic index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings). See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). +If your changes include modifications to existing field mappings or [static index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md), a reindex is often required to apply the changes to a data stream’s backing indices. If you are already performing a reindex, you can use the same process to add new field mappings and change [dynamic index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md). See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). :::: @@ -173,7 +173,7 @@ If you need to change the mapping of an existing field, create a new data stream ### Change a dynamic index setting for a data stream [change-dynamic-index-setting-for-a-data-stream] -To change a [dynamic index setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings) for a data stream, follow these steps: +To change a [dynamic index setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md) for a data stream, follow these steps: 1. Update the index template used by the data stream. This ensures the setting is applied to future backing indices created for the stream. @@ -219,7 +219,7 @@ To change the `index.lifecycle.name` setting, first use the [remove policy API]( ### Change a static index setting for a data stream [change-static-index-setting-for-a-data-stream] -[Static index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings) can only be set when a backing index is created. You cannot update static index settings using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). +[Static index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md) can only be set when a backing index is created. You cannot update static index settings using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). To apply a new static setting to future backing indices, update the index template used by the data stream. The setting is automatically applied to any backing index created after the update. diff --git a/manage-data/data-store/data-streams/reindex-tsds.md b/manage-data/data-store/data-streams/reindex-tsds.md index 68e9d201c1..0a6eff68b2 100644 --- a/manage-data/data-store/data-streams/reindex-tsds.md +++ b/manage-data/data-store/data-streams/reindex-tsds.md @@ -267,5 +267,5 @@ POST /k9s/_rollover/ This creates a new backing index with the updated index settings. The destination data stream is now ready to accept new documents. -Note that the initial backing index can still accept documents within the range of timestamps derived from the source data stream. If this is not desired, mark it as [read-only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block-settings.md#index-blocks-read-only) explicitly. +Note that the initial backing index can still accept documents within the range of timestamps derived from the source data stream. If this is not desired, mark it as [read-only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md#index-blocks-read-only) explicitly. diff --git a/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md b/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md index bc5e69235e..879c281c09 100644 --- a/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md +++ b/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md @@ -317,7 +317,7 @@ POST /datastream/_rollover/ ## View downsampling results [downsampling-dsl-view-results] -By default, data stream lifecycle actions are executed every five minutes. Downsampling takes place after the index is rolled over and the [index time series end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-time-series-end-time) has lapsed as the source index is still expected to receive major writes until then. Index is now rolled over after previous step but its time series range end is likely still in the future. Once index time series range is in the past, re-run the `GET _data_stream` request. +By default, data stream lifecycle actions are executed every five minutes. Downsampling takes place after the index is rolled over and the [index time series end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has lapsed as the source index is still expected to receive major writes until then. Index is now rolled over after previous step but its time series range end is likely still in the future. Once index time series range is in the past, re-run the `GET _data_stream` request. ```console GET _data_stream diff --git a/manage-data/data-store/data-streams/run-downsampling-with-ilm.md b/manage-data/data-store/data-streams/run-downsampling-with-ilm.md index 21ff209aee..8546f72dd2 100644 --- a/manage-data/data-store/data-streams/run-downsampling-with-ilm.md +++ b/manage-data/data-store/data-streams/run-downsampling-with-ilm.md @@ -34,7 +34,7 @@ Create an ILM policy for your time series data. While not required, an ILM polic To enable downsampling, add a [Downsample action](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) and set [`fixed_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md#ilm-downsample-options) to the downsampling interval at which you want to aggregate the original time series data. -In this example, an ILM policy is configured for the `hot` phase. The downsample takes place after the index is rolled over and the [index time series end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-time-series-end-time) has lapsed as the source index is still expected to receive major writes until then. {{ilm-cap}} will not proceed with any action that expects the index to not receive writes anymore until the [index’s end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-time-series-end-time) has passed. The {{ilm-cap}} actions that wait on the end time before proceeding are: - [Delete](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - [Force merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - [Read only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - [Searchable snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - [Shrink](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) +In this example, an ILM policy is configured for the `hot` phase. The downsample takes place after the index is rolled over and the [index time series end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has lapsed as the source index is still expected to receive major writes until then. {{ilm-cap}} will not proceed with any action that expects the index to not receive writes anymore until the [index’s end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed. The {{ilm-cap}} actions that wait on the end time before proceeding are: - [Delete](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - [Force merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - [Read only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - [Searchable snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - [Shrink](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) ```console PUT _ilm/policy/datastream_policy diff --git a/manage-data/data-store/data-streams/set-up-tsds.md b/manage-data/data-store/data-streams/set-up-tsds.md index 665304f81e..5aebe1cf74 100644 --- a/manage-data/data-store/data-streams/set-up-tsds.md +++ b/manage-data/data-store/data-streams/set-up-tsds.md @@ -105,7 +105,7 @@ To setup a TSDS create an index template with the following details: * Set `index.mode` setting to `time_series`. * Your lifecycle policy in the `index.lifecycle.name` index setting. - * Optional: Other index settings, such as [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-number-of-replicas), for your TSDS’s backing indices. + * Optional: Other index settings, such as [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas), for your TSDS’s backing indices. * A priority higher than `200` to avoid collisions with built-in templates. See [Avoid index pattern collisions](../templates.md#avoid-index-pattern-collisions). * Optional: Component templates containing your mappings and other index settings. diff --git a/manage-data/data-store/data-streams/time-series-data-stream-tsds.md b/manage-data/data-store/data-streams/time-series-data-stream-tsds.md index 7b5d4da90e..24d2ea729d 100644 --- a/manage-data/data-store/data-streams/time-series-data-stream-tsds.md +++ b/manage-data/data-store/data-streams/time-series-data-stream-tsds.md @@ -32,7 +32,7 @@ A TSDS works like a regular data stream with some key differences: * {{es}} generates a hidden [`_tsid`](#tsid) metadata field for each document in a TSDS. * A TSDS uses [time-bound backing indices](#time-bound-indices) to store data from the same time period in the same backing index. * The matching index template for a TSDS must contain the `index.routing_path` index setting. A TSDS uses this setting to perform [dimension-based routing](#dimension-based-routing). -* A TSDS uses internal [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-sorting-settings.md) to order shard segments by `_tsid` and `@timestamp`. +* A TSDS uses internal [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) to order shard segments by `_tsid` and `@timestamp`. * TSDS documents only support auto-generated document `_id` values. For TSDS documents, the document `_id` is a hash of the document’s dimensions and `@timestamp`. A TSDS doesn’t support custom document `_id` values. * A TSDS uses [synthetic `_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), and as a result is subject to some [restrictions](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-restrictions) and [modifications](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications) applied to the `_source` field. @@ -123,7 +123,7 @@ Due to the cumulative nature of counter fields, the following aggregations are s ## Time series mode [time-series-mode] -The matching index template for a TSDS must contain a `data_stream` object with the `index_mode: time_series` option. This option ensures the TSDS creates backing indices with an [`index.mode`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-mode) setting of `time_series`. This setting enables most TSDS-related functionality in the backing indices. +The matching index template for a TSDS must contain a `data_stream` object with the `index_mode: time_series` option. This option ensures the TSDS creates backing indices with an [`index.mode`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-mode) setting of `time_series`. This setting enables most TSDS-related functionality in the backing indices. If you convert an existing data stream to a TSDS, only backing indices created after the conversion have an `index.mode` of `time_series`. You can’t change the `index.mode` of an existing backing index. @@ -142,7 +142,7 @@ The format of the `_tsid` field shouldn’t be relied upon. It may change from v ### Time-bound indices [time-bound-indices] -In a TSDS, each backing index, including the most recent backing index, has a range of accepted `@timestamp` values. This range is defined by the [`index.time_series.start_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-time-series-start-time) and [`index.time_series.end_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-time-series-end-time) index settings. +In a TSDS, each backing index, including the most recent backing index, has a range of accepted `@timestamp` values. This range is defined by the [`index.time_series.start_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-start-time) and [`index.time_series.end_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) index settings. When you add a document to a TSDS, {{es}} adds the document to the appropriate backing index based on its `@timestamp` value. As a result, a TSDS can add documents to any TSDS backing index that can receive writes. This applies even if the index isn’t the most recent backing index. @@ -151,7 +151,7 @@ When you add a document to a TSDS, {{es}} adds the document to the appropriate b ::: ::::{tip} -Some {{ilm-init}} actions mark the source index as read-only, or expect the index to not be actively written anymore in order to provide good performance. These actions are: - [Delete](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - [Force merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - [Read only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - [Searchable snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - [Shrink](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) {{ilm-cap}} will **not** proceed with executing these actions until the upper time-bound for accepting writes, represented by the [`index.time_series.end_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-time-series-end-time) index setting, has lapsed. +Some {{ilm-init}} actions mark the source index as read-only, or expect the index to not be actively written anymore in order to provide good performance. These actions are: - [Delete](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - [Force merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - [Read only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - [Searchable snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - [Shrink](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) {{ilm-cap}} will **not** proceed with executing these actions until the upper time-bound for accepting writes, represented by the [`index.time_series.end_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) index setting, has lapsed. :::: @@ -162,7 +162,7 @@ If no backing index can accept a document’s `@timestamp` value, {{es}} rejects ### Look-ahead time [tsds-look-ahead-time] -Use the [`index.look_ahead_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-look-ahead-time) index setting to configure how far into the future you can add documents to an index. When you create a new write index for a TSDS, {{es}} calculates the index’s `index.time_series.end_time` value as: +Use the [`index.look_ahead_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-look-ahead-time) index setting to configure how far into the future you can add documents to an index. When you create a new write index for a TSDS, {{es}} calculates the index’s `index.time_series.end_time` value as: `now + index.look_ahead_time` @@ -175,7 +175,7 @@ This process continues until the write index rolls over. When the index rolls ov ### Look-back time [tsds-look-back-time] -Use the [`index.look_back_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-look-back-time) index setting to configure how far in the past you can add documents to an index. When you create a data stream for a TSDS, {{es}} calculates the index’s `index.time_series.start_time` value as: +Use the [`index.look_back_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-look-back-time) index setting to configure how far in the past you can add documents to an index. When you create a data stream for a TSDS, {{es}} calculates the index’s `index.time_series.start_time` value as: `now - index.look_back_time` @@ -196,7 +196,7 @@ You can use the [get data stream API](https://www.elastic.co/docs/api/doc/elasti ### Dimension-based routing [dimension-based-routing] -Within each TSDS backing index, {{es}} uses the [`index.routing_path`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series-index-settings.md#index-routing-path) index setting to route documents with the same dimensions to the same shards. +Within each TSDS backing index, {{es}} uses the [`index.routing_path`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-routing-path) index setting to route documents with the same dimensions to the same shards. When you create the matching index template for a TSDS, you must specify one or more dimensions in the `index.routing_path` setting. Each document in a TSDS must contain one or more dimensions that match the `index.routing_path` setting. @@ -209,11 +209,11 @@ TSDS documents don’t support a custom `_routing` value. Similarly, you can’t ### Index sorting [tsds-index-sorting] -{{es}} uses [compression algorithms](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-codec) to compress repeated values. This compression works best when repeated values are stored near each other — in the same index, on the same shard, and side-by-side in the same shard segment. +{{es}} uses [compression algorithms](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-codec) to compress repeated values. This compression works best when repeated values are stored near each other — in the same index, on the same shard, and side-by-side in the same shard segment. Most time series data contains repeated values. Dimensions are repeated across documents in the same time series. The metric values of a time series may also change slowly over time. -Internally, each TSDS backing index uses [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-sorting-settings.md) to order its shard segments by `_tsid` and `@timestamp`. This makes it more likely that these repeated values are stored near each other for better compression. A TSDS doesn’t support any [`index.sort.*`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-sorting-settings.md) index settings. +Internally, each TSDS backing index uses [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) to order its shard segments by `_tsid` and `@timestamp`. This makes it more likely that these repeated values are stored near each other for better compression. A TSDS doesn’t support any [`index.sort.*`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) index settings. ## What’s next? [tsds-whats-next] diff --git a/manage-data/data-store/index-basics.md b/manage-data/data-store/index-basics.md index 48a0aaade4..0590615dd5 100644 --- a/manage-data/data-store/index-basics.md +++ b/manage-data/data-store/index-basics.md @@ -82,7 +82,7 @@ Investigate your indices and perform operations from the **Indices** view. * To show details and perform operations, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices). * To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). -* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. +* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. * To create new indices, use the **Create index** wizard. ### Manage data streams diff --git a/manage-data/data-store/mapping.md b/manage-data/data-store/mapping.md index 7b8f3d5bb6..115cdf8333 100644 --- a/manage-data/data-store/mapping.md +++ b/manage-data/data-store/mapping.md @@ -100,4 +100,4 @@ Defining too many fields in an index can lead to a mapping explosion, which can Consider a situation where every new document inserted introduces new fields, such as with [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md). Each new field is added to the index mapping, which can become a problem as the mapping grows. -Use the [mapping limit settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit-settings.md) to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. \ No newline at end of file +Use the [mapping limit settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit.md) to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. \ No newline at end of file diff --git a/manage-data/ingest/transform-enrich/ingest-pipelines.md b/manage-data/ingest/transform-enrich/ingest-pipelines.md index 85b30cbb92..ef7b1eb018 100644 --- a/manage-data/ingest/transform-enrich/ingest-pipelines.md +++ b/manage-data/ingest/transform-enrich/ingest-pipelines.md @@ -228,12 +228,12 @@ POST _reindex ## Set a default pipeline [set-default-pipeline] -Use the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-default-pipeline) index setting to set a default pipeline. {{es}} applies this pipeline to indexing requests if no `pipeline` parameter is specified. +Use the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) index setting to set a default pipeline. {{es}} applies this pipeline to indexing requests if no `pipeline` parameter is specified. ## Set a final pipeline [set-final-pipeline] -Use the [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-final-pipeline) index setting to set a final pipeline. {{es}} applies this pipeline after the request or default pipeline, even if neither is specified. +Use the [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting to set a final pipeline. {{es}} applies this pipeline after the request or default pipeline, even if neither is specified. ## Pipelines for {{beats}} [pipelines-for-beats] @@ -270,7 +270,7 @@ $$$pipeline-custom-logs-index-template$$$ } ``` -2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-default-pipeline) or [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs--*`. +2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs--*`. You can create this template using {{kib}}'s [**Index Management**](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). @@ -345,7 +345,7 @@ $$$pipeline-custom-logs-configuration$$$ **{{agent}} standalone** -If you run {{agent}} standalone, you can apply pipelines using an [index template](../../data-store/templates.md) that includes the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-default-pipeline) or [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-final-pipeline) index setting. Alternatively, you can specify the `pipeline` policy setting in your `elastic-agent.yml` configuration. See [Install standalone {{agent}}s](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md). +If you run {{agent}} standalone, you can apply pipelines using an [index template](../../data-store/templates.md) that includes the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Alternatively, you can specify the `pipeline` policy setting in your `elastic-agent.yml` configuration. See [Install standalone {{agent}}s](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md). ## Pipelines for search indices [pipelines-in-enterprise-search] diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index 44e8b12f55..03f67d3b82 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -436,7 +436,7 @@ We recommend you use [dedicated nodes](/deploy-manage/distributed-architecture/c ## Data tier index allocation [data-tier-allocation] -The [`index.routing.allocation.include._tier_preference`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation-settings.md#tier-preference-allocation-filter) setting determines which tier the index should be allocated to. +The [`index.routing.allocation.include._tier_preference`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation.md#tier-preference-allocation-filter) setting determines which tier the index should be allocated to. When you create an index, by default {{es}} sets the `_tier_preference` to `data_content` to automatically allocate the index shards to the content tier. diff --git a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md index 8f277c9c77..ae3be01f4d 100644 --- a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md +++ b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md @@ -35,7 +35,7 @@ Investigate your indices and perform operations from the **Indices** view. * To show details and perform operations such as close, forcemerge, and flush, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices). * To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). -* To drill down into the index [mappings](../../data-store/mapping.md), [settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. +* To drill down into the index [mappings](../../data-store/mapping.md), [settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. :::{image} ../../../images/elasticsearch-reference-management_index_details.png :alt: Index Management UI diff --git a/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md b/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md index e25ef5cfa6..71be7632e1 100644 --- a/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md +++ b/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md @@ -103,7 +103,7 @@ To completely avoid the issues that raise when mixing the tier preference and cu To enable {{ilm-init}} to move an *existing* managed index through the data tiers, update the index settings to: 1. Remove the custom allocation filter by setting it to `null`. -2. Set the [tier preference](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation-settings.md#tier-preference-allocation-filter). +2. Set the [tier preference](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation.md#tier-preference-allocation-filter). For example, if your old template set the `data` attribute to `hot` to allocate shards to the hot tier, set the `data` attribute to `null` and set the `_tier_preference` to `data_hot`. diff --git a/raw-migrated-files/cloud-on-k8s/cloud-on-k8s/k8s-orchestration.md b/raw-migrated-files/cloud-on-k8s/cloud-on-k8s/k8s-orchestration.md index fb66c81a65..5aca84f139 100644 --- a/raw-migrated-files/cloud-on-k8s/cloud-on-k8s/k8s-orchestration.md +++ b/raw-migrated-files/cloud-on-k8s/cloud-on-k8s/k8s-orchestration.md @@ -169,7 +169,7 @@ Advanced users may force an upgrade by manually deleting Pods themselves. The de Operations that reduce the number of nodes in the cluster cannot make progress without user intervention, if the Elasticsearch index replica settings are incompatible with the intended downscale. Specifically, if the Elasticsearch index settings demand a higher number of shard copies than data nodes in the cluster after the downscale operation, ECK cannot migrate the data away from the node about to be removed. You can address this in the following ways: * Adjust the Elasticsearch [index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to a number of replicas that allow the desired node removal. -* Use [`auto_expand_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-settings) to automatically adjust the replicas to the number of data nodes in the cluster. +* Use [`auto_expand_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-settings) to automatically adjust the replicas to the number of data nodes in the cluster. ## Advanced control during rolling upgrades [k8s-advanced-upgrade-control] diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md b/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md index f139106b1c..20aa2ba01f 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md @@ -114,7 +114,7 @@ The following circuit breaker settings are supported: The following indexing pressure settings are supported: `indexing_pressure.memory.limit` -: Configures [the indexing pressure settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/indexing-pressure-settings.md#indexing-pressure-settings). +: Configures [the indexing pressure settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/pressure.md). ### X-Pack [echx_pack] diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md b/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md index 62482ced4e..34dc5ff168 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md @@ -173,7 +173,7 @@ When shipping logs to a monitoring deployment there are more logging features av #### For {{es}}: [ech-extra-logging-features-elasticsearch] * [Audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log-settings.md) - helps find and debug slow queries and indexing +* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing * Verbose logging - helps debug stack issues by increasing component logs After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable these features. diff --git a/raw-migrated-files/cloud/cloud/ec-add-user-settings.md b/raw-migrated-files/cloud/cloud/ec-add-user-settings.md index 8e76aae980..4952b05775 100644 --- a/raw-migrated-files/cloud/cloud/ec-add-user-settings.md +++ b/raw-migrated-files/cloud/cloud/ec-add-user-settings.md @@ -114,7 +114,7 @@ The following circuit breaker settings are supported: The following indexing pressure settings are supported: `indexing_pressure.memory.limit` -: Configures [the indexing pressure settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/indexing-pressure-settings.md#indexing-pressure-settings). +: Configures [the indexing pressure settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/pressure.md). ### X-Pack [ec_x_pack] diff --git a/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md b/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md index 7957951bab..aae765e7a5 100644 --- a/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md +++ b/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md @@ -173,7 +173,7 @@ When shipping logs to a monitoring deployment there are more logging features av #### For {{es}}: [ec-extra-logging-features-elasticsearch] * [Audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log-settings.md) - helps find and debug slow queries and indexing +* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing * Verbose logging - helps debug stack issues by increasing component logs After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable these features. diff --git a/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md b/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md index 3ebba5470a..0c980451e4 100644 --- a/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md +++ b/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md @@ -88,7 +88,7 @@ When attempting to use an unavailable API, you’ll receive a clear error messag ## Settings availability [elasticsearch-differences-serverless-settings-availability] -In {{es-serverless}}, you can only configure [index-level settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. +In {{es-serverless}}, you can only configure [index-level settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. Available settings : **Index-level settings**: Settings that control how {{es}} documents are processed, stored, and searched are available to end users. These include: diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/index-modules-allocation.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/index-modules-allocation.md index 05a7d4089f..fa7a52651e 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/index-modules-allocation.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/index-modules-allocation.md @@ -5,7 +5,7 @@ This module provides per-index settings to control the allocation of shards to n * [Shard allocation filtering](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md): Controlling which shards are allocated to which nodes. * [Delayed allocation](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/delaying-allocation-when-node-leaves.md): Delaying allocation of unassigned shards caused by a node leaving. * [Total shards per node](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/total-shards-per-node.md): A hard limit on the number of shards from the same index per node. -* [Data tier allocation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation-settings.md): Controls the allocation of indices to [data tiers](../../../manage-data/lifecycle/data-tiers.md). +* [Data tier allocation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation.md): Controls the allocation of indices to [data tiers](../../../manage-data/lifecycle/data-tiers.md). diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/shard-allocation-filtering.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/shard-allocation-filtering.md index fe3ac682ad..03ff711952 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/shard-allocation-filtering.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/shard-allocation-filtering.md @@ -84,7 +84,7 @@ The index allocation settings support the following built-in attributes: : Match nodes by node id `_tier` -: Match nodes by the node’s [data tier](../../../manage-data/lifecycle/data-tiers.md) role. For more details see [data tier allocation filtering](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation-settings.md) +: Match nodes by the node’s [data tier](../../../manage-data/lifecycle/data-tiers.md) role. For more details see [data tier allocation filtering](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation.md) ::::{note} `_tier` filtering is based on [node](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md) roles. Only a subset of roles are [data tier](../../../manage-data/lifecycle/data-tiers.md) roles, and the generic [data role](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#data-node-role) will match any tier filtering. diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md index 72d14bad5e..6c7dd593d8 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md @@ -23,7 +23,7 @@ This guide also provides tips for [restoring to another cluster](../../../deploy * You can only restore a snapshot to a running cluster with an elected [master node](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#master-node-role). The snapshot’s repository must be [registered](../../../deploy-manage/tools/snapshot-and-restore/self-managed.md) and available to the cluster. * The snapshot and cluster versions must be compatible. See [Snapshot compatibility](../../../deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility). -* To restore a snapshot, the cluster’s global metadata must be writable. Ensure there aren’t any [cluster blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-read-only) that prevent writes. The restore operation ignores [index blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block-settings.md). +* To restore a snapshot, the cluster’s global metadata must be writable. Ensure there aren’t any [cluster blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-read-only) that prevent writes. The restore operation ignores [index blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md). * Before you restore a data stream, ensure the cluster contains a [matching index template](../../../manage-data/data-store/data-streams/set-up-data-stream.md#create-index-template) with data stream enabled. To check, use {{kib}}'s [**Index Management**](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [get index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-index-template): ```console @@ -466,7 +466,7 @@ Before you start a restore operation, ensure the new cluster has enough capacity * Add nodes or upgrade your hardware to increase capacity. * Restore fewer indices and data streams. -* Reduce the [number of replicas](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-number-of-replicas) for restored indices. +* Reduce the [number of replicas](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas) for restored indices. For example, the following restore snapshot API request uses the `index_settings` option to set `index.number_of_replicas` to `1`. diff --git a/solutions/observability/apps/view-elasticsearch-index-template.md b/solutions/observability/apps/view-elasticsearch-index-template.md index 40ffeaa621..aa9e612fe0 100644 --- a/solutions/observability/apps/view-elasticsearch-index-template.md +++ b/solutions/observability/apps/view-elasticsearch-index-template.md @@ -28,7 +28,7 @@ Add any custom metadata, index settings, or mappings. ### Index settings [apm-custom-index-template-index-settings] -In the **Index settings** step, you can specify custom [index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings). For example, you could: +In the **Index settings** step, you can specify custom [index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md). For example, you could: * Customize the index lifecycle policy applied to a data stream. See [custom index lifecycle policies](index-lifecycle-management.md#apm-data-streams-custom-policy) for a walk-through. * Change the number of [shards](../../../deploy-manage/index.md) per index. Specify the number of primary shards: diff --git a/solutions/search/full-text/how-full-text-works.md b/solutions/search/full-text/how-full-text-works.md index a077017d29..cb9a77cded 100644 --- a/solutions/search/full-text/how-full-text-works.md +++ b/solutions/search/full-text/how-full-text-works.md @@ -26,7 +26,7 @@ Refer to [Test an analyzer](../../../manage-data/data-store/text-analysis/test-a * **Relevance scoring**: Results are ranked by how relevant they are to the given query. The relevance score of each document is represented by a positive floating-point number called the `_score`. The higher the `_score`, the more relevant the document. - The default [similarity algorithm](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/similarity-settings.md) {{es}} uses for calculating relevance scores is [Okapi BM25](https://en.wikipedia.org/wiki/Okapi_BM25), a variation of the [TF-IDF algorithm](https://en.wikipedia.org/wiki/Tf–idf). BM25 calculates relevance scores based on term frequency, document frequency, and document length. Refer to this [technical blog post](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables) for a deep dive into BM25. + The default [similarity algorithm](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/similarity.md) {{es}} uses for calculating relevance scores is [Okapi BM25](https://en.wikipedia.org/wiki/Okapi_BM25), a variation of the [TF-IDF algorithm](https://en.wikipedia.org/wiki/Tf–idf). BM25 calculates relevance scores based on term frequency, document frequency, and document length. Refer to this [technical blog post](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables) for a deep dive into BM25. * **Full-text search query**: Query text is analyzed [the same way as the indexed text](../../../manage-data/data-store/text-analysis/index-search-analysis.md), and the resulting tokens are used to search the inverted index. diff --git a/troubleshoot/elasticsearch/allow-all-index-allocation.md b/troubleshoot/elasticsearch/allow-all-index-allocation.md index 53e3c20d42..0da28da379 100644 --- a/troubleshoot/elasticsearch/allow-all-index-allocation.md +++ b/troubleshoot/elasticsearch/allow-all-index-allocation.md @@ -9,7 +9,7 @@ mapped_pages: # Allow Elasticsearch to allocate the index [allow-all-index-allocation] -The allocation of data can be controlled using the [enable allocation configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-routing-allocation-enable-setting). In certain circumstances users might want to temporarily disable or restrict the allocation of data. +The allocation of data can be controlled using the [enable allocation configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-routing-allocation-enable-setting). In certain circumstances users might want to temporarily disable or restrict the allocation of data. Forgetting to re-allow all data allocation can lead to unassigned shards. @@ -18,7 +18,7 @@ In order to (re)allow all data to be allocated follow these steps: :::::::{tab-set} ::::::{tab-item} Elasticsearch Service -In order to get the shards assigned we’ll need to change the value of the [configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-routing-allocation-enable-setting) that restricts the assignemnt of the shards to `all`. +In order to get the shards assigned we’ll need to change the value of the [configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-routing-allocation-enable-setting) that restricts the assignemnt of the shards to `all`. **Use {{kib}}** @@ -56,7 +56,7 @@ In order to get the shards assigned we’ll need to change the value of the [con 1. Represents the current configured value that controls if the index is allowed to be partially or totally allocated. -5. [Change](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) the [configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-routing-allocation-enable-setting) value to allow the index to be fully allocated: +5. [Change](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) the [configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-routing-allocation-enable-setting) value to allow the index to be fully allocated: ```console PUT /my-index-000001/_settings @@ -71,7 +71,7 @@ In order to get the shards assigned we’ll need to change the value of the [con :::::: ::::::{tab-item} Self-managed -In order to get the shards assigned we’ll need to change the value of the [configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-routing-allocation-enable-setting) that restricts the assignemnt of the shards to `all`. +In order to get the shards assigned we’ll need to change the value of the [configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-routing-allocation-enable-setting) that restricts the assignemnt of the shards to `all`. 1. Inspect the `index.routing.allocation.enable` [index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) for the index with unassigned shards: @@ -93,7 +93,7 @@ In order to get the shards assigned we’ll need to change the value of the [con 1. Represents the current configured value that controls if the index is allowed to be partially or totally allocated. -2. [Change](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) the [configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-routing-allocation-enable-setting) value to allow the index to be fully allocated: +2. [Change](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) the [configuration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-routing-allocation-enable-setting) value to allow the index to be fully allocated: ```console PUT /my-index-000001/_settings diff --git a/troubleshoot/elasticsearch/fix-watermark-errors.md b/troubleshoot/elasticsearch/fix-watermark-errors.md index df63336c65..a0b9d91104 100644 --- a/troubleshoot/elasticsearch/fix-watermark-errors.md +++ b/troubleshoot/elasticsearch/fix-watermark-errors.md @@ -11,7 +11,7 @@ mapped_pages: When a data node is critically low on disk space and has reached the [flood-stage disk usage watermark](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-flood-stage), the following error is logged: `Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block`. -To prevent a full disk, when a node reaches this watermark, {{es}} [blocks writes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block-settings.md#index-block-settings) to any index with a shard on the node. If the block affects related system indices, {{kib}} and other {{stack}} features may become unavailable. For example, this could induce {{kib}}'s `Kibana Server is not Ready yet` [error message](/troubleshoot/kibana/error-server-not-ready.md). +To prevent a full disk, when a node reaches this watermark, {{es}} [blocks writes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md) to any index with a shard on the node. If the block affects related system indices, {{kib}} and other {{stack}} features may become unavailable. For example, this could induce {{kib}}'s `Kibana Server is not Ready yet` [error message](/troubleshoot/kibana/error-server-not-ready.md). {{es}} will automatically remove the write block when the affected node’s disk usage falls below the [high disk watermark](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-watermark-high). To achieve this, {{es}} attempts to rebalance some of the affected node’s shards to other nodes in the same data tier. @@ -45,7 +45,7 @@ GET _cluster/allocation/explain ## Temporary Relief [fix-watermark-errors-temporary] -To immediately restore write operations, you can temporarily increase [disk watermarks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation) and remove the [write block](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block-settings.md#index-block-settings). +To immediately restore write operations, you can temporarily increase [disk watermarks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation) and remove the [write block](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md). ```console PUT _cluster/settings diff --git a/troubleshoot/elasticsearch/high-jvm-memory-pressure.md b/troubleshoot/elasticsearch/high-jvm-memory-pressure.md index 2bf98399a8..f9342b550e 100644 --- a/troubleshoot/elasticsearch/high-jvm-memory-pressure.md +++ b/troubleshoot/elasticsearch/high-jvm-memory-pressure.md @@ -71,11 +71,11 @@ Every shard uses memory. In most cases, a small set of large shards uses fewer r $$$avoid-expensive-searches$$$ **Avoid expensive searches** -Expensive searches can use large amounts of memory. To better track expensive searches on your cluster, enable [slow logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log-settings.md). +Expensive searches can use large amounts of memory. To better track expensive searches on your cluster, enable [slow logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md). Expensive searches may have a large [`size` argument](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/paginate-search-results.md), use aggregations with a large number of buckets, or include [expensive queries](../../explore-analyze/query-filter/languages/querydsl.md#query-dsl-allow-expensive-queries). To prevent expensive searches, consider the following setting changes: -* Lower the `size` limit using the [`index.max_result_window`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-max-result-window) index setting. +* Lower the `size` limit using the [`index.max_result_window`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-max-result-window) index setting. * Decrease the maximum number of allowed aggregation buckets using the [search.max_buckets](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/search-settings.md#search-settings-max-buckets) cluster setting. * Disable expensive queries using the [`search.allow_expensive_queries`](../../explore-analyze/query-filter/languages/querydsl.md#query-dsl-allow-expensive-queries) cluster setting. * Set a default search timeout using the [`search.default_search_timeout`](../../solutions/search/the-search-api.md#search-timeout) cluster setting. @@ -97,7 +97,7 @@ PUT _cluster/settings **Prevent mapping explosions** -Defining too many fields or nesting fields too deeply can lead to [mapping explosions](../../manage-data/data-store/mapping.md#mapping-limit-settings) that use large amounts of memory. To prevent mapping explosions, use the [mapping limit settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit-settings.md) to limit the number of field mappings. +Defining too many fields or nesting fields too deeply can lead to [mapping explosions](../../manage-data/data-store/mapping.md#mapping-limit-settings) that use large amounts of memory. To prevent mapping explosions, use the [mapping limit settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit.md) to limit the number of field mappings. **Spread out bulk requests** diff --git a/troubleshoot/elasticsearch/increase-tier-capacity.md b/troubleshoot/elasticsearch/increase-tier-capacity.md index 8861fc415e..92cae5336a 100644 --- a/troubleshoot/elasticsearch/increase-tier-capacity.md +++ b/troubleshoot/elasticsearch/increase-tier-capacity.md @@ -8,7 +8,7 @@ mapped_pages: Distributing copies of the data (index shard replicas) on different nodes can parallelize processing requests thus speeding up search queries. This can be achieved by increasing the number of replica shards up to the maximum value (total number of nodes minus one) which also serves the purpose to protect against hardware failure. If the index has a preferred tier, Elasticsearch will only place the copies of the data for that index on nodes in the target tier. -If a warning is encountered with not enough nodes to allocate all shard replicas, you can influence this behavior by adding more nodes to the cluster (or tier if tiers are in use), or by reducing the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-number-of-replicas) index setting. +If a warning is encountered with not enough nodes to allocate all shard replicas, you can influence this behavior by adding more nodes to the cluster (or tier if tiers are in use), or by reducing the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas) index setting. In order to fix this follow the next steps: @@ -71,10 +71,10 @@ Now that you know the tier, you want to increase the number of nodes in that tie * Find the **Availability zones** selection. If it is less than 3, you can select a higher number of availability zones for that tier. -If it is not possible to increase the size per zone or the number of availability zones, you can reduce the number of replicas of your index data. We’ll achieve this by inspecting the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-number-of-replicas) index setting index setting and decreasing the configured value. +If it is not possible to increase the size per zone or the number of availability zones, you can reduce the number of replicas of your index data. We’ll achieve this by inspecting the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas) index setting index setting and decreasing the configured value. 1. Access {{kib}} as described above. -2. Inspect the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-number-of-replicas) index setting. +2. Inspect the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas) index setting. ```console GET /my-index-000001/_settings/index.number_of_replicas @@ -150,9 +150,9 @@ The response will look like this: 1. Represents a comma separated list of data tier node roles this index is allowed to be allocated on, the first one in the list being the one with the higher priority i.e. the tier the index is targeting. e.g. in this example the tier preference is `data_warm,data_hot` so the index is targeting the `warm` tier and more nodes with the `data_warm` role are needed in the {{es}} cluster. -Alternatively, if adding more nodes to the {{es}} cluster is not desired, inspect the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-number-of-replicas) index setting and decrease the configured value: +Alternatively, if adding more nodes to the {{es}} cluster is not desired, inspect the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas) index setting and decrease the configured value: -1. Inspect the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-number-of-replicas) index setting for the index with unassigned replica shards: +1. Inspect the [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas) index setting for the index with unassigned replica shards: ```console GET /my-index-000001/_settings/index.number_of_replicas diff --git a/troubleshoot/elasticsearch/mapping-explosion.md b/troubleshoot/elasticsearch/mapping-explosion.md index d9e324c97c..a43f1f0c48 100644 --- a/troubleshoot/elasticsearch/mapping-explosion.md +++ b/troubleshoot/elasticsearch/mapping-explosion.md @@ -20,11 +20,11 @@ Mapping explosion may surface as the following performance symptoms: ## Prevent or prepare [prevent] -[Mappings](../../manage-data/data-store/mapping.md) cannot be field-reduced once initialized. {{es}} indices default to [dynamic mappings](../../manage-data/data-store/mapping.md) which doesn’t normally cause problems unless it’s combined with overriding [`index.mapping.total_fields.limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit-settings.md). The default `1000` limit is considered generous, though overriding to `10000` doesn’t cause noticeable impact depending on use case. However, to give a bad example, overriding to `100000` and this limit being hit by mapping totals would usually have strong performance implications. +[Mappings](../../manage-data/data-store/mapping.md) cannot be field-reduced once initialized. {{es}} indices default to [dynamic mappings](../../manage-data/data-store/mapping.md) which doesn’t normally cause problems unless it’s combined with overriding [`index.mapping.total_fields.limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit.md). The default `1000` limit is considered generous, though overriding to `10000` doesn’t cause noticeable impact depending on use case. However, to give a bad example, overriding to `100000` and this limit being hit by mapping totals would usually have strong performance implications. If your index mapped fields expect to contain a large, arbitrary set of keys, you may instead consider: -* Setting [`index.mapping.total_fields.ignore_dynamic_beyond_limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit-settings.md) to `true`. Instead of rejecting documents that exceed the field limit, this will ignore dynamic fields once the limit is reached. +* Setting [`index.mapping.total_fields.ignore_dynamic_beyond_limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit.md) to `true`. Instead of rejecting documents that exceed the field limit, this will ignore dynamic fields once the limit is reached. * Using the [flattened](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/flattened.md) data type. Please note, however, that flattened objects is [not fully supported in {{kib}}](https://github.com/elastic/kibana/issues/25820) yet. For example, this could apply to sub-mappings like { `host.name` , `host.os`, `host.version` }. Desired fields are still accessed by [runtime fields](../../manage-data/data-store/mapping/define-runtime-fields-in-search-request.md). * Disable [dynamic mappings](../../manage-data/data-store/mapping.md). This cannot effect current index mapping, but can apply going forward via an [index template](../../manage-data/data-store/templates.md). diff --git a/troubleshoot/elasticsearch/red-yellow-cluster-status.md b/troubleshoot/elasticsearch/red-yellow-cluster-status.md index bc6935bc9e..288442769a 100644 --- a/troubleshoot/elasticsearch/red-yellow-cluster-status.md +++ b/troubleshoot/elasticsearch/red-yellow-cluster-status.md @@ -63,7 +63,7 @@ A shard can become unassigned for several reasons. The following tips outline th ### Single node cluster [fix-cluster-status-only-one-node] -{{es}} will never assign a replica to the same node as the primary shard. A single-node cluster will always have yellow status. To change to green, set [number_of_replicas](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#dynamic-index-number-of-replicas) to 0 for all indices. +{{es}} will never assign a replica to the same node as the primary shard. A single-node cluster will always have yellow status. To change to green, set [number_of_replicas](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas) to 0 for all indices. Therefore, if the number of replicas equals or exceeds the number of nodes, some shards won’t be allocated. diff --git a/troubleshoot/elasticsearch/rejected-requests.md b/troubleshoot/elasticsearch/rejected-requests.md index e3625bf34a..db1e3b682c 100644 --- a/troubleshoot/elasticsearch/rejected-requests.md +++ b/troubleshoot/elasticsearch/rejected-requests.md @@ -9,10 +9,10 @@ When {{es}} rejects a request, it stops the operation and returns an error with * A [depleted thread pool](high-cpu-usage.md). A depleted `search` or `write` thread pool returns a `TOO_MANY_REQUESTS` error message. * A [circuit breaker error](circuit-breaker-errors.md). -* High [indexing pressure](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/indexing-pressure-settings.md) that exceeds the [`indexing_pressure.memory.limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/indexing-pressure-settings.md#memory-limits). +* High [indexing pressure](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/pressure.md) that exceeds the [`indexing_pressure.memory.limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/pressure.md#memory-limits). ::::{tip} -If you’re using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. For more information, refer to [Monitor with AutoOps](/deploy-manage/monitor/autoops.md). +If you're using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. For more information, refer to [Monitor with AutoOps](/deploy-manage/monitor/autoops.md). :::: @@ -48,7 +48,7 @@ See [this video](https://www.youtube.com/watch?v=k3wYlRVbMSw) for a walkthrough ## Check indexing pressure [check-indexing-pressure] -To check the number of [indexing pressure](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/indexing-pressure-settings.md) rejections, use the [node stats API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-stats). +To check the number of [indexing pressure](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/pressure.md) rejections, use the [node stats API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-stats). ```console GET _nodes/stats?human&filter_path=nodes.*.indexing_pressure @@ -58,7 +58,7 @@ These stats are cumulative from node startup. Indexing pressure rejections appear as an `EsRejectedExecutionException`, and indicate that they were rejected due to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`. -These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target’s [`refresh_interval` setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md). +These errors are often related to [backlogged tasks](task-queue-backlog.md), [bulk index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) sizing, or the ingest target's [`refresh_interval` setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md). See [this video](https://www.youtube.com/watch?v=QuV8QqSfc0c) for a walkthrough of diagnosing indexing pressure rejections. diff --git a/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md b/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md index 6e3ee42569..4841f0fba2 100644 --- a/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md +++ b/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md @@ -83,7 +83,7 @@ In order to get the shards assigned we need to call the [migrate to data tiers r ``` 1. The ILM policies that were updated. - 2. The indices that were migrated to [tier preference](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation-settings.md#tier-preference-allocation-filter) routing. + 2. The indices that were migrated to [tier preference](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation.md#tier-preference-allocation-filter) routing. 3. The legacy index templates that were updated to not contain custom routing settings for the provided data attribute. 4. The composable index templates that were updated to not contain custom routing settings for the provided data attribute. 5. The component templates that were updated to not contain custom routing settings for the provided data attribute. @@ -160,7 +160,7 @@ In order to get the shards assigned we need to make sure the deployment is using ``` 1. The ILM policies that were updated. - 2. The indices that were migrated to [tier preference](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation-settings.md#tier-preference-allocation-filter) routing. + 2. The indices that were migrated to [tier preference](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation.md#tier-preference-allocation-filter) routing. 3. The legacy index templates that were updated to not contain custom routing settings for the provided data attribute. 4. The composable index templates that were updated to not contain custom routing settings for the provided data attribute. 5. The component templates that were updated to not contain custom routing settings for the provided data attribute. diff --git a/troubleshoot/elasticsearch/troubleshooting-searches.md b/troubleshoot/elasticsearch/troubleshooting-searches.md index 6af0aa1440..ab40ac2301 100644 --- a/troubleshoot/elasticsearch/troubleshooting-searches.md +++ b/troubleshoot/elasticsearch/troubleshooting-searches.md @@ -211,7 +211,7 @@ To troubleshoot queries in {{kib}}, select **Inspect** in the toolbar. Next, sel ## Check index settings [troubleshooting-searches-settings] -[Index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings) can influence search results. For example, the `index.query.default_field` setting, which determines the field that is queried when a query specifies no explicit field. Use the [get index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) to retrieve the settings for an index: +[Index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md) can influence search results. For example, the `index.query.default_field` setting, which determines the field that is queried when a query specifies no explicit field. Use the [get index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) to retrieve the settings for an index: ```console GET /my-index-000001/_settings @@ -224,7 +224,7 @@ For static settings, you need to create a new index with the correct settings. N ## Find slow queries [troubleshooting-slow-searches] -[Slow logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log-settings.md) can help pinpoint slow performing search requests. Enabling [audit logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md) on top can help determine query source. Add the following settings to the `elasticsearch.yml` configuration file to trace queries. The resulting logging is verbose, so disable these settings when not troubleshooting. +[Slow logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md) can help pinpoint slow performing search requests. Enabling [audit logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md) on top can help determine query source. Add the following settings to the `elasticsearch.yml` configuration file to trace queries. The resulting logging is verbose, so disable these settings when not troubleshooting. ```yaml xpack.security.audit.enabled: true diff --git a/troubleshoot/kibana/error-server-not-ready.md b/troubleshoot/kibana/error-server-not-ready.md index aa3993641e..aa9680d7a4 100644 --- a/troubleshoot/kibana/error-server-not-ready.md +++ b/troubleshoot/kibana/error-server-not-ready.md @@ -20,7 +20,7 @@ To troubleshoot the `Kibana server is not ready yet` error, try these steps: curl -XGET elasticsearch_ip_or_hostname:9200/_cat/indices/.kibana,.kibana_task_manager,.kibana_security_session?v=true ``` - These {{kib}}-backing indices must also not have [index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) flagging `read_only_allow_delete` or `write` [index blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block-settings.md). + These {{kib}}-backing indices must also not have [index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-settings) flagging `read_only_allow_delete` or `write` [index blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md). 3. [Shut down all {{kib}} nodes](../../deploy-manage/maintenance/start-stop-services/start-stop-kibana.md). 4. Choose any {{kib}} node, then update the config to set the [debug logging](../../deploy-manage/monitor/logging-configuration/kibana-log-settings-examples.md#change-overall-log-level). diff --git a/troubleshoot/kibana/trace-elasticsearch-query-to-the-origin-in-kibana.md b/troubleshoot/kibana/trace-elasticsearch-query-to-the-origin-in-kibana.md index d9af48f7f5..2760d18019 100644 --- a/troubleshoot/kibana/trace-elasticsearch-query-to-the-origin-in-kibana.md +++ b/troubleshoot/kibana/trace-elasticsearch-query-to-the-origin-in-kibana.md @@ -6,7 +6,7 @@ mapped_pages: # Trace an {{es}} query in {{kib}} [kibana-troubleshooting-trace-query] -Sometimes the {{es}} server might be slowed down by the execution of an expensive query. Such queries are logged to {{es}}'s [search slow log](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log-settings.md#search-slow-log) file. But there is a problem: it’s impossible to say what triggered a slow search request—a {{kib}} instance or a user accessing an {{es}} endpoint directly. To simplify the investigation of such cases, the search slow log file includes the `x-opaque-id` header, which might provide additional information about a request if it originated from {{kib}}. +Sometimes the {{es}} server might be slowed down by the execution of an expensive query. Such queries are logged to {{es}}'s [search slow log](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md#search-slow-log) file. But there is a problem: it’s impossible to say what triggered a slow search request—a {{kib}} instance or a user accessing an {{es}} endpoint directly. To simplify the investigation of such cases, the search slow log file includes the `x-opaque-id` header, which might provide additional information about a request if it originated from {{kib}}. ::::{warning} At the moment, {{kib}} can only highlight cases where a slow query originated from a {{kib}} visualization, **Lens**, **Discover**, **Maps**, or **Alerting**.