From 0f814a8f15f2b4f0e9829fe89fc5b0d77b9da1c0 Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Wed, 18 Dec 2024 15:45:27 -0500 Subject: [PATCH 01/11] pull some conceptual content out of reference areas, do some cleanup --- docs/reference/data-management.asciidoc | 39 ++++++++++++------- .../data-store-architecture.asciidoc | 16 ++++++++ docs/reference/docs.asciidoc | 6 +-- docs/reference/docs/data-replication.asciidoc | 2 +- docs/reference/high-availability.asciidoc | 38 +++++++++--------- docs/reference/index.asciidoc | 4 ++ docs/reference/modules/shard-ops.asciidoc | 26 ++++++++----- docs/reference/setup.asciidoc | 2 - 8 files changed, 82 insertions(+), 51 deletions(-) create mode 100644 docs/reference/data-store-architecture.asciidoc diff --git a/docs/reference/data-management.asciidoc b/docs/reference/data-management.asciidoc index 4245227a1524d..6c2a9d787cf4c 100644 --- a/docs/reference/data-management.asciidoc +++ b/docs/reference/data-management.asciidoc @@ -6,29 +6,26 @@ -- The data you store in {es} generally falls into one of two categories: -* Content: a collection of items you want to search, such as a catalog of products -* Time series data: a stream of continuously-generated timestamped data, such as log entries - -Content might be frequently updated, +* *Content*: a collection of items you want to search, such as a catalog of products +* *Time series data*: a stream of continuously-generated timestamped data, such as log entries +*Content* might be frequently updated, but the value of the content remains relatively constant over time. You want to be able to retrieve items quickly regardless of how old they are. -Time series data keeps accumulating over time, so you need strategies for +*Time series data* keeps accumulating over time, so you need strategies for balancing the value of the data against the cost of storing it. As it ages, it tends to become less important and less-frequently accessed, so you can move it to less expensive, less performant hardware. For your oldest data, what matters is that you have access to the data. It's ok if queries take longer to complete. -To help you manage your data, {es} offers you: - -* <> ({ilm-init}) to manage both indices and data streams and it is fully customisable, and -* <> which is the built-in lifecycle of data streams and addresses the most -common lifecycle management needs. +To help you manage your data, {es} offers you the following options: -preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but this feature is not subject to the support SLA of official GA features."] +* <> +* <> +* {curator-ref-current}/about.html[Elastic Curator] -**{ilm-init}** can be used to manage both indices and data streams and it allows you to: +**{ilm-init}** can be used to manage both indices and data streams. It allows you to do the following: * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}. Data older than this period can be deleted by {es}. @@ -38,12 +35,24 @@ Data older than this period can be deleted by {es}. for your older indices while reducing operating costs and maintaining search performance. * Perform <> of data stored on less-performant hardware. -**Data stream lifecycle** is less feature rich but is focused on simplicity, so it allows you to easily: +**Data stream lifecycle** is less feature rich but is focused on simplicity. It allows you to do the following: * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}. Data older than this period can be deleted by {es} at a later time. -* Improve the performance of your data stream by performing background operations that will optimise the way your data -stream is stored. +* Improve the performance of your data stream by performing background operations that will optimise the way your data stream is stored. + +**Elastic Curator** is a tool that allows you to manage your indices and snapshots using user-defined filters and predefined actions. If ILM provides the functionality to manage your index lifecycle, and you have at least a Basic license, consider using ILM in place of Curator. Many stack components make use of ILM by default. {curator-ref-current}/ilm.html[Learn more]. + +NOTE: <> is a deprecated Elasticsearch feature that allows you to manage the amount of data that is stored in your cluster, similar to the downsampling functionality of {ilm-init} and data stream lifecycle. This feature should not be used for new deployments. + +[TIP] +==== +{ilm-init} is not available on {es-serverless}. + +In an {cloud} or self-managed environment, ILM lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {es-serverless} eliminates this complexity by optimizing your cluster performance for you. + +Data stream lifecycle is an optimized lifecycle tool that lets you focus on the most common lifecycle management needs, without unnecessary hardware-centric concepts like data tiers. +==== -- include::ilm/index.asciidoc[] diff --git a/docs/reference/data-store-architecture.asciidoc b/docs/reference/data-store-architecture.asciidoc new file mode 100644 index 0000000000000..c6c0564686abb --- /dev/null +++ b/docs/reference/data-store-architecture.asciidoc @@ -0,0 +1,16 @@ += Data store architecture + +[partintro] +-- + +{es} is a distributed document store. Instead of storing information as rows of columnar data, {es} stores complex data structures that have been serialized as JSON documents. When you have multiple {es} nodes in a cluster, stored documents are distributed across the cluster and can be accessed immediately +from any node. + +The topics in this section provides information about the architecture of {es} and how it stores and retrieves data: + +* <>: Learn about the basic building blocks of an {es} cluster, including nodes, shards, primaries, and replicas. +* <>: Learn how {es} replicates read and write operations across shards and shard copies. +* <>: Learn how {es} allocates and balances shards across nodes. +-- +include::docs/data-replication.asciidoc[leveloffset=-1] +include::modules/shard-ops.asciidoc[] \ No newline at end of file diff --git a/docs/reference/docs.asciidoc b/docs/reference/docs.asciidoc index ff2d823410a6d..1703f033ad953 100644 --- a/docs/reference/docs.asciidoc +++ b/docs/reference/docs.asciidoc @@ -1,9 +1,7 @@ [[docs]] == Document APIs -This section starts with a short introduction to {es}'s <>, followed by a detailed description of the following CRUD -APIs: +This section describes the following CRUD APIs: .Single document APIs * <> @@ -18,8 +16,6 @@ APIs: * <> * <> -include::docs/data-replication.asciidoc[] - include::docs/index_.asciidoc[] include::docs/get.asciidoc[] diff --git a/docs/reference/docs/data-replication.asciidoc b/docs/reference/docs/data-replication.asciidoc index 2c1a16c81d011..6ee266070e727 100644 --- a/docs/reference/docs/data-replication.asciidoc +++ b/docs/reference/docs/data-replication.asciidoc @@ -1,6 +1,6 @@ [[docs-replication]] -=== Reading and Writing documents +=== Reading and writing documents [discrete] ==== Introduction diff --git a/docs/reference/high-availability.asciidoc b/docs/reference/high-availability.asciidoc index 2f34b6bc1bb21..37e2a38aa0f2c 100644 --- a/docs/reference/high-availability.asciidoc +++ b/docs/reference/high-availability.asciidoc @@ -3,28 +3,28 @@ [partintro] -- -Your data is important to you. Keeping it safe and available is important -to {es}. Sometimes your cluster may experience hardware failure or a power -loss. To help you plan for this, {es} offers a number of features -to achieve high availability despite failures. +Your data is important to you. Keeping it safe and available is important to Elastic. Sometimes your cluster may experience hardware failure or a power loss. To help you plan for this, {es} offers a number of features to achieve high availability despite failures. Depending on your deployment type, you might need to provision servers in different zones or configure external repositories to meet your organization's availability needs. -* With proper planning, a cluster can be - <> to many of the - things that commonly go wrong, from the loss of a single node or network - connection right up to a zone-wide outage such as power loss. +* *<>* ++ +Distributed systems like Elasticsearch are designed to keep working even if some of their components have failed. An Elasticsearch cluster can continue operating normally if some of its nodes are unavailable or disconnected, as long as there are enough well-connected nodes to take over the unavailable node's responsibilities. ++ +If you're designing a smaller cluster, you might focus on making your cluster resilient to single-node failures. Designers of larger clusters must also consider cases where multiple nodes fail at the same time. +// need to improve connections to ECE, EC hosted, ECK pod/zone docs in the child topics -* You can use <> to replicate data to a remote _follower_ - cluster which may be in a different data centre or even on a different - continent from the leader cluster. The follower cluster acts as a hot - standby, ready for you to fail over in the event of a disaster so severe that - the leader cluster fails. The follower cluster can also act as a geo-replica - to serve searches from nearby clients. +* *<>* ++ +To effectively distribute read and write operations across nodes, the nodes in a cluster need good, reliable connections to each other. To provide better connections, you typically co-locate the nodes in the same data center or nearby data centers. ++ +Co-locating nodes in a single location exposes you to the risk of a single outage taking your entire cluster offline. To maintain high availability, you can prepare a second cluster that can take over in case of disaster by implementing {ccr} (CCR). ++ +CCR provides a way to automatically synchronize indices from a leader cluster to a follower cluster. This cluster could be in a different data center or even a different content from the leader cluster. If the primary cluster fails, the secondary cluster can take over. ++ +TIP: You can also use CCR to create secondary clusters to serve read requests in geo-proximity to your users. -* The last line of defence against data loss is to take - <> of your cluster so that you can - restore a completely fresh copy of it elsewhere if needed. +* *<>* ++ +Take snapshots of your cluster that can be restored in case of failure. -- include::high-availability/cluster-design.asciidoc[] - -include::ccr/index.asciidoc[] diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index 18052cfb64e8f..8e1c211eb9426 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -76,8 +76,12 @@ include::autoscaling/index.asciidoc[] include::snapshot-restore/index.asciidoc[] +include::ccr/index.asciidoc[leveloffset=-1] + // reference +include::data-store-architecture.asciidoc[] + include::rest-api/index.asciidoc[] include::commands/index.asciidoc[] diff --git a/docs/reference/modules/shard-ops.asciidoc b/docs/reference/modules/shard-ops.asciidoc index c0e5ee6a220f0..66ceebcfa0319 100644 --- a/docs/reference/modules/shard-ops.asciidoc +++ b/docs/reference/modules/shard-ops.asciidoc @@ -1,5 +1,5 @@ [[shard-allocation-relocation-recovery]] -=== Shard allocation, relocation, and recovery +== Shard allocation, relocation, and recovery Each <> in Elasticsearch is divided into one or more <>. Each document in an index belongs to a single shard. @@ -12,14 +12,16 @@ Over the course of normal operation, Elasticsearch allocates shard copies to nod TIP: To learn about optimizing the number and size of shards in your cluster, refer to <>. To learn about how read and write operations are replicated across shards and shard copies, refer to <>. +[discrete] [[shard-allocation]] -==== Shard allocation +=== Shard allocation include::{es-ref-dir}/modules/shard-allocation-desc.asciidoc[] By default, the primary and replica shard copies for an index can be allocated to any node in the cluster, and may be relocated to rebalance the cluster. -===== Adjust shard allocation settings +[discrete] +==== Adjust shard allocation settings You can control how shard copies are allocated using the following settings: @@ -27,7 +29,8 @@ You can control how shard copies are allocated using the following settings: - <>: Use these settings to control how the shard copies for a specific index are allocated. For example, you might want to allocate an index to a node in a specific data tier, or to an node with specific attributes. -===== Monitor shard allocation +[discrete] +==== Monitor shard allocation If a shard copy is unassigned, it means that the shard copy is not allocated to any node in the cluster. This can happen if there are not enough nodes in the cluster to allocate the shard copy, or if the shard copy can't be allocated to any node that satisfies the shard allocation filtering rules. When a shard copy is unassigned, your cluster is considered unhealthy and returns a yellow or red cluster health status. @@ -39,12 +42,14 @@ You can use the following APIs to monitor shard allocation: <>. +[discrete] [[shard-recovery]] -==== Shard recovery +=== Shard recovery include::{es-ref-dir}/modules/shard-recovery-desc.asciidoc[] -===== Adjust shard recovery settings +[discrete] +==== Adjust shard recovery settings To control how shards are recovered, for example the resources that can be used by recovery operations, and which indices should be prioritized for recovery, you can adjust the following settings: @@ -54,21 +59,24 @@ To control how shards are recovered, for example the resources that can be used Shard recovery operations also respect general shard allocation settings. -===== Monitor shard recovery +[discrete] +==== Monitor shard recovery You can use the following APIs to monitor shard allocation: - View a list of in-progress and completed recoveries using the <> - View detailed information about a specific recovery using the <> +[discrete] [[shard-relocation]] -==== Shard relocation +=== Shard relocation Shard relocation is the process of moving shard copies from one node to another. This can happen when a node joins or leaves the cluster, or when the cluster is rebalancing. When a shard copy is relocated, it is created as a new shard copy on the target node. When the shard copy is fully allocated and recovered, the old shard copy is deleted. If the shard copy being relocated is a primary, then the new shard copy is marked as primary before the old shard copy is deleted. -===== Adjust shard relocation settings +[discrete] +==== Adjust shard relocation settings You can control how and when shard copies are relocated. For example, you can adjust the rebalancing settings that control when shard copies are relocated to balance the cluster, or the high watermark for disk-based shard allocation that can trigger relocation. These settings are part of the <>. diff --git a/docs/reference/setup.asciidoc b/docs/reference/setup.asciidoc index a284e563917c3..80828fdbfbb02 100644 --- a/docs/reference/setup.asciidoc +++ b/docs/reference/setup.asciidoc @@ -83,8 +83,6 @@ include::modules/indices/search-settings.asciidoc[] include::settings/security-settings.asciidoc[] -include::modules/shard-ops.asciidoc[] - include::modules/indices/request_cache.asciidoc[] include::settings/snapshot-settings.asciidoc[] From f3da5842c3125d3342987b705614c37d827170d4 Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Wed, 18 Dec 2024 15:57:05 -0500 Subject: [PATCH 02/11] more --- .../data-store-architecture.asciidoc | 4 +- docs/reference/intro.asciidoc | 41 ++---------------- docs/reference/nodes-shards.asciidoc | 43 +++++++++++++++++++ 3 files changed, 50 insertions(+), 38 deletions(-) create mode 100644 docs/reference/nodes-shards.asciidoc diff --git a/docs/reference/data-store-architecture.asciidoc b/docs/reference/data-store-architecture.asciidoc index c6c0564686abb..4ee75c15562ea 100644 --- a/docs/reference/data-store-architecture.asciidoc +++ b/docs/reference/data-store-architecture.asciidoc @@ -8,9 +8,11 @@ from any node. The topics in this section provides information about the architecture of {es} and how it stores and retrieves data: -* <>: Learn about the basic building blocks of an {es} cluster, including nodes, shards, primaries, and replicas. +* <>: Learn about the basic building blocks of an {es} cluster, including nodes, shards, primaries, and replicas. * <>: Learn how {es} replicates read and write operations across shards and shard copies. * <>: Learn how {es} allocates and balances shards across nodes. -- + +include::nodes-shards.asciidoc[] include::docs/data-replication.asciidoc[leveloffset=-1] include::modules/shard-ops.asciidoc[] \ No newline at end of file diff --git a/docs/reference/intro.asciidoc b/docs/reference/intro.asciidoc index e0100b1c5640b..391439df2ae85 100644 --- a/docs/reference/intro.asciidoc +++ b/docs/reference/intro.asciidoc @@ -397,51 +397,18 @@ geographic location of your users and your resources. [[use-multiple-nodes-shards]] ==== Use multiple nodes and shards -[NOTE] -==== -Nodes and shards are what make {es} distributed and scalable. +When you move to production, you need to introduce multiple nodes and shards to your cluster. Nodes and shards are what make {es} distributed and scalable. The size and number of these nodes and shards depends on your data, your use case, and your budget. -These concepts aren’t essential if you’re just getting started. How you <> in production determines what you need to know: +These concepts aren't essential if you're just getting started. How you <> in production determines what you need to know: * *Self-managed {es}*: You are responsible for setting up and managing nodes, clusters, shards, and replicas. This includes managing the underlying infrastructure, scaling, and ensuring high availability through failover and backup strategies. * *Elastic Cloud*: Elastic can autoscale resources in response to workload changes. Choose from different deployment types to apply sensible defaults for your use case. A basic understanding of nodes, shards, and replicas is still important. -* *Elastic Cloud Serverless*: You don’t need to worry about nodes, shards, or replicas. These resources are 100% automated +* *Elastic Cloud Serverless*: You don't need to worry about nodes, shards, or replicas. These resources are 100% automated on the serverless platform, which is designed to scale with your workload. -==== - -You can add servers (_nodes_) to a cluster to increase capacity, and {es} automatically distributes your data and query load -across all of the available nodes. - -Elastic is able to distribute your data across nodes by subdividing an index into _shards_. Each index in {es} is a grouping -of one or more physical shards, where each shard is a self-contained Lucene index containing a subset of the documents in -the index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple -nodes, {es} increases indexing and query capacity. - -There are two types of shards: _primaries_ and _replicas_. Each document in an index belongs to one primary shard. A replica -shard is a copy of a primary shard. Replicas maintain redundant copies of your data across the nodes in your cluster. -This protects against hardware failure and increases capacity to serve read requests like searching or retrieving a document. - -[TIP] -==== -The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can -be changed at any time, without interrupting indexing or query operations. -==== - -Shard copies in your cluster are automatically balanced across nodes to provide scale and high availability. All nodes are -aware of all the other nodes in the cluster and can forward client requests to the appropriate node. This allows {es} -to distribute indexing and query load across the cluster. - -If you’re exploring {es} for the first time or working in a development environment, then you can use a cluster with a single node and create indices -with only one shard. However, in a production environment, you should build a cluster with multiple nodes and indices -with multiple shards to increase performance and resilience. - -// TODO - diagram -To learn about optimizing the number and size of shards in your cluster, refer to <>. -To learn about how read and write operations are replicated across shards and shard copies, refer to <>. -To adjust how shards are allocated and balanced across nodes, refer to <>. +Learn more about <>. [discrete] [[ccr-disaster-recovery-geo-proximity]] diff --git a/docs/reference/nodes-shards.asciidoc b/docs/reference/nodes-shards.asciidoc new file mode 100644 index 0000000000000..11095ed7b7eb3 --- /dev/null +++ b/docs/reference/nodes-shards.asciidoc @@ -0,0 +1,43 @@ +[[nodes-shards]] +== Nodes and shards + +[NOTE] +==== +Nodes and shards are what make {es} distributed and scalable. +These concepts aren't essential if you're just getting started. How you <> in production determines what you need to know: + +* *Self-managed {es}*: You are responsible for setting up and managing nodes, clusters, shards, and replicas. This includes managing the underlying infrastructure, scaling, and ensuring high availability through failover and backup strategies. +* *Elastic Cloud*: Elastic can autoscale resources in response to workload changes. Choose from different deployment types to apply sensible defaults for your use case. A basic understanding of nodes, shards, and replicas is still important. +* *Elastic Cloud Serverless*: You don't need to worry about nodes, shards, or replicas. These resources are 100% automated on the serverless platform, which is designed to scale with your workload. +==== + +You can add servers (_nodes_) to a cluster to increase capacity, and {es} automatically distributes your data and query load across all of the available nodes. + +Elastic is able to distribute your data across nodes by subdividing an index into _shards_. Each index in {es} is a grouping +of one or more physical shards, where each shard is a self-contained Lucene index containing a subset of the documents in +the index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple +nodes, {es} increases indexing and query capacity. + +There are two types of shards: _primaries_ and _replicas_. Each document in an index belongs to one primary shard. A replica +shard is a copy of a primary shard. Replicas maintain redundant copies of your data across the nodes in your cluster. +This protects against hardware failure and increases capacity to serve read requests like searching or retrieving a document. + +[TIP] +==== +The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can +be changed at any time, without interrupting indexing or query operations. +==== + +Shard copies in your cluster are automatically balanced across nodes to provide scale and high availability. All nodes are +aware of all the other nodes in the cluster and can forward client requests to the appropriate node. This allows {es} +to distribute indexing and query load across the cluster. + +If you're exploring {es} for the first time or working in a development environment, then you can use a cluster with a single node and create indices +with only one shard. However, in a production environment, you should build a cluster with multiple nodes and indices +with multiple shards to increase performance and resilience. + +// TODO - diagram + +* To learn about optimizing the number and size of shards in your cluster, refer to <>. +* To learn about how read and write operations are replicated across shards and shard copies, refer to <>. +* To adjust how shards are allocated and balanced across nodes, refer to <>. \ No newline at end of file From b3b762eff429744e913d408d4856918852f25097 Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Mon, 30 Dec 2024 14:51:18 -0500 Subject: [PATCH 03/11] dangling-indices-move --- docs/reference/commands/node-tool.asciidoc | 2 +- .../data-streams/downsampling.asciidoc | 39 ++++++++++++++++ .../indices/downsample-data-stream.asciidoc | 44 ++----------------- docs/reference/modules/gateway.asciidoc | 15 +------ .../modules/indices/fielddata.asciidoc | 18 +++----- docs/reference/modules/node.asciidoc | 2 +- docs/reference/monitoring/index.asciidoc | 3 ++ docs/reference/redirects.asciidoc | 2 +- docs/reference/setup.asciidoc | 2 - 9 files changed, 54 insertions(+), 73 deletions(-) diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index cdd2bb8f0f9d7..dc0e730c6cb48 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -309,7 +309,7 @@ Unsafe cluster bootstrapping is only possible if there is at least one surviving master-eligible node. If there are no remaining master-eligible nodes then the cluster metadata is completely lost. However, the individual data nodes also contain a copy of the index metadata corresponding with their shards. This sometimes allows a new cluster to import these shards as -<>. You can sometimes +<>. You can sometimes recover some indices after the loss of all main-eligible nodes in a cluster by creating a new cluster and then using the `elasticsearch-node detach-cluster` command to move any surviving nodes into this new cluster. Once the new cluster is fully formed, diff --git a/docs/reference/data-streams/downsampling.asciidoc b/docs/reference/data-streams/downsampling.asciidoc index 0b08b0972f9a1..10a0241cf0732 100644 --- a/docs/reference/data-streams/downsampling.asciidoc +++ b/docs/reference/data-streams/downsampling.asciidoc @@ -72,6 +72,45 @@ the granularity of `cold` archival data to monthly or less. .Downsampled metrics series image::images/data-streams/time-series-downsampled.png[align="center"] +[discrete] +[[downsample-api-process]] +==== The downsampling process + +The downsampling operation traverses the source TSDS index and performs the +following steps: + +. Creates a new document for each value of the `_tsid` field and each +`@timestamp` value, rounded to the `fixed_interval` defined in the downsample +configuration. +. For each new document, copies all <> from the source index to the target index. Dimensions in a +TSDS are constant, so this is done only once per bucket. +. For each <> field, computes aggregations +for all documents in the bucket. Depending on the metric type of each metric +field a different set of pre-aggregated results is stored: + +** `gauge`: The `min`, `max`, `sum`, and `value_count` are stored; `value_count` +is stored as type `aggregate_metric_double`. +** `counter`: The `last_value` is stored. +. For all other fields, the most recent value is copied to the target index. + +[discrete] +[[downsample-api-mappings]] +==== Source and target index field mappings + +Fields in the target, downsampled index are created based on fields in the +original source index, as follows: + +. All fields mapped with the `time-series-dimension` parameter are created in +the target downsample index with the same mapping as in the source index. +. All fields mapped with the `time_series_metric` parameter are created +in the target downsample index with the same mapping as in the source +index. An exception is that for fields mapped as `time_series_metric: gauge` +the field type is changed to `aggregate_metric_double`. +. All other fields that are neither dimensions nor metrics (that is, label +fields), are created in the target downsample index with the same mapping +that they had in the source index. + [discrete] [[running-downsampling]] === Running downsampling on time series data diff --git a/docs/reference/indices/downsample-data-stream.asciidoc b/docs/reference/indices/downsample-data-stream.asciidoc index 5ace4e03dfb66..39db7f8f4f046 100644 --- a/docs/reference/indices/downsample-data-stream.asciidoc +++ b/docs/reference/indices/downsample-data-stream.asciidoc @@ -75,6 +75,8 @@ DELETE _index_template/* //// // end::downsample-example[] +Check the <> documentation for an overview, details about the downsampling process, and examples of running downsampling manually and as part of an ILM policy. + [[downsample-api-request]] ==== {api-request-title} @@ -115,44 +117,4 @@ to aggregate the original time series index. For example, `60m` produces a document for each 60 minute (hourly) interval. This follows standard time formatting syntax as used elsewhere in {es}. + -NOTE: Smaller, more granular intervals take up proportionally more space. - -[[downsample-api-process]] -==== The downsampling process - -The downsampling operation traverses the source TSDS index and performs the -following steps: - -. Creates a new document for each value of the `_tsid` field and each -`@timestamp` value, rounded to the `fixed_interval` defined in the downsample -configuration. -. For each new document, copies all <> from the source index to the target index. Dimensions in a -TSDS are constant, so this is done only once per bucket. -. For each <> field, computes aggregations -for all documents in the bucket. Depending on the metric type of each metric -field a different set of pre-aggregated results is stored: - -** `gauge`: The `min`, `max`, `sum`, and `value_count` are stored; `value_count` -is stored as type `aggregate_metric_double`. -** `counter`: The `last_value` is stored. -. For all other fields, the most recent value is copied to the target index. - -[[downsample-api-mappings]] -==== Source and target index field mappings - -Fields in the target, downsampled index are created based on fields in the -original source index, as follows: - -. All fields mapped with the `time-series-dimension` parameter are created in -the target downsample index with the same mapping as in the source index. -. All fields mapped with the `time_series_metric` parameter are created -in the target downsample index with the same mapping as in the source -index. An exception is that for fields mapped as `time_series_metric: gauge` -the field type is changed to `aggregate_metric_double`. -. All other fields that are neither dimensions nor metrics (that is, label -fields), are created in the target downsample index with the same mapping -that they had in the source index. - -Check the <> documentation for an overview and -examples of running downsampling manually and as part of an ILM policy. +NOTE: Smaller, more granular intervals take up proportionally more space. \ No newline at end of file diff --git a/docs/reference/modules/gateway.asciidoc b/docs/reference/modules/gateway.asciidoc index bf7e6de64f093..12490a3d43216 100644 --- a/docs/reference/modules/gateway.asciidoc +++ b/docs/reference/modules/gateway.asciidoc @@ -36,17 +36,4 @@ These settings can be configured in `elasticsearch.yml` as follows: gateway.expected_data_nodes: 3 gateway.recover_after_time: 600s gateway.recover_after_data_nodes: 3 --------------------------------------------------- - -[[dangling-indices]] -==== Dangling indices - -When a node joins the cluster, if it finds any shards stored in its local -data directory that do not already exist in the cluster, it will consider -those shards to belong to a "dangling" index. You can list, import or -delete dangling indices using the <>. - -NOTE: The API cannot offer any guarantees as to whether the imported data -truly represents the latest state of the data when the index was still part -of the cluster. +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/modules/indices/fielddata.asciidoc b/docs/reference/modules/indices/fielddata.asciidoc index 1383bf74d6d4c..688685c0a2247 100644 --- a/docs/reference/modules/indices/fielddata.asciidoc +++ b/docs/reference/modules/indices/fielddata.asciidoc @@ -5,10 +5,6 @@ The field data cache contains <> and <>. This behavior can be configured. @@ -20,16 +16,12 @@ at the cost of rebuilding the cache as needed. If the circuit breaker limit is reached, further requests that increase the cache size will be prevented. In this case you should manually <>. +TIP: You can monitor memory usage for field data as well as the field data circuit +breaker using +the <> or the <>. + `indices.fielddata.cache.size`:: (<>) The max size of the field data cache, eg `38%` of node heap space, or an absolute value, eg `12GB`. Defaults to unbounded. If you choose to set it, -it should be smaller than <> limit. - -[discrete] -[[fielddata-monitoring]] -==== Monitoring field data - -You can monitor memory usage for field data as well as the field data circuit -breaker using -the <> or the <>. +it should be smaller than <> limit. \ No newline at end of file diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index 022e8b5d1e2fe..e57297dd35394 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -420,7 +420,7 @@ Similarly, each master-eligible node maintains the following data on disk: Each node checks the contents of its data path at startup. If it discovers unexpected data then it will refuse to start. This is to avoid importing -unwanted <> which can lead +unwanted <> which can lead to a red cluster health. To be more precise, nodes without the `data` role will refuse to start if they find any shard data on disk at startup, and nodes without both the `master` and `data` roles will refuse to start if they have any diff --git a/docs/reference/monitoring/index.asciidoc b/docs/reference/monitoring/index.asciidoc index 1b83f4c11ba54..c6380ab676a23 100644 --- a/docs/reference/monitoring/index.asciidoc +++ b/docs/reference/monitoring/index.asciidoc @@ -9,6 +9,7 @@ performance of your {es} cluster. * <> * <> +* <> * <> * <> * <> @@ -23,6 +24,8 @@ include::overview.asciidoc[] include::how-monitoring-works.asciidoc[] +include::setup/logging-config.asciidoc[] + include::production.asciidoc[] include::configuring-elastic-agent.asciidoc[] diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index c3bf84fa600d2..a7bbf41c849e9 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -1215,7 +1215,7 @@ See <>. [role="exclude",id="modules-gateway-dangling-indices"] === Dangling indices -See <>. +See <>. [role="exclude",id="shards-allocation"] === Cluster-level shard allocation diff --git a/docs/reference/setup.asciidoc b/docs/reference/setup.asciidoc index 80828fdbfbb02..5bce5da19c055 100644 --- a/docs/reference/setup.asciidoc +++ b/docs/reference/setup.asciidoc @@ -65,8 +65,6 @@ include::settings/license-settings.asciidoc[] include::modules/gateway.asciidoc[] -include::setup/logging-config.asciidoc[] - include::settings/ml-settings.asciidoc[] include::settings/inference-settings.asciidoc[] From 34d60281c6e84bbb5f4672217ccbc7e18531c32d Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Mon, 30 Dec 2024 16:08:13 -0500 Subject: [PATCH 04/11] shard request cache --- docs/reference/commands/node-tool.asciidoc | 2 +- .../data-store-architecture.asciidoc | 6 +- .../modules/discovery/publishing.asciidoc | 33 +- .../modules/indices/request_cache.asciidoc | 137 +----- docs/reference/modules/node.asciidoc | 389 +--------------- docs/reference/monitoring/index.asciidoc | 4 +- docs/reference/node-roles.asciidoc | 437 ++++++++++++++++++ docs/reference/redirects.asciidoc | 2 +- docs/reference/settings/ml-settings.asciidoc | 13 +- docs/reference/setup/logging-config.asciidoc | 2 +- docs/reference/shard-request-cache.asciidoc | 134 ++++++ 11 files changed, 631 insertions(+), 528 deletions(-) create mode 100644 docs/reference/node-roles.asciidoc create mode 100644 docs/reference/shard-request-cache.asciidoc diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index dc0e730c6cb48..cdd2bb8f0f9d7 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -309,7 +309,7 @@ Unsafe cluster bootstrapping is only possible if there is at least one surviving master-eligible node. If there are no remaining master-eligible nodes then the cluster metadata is completely lost. However, the individual data nodes also contain a copy of the index metadata corresponding with their shards. This sometimes allows a new cluster to import these shards as -<>. You can sometimes +<>. You can sometimes recover some indices after the loss of all main-eligible nodes in a cluster by creating a new cluster and then using the `elasticsearch-node detach-cluster` command to move any surviving nodes into this new cluster. Once the new cluster is fully formed, diff --git a/docs/reference/data-store-architecture.asciidoc b/docs/reference/data-store-architecture.asciidoc index 4ee75c15562ea..dc99c64a9b7f3 100644 --- a/docs/reference/data-store-architecture.asciidoc +++ b/docs/reference/data-store-architecture.asciidoc @@ -9,10 +9,14 @@ from any node. The topics in this section provides information about the architecture of {es} and how it stores and retrieves data: * <>: Learn about the basic building blocks of an {es} cluster, including nodes, shards, primaries, and replicas. +* <>: Learn about the different roles that nodes can have in an {es} cluster. * <>: Learn how {es} replicates read and write operations across shards and shard copies. * <>: Learn how {es} allocates and balances shards across nodes. +* <>: Learn how {es} caches search requests to improve performance. -- include::nodes-shards.asciidoc[] +include::node-roles.asciidoc[] include::docs/data-replication.asciidoc[leveloffset=-1] -include::modules/shard-ops.asciidoc[] \ No newline at end of file +include::modules/shard-ops.asciidoc[] +include::shard-request-cache.asciidoc[leveloffset=-1] \ No newline at end of file diff --git a/docs/reference/modules/discovery/publishing.asciidoc b/docs/reference/modules/discovery/publishing.asciidoc index af664585085c2..22fed9528e615 100644 --- a/docs/reference/modules/discovery/publishing.asciidoc +++ b/docs/reference/modules/discovery/publishing.asciidoc @@ -1,5 +1,23 @@ +[[cluster-state-overview]] +=== Cluster state + +The _cluster state_ is an internal data structure which keeps track of a +variety of information needed by every node, including: + +* The identity and attributes of the other nodes in the cluster + +* Cluster-wide settings + +* Index metadata, including the mapping and settings for each index + +* The location and status of every shard copy in the cluster + +The elected master node ensures that every node in the cluster has a copy of +the same cluster state. The <> lets you retrieve a +representation of this internal state for debugging or diagnostic purposes. + [[cluster-state-publishing]] -=== Publishing the cluster state +==== Publishing the cluster state The elected master node is the only node in a cluster that can make changes to the cluster state. The elected master node processes one batch of cluster state @@ -58,3 +76,16 @@ speed of the storage on each master-eligible node, as well as the reliability and latency of the network interconnections between all nodes in the cluster. You must therefore ensure that the storage and networking available to the nodes in your cluster are good enough to meet your performance goals. + +[[dangling-indices]] +==== Dangling indices + +When a node joins the cluster, if it finds any shards stored in its local +data directory that do not already exist in the cluster state, it will consider +those shards to belong to a "dangling" index. You can list, import or +delete dangling indices using the <>. + +NOTE: The API cannot offer any guarantees as to whether the imported data +truly represents the latest state of the data when the index was still part +of the cluster. \ No newline at end of file diff --git a/docs/reference/modules/indices/request_cache.asciidoc b/docs/reference/modules/indices/request_cache.asciidoc index 4d4d349c685a1..088adfcacfcbf 100644 --- a/docs/reference/modules/indices/request_cache.asciidoc +++ b/docs/reference/modules/indices/request_cache.asciidoc @@ -1,4 +1,4 @@ -[[shard-request-cache]] +[[shard-request-cache-settings]] === Shard request cache settings When a search request is run against an index or against many indices, each @@ -10,139 +10,16 @@ The shard-level request cache module caches the local results on each shard. This allows frequently used (and potentially heavy) search requests to return results almost instantly. The requests cache is a very good fit for the logging use case, where only the most recent index is being actively updated -- -results from older indices will be served directly from the cache. +results from older indices will be served directly from the cache. You can use shard request cache settings to control the size and expiration of the cache. -[IMPORTANT] -=================================== - -By default, the requests cache will only cache the results of search requests -where `size=0`, so it will not cache `hits`, -but it will cache `hits.total`, <>, and -<>. - -Most queries that use `now` (see <>) cannot be cached. - -Scripted queries that use the API calls which are non-deterministic, such as -`Math.random()` or `new Date()` are not cached. -=================================== - -[discrete] -==== Cache invalidation - -The cache is smart -- it keeps the same _near real-time_ promise as uncached -search. - -Cached results are invalidated automatically whenever the shard refreshes to -pick up changes to the documents or when you update the mapping. In other -words you will always get the same results from the cache as you would for an -uncached search request. - -The longer the refresh interval, the longer that cached entries will remain -valid even if there are changes to the documents. If the cache is full, the -least recently used cache keys will be evicted. - -The cache can be expired manually with the <>: - -[source,console] ------------------------- -POST /my-index-000001,my-index-000002/_cache/clear?request=true ------------------------- -// TEST[s/^/PUT my-index-000001\nPUT my-index-000002\n/] - -[discrete] -==== Enabling and disabling caching - -The cache is enabled by default, but can be disabled when creating a new -index as follows: - -[source,console] ------------------------------ -PUT /my-index-000001 -{ - "settings": { - "index.requests.cache.enable": false - } -} ------------------------------ - -It can also be enabled or disabled dynamically on an existing index with the -<> API: - -[source,console] ------------------------------ -PUT /my-index-000001/_settings -{ "index.requests.cache.enable": true } ------------------------------ -// TEST[continued] - - -[discrete] -==== Enabling and disabling caching per request - -The `request_cache` query-string parameter can be used to enable or disable -caching on a *per-request* basis. If set, it overrides the index-level setting: - -[source,console] ------------------------------ -GET /my-index-000001/_search?request_cache=true -{ - "size": 0, - "aggs": { - "popular_colors": { - "terms": { - "field": "colors" - } - } - } -} ------------------------------ -// TEST[continued] - -Requests where `size` is greater than 0 will not be cached even if the request cache is -enabled in the index settings. To cache these requests you will need to use the -query-string parameter detailed here. - -[discrete] -==== Cache key - -A hash of the whole JSON body is used as the cache key. This means that if the JSON -changes -- for instance if keys are output in a different order -- then the -cache key will not be recognised. - -TIP: Most JSON libraries support a _canonical_ mode which ensures that JSON -keys are always emitted in the same order. This canonical mode can be used in -the application to ensure that a request is always serialized in the same way. +To learn more about the shard request cache, see <>. [discrete] ==== Cache settings -The cache is managed at the node level, and has a default maximum size of `1%` -of the heap. This can be changed in the `config/elasticsearch.yml` file with: - -[source,yaml] --------------------------------- -indices.requests.cache.size: 2% --------------------------------- - -Also, you can use the +indices.requests.cache.expire+ setting to specify a TTL -for cached results, but there should be no reason to do so. Remember that -stale results are automatically invalidated when the index is refreshed. This -setting is provided for completeness' sake only. - -[discrete] -==== Monitoring cache usage - -The size of the cache (in bytes) and the number of evictions can be viewed -by index, with the <> API: - -[source,console] ------------------------- -GET /_stats/request_cache?human ------------------------- +indices.requests.cache.size:: +(<>) The maximum size of the cache, as a percentage of the heap. Default: `1%`. -or by node with the <> API: +indices.requests.cache.expire:: +(<>) The TTL for cached results. Stale results are automatically invalidated when the index is refreshed, so you shouldn't need to use this setting. -[source,console] ------------------------- -GET /_nodes/stats/indices/request_cache?human ------------------------- diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index e57297dd35394..8f0617b30c6a8 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -1,5 +1,5 @@ [[modules-node]] -=== Nodes +=== Node settings Any time that you start an instance of {es}, you are starting a _node_. A collection of connected nodes is called a <>. If you @@ -19,7 +19,7 @@ Review our recommendations for optimizing your storage for <>. [[node-roles]] -==== Node roles +==== Node role settings You define a node's roles by setting `node.roles` in `elasticsearch.yml`. If you set `node.roles`, the node is only assigned the roles you specify. If you don't @@ -37,6 +37,10 @@ set `node.roles`, the node is assigned the following roles: * `remote_cluster_client` * `transform` +The following additional roles are available: + +* `voting_only` + [IMPORTANT] ==== If you set `node.roles`, ensure you specify every node role your cluster needs. @@ -65,386 +69,7 @@ As the cluster grows and in particular if you have large {ml} jobs or {ctransforms}, consider separating dedicated master-eligible nodes from dedicated data nodes, {ml} nodes, and {transform} nodes. -<>:: - -A node that has the `master` role, which makes it eligible to be -<>, which controls the cluster. - -<>:: - -A node that has one of several data roles. Data nodes hold data and perform data -related operations such as CRUD, search, and aggregations. A node with a generic `data` role can fill any of the specialized data node roles. - -<>:: - -A node that has the `ingest` role. Ingest nodes are able to apply an -<> to a document in order to transform and enrich the -document before indexing. With a heavy ingest load, it makes sense to use -dedicated ingest nodes and to not include the `ingest` role from nodes that have -the `master` or `data` roles. - -<>:: - -A node that has the `remote_cluster_client` role, which makes it eligible to act -as a remote client. - -<>:: - -A node that has the `ml` role. If you want to use {ml-features}, there must be -at least one {ml} node in your cluster. For more information, see -<> and {ml-docs}/index.html[Machine learning in the {stack}]. - -<>:: - -A node that has the `transform` role. If you want to use {transforms}, there -must be at least one {transform} node in your cluster. For more information, see -<> and <>. - -[NOTE] -[[coordinating-node]] -.Coordinating node -=============================================== - -Requests like search requests or bulk-indexing requests may involve data held -on different data nodes. A search request, for example, is executed in two -phases which are coordinated by the node which receives the client request -- -the _coordinating node_. - -In the _scatter_ phase, the coordinating node forwards the request to the data -nodes which hold the data. Each data node executes the request locally and -returns its results to the coordinating node. In the _gather_ phase, the -coordinating node reduces each data node's results into a single global -result set. - -Every node is implicitly a coordinating node. This means that a node that has -an explicit empty list of roles via `node.roles` will only act as a coordinating -node, which cannot be disabled. As a result, such a node needs to have enough -memory and CPU in order to deal with the gather phase. - -=============================================== - -[[master-node]] -==== Master-eligible node - -The master node is responsible for lightweight cluster-wide actions such as -creating or deleting an index, tracking which nodes are part of the cluster, -and deciding which shards to allocate to which nodes. It is important for -cluster health to have a stable master node. - -Any master-eligible node that is not a <> may -be elected to become the master node by the <>. - -IMPORTANT: Master nodes must have a `path.data` directory whose contents -persist across restarts, just like data nodes, because this is where the -cluster metadata is stored. The cluster metadata describes how to read the data -stored on the data nodes, so if it is lost then the data stored on the data -nodes cannot be read. - -[[dedicated-master-node]] -===== Dedicated master-eligible node - -It is important for the health of the cluster that the elected master node has -the resources it needs to fulfill its responsibilities. If the elected master -node is overloaded with other tasks then the cluster will not operate well. The -most reliable way to avoid overloading the master with other tasks is to -configure all the master-eligible nodes to be _dedicated master-eligible nodes_ -which only have the `master` role, allowing them to focus on managing the -cluster. Master-eligible nodes will still also behave as -<> that route requests from clients to -the other nodes in the cluster, but you should _not_ use dedicated master nodes -for this purpose. - -A small or lightly-loaded cluster may operate well if its master-eligible nodes -have other roles and responsibilities, but once your cluster comprises more -than a handful of nodes it usually makes sense to use dedicated master-eligible -nodes. - -To create a dedicated master-eligible node, set: - -[source,yaml] -------------------- -node.roles: [ master ] -------------------- - -[[voting-only-node]] -===== Voting-only master-eligible node - -A voting-only master-eligible node is a node that participates in -<> but which will not act as the cluster's -elected master node. In particular, a voting-only node can serve as a tiebreaker -in elections. - -It may seem confusing to use the term "master-eligible" to describe a -voting-only node since such a node is not actually eligible to become the master -at all. This terminology is an unfortunate consequence of history: -master-eligible nodes are those nodes that participate in elections and perform -certain tasks during cluster state publications, and voting-only nodes have the -same responsibilities even if they can never become the elected master. - -To configure a master-eligible node as a voting-only node, include `master` and -`voting_only` in the list of roles. For example to create a voting-only data -node: - -[source,yaml] -------------------- -node.roles: [ data, master, voting_only ] -------------------- - -IMPORTANT: Only nodes with the `master` role can be marked as having the -`voting_only` role. - -High availability (HA) clusters require at least three master-eligible nodes, at -least two of which are not voting-only nodes. Such a cluster will be able to -elect a master node even if one of the nodes fails. - -Voting-only master-eligible nodes may also fill other roles in your cluster. -For instance, a node may be both a data node and a voting-only master-eligible -node. A _dedicated_ voting-only master-eligible nodes is a voting-only -master-eligible node that fills no other roles in the cluster. To create a -dedicated voting-only master-eligible node, set: - -[source,yaml] -------------------- -node.roles: [ master, voting_only ] -------------------- - -Since dedicated voting-only nodes never act as the cluster's elected master, -they may require less heap and a less powerful CPU than the true master nodes. -However all master-eligible nodes, including voting-only nodes, are on the -critical path for <>. Cluster state updates are usually independent of -performance-critical workloads such as indexing or searches, but they are -involved in management activities such as index creation and rollover, mapping -updates, and recovery after a failure. The performance characteristics of these -activities are a function of the speed of the storage on each master-eligible -node, as well as the reliability and latency of the network interconnections -between the elected master node and the other nodes in the cluster. You must -therefore ensure that the storage and networking available to the nodes in your -cluster are good enough to meet your performance goals. - -[[data-node]] -==== Data nodes - -Data nodes hold the shards that contain the documents you have indexed. Data -nodes handle data related operations like CRUD, search, and aggregations. -These operations are I/O-, memory-, and CPU-intensive. It is important to -monitor these resources and to add more data nodes if they are overloaded. - -The main benefit of having dedicated data nodes is the separation of the master -and data roles. - -In a multi-tier deployment architecture, you use specialized data roles to -assign data nodes to specific tiers: `data_content`,`data_hot`, `data_warm`, -`data_cold`, or `data_frozen`. A node can belong to multiple tiers. - -If you want to include a node in all tiers, or if your cluster does not use multiple tiers, then you can use the generic `data` role. - -include::../how-to/shard-limits.asciidoc[] - -WARNING: If you assign a node to a specific tier using a specialized data role, then you shouldn't also assign it the generic `data` role. The generic `data` role takes precedence over specialized data roles. - -[[generic-data-node]] -===== Generic data node - -Generic data nodes are included in all content tiers. - -To create a dedicated generic data node, set: -[source,yaml] ----- -node.roles: [ data ] ----- - -[[data-content-node]] -===== Content data node - -Content data nodes are part of the content tier. -include::{es-ref-dir}/datatiers.asciidoc[tag=content-tier] - -To create a dedicated content node, set: -[source,yaml] ----- -node.roles: [ data_content ] ----- - -[[data-hot-node]] -===== Hot data node - -Hot data nodes are part of the hot tier. -include::{es-ref-dir}/datatiers.asciidoc[tag=hot-tier] - -To create a dedicated hot node, set: -[source,yaml] ----- -node.roles: [ data_hot ] ----- - -[[data-warm-node]] -===== Warm data node - -Warm data nodes are part of the warm tier. -include::{es-ref-dir}/datatiers.asciidoc[tag=warm-tier] - -To create a dedicated warm node, set: -[source,yaml] ----- -node.roles: [ data_warm ] ----- - -[[data-cold-node]] -===== Cold data node - -Cold data nodes are part of the cold tier. -include::{es-ref-dir}/datatiers.asciidoc[tag=cold-tier] - -To create a dedicated cold node, set: -[source,yaml] ----- -node.roles: [ data_cold ] ----- - -[[data-frozen-node]] -===== Frozen data node - -Frozen data nodes are part of the frozen tier. -include::{es-ref-dir}/datatiers.asciidoc[tag=frozen-tier] - -To create a dedicated frozen node, set: -[source,yaml] ----- -node.roles: [ data_frozen ] ----- - -[[node-ingest-node]] -==== Ingest node - -Ingest nodes can execute pre-processing pipelines, composed of one or more -ingest processors. Depending on the type of operations performed by the ingest -processors and the required resources, it may make sense to have dedicated -ingest nodes, that will only perform this specific task. - -To create a dedicated ingest node, set: - -[source,yaml] ----- -node.roles: [ ingest ] ----- - -[[coordinating-only-node]] -==== Coordinating only node - -If you take away the ability to be able to handle master duties, to hold data, -and pre-process documents, then you are left with a _coordinating_ node that -can only route requests, handle the search reduce phase, and distribute bulk -indexing. Essentially, coordinating only nodes behave as smart load balancers. - -Coordinating only nodes can benefit large clusters by offloading the -coordinating node role from data and master-eligible nodes. They join the -cluster and receive the full <>, like every other -node, and they use the cluster state to route requests directly to the -appropriate place(s). - -WARNING: Adding too many coordinating only nodes to a cluster can increase the -burden on the entire cluster because the elected master node must await -acknowledgement of cluster state updates from every node! The benefit of -coordinating only nodes should not be overstated -- data nodes can happily -serve the same purpose. - -To create a dedicated coordinating node, set: - -[source,yaml] ----- -node.roles: [ ] ----- - -[[remote-node]] -==== Remote-eligible node - -A remote-eligible node acts as a cross-cluster client and connects to -<>. Once connected, you can search -remote clusters using <>. You can also sync -data between clusters using <>. - -[source,yaml] ----- -node.roles: [ remote_cluster_client ] ----- - -[[ml-node]] -==== [xpack]#Machine learning node# - -{ml-cap} nodes run jobs and handle {ml} API requests. For more information, see -<>. - -To create a dedicated {ml} node, set: - -[source,yaml] ----- -node.roles: [ ml, remote_cluster_client] ----- - -The `remote_cluster_client` role is optional but strongly recommended. -Otherwise, {ccs} fails when used in {ml} jobs or {dfeeds}. If you use {ccs} in -your {anomaly-jobs}, the `remote_cluster_client` role is also required on all -master-eligible nodes. Otherwise, the {dfeed} cannot start. See <>. - -[[transform-node]] -==== [xpack]#{transform-cap} node# - -{transform-cap} nodes run {transforms} and handle {transform} API requests. For -more information, see <>. - -To create a dedicated {transform} node, set: - -[source,yaml] ----- -node.roles: [ transform, remote_cluster_client ] ----- - -The `remote_cluster_client` role is optional but strongly recommended. -Otherwise, {ccs} fails when used in {transforms}. See <>. - -[[change-node-role]] -==== Changing the role of a node - -Each data node maintains the following data on disk: - -* the shard data for every shard allocated to that node, -* the index metadata corresponding with every shard allocated to that node, and -* the cluster-wide metadata, such as settings and index templates. - -Similarly, each master-eligible node maintains the following data on disk: - -* the index metadata for every index in the cluster, and -* the cluster-wide metadata, such as settings and index templates. - -Each node checks the contents of its data path at startup. If it discovers -unexpected data then it will refuse to start. This is to avoid importing -unwanted <> which can lead -to a red cluster health. To be more precise, nodes without the `data` role will -refuse to start if they find any shard data on disk at startup, and nodes -without both the `master` and `data` roles will refuse to start if they have any -index metadata on disk at startup. - -It is possible to change the roles of a node by adjusting its -`elasticsearch.yml` file and restarting it. This is known as _repurposing_ a -node. In order to satisfy the checks for unexpected data described above, you -must perform some extra steps to prepare a node for repurposing when starting -the node without the `data` or `master` roles. - -* If you want to repurpose a data node by removing the `data` role then you - should first use an <> to safely - migrate all the shard data onto other nodes in the cluster. - -* If you want to repurpose a node to have neither the `data` nor `master` roles - then it is simplest to start a brand-new node with an empty data path and the - desired roles. You may find it safest to use an - <> to migrate the shard data elsewhere - in the cluster first. - -If it is not possible to follow these extra steps then you may be able to use -the <> tool to delete any -excess data that prevents a node from starting. +To learn more about the available node roles, see <>. [discrete] === Node data path settings diff --git a/docs/reference/monitoring/index.asciidoc b/docs/reference/monitoring/index.asciidoc index c6380ab676a23..82e1447ba8a1f 100644 --- a/docs/reference/monitoring/index.asciidoc +++ b/docs/reference/monitoring/index.asciidoc @@ -9,7 +9,7 @@ performance of your {es} cluster. * <> * <> -* <> +* <> * <> * <> * <> @@ -24,7 +24,7 @@ include::overview.asciidoc[] include::how-monitoring-works.asciidoc[] -include::setup/logging-config.asciidoc[] +include::{es-ref-dir}/setup/logging-config.asciidoc[] include::production.asciidoc[] diff --git a/docs/reference/node-roles.asciidoc b/docs/reference/node-roles.asciidoc new file mode 100644 index 0000000000000..e414dd7ffcdfa --- /dev/null +++ b/docs/reference/node-roles.asciidoc @@ -0,0 +1,437 @@ +[[node-roles-overview]] +== Node roles + +Any time that you start an instance of {es}, you are starting a _node_. A +collection of connected nodes is called a <>. If you +are running a single node of {es}, then you have a cluster of one node. All nodes know about all the other nodes in the cluster and can forward client +requests to the appropriate node. + +Each node performs one or more roles. Roles control the behavior of the node in the cluster. + +[discrete] +[[set-node-roles]] +=== Set node roles + +You define a node's roles by setting `node.roles` in <>. If you set `node.roles`, the node is only assigned the roles you specify. If you don't set `node.roles`, the node is assigned the following roles: + +* `master` +* `data` +* `data_content` +* `data_hot` +* `data_warm` +* `data_cold` +* `data_frozen` +* `ingest` +* `ml` +* `remote_cluster_client` +* `transform` + +[IMPORTANT] +==== +If you set `node.roles`, ensure you specify every node role your cluster needs. +Every cluster requires the following node roles: + +* `master` +* {blank} ++ +-- +`data_content` and `data_hot` + +OR + +`data` +-- + +Some {stack} features also require specific node roles: + +- {ccs-cap} and {ccr} require the `remote_cluster_client` role. +- {stack-monitor-app} and ingest pipelines require the `ingest` role. +- {fleet}, the {security-app}, and {transforms} require the `transform` role. + The `remote_cluster_client` role is also required to use {ccs} with these + features. +- {ml-cap} features, such as {anomaly-detect}, require the `ml` role. +==== + +As the cluster grows and in particular if you have large {ml} jobs or +{ctransforms}, consider separating dedicated master-eligible nodes from +dedicated data nodes, {ml} nodes, and {transform} nodes. + +[discrete] +[[change-node-role]] +=== Change the role of a node + +Each data node maintains the following data on disk: + +* the shard data for every shard allocated to that node, +* the index metadata corresponding with every shard allocated to that node, and +* the cluster-wide metadata, such as settings and index templates. + +Similarly, each master-eligible node maintains the following data on disk: + +* the index metadata for every index in the cluster, and +* the cluster-wide metadata, such as settings and index templates. + +Each node checks the contents of its data path at startup. If it discovers +unexpected data then it will refuse to start. This is to avoid importing +unwanted <> which can lead +to a red cluster health. To be more precise, nodes without the `data` role will +refuse to start if they find any shard data on disk at startup, and nodes +without both the `master` and `data` roles will refuse to start if they have any +index metadata on disk at startup. + +It is possible to change the roles of a node by adjusting its +`elasticsearch.yml` file and restarting it. This is known as _repurposing_ a +node. In order to satisfy the checks for unexpected data described above, you +must perform some extra steps to prepare a node for repurposing when starting +the node without the `data` or `master` roles. + +* If you want to repurpose a data node by removing the `data` role then you + should first use an <> to safely + migrate all the shard data onto other nodes in the cluster. + +* If you want to repurpose a node to have neither the `data` nor `master` roles + then it is simplest to start a brand-new node with an empty data path and the + desired roles. You may find it safest to use an + <> to migrate the shard data elsewhere + in the cluster first. + +If it is not possible to follow these extra steps then you may be able to use +the <> tool to delete any +excess data that prevents a node from starting. + +[discrete] +[[node-roles-list]] +=== Available node roles + +The following is a list of the roles that a node can perform in a cluster. A node can have one or more roles. + +* <> (`master`): A node that is eligible to be +<>, which controls the cluster. + +* <> (`data`, `data_content`, `data_hot`, `data_warm`, `data_cold`, `data_frozen`): A node that has one of several data roles. Data nodes hold data and perform data related operations such as CRUD, search, and aggregations. You might use multiple data roles in a cluster so you can implement <>. + +* <> (`ingest`): Ingest nodes are able to apply an <> to a document in order to transform and enrich the document before indexing. With a heavy ingest load, it makes sense to use dedicated ingest nodes and to not include the `ingest` role from nodes that have the `master` or `data` roles. + +* <> (`remote_cluster_client`): A node that is eligible to act as a remote client. + +* <> (`ml`): A node that can run {ml-features}. If you want to use {ml-features}, there must be at least one {ml} node in your cluster. For more information, see <> and {ml-docs}/index.html[Machine learning in the {stack}]. + +* <> (`transform`): A node that can perform {transforms}. If you want to use {transforms}, there must be at least one {transform} node in your cluster. For more information, see <> and <>. + +[NOTE] +[[coordinating-node]] +.Coordinating node +=============================================== + +Requests like search requests or bulk-indexing requests may involve data held +on different data nodes. A search request, for example, is executed in two +phases which are coordinated by the node which receives the client request -- +the _coordinating node_. + +In the _scatter_ phase, the coordinating node forwards the request to the data +nodes which hold the data. Each data node executes the request locally and +returns its results to the coordinating node. In the _gather_ phase, the +coordinating node reduces each data node's results into a single global +result set. + +Every node is implicitly a coordinating node. This means that a node that has +an explicit empty list of roles in the `node.roles` setting will only act as a coordinating +node, which cannot be disabled. As a result, such a node needs to have enough +memory and CPU in order to deal with the gather phase. + +=============================================== + +[discrete] + +[[master-node]] +==== Master-eligible node + +The master node is responsible for lightweight cluster-wide actions such as +creating or deleting an index, tracking which nodes are part of the cluster, +and deciding which shards to allocate to which nodes. It is important for +cluster health to have a stable master node. + +Any master-eligible node that is not a <> may +be elected to become the master node by the <>. + +IMPORTANT: Master nodes must have a `path.data` directory whose contents +persist across restarts, just like data nodes, because this is where the +cluster metadata is stored. The cluster metadata describes how to read the data +stored on the data nodes, so if it is lost then the data stored on the data +nodes cannot be read. + +[discrete] +[[dedicated-master-node]] +===== Dedicated master-eligible node + +It is important for the health of the cluster that the elected master node has +the resources it needs to fulfill its responsibilities. If the elected master +node is overloaded with other tasks then the cluster will not operate well. The +most reliable way to avoid overloading the master with other tasks is to +configure all the master-eligible nodes to be _dedicated master-eligible nodes_ +which only have the `master` role, allowing them to focus on managing the +cluster. Master-eligible nodes will still also behave as +<> that route requests from clients to +the other nodes in the cluster, but you should _not_ use dedicated master nodes +for this purpose. + +A small or lightly-loaded cluster may operate well if its master-eligible nodes +have other roles and responsibilities, but once your cluster comprises more +than a handful of nodes it usually makes sense to use dedicated master-eligible +nodes. + +To create a dedicated master-eligible node, set: + +[source,yaml] +------------------- +node.roles: [ master ] +------------------- + +[discrete] +[[voting-only-node]] +===== Voting-only master-eligible node + +A voting-only master-eligible node is a node that participates in +<> but which will not act as the cluster's +elected master node. In particular, a voting-only node can serve as a tiebreaker +in elections. + +It may seem confusing to use the term "master-eligible" to describe a +voting-only node since such a node is not actually eligible to become the master +at all. This terminology is an unfortunate consequence of history: +master-eligible nodes are those nodes that participate in elections and perform +certain tasks during cluster state publications, and voting-only nodes have the +same responsibilities even if they can never become the elected master. + +To configure a master-eligible node as a voting-only node, include `master` and +`voting_only` in the list of roles. For example to create a voting-only data +node: + +[source,yaml] +------------------- +node.roles: [ data, master, voting_only ] +------------------- + +IMPORTANT: Only nodes with the `master` role can be marked as having the +`voting_only` role. + +High availability (HA) clusters require at least three master-eligible nodes, at +least two of which are not voting-only nodes. Such a cluster will be able to +elect a master node even if one of the nodes fails. + +Voting-only master-eligible nodes may also fill other roles in your cluster. +For instance, a node may be both a data node and a voting-only master-eligible +node. A _dedicated_ voting-only master-eligible nodes is a voting-only +master-eligible node that fills no other roles in the cluster. To create a +dedicated voting-only master-eligible node, set: + +[source,yaml] +------------------- +node.roles: [ master, voting_only ] +------------------- + +Since dedicated voting-only nodes never act as the cluster's elected master, +they may require less heap and a less powerful CPU than the true master nodes. +However all master-eligible nodes, including voting-only nodes, are on the +critical path for <>. Cluster state updates are usually independent of +performance-critical workloads such as indexing or searches, but they are +involved in management activities such as index creation and rollover, mapping +updates, and recovery after a failure. The performance characteristics of these +activities are a function of the speed of the storage on each master-eligible +node, as well as the reliability and latency of the network interconnections +between the elected master node and the other nodes in the cluster. You must +therefore ensure that the storage and networking available to the nodes in your +cluster are good enough to meet your performance goals. + +[discrete] +[[data-node]] +==== Data nodes + +Data nodes hold the shards that contain the documents you have indexed. Data +nodes handle data related operations like CRUD, search, and aggregations. +These operations are I/O-, memory-, and CPU-intensive. It is important to +monitor these resources and to add more data nodes if they are overloaded. + +The main benefit of having dedicated data nodes is the separation of the master +and data roles. + +In a multi-tier deployment architecture, you use specialized data roles to +assign data nodes to specific tiers: `data_content`,`data_hot`, `data_warm`, +`data_cold`, or `data_frozen`. A node can belong to multiple tiers. + +If you want to include a node in all tiers, or if your cluster does not use multiple tiers, then you can use the generic `data` role. + +include::{es-ref-dir}/how-to/shard-limits.asciidoc[] + +WARNING: If you assign a node to a specific tier using a specialized data role, then you shouldn't also assign it the generic `data` role. The generic `data` role takes precedence over specialized data roles. + +[discrete] +[[generic-data-node]] +===== Generic data node + +Generic data nodes are included in all content tiers. A node with a generic `data` role can fill any of the specialized data node roles. + +To create a dedicated generic data node, set: +[source,yaml] +---- +node.roles: [ data ] +---- + +[discrete] +[[data-content-node]] +===== Content data node + +Content data nodes are part of the content tier. +include::{es-ref-dir}/datatiers.asciidoc[tag=content-tier] + +To create a dedicated content node, set: +[source,yaml] +---- +node.roles: [ data_content ] +---- + +[discrete] +[[data-hot-node]] +===== Hot data node + +Hot data nodes are part of the hot tier. +include::{es-ref-dir}/datatiers.asciidoc[tag=hot-tier] + +To create a dedicated hot node, set: +[source,yaml] +---- +node.roles: [ data_hot ] +---- + +[discrete] +[[data-warm-node]] +===== Warm data node + +Warm data nodes are part of the warm tier. +include::{es-ref-dir}/datatiers.asciidoc[tag=warm-tier] + +To create a dedicated warm node, set: +[source,yaml] +---- +node.roles: [ data_warm ] +---- + +[discrete] +[[data-cold-node]] +===== Cold data node + +Cold data nodes are part of the cold tier. +include::{es-ref-dir}/datatiers.asciidoc[tag=cold-tier] + +To create a dedicated cold node, set: +[source,yaml] +---- +node.roles: [ data_cold ] +---- + +[discrete] +[[data-frozen-node]] +===== Frozen data node + +Frozen data nodes are part of the frozen tier. +include::{es-ref-dir}/datatiers.asciidoc[tag=frozen-tier] + +To create a dedicated frozen node, set: +[source,yaml] +---- +node.roles: [ data_frozen ] +---- + +[discrete] +[[node-ingest-node]] +==== Ingest node + +Ingest nodes can execute pre-processing pipelines, composed of one or more +ingest processors. Depending on the type of operations performed by the ingest +processors and the required resources, it may make sense to have dedicated +ingest nodes, that will only perform this specific task. + +To create a dedicated ingest node, set: + +[source,yaml] +---- +node.roles: [ ingest ] +---- + +[discrete] +[[coordinating-only-node]] +==== Coordinating only node + +If you take away the ability to be able to handle master duties, to hold data, +and pre-process documents, then you are left with a _coordinating_ node that +can only route requests, handle the search reduce phase, and distribute bulk +indexing. Essentially, coordinating only nodes behave as smart load balancers. + +Coordinating only nodes can benefit large clusters by offloading the +coordinating node role from data and master-eligible nodes. They join the +cluster and receive the full <>, like every other +node, and they use the cluster state to route requests directly to the +appropriate place(s). + +WARNING: Adding too many coordinating only nodes to a cluster can increase the +burden on the entire cluster because the elected master node must await +acknowledgement of cluster state updates from every node! The benefit of +coordinating only nodes should not be overstated -- data nodes can happily +serve the same purpose. + +To create a dedicated coordinating node, set: + +[source,yaml] +---- +node.roles: [ ] +---- + +[discrete] +[[remote-node]] +==== Remote-eligible node + +A remote-eligible node acts as a cross-cluster client and connects to +<>. Once connected, you can search +remote clusters using <>. You can also sync +data between clusters using <>. + +[source,yaml] +---- +node.roles: [ remote_cluster_client ] +---- + +[discrete] +[[ml-node]] +==== [xpack]#Machine learning node# + +{ml-cap} nodes run jobs and handle {ml} API requests. For more information, see +<>. + +To create a dedicated {ml} node, set: + +[source,yaml] +---- +node.roles: [ ml, remote_cluster_client] +---- + +The `remote_cluster_client` role is optional but strongly recommended. +Otherwise, {ccs} fails when used in {ml} jobs or {dfeeds}. If you use {ccs} in +your {anomaly-jobs}, the `remote_cluster_client` role is also required on all +master-eligible nodes. Otherwise, the {dfeed} cannot start. See <>. + +[discrete] +[[transform-node]] +==== [xpack]#{transform-cap} node# + +{transform-cap} nodes run {transforms} and handle {transform} API requests. For +more information, see <>. + +To create a dedicated {transform} node, set: + +[source,yaml] +---- +node.roles: [ transform, remote_cluster_client ] +---- + +The `remote_cluster_client` role is optional but strongly recommended. +Otherwise, {ccs} fails when used in {transforms}. See <>. \ No newline at end of file diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index a7bbf41c849e9..c3bf84fa600d2 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -1215,7 +1215,7 @@ See <>. [role="exclude",id="modules-gateway-dangling-indices"] === Dangling indices -See <>. +See <>. [role="exclude",id="shards-allocation"] === Cluster-level shard allocation diff --git a/docs/reference/settings/ml-settings.asciidoc b/docs/reference/settings/ml-settings.asciidoc index 1077a63b00249..08163b1391f2d 100644 --- a/docs/reference/settings/ml-settings.asciidoc +++ b/docs/reference/settings/ml-settings.asciidoc @@ -19,6 +19,8 @@ at all. // end::ml-settings-description-tag[] +TIP: To control memory usage used by {ml} jobs, you can use the <>. + [discrete] [[general-ml-settings]] ==== General machine learning settings @@ -67,7 +69,7 @@ limitations as described <>. The inference cache exists in the JVM heap on each ingest node. The cache affords faster processing times for the `inference` processor. The value can be a static byte sized value (such as `2gb`) or a percentage of total allocated -heap. Defaults to `40%`. See also <>. +heap. Defaults to `40%`. See also <>. [[xpack-interference-model-ttl]] // tag::interference-model-ttl-tag[] @@ -249,11 +251,4 @@ nodes in your cluster, you shouldn't use this setting. + If this setting is `true` it also affects the default value for `xpack.ml.max_model_memory_limit`. In this case `xpack.ml.max_model_memory_limit` -defaults to the largest size that could be assigned in the current cluster. - -[discrete] -[[model-inference-circuit-breaker]] -==== {ml-cap} circuit breaker settings - -The relevant circuit breaker settings can be found in the <>. - +defaults to the largest size that could be assigned in the current cluster. \ No newline at end of file diff --git a/docs/reference/setup/logging-config.asciidoc b/docs/reference/setup/logging-config.asciidoc index e382bbdacb464..81a28e6c51335 100644 --- a/docs/reference/setup/logging-config.asciidoc +++ b/docs/reference/setup/logging-config.asciidoc @@ -11,7 +11,7 @@ If you run {es} from the command line, {es} prints logs to the standard output (`stdout`). [discrete] -[[loggin-configuration]] +[[logging-configuration]] === Logging configuration IMPORTANT: Elastic strongly recommends using the Log4j 2 configuration that is shipped by default. diff --git a/docs/reference/shard-request-cache.asciidoc b/docs/reference/shard-request-cache.asciidoc new file mode 100644 index 0000000000000..ec79dfb531bdb --- /dev/null +++ b/docs/reference/shard-request-cache.asciidoc @@ -0,0 +1,134 @@ +[[shard-request-cache]] +=== The shard request cache + +When a search request is run against an index or against many indices, each +involved shard executes the search locally and returns its local results to +the _coordinating node_, which combines these shard-level results into a +``global'' result set. + +The shard-level request cache module caches the local results on each shard. +This allows frequently used (and potentially heavy) search requests to return +results almost instantly. The requests cache is a very good fit for the logging +use case, where only the most recent index is being actively updated -- +results from older indices will be served directly from the cache. + +You can control the size and expiration of the cache at the node level using the <>. + +[IMPORTANT] +=================================== + +By default, the requests cache will only cache the results of search requests +where `size=0`, so it will not cache `hits`, +but it will cache `hits.total`, <>, and +<>. + +Most queries that use `now` (see <>) cannot be cached. + +Scripted queries that use the API calls which are non-deterministic, such as +`Math.random()` or `new Date()` are not cached. +=================================== + +[discrete] +==== Cache invalidation + +The cache is smart -- it keeps the same _near real-time_ promise as uncached +search. + +Cached results are invalidated automatically whenever the shard refreshes to +pick up changes to the documents or when you update the mapping. In other +words you will always get the same results from the cache as you would for an +uncached search request. + +The longer the refresh interval, the longer that cached entries will remain +valid even if there are changes to the documents. If the cache is full, the +least recently used cache keys will be evicted. + +The cache can be expired manually with the <>: + +[source,console] +------------------------ +POST /my-index-000001,my-index-000002/_cache/clear?request=true +------------------------ +// TEST[s/^/PUT my-index-000001\nPUT my-index-000002\n/] + +[discrete] +==== Enabling and disabling caching + +The cache is enabled by default, but can be disabled when creating a new +index as follows: + +[source,console] +----------------------------- +PUT /my-index-000001 +{ + "settings": { + "index.requests.cache.enable": false + } +} +----------------------------- + +It can also be enabled or disabled dynamically on an existing index with the +<> API: + +[source,console] +----------------------------- +PUT /my-index-000001/_settings +{ "index.requests.cache.enable": true } +----------------------------- +// TEST[continued] + + +[discrete] +==== Enabling and disabling caching per request + +The `request_cache` query-string parameter can be used to enable or disable +caching on a *per-request* basis. If set, it overrides the index-level setting: + +[source,console] +----------------------------- +GET /my-index-000001/_search?request_cache=true +{ + "size": 0, + "aggs": { + "popular_colors": { + "terms": { + "field": "colors" + } + } + } +} +----------------------------- +// TEST[continued] + +Requests where `size` is greater than 0 will not be cached even if the request cache is +enabled in the index settings. To cache these requests you will need to use the +query-string parameter detailed here. + +[discrete] +==== Cache key + +A hash of the whole JSON body is used as the cache key. This means that if the JSON +changes -- for instance if keys are output in a different order -- then the +cache key will not be recognised. + +TIP: Most JSON libraries support a _canonical_ mode which ensures that JSON +keys are always emitted in the same order. This canonical mode can be used in +the application to ensure that a request is always serialized in the same way. + +[discrete] +==== Monitoring cache usage + +The size of the cache (in bytes) and the number of evictions can be viewed +by index, with the <> API: + +[source,console] +------------------------ +GET /_stats/request_cache?human +------------------------ + +or by node with the <> API: + +[source,console] +------------------------ +GET /_nodes/stats/indices/request_cache?human +------------------------ From b32842ee7fea3ba782332e1b0c620606123dbc50 Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Mon, 30 Dec 2024 16:12:03 -0500 Subject: [PATCH 05/11] remove merge artifact --- docs/reference/docs.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/docs.asciidoc b/docs/reference/docs.asciidoc index 1dda99842d657..ccdbaaffb2b77 100644 --- a/docs/reference/docs.asciidoc +++ b/docs/reference/docs.asciidoc @@ -1,7 +1,6 @@ [[docs]] == Document APIs -======= .New API reference [sidebar] -- From a885a0fc785c4c379ee352db631479df0c8ff29c Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Mon, 30 Dec 2024 17:21:43 -0500 Subject: [PATCH 06/11] clean up cluster-level shard allocation and routing settings --- docs/reference/cat/nodeattrs.asciidoc | 2 +- docs/reference/cluster.asciidoc | 2 +- .../migrate-index-allocation-filters.asciidoc | 2 +- .../data-store-architecture.asciidoc | 2 + .../ilm/apis/migrate-to-data-tiers.asciidoc | 2 +- .../allocation/filtering.asciidoc | 2 +- docs/reference/modules/cluster.asciidoc | 18 ++++- .../cluster/allocation_awareness.asciidoc | 15 ++-- .../cluster/allocation_filtering.asciidoc | 2 +- .../modules/cluster/disk_allocator.asciidoc | 76 ++++++------------- docs/reference/modules/node.asciidoc | 21 ++++- docs/reference/modules/shard-ops.asciidoc | 4 +- 12 files changed, 74 insertions(+), 74 deletions(-) diff --git a/docs/reference/cat/nodeattrs.asciidoc b/docs/reference/cat/nodeattrs.asciidoc index e8b1128e0c211..eb998ae2216c0 100644 --- a/docs/reference/cat/nodeattrs.asciidoc +++ b/docs/reference/cat/nodeattrs.asciidoc @@ -17,7 +17,7 @@ console. They are _not_ intended for use by applications. For application consumption, use the <>. ==== -Returns information about <>. +Returns information about <>. [[cat-nodeattrs-api-request]] ==== {api-request-title} diff --git a/docs/reference/cluster.asciidoc b/docs/reference/cluster.asciidoc index b4359c2bf4fbb..398ece616fe07 100644 --- a/docs/reference/cluster.asciidoc +++ b/docs/reference/cluster.asciidoc @@ -35,7 +35,7 @@ one of the following: master-eligible nodes, all data nodes, all ingest nodes, all voting-only nodes, all machine learning nodes, and all coordinating-only nodes. * a pair of patterns, using `*` wildcards, of the form `attrname:attrvalue`, - which adds to the subset all nodes with a custom node attribute whose name + which adds to the subset all nodes with a <> whose name and value match the respective patterns. Custom node attributes are configured by setting properties in the configuration file of the form `node.attr.attrname: attrvalue`. diff --git a/docs/reference/data-management/migrate-index-allocation-filters.asciidoc b/docs/reference/data-management/migrate-index-allocation-filters.asciidoc index 85d42e4105a92..ee7d5640d53df 100644 --- a/docs/reference/data-management/migrate-index-allocation-filters.asciidoc +++ b/docs/reference/data-management/migrate-index-allocation-filters.asciidoc @@ -2,7 +2,7 @@ [[migrate-index-allocation-filters]] == Migrate index allocation filters to node roles -If you currently use custom node attributes and +If you currently use <> and <> to move indices through <> in a https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management[hot-warm-cold architecture], diff --git a/docs/reference/data-store-architecture.asciidoc b/docs/reference/data-store-architecture.asciidoc index dc99c64a9b7f3..4119c0816df7c 100644 --- a/docs/reference/data-store-architecture.asciidoc +++ b/docs/reference/data-store-architecture.asciidoc @@ -12,6 +12,7 @@ The topics in this section provides information about the architecture of {es} a * <>: Learn about the different roles that nodes can have in an {es} cluster. * <>: Learn how {es} replicates read and write operations across shards and shard copies. * <>: Learn how {es} allocates and balances shards across nodes. +** <>: Learn how to use custom node attributes to distribute shards across different racks or availability zones. * <>: Learn how {es} caches search requests to improve performance. -- @@ -19,4 +20,5 @@ include::nodes-shards.asciidoc[] include::node-roles.asciidoc[] include::docs/data-replication.asciidoc[leveloffset=-1] include::modules/shard-ops.asciidoc[] +include::modules/cluster/allocation_awareness.asciidoc[leveloffset=+1] include::shard-request-cache.asciidoc[leveloffset=-1] \ No newline at end of file diff --git a/docs/reference/ilm/apis/migrate-to-data-tiers.asciidoc b/docs/reference/ilm/apis/migrate-to-data-tiers.asciidoc index 8ba57120a8a65..8e775a55b5eca 100644 --- a/docs/reference/ilm/apis/migrate-to-data-tiers.asciidoc +++ b/docs/reference/ilm/apis/migrate-to-data-tiers.asciidoc @@ -5,7 +5,7 @@ Migrate indices, ILM policies, and legacy, composable and component templates to data tiers routing ++++ -Switches the indices, ILM policies, and legacy, composable and component templates from using custom node attributes and +Switches the indices, ILM policies, and legacy, composable and component templates from using <> and <> to using <>, and optionally deletes one legacy index template. Using node roles enables {ilm-init} to <> between diff --git a/docs/reference/index-modules/allocation/filtering.asciidoc b/docs/reference/index-modules/allocation/filtering.asciidoc index 07a2455ca1eff..042b30dc32229 100644 --- a/docs/reference/index-modules/allocation/filtering.asciidoc +++ b/docs/reference/index-modules/allocation/filtering.asciidoc @@ -6,7 +6,7 @@ a particular index. These per-index filters are applied in conjunction with <> and <>. -Shard allocation filters can be based on custom node attributes or the built-in +Shard allocation filters can be based on <> or the built-in `_name`, `_host_ip`, `_publish_ip`, `_ip`, `_host`, `_id`, `_tier` and `_tier_preference` attributes. <> uses filters based on custom node attributes to determine how to reallocate shards when moving diff --git a/docs/reference/modules/cluster.asciidoc b/docs/reference/modules/cluster.asciidoc index b3eaa5b47c238..cf8e97de5e188 100644 --- a/docs/reference/modules/cluster.asciidoc +++ b/docs/reference/modules/cluster.asciidoc @@ -27,7 +27,23 @@ include::cluster/shards_allocation.asciidoc[] include::cluster/disk_allocator.asciidoc[] -include::cluster/allocation_awareness.asciidoc[] +[[shard-allocation-awareness-settings]] +==== Shard allocation awareness settings + +You can use <> as _awareness attributes_ to enable {es} +to take your physical hardware configuration into account when allocating shards. +If {es} knows which nodes are on the same physical server, in the same rack, or +in the same zone, it can distribute the primary shard and its replica shards to +minimize the risk of losing all shard copies in the event of a failure. <>. + +`cluster.routing.allocation.awareness.attributes`:: + (<>) + The node attributes that {es} should use as awareness attributes. For example, if you have a `rack_id` attribute that specifies the rack in which each node resides, you can set this setting to `rack_id` to ensure that primary and replica shards are not allocated on the same rack. You can specify multiple attributes as a comma-separated list. + +`cluster.routing.allocation.awareness.force.*`:: + (<>) + The shard allocation awareness values that must exist for shards to be reallocated in case of location failure. Learn more about <>. + include::cluster/allocation_filtering.asciidoc[] diff --git a/docs/reference/modules/cluster/allocation_awareness.asciidoc b/docs/reference/modules/cluster/allocation_awareness.asciidoc index 9c6197f9ba40d..34164cd364fc5 100644 --- a/docs/reference/modules/cluster/allocation_awareness.asciidoc +++ b/docs/reference/modules/cluster/allocation_awareness.asciidoc @@ -1,5 +1,5 @@ [[shard-allocation-awareness]] -==== Shard allocation awareness +== Shard allocation awareness You can use custom node attributes as _awareness attributes_ to enable {es} to take your physical hardware configuration into account when allocating shards. @@ -7,12 +7,7 @@ If {es} knows which nodes are on the same physical server, in the same rack, or in the same zone, it can distribute the primary shard and its replica shards to minimize the risk of losing all shard copies in the event of a failure. -When shard allocation awareness is enabled with the -<> -`cluster.routing.allocation.awareness.attributes` setting, shards are only -allocated to nodes that have values set for the specified awareness attributes. -If you use multiple awareness attributes, {es} considers each attribute -separately when allocating shards. +When shard allocation awareness is enabled with the `cluster.routing.allocation.awareness.attributes` setting, shards are only allocated to nodes that have values set for the specified awareness attributes. If you use multiple awareness attributes, {es} considers each attribute separately when allocating shards. NOTE: The number of attribute values determines how many shard copies are allocated in each location. If the number of nodes in each location is @@ -22,11 +17,11 @@ unassigned. TIP: Learn more about <>. [[enabling-awareness]] -===== Enabling shard allocation awareness +=== Enabling shard allocation awareness To enable shard allocation awareness: -. Specify the location of each node with a custom node attribute. For example, +. Specify the location of each node with a <>. For example, if you want Elasticsearch to distribute shards across different racks, you might use an awareness attribute called `rack_id`. + @@ -94,7 +89,7 @@ copies of a particular shard from being allocated in the same location, you can enable forced awareness. [[forced-awareness]] -===== Forced awareness +=== Forced awareness By default, if one location fails, {es} spreads its shards across the remaining locations. This might be undesirable if the cluster does not have sufficient diff --git a/docs/reference/modules/cluster/allocation_filtering.asciidoc b/docs/reference/modules/cluster/allocation_filtering.asciidoc index e70e43682973b..e6705dc10361b 100644 --- a/docs/reference/modules/cluster/allocation_filtering.asciidoc +++ b/docs/reference/modules/cluster/allocation_filtering.asciidoc @@ -6,7 +6,7 @@ allocates shards from any index. These cluster wide filters are applied in conjunction with <> and <>. -Shard allocation filters can be based on custom node attributes or the built-in +Shard allocation filters can be based on <> or the built-in `_name`, `_host_ip`, `_publish_ip`, `_ip`, `_host`, `_id` and `_tier` attributes. The `cluster.routing.allocation` settings are <>, enabling live indices to diff --git a/docs/reference/modules/cluster/disk_allocator.asciidoc b/docs/reference/modules/cluster/disk_allocator.asciidoc index 02cc48c6e27fc..8efe4c0132e86 100644 --- a/docs/reference/modules/cluster/disk_allocator.asciidoc +++ b/docs/reference/modules/cluster/disk_allocator.asciidoc @@ -41,6 +41,23 @@ on the affected node drops below the high watermark, {es} automatically removes the write block. Refer to <> to resolve persistent watermark errors. +[NOTE] +.Max headroom settings +=================================================== + +Max headroom settings apply only when watermark settings are percentages or ratios. + +A max headroom value is intended to cap the required free disk space before hitting +the respective watermark. This is useful for servers with larger disks, where a percentage or ratio watermark could translate to an overly large free disk space requirement. In this case, the max headroom can be used to cap the required free disk space amount. + +For example, where `cluster.routing.allocation.disk.watermark.flood_stage` is 95% and `cluster.routing.allocation.disk.watermark.flood_stage.max_headroom` is 100GB, this means that: + +* For a smaller disk, e.g., of 100GB, the flood watermark will hit at 95%, meaning at 5GB of free space, since 5GB is smaller than the 100GB max headroom value. +* For a larger disk, e.g., of 100TB, the flood watermark will hit at 100GB of free space. That is because the 95% flood watermark alone would require 5TB of free disk space, but is capped by the max headroom setting to 100GB. + +Max headroom settings have their default values only if their respective watermark settings are not explicitly set. If watermarks are explicitly set, then the max headroom settings do not have their default values, and need to be explicitly set if they are needed. +=================================================== + [[disk-based-shard-allocation-does-not-balance]] [TIP] ==== @@ -100,18 +117,7 @@ is now `true`. The setting will be removed in a future release. + -- (<>) -Controls the flood stage watermark, which defaults to 95%. {es} enforces a read-only index block (`index.blocks.read_only_allow_delete`) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark. Similarly to the low and high watermark values, it can alternatively be set to a ratio value, e.g., `0.95`, or an absolute byte value. - -An example of resetting the read-only index block on the `my-index-000001` index: - -[source,console] --------------------------------------------------- -PUT /my-index-000001/_settings -{ - "index.blocks.read_only_allow_delete": null -} --------------------------------------------------- -// TEST[setup:my_index] +Controls the flood stage watermark, which defaults to 95%. {es} enforces a read-only index block (<>) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark. Similarly to the low and high watermark values, it can alternatively be set to a ratio value, e.g., `0.95`, or an absolute byte value. -- // end::cluster-routing-flood-stage-tag[] @@ -121,10 +127,10 @@ Defaults to 100GB when `cluster.routing.allocation.disk.watermark.flood_stage` is not explicitly set. This caps the amount of free space required. -NOTE: You cannot mix the usage of percentage/ratio values and byte values across +NOTE: You can't mix the usage of percentage/ratio values and byte values across the `cluster.routing.allocation.disk.watermark.low`, `cluster.routing.allocation.disk.watermark.high`, and `cluster.routing.allocation.disk.watermark.flood_stage` settings. Either all values -are set to percentage/ratio values, or all are set to byte values. This enforcement is +must be set to percentage/ratio values, or all must be set to byte values. This is required so that {es} can validate that the settings are internally consistent, ensuring that the low disk threshold is less than the high disk threshold, and the high disk threshold is less than the flood stage threshold. A similar comparison check is done for the max @@ -150,44 +156,6 @@ set. This caps the amount of free space required on dedicated frozen nodes. cluster. Defaults to `30s`. NOTE: Percentage values refer to used disk space, while byte values refer to -free disk space. This can be confusing, since it flips the meaning of high and +free disk space. This can be confusing, because it flips the meaning of high and low. For example, it makes sense to set the low watermark to 10gb and the high -watermark to 5gb, but not the other way around. - -An example of updating the low watermark to at least 100 gigabytes free, a high -watermark of at least 50 gigabytes free, and a flood stage watermark of 10 -gigabytes free, and updating the information about the cluster every minute: - -[source,console] --------------------------------------------------- -PUT _cluster/settings -{ - "persistent": { - "cluster.routing.allocation.disk.watermark.low": "100gb", - "cluster.routing.allocation.disk.watermark.high": "50gb", - "cluster.routing.allocation.disk.watermark.flood_stage": "10gb", - "cluster.info.update.interval": "1m" - } -} --------------------------------------------------- - -Concerning the max headroom settings for the watermarks, please note -that these apply only in the case that the watermark settings are percentages/ratios. -The aim of a max headroom value is to cap the required free disk space before hitting -the respective watermark. This is especially useful for servers with larger -disks, where a percentage/ratio watermark could translate to a big free disk space requirement, -and the max headroom can be used to cap the required free disk space amount. -As an example, let us take the default settings for the flood watermark. -It has a 95% default value, and the flood max headroom setting has a default value of 100GB. -This means that: - -* For a smaller disk, e.g., of 100GB, the flood watermark will hit at 95%, meaning at 5GB -of free space, since 5GB is smaller than the 100GB max headroom value. -* For a larger disk, e.g., of 100TB, the flood watermark will hit at 100GB of free space. -That is because the 95% flood watermark alone would require 5TB of free disk space, but -that is capped by the max headroom setting to 100GB. - -Finally, the max headroom settings have their default values only if their respective watermark -settings are not explicitly set (thus, they have their default percentage values). -If watermarks are explicitly set, then the max headroom settings do not have their default values, -and would need to be explicitly set if they are desired. +watermark to 5gb, but not the other way around. \ No newline at end of file diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index 8f0617b30c6a8..297e777ab738a 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -120,6 +120,25 @@ modify the contents of the data directory. The data directory contains no executables so a virus scan will only find false positives. // end::modules-node-data-path-warning-tag[] +[[custom-node-attributes]] +==== Custom node attributes + +If needed, you can add custom attributes to a node. These attributes can be used to <>, or to group nodes together for <>. + +[TIP] +=============================================== +You can also set a node attribute using the `-E` command line argument when you start a node: + +[source,sh] +-------------------------------------------------------- +./bin/elasticsearch -Enode.attr.rack_id=rack_one +-------------------------------------------------------- +=============================================== + +`node.attr.`:: + (<>) + A custom attribute that you can assign to a node. For example, you might assign a `rack_id` attribute to each node to ensure that primary and replica shards are not allocated on the same rack. You can specify multiple attributes as a comma-separated list. + [discrete] [[other-node-settings]] === Other node settings @@ -129,4 +148,4 @@ including: * <> * <> -* <> +* <> \ No newline at end of file diff --git a/docs/reference/modules/shard-ops.asciidoc b/docs/reference/modules/shard-ops.asciidoc index 66ceebcfa0319..93d6b6d3468f8 100644 --- a/docs/reference/modules/shard-ops.asciidoc +++ b/docs/reference/modules/shard-ops.asciidoc @@ -25,7 +25,7 @@ By default, the primary and replica shard copies for an index can be allocated t You can control how shard copies are allocated using the following settings: -- <>: Use these settings to control how shard copies are allocated and balanced across the entire cluster. For example, you might want to allocate nodes availability zones, or prevent certain nodes from being used so you can perform maintenance. +- <>: Use these settings to control how shard copies are allocated and balanced across the entire cluster. For example, you might want to <>, or prevent certain nodes from being used so you can perform maintenance. - <>: Use these settings to control how the shard copies for a specific index are allocated. For example, you might want to allocate an index to a node in a specific data tier, or to an node with specific attributes. @@ -80,4 +80,4 @@ When a shard copy is relocated, it is created as a new shard copy on the target You can control how and when shard copies are relocated. For example, you can adjust the rebalancing settings that control when shard copies are relocated to balance the cluster, or the high watermark for disk-based shard allocation that can trigger relocation. These settings are part of the <>. -Shard relocation operations also respect shard allocation and recovery settings. \ No newline at end of file +Shard relocation operations also respect shard allocation and recovery settings. \ No newline at end of file From 54737617f6d8c49252c382a6d27536cde856aea8 Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Tue, 31 Dec 2024 11:47:54 -0500 Subject: [PATCH 07/11] make sure important settings are represented in reference content --- docs/reference/modules/cluster/misc.asciidoc | 3 + docs/reference/modules/network.asciidoc | 19 +++ docs/reference/modules/node.asciidoc | 5 + .../reference/path-settings-overview.asciidoc | 112 +++++++++++++++++ docs/reference/setup.asciidoc | 16 ++- .../setup/advanced-configuration.asciidoc | 35 ++---- .../setup/important-settings.asciidoc | 8 ++ .../important-settings/cluster-name.asciidoc | 1 - .../important-settings/error-file.asciidoc | 1 - .../important-settings/gc-logging.asciidoc | 4 +- .../heap-dump-path.asciidoc | 1 - .../important-settings/node-name.asciidoc | 1 - .../important-settings/path-settings.asciidoc | 115 +----------------- 13 files changed, 169 insertions(+), 152 deletions(-) create mode 100644 docs/reference/path-settings-overview.asciidoc diff --git a/docs/reference/modules/cluster/misc.asciidoc b/docs/reference/modules/cluster/misc.asciidoc index 75eaca88c66b1..b66ac1fdb0cca 100644 --- a/docs/reference/modules/cluster/misc.asciidoc +++ b/docs/reference/modules/cluster/misc.asciidoc @@ -1,6 +1,9 @@ [[misc-cluster-settings]] === Miscellaneous cluster settings +[[cluster-name]] +include::{es-ref-dir}/setup/important-settings/cluster-name.asciidoc[] + [discrete] [[cluster-read-only]] ==== Metadata diff --git a/docs/reference/modules/network.asciidoc b/docs/reference/modules/network.asciidoc index 1e4c5a21d386c..2ea4dcb9b18f5 100644 --- a/docs/reference/modules/network.asciidoc +++ b/docs/reference/modules/network.asciidoc @@ -286,3 +286,22 @@ include::remote-cluster-network.asciidoc[] include::network/tracers.asciidoc[] include::network/threading.asciidoc[] + +[[readiness-tcp-port]] +==== TCP readiness port + +preview::[] + +If configured, a node can open a TCP port when the node is in a ready state. A node is deemed +ready when it has successfully joined a cluster. In a single node configuration, the node is +said to be ready, when it's able to accept requests. + +To enable the readiness TCP port, use the `readiness.port` setting. The readiness service will bind to +all host addresses. + +If the node leaves the cluster, or the <> is used to mark the node +for shutdown, the readiness port is immediately closed. + +A successful connection to the readiness TCP port signals that the {es} node is ready. When a client +connects to the readiness port, the server simply terminates the socket connection. No data is sent back +to the client. If a client cannot connect to the readiness port, the node is not ready. \ No newline at end of file diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index 297e777ab738a..0e99a9cd90ab3 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -18,6 +18,11 @@ TIP: The performance of an {es} node is often limited by the performance of the Review our recommendations for optimizing your storage for <> and <>. +[[node-name-settings]] +==== Node name setting + +include::{es-ref-dir}/setup/important-settings/node-name.asciidoc[] + [[node-roles]] ==== Node role settings diff --git a/docs/reference/path-settings-overview.asciidoc b/docs/reference/path-settings-overview.asciidoc new file mode 100644 index 0000000000000..0740b9769c9b2 --- /dev/null +++ b/docs/reference/path-settings-overview.asciidoc @@ -0,0 +1,112 @@ +[[path-settings-overview]] +=== Path settings + +include::{es-ref-dir}/setup/important-settings/path-settings.asciidoc[] + +[[multiple-data-paths]] +==== Multiple data paths +deprecated::[7.13.0] + +If needed, you can specify multiple paths in `path.data`. {es} stores the node's +data across all provided paths but keeps each shard's data on the same path. + +{es} does not balance shards across a node's data paths. High disk +usage in a single path can trigger a <> for the entire node. If triggered, {es} will not add shards to +the node, even if the node’s other paths have available disk space. If you need +additional disk space, we recommend you add a new node rather than additional +data paths. + +include::{es-ref-dir}/tab-widgets/multi-data-path-widget.asciidoc[] + +[[mdp-migrate]] +===== Migrate from multiple data paths + +Support for multiple data paths was deprecated in 7.13 and will be removed +in a future release. + +As an alternative to multiple data paths, you can create a filesystem which +spans multiple disks with a hardware virtualisation layer such as RAID, or a +software virtualisation layer such as Logical Volume Manager (LVM) on Linux or +Storage Spaces on Windows. If you wish to use multiple data paths on a single +machine then you must run one node for each data path. + +If you currently use multiple data paths in a +{ref}/high-availability-cluster-design.html[highly available cluster] then you +can migrate to a setup that uses a single path for each node without downtime +using a process similar to a +{ref}/restart-cluster.html#restart-cluster-rolling[rolling restart]: shut each +node down in turn and replace it with one or more nodes each configured to use +a single data path. In more detail, for each node that currently has multiple +data paths you should follow the following process. In principle you can +perform this migration during a rolling upgrade to 8.0, but we recommend +migrating to a single-data-path setup before starting to upgrade. + +1. Take a snapshot to protect your data in case of disaster. + +2. Optionally, migrate the data away from the target node by using an +{ref}/modules-cluster.html#cluster-shard-allocation-filtering[allocation filter]: ++ +[source,console] +-------------------------------------------------- +PUT _cluster/settings +{ + "persistent": { + "cluster.routing.allocation.exclude._name": "target-node-name" + } +} +-------------------------------------------------- ++ +You can use the {ref}/cat-allocation.html[cat allocation API] to track progress +of this data migration. If some shards do not migrate then the +{ref}/cluster-allocation-explain.html[cluster allocation explain API] will help +you to determine why. + +3. Follow the steps in the +{ref}/restart-cluster.html#restart-cluster-rolling[rolling restart process] +up to and including shutting the target node down. + +4. Ensure your cluster health is `yellow` or `green`, so that there is a copy +of every shard assigned to at least one of the other nodes in your cluster. + +5. If applicable, remove the allocation filter applied in the earlier step. ++ +[source,console] +-------------------------------------------------- +PUT _cluster/settings +{ + "persistent": { + "cluster.routing.allocation.exclude._name": null + } +} +-------------------------------------------------- + +6. Discard the data held by the stopped node by deleting the contents of its +data paths. + +7. Reconfigure your storage. For instance, combine your disks into a single +filesystem using LVM or Storage Spaces. Ensure that your reconfigured storage +has sufficient space for the data that it will hold. + +8. Reconfigure your node by adjusting the `path.data` setting in its +`elasticsearch.yml` file. If needed, install more nodes each with their own +`path.data` setting pointing at a separate data path. + +9. Start the new nodes and follow the rest of the +{ref}/restart-cluster.html#restart-cluster-rolling[rolling restart process] for +them. + +10. Ensure your cluster health is `green`, so that every shard has been +assigned. + +You can alternatively add some number of single-data-path nodes to your +cluster, migrate all your data over to these new nodes using +{ref}/modules-cluster.html#cluster-shard-allocation-filtering[allocation filters], +and then remove the old nodes from the cluster. This approach will temporarily +double the size of your cluster so it will only work if you have the capacity to +expand your cluster like this. + +If you currently use multiple data paths but your cluster is not highly +available then you can migrate to a non-deprecated configuration by taking +a snapshot, creating a new cluster with the desired configuration and restoring +the snapshot into it. diff --git a/docs/reference/setup.asciidoc b/docs/reference/setup.asciidoc index 5bce5da19c055..922f1bdba4d1f 100644 --- a/docs/reference/setup.asciidoc +++ b/docs/reference/setup.asciidoc @@ -27,6 +27,8 @@ the only resource-intensive application on the host or container. For example, you might run {metricbeat} alongside {es} for cluster statistics, but a resource-heavy {ls} deployment should be on its own host. +// alphabetized + include::run-elasticsearch-locally.asciidoc[] include::setup/install.asciidoc[] @@ -47,28 +49,28 @@ include::settings/ccr-settings.asciidoc[] include::modules/discovery/discovery-settings.asciidoc[] +include::settings/data-stream-lifecycle-settings.asciidoc[] + include::modules/indices/fielddata.asciidoc[] +include::modules/gateway.asciidoc[] + include::settings/health-diagnostic-settings.asciidoc[] include::settings/ilm-settings.asciidoc[] -include::settings/data-stream-lifecycle-settings.asciidoc[] - include::modules/indices/index_management.asciidoc[] include::modules/indices/recovery.asciidoc[] include::modules/indices/indexing_buffer.asciidoc[] -include::settings/license-settings.asciidoc[] +include::settings/inference-settings.asciidoc[] -include::modules/gateway.asciidoc[] +include::settings/license-settings.asciidoc[] include::settings/ml-settings.asciidoc[] -include::settings/inference-settings.asciidoc[] - include::settings/monitoring-settings.asciidoc[] include::modules/node.asciidoc[] @@ -77,6 +79,8 @@ include::modules/network.asciidoc[] include::modules/indices/query_cache.asciidoc[] +include::{es-ref-dir}/path-settings-overview.asciidoc[] + include::modules/indices/search-settings.asciidoc[] include::settings/security-settings.asciidoc[] diff --git a/docs/reference/setup/advanced-configuration.asciidoc b/docs/reference/setup/advanced-configuration.asciidoc index 2a7ccc56742de..555650ff447e3 100644 --- a/docs/reference/setup/advanced-configuration.asciidoc +++ b/docs/reference/setup/advanced-configuration.asciidoc @@ -1,10 +1,3 @@ -[[advanced-configuration]] -=== Advanced configuration - -Modifying advanced settings is generally not recommended and could negatively -impact performance and stability. Using the {es}-provided defaults -is recommended in most circumstances. - [[set-jvm-options]] ==== Set JVM options @@ -21,6 +14,10 @@ Where you put the JVM options files depends on the type of installation: * Docker: Bind mount custom JVM options files into `/usr/share/elasticsearch/config/jvm.options.d/`. +CAUTION: Setting your own JVM options is generally not recommended and could negatively +impact performance and stability. Using the {es}-provided defaults +is recommended in most circumstances. + NOTE: Do not modify the root `jvm.options` file. Use files in `jvm.options.d/` instead. [[jvm-options-syntax]] @@ -155,23 +152,11 @@ options. We do not recommend using `ES_JAVA_OPTS` in production. NOTE: If you are running {es} as a Windows service, you can change the heap size using the service manager. See <>. -[[readiness-tcp-port]] -===== Enable the Elasticsearch TCP readiness port - -preview::[] - -If configured, a node can open a TCP port when the node is in a ready state. A node is deemed -ready when it has successfully joined a cluster. In a single node configuration, the node is -said to be ready, when it's able to accept requests. - -To enable the readiness TCP port, use the `readiness.port` setting. The readiness service will bind to -all host addresses. - -If the node leaves the cluster, or the <> is used to mark the node -for shutdown, the readiness port is immediately closed. - -A successful connection to the readiness TCP port signals that the {es} node is ready. When a client -connects to the readiness port, the server simply terminates the socket connection. No data is sent back -to the client. If a client cannot connect to the readiness port, the node is not ready. +[[heap-dump-path]] +include::important-settings/heap-dump-path.asciidoc[] +[[gc-logging]] +include::important-settings/gc-logging.asciidoc[] +[[error-file-path]] +include::important-settings/error-file.asciidoc[] \ No newline at end of file diff --git a/docs/reference/setup/important-settings.asciidoc b/docs/reference/setup/important-settings.asciidoc index 03c891af70743..f90d8a0503bb5 100644 --- a/docs/reference/setup/important-settings.asciidoc +++ b/docs/reference/setup/important-settings.asciidoc @@ -19,10 +19,18 @@ of items which *must* be considered before using your cluster in production: Our {ess-trial}[{ecloud}] service configures these items automatically, making your cluster production-ready by default. +[[path-settings]] +[discrete] +==== Path settings + include::important-settings/path-settings.asciidoc[] +Elasticsearch offers a deprecated setting that allows you to specify multiple paths in `path.data`. +To learn about this setting, and how to migrate away from it, refer to <>. + include::important-settings/cluster-name.asciidoc[] +[[node-name]] include::important-settings/node-name.asciidoc[] include::important-settings/network-host.asciidoc[] diff --git a/docs/reference/setup/important-settings/cluster-name.asciidoc b/docs/reference/setup/important-settings/cluster-name.asciidoc index 3f1516f21de1e..6d489eee76cf6 100644 --- a/docs/reference/setup/important-settings/cluster-name.asciidoc +++ b/docs/reference/setup/important-settings/cluster-name.asciidoc @@ -1,4 +1,3 @@ -[[cluster-name]] [discrete] ==== Cluster name setting diff --git a/docs/reference/setup/important-settings/error-file.asciidoc b/docs/reference/setup/important-settings/error-file.asciidoc index ca95ded78d53f..2f654002d51f8 100644 --- a/docs/reference/setup/important-settings/error-file.asciidoc +++ b/docs/reference/setup/important-settings/error-file.asciidoc @@ -1,4 +1,3 @@ -[[error-file-path]] [discrete] ==== JVM fatal error log setting diff --git a/docs/reference/setup/important-settings/gc-logging.asciidoc b/docs/reference/setup/important-settings/gc-logging.asciidoc index 3534e1335c9fd..873c85d58d914 100644 --- a/docs/reference/setup/important-settings/gc-logging.asciidoc +++ b/docs/reference/setup/important-settings/gc-logging.asciidoc @@ -1,4 +1,3 @@ -[[gc-logging]] [discrete] ==== GC logging settings @@ -20,9 +19,8 @@ To see further options not contained in the original JEP, see https://docs.oracle.com/en/java/javase/13/docs/specs/man/java.html#enable-logging-with-the-jvm-unified-logging-framework[Enable Logging with the JVM Unified Logging Framework]. -[[gc-logging-examples]] [discrete] -==== Examples +===== Examples Change the default GC log output location to `/opt/my-app/gc.log` by creating `$ES_HOME/config/jvm.options.d/gc.options` with some sample diff --git a/docs/reference/setup/important-settings/heap-dump-path.asciidoc b/docs/reference/setup/important-settings/heap-dump-path.asciidoc index 8f01379842a90..8b06ae752f360 100644 --- a/docs/reference/setup/important-settings/heap-dump-path.asciidoc +++ b/docs/reference/setup/important-settings/heap-dump-path.asciidoc @@ -1,4 +1,3 @@ -[[heap-dump-path]] [discrete] ==== JVM heap dump path setting diff --git a/docs/reference/setup/important-settings/node-name.asciidoc b/docs/reference/setup/important-settings/node-name.asciidoc index eda3052d119c9..e1aeb1ee8ca0e 100644 --- a/docs/reference/setup/important-settings/node-name.asciidoc +++ b/docs/reference/setup/important-settings/node-name.asciidoc @@ -1,4 +1,3 @@ -[[node-name]] [discrete] ==== Node name setting diff --git a/docs/reference/setup/important-settings/path-settings.asciidoc b/docs/reference/setup/important-settings/path-settings.asciidoc index a0a444ca5090a..002e08e2dc746 100644 --- a/docs/reference/setup/important-settings/path-settings.asciidoc +++ b/docs/reference/setup/important-settings/path-settings.asciidoc @@ -1,7 +1,3 @@ -[[path-settings]] -[discrete] -==== Path settings - {es} writes the data you index to indices and data streams to a `data` directory. {es} writes its own application logs, which contain information about cluster health and operations, to a `logs` directory. @@ -20,113 +16,4 @@ Supported `path.data` and `path.logs` values vary by platform: include::{es-ref-dir}/tab-widgets/customize-data-log-path-widget.asciidoc[] -include::{es-ref-dir}/modules/node.asciidoc[tag=modules-node-data-path-warning-tag] - -[discrete] -==== Multiple data paths -deprecated::[7.13.0] - -If needed, you can specify multiple paths in `path.data`. {es} stores the node's -data across all provided paths but keeps each shard's data on the same path. - -{es} does not balance shards across a node's data paths. High disk -usage in a single path can trigger a <> for the entire node. If triggered, {es} will not add shards to -the node, even if the node’s other paths have available disk space. If you need -additional disk space, we recommend you add a new node rather than additional -data paths. - -include::{es-ref-dir}/tab-widgets/multi-data-path-widget.asciidoc[] - -[discrete] -[[mdp-migrate]] -==== Migrate from multiple data paths - -Support for multiple data paths was deprecated in 7.13 and will be removed -in a future release. - -As an alternative to multiple data paths, you can create a filesystem which -spans multiple disks with a hardware virtualisation layer such as RAID, or a -software virtualisation layer such as Logical Volume Manager (LVM) on Linux or -Storage Spaces on Windows. If you wish to use multiple data paths on a single -machine then you must run one node for each data path. - -If you currently use multiple data paths in a -{ref}/high-availability-cluster-design.html[highly available cluster] then you -can migrate to a setup that uses a single path for each node without downtime -using a process similar to a -{ref}/restart-cluster.html#restart-cluster-rolling[rolling restart]: shut each -node down in turn and replace it with one or more nodes each configured to use -a single data path. In more detail, for each node that currently has multiple -data paths you should follow the following process. In principle you can -perform this migration during a rolling upgrade to 8.0, but we recommend -migrating to a single-data-path setup before starting to upgrade. - -1. Take a snapshot to protect your data in case of disaster. - -2. Optionally, migrate the data away from the target node by using an -{ref}/modules-cluster.html#cluster-shard-allocation-filtering[allocation filter]: -+ -[source,console] --------------------------------------------------- -PUT _cluster/settings -{ - "persistent": { - "cluster.routing.allocation.exclude._name": "target-node-name" - } -} --------------------------------------------------- -+ -You can use the {ref}/cat-allocation.html[cat allocation API] to track progress -of this data migration. If some shards do not migrate then the -{ref}/cluster-allocation-explain.html[cluster allocation explain API] will help -you to determine why. - -3. Follow the steps in the -{ref}/restart-cluster.html#restart-cluster-rolling[rolling restart process] -up to and including shutting the target node down. - -4. Ensure your cluster health is `yellow` or `green`, so that there is a copy -of every shard assigned to at least one of the other nodes in your cluster. - -5. If applicable, remove the allocation filter applied in the earlier step. -+ -[source,console] --------------------------------------------------- -PUT _cluster/settings -{ - "persistent": { - "cluster.routing.allocation.exclude._name": null - } -} --------------------------------------------------- - -6. Discard the data held by the stopped node by deleting the contents of its -data paths. - -7. Reconfigure your storage. For instance, combine your disks into a single -filesystem using LVM or Storage Spaces. Ensure that your reconfigured storage -has sufficient space for the data that it will hold. - -8. Reconfigure your node by adjusting the `path.data` setting in its -`elasticsearch.yml` file. If needed, install more nodes each with their own -`path.data` setting pointing at a separate data path. - -9. Start the new nodes and follow the rest of the -{ref}/restart-cluster.html#restart-cluster-rolling[rolling restart process] for -them. - -10. Ensure your cluster health is `green`, so that every shard has been -assigned. - -You can alternatively add some number of single-data-path nodes to your -cluster, migrate all your data over to these new nodes using -{ref}/modules-cluster.html#cluster-shard-allocation-filtering[allocation filters], -and then remove the old nodes from the cluster. This approach will temporarily -double the size of your cluster so it will only work if you have the capacity to -expand your cluster like this. - -If you currently use multiple data paths but your cluster is not highly -available then you can migrate to a non-deprecated configuration by taking -a snapshot, creating a new cluster with the desired configuration and restoring -the snapshot into it. +include::{es-ref-dir}/modules/node.asciidoc[tag=modules-node-data-path-warning-tag] \ No newline at end of file From 83cb19c43f31ef7117ce1c39d0f205925cd8190e Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Tue, 31 Dec 2024 12:36:13 -0500 Subject: [PATCH 08/11] work around anchor problem --- docs/plugins/discovery-ec2.asciidoc | 4 ++-- docs/reference/commands/node-tool.asciidoc | 4 ++-- docs/reference/datatiers.asciidoc | 2 +- .../high-availability/cluster-design.asciidoc | 10 +++++----- .../allocation/data_tier_allocation.asciidoc | 2 +- .../index-modules/allocation/filtering.asciidoc | 2 +- .../cluster/allocation_filtering.asciidoc | 4 ++-- .../modules/discovery/bootstrapping.asciidoc | 2 +- docs/reference/modules/discovery/voting.asciidoc | 2 +- docs/reference/modules/gateway.asciidoc | 2 +- .../modules/indices/request_cache.asciidoc | 4 ++-- docs/reference/modules/node.asciidoc | 4 ++++ docs/reference/modules/remote-clusters.asciidoc | 2 +- docs/reference/node-roles.asciidoc | 16 ++++++++-------- docs/reference/redirects.asciidoc | 5 +++++ .../search/multi-search-template-api.asciidoc | 2 +- docs/reference/search/multi-search.asciidoc | 2 +- docs/reference/setup/add-nodes.asciidoc | 2 +- .../setup/advanced-configuration.asciidoc | 12 ++++++------ docs/reference/setup/important-settings.asciidoc | 2 ++ .../setup/important-settings/node-name.asciidoc | 3 --- .../apis/restore-snapshot-api.asciidoc | 2 +- .../snapshot-restore/take-snapshot.asciidoc | 2 +- docs/reference/transform/setup.asciidoc | 2 +- .../upgrade/disable-shard-alloc.asciidoc | 2 +- 25 files changed, 52 insertions(+), 44 deletions(-) diff --git a/docs/plugins/discovery-ec2.asciidoc b/docs/plugins/discovery-ec2.asciidoc index 44acba4752aaa..164e3398d7a4f 100644 --- a/docs/plugins/discovery-ec2.asciidoc +++ b/docs/plugins/discovery-ec2.asciidoc @@ -241,7 +241,7 @@ The `discovery-ec2` plugin can automatically set the `aws_availability_zone` node attribute to the availability zone of each node. This node attribute allows you to ensure that each shard has copies allocated redundantly across multiple availability zones by using the -{ref}/modules-cluster.html#shard-allocation-awareness[Allocation Awareness] +{ref}/shard-allocation-awareness.html#[Allocation Awareness] feature. In order to enable the automatic definition of the `aws_availability_zone` @@ -333,7 +333,7 @@ labelled as `Moderate` or `Low`. * It is a good idea to distribute your nodes across multiple https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[availability -zones] and use {ref}/modules-cluster.html#shard-allocation-awareness[shard +zones] and use {ref}/shard-allocation-awareness.html[shard allocation awareness] to ensure that each shard has copies in more than one availability zone. diff --git a/docs/reference/commands/node-tool.asciidoc b/docs/reference/commands/node-tool.asciidoc index cdd2bb8f0f9d7..265006aa3df17 100644 --- a/docs/reference/commands/node-tool.asciidoc +++ b/docs/reference/commands/node-tool.asciidoc @@ -23,8 +23,8 @@ bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster|override-versio This tool has a number of modes: * `elasticsearch-node repurpose` can be used to delete unwanted data from a - node if it used to be a <> or a - <> but has been repurposed not to have one + node if it used to be a <> or a + <> but has been repurposed not to have one or other of these roles. * `elasticsearch-node remove-settings` can be used to remove persistent settings diff --git a/docs/reference/datatiers.asciidoc b/docs/reference/datatiers.asciidoc index 65e029d876e6f..066765368ec5e 100644 --- a/docs/reference/datatiers.asciidoc +++ b/docs/reference/datatiers.asciidoc @@ -189,7 +189,7 @@ tier]. [[configure-data-tiers-on-premise]] ==== Self-managed deployments -For self-managed deployments, each node's <> is configured +For self-managed deployments, each node's <> is configured in `elasticsearch.yml`. For example, the highest-performance nodes in a cluster might be assigned to both the hot and content tiers: diff --git a/docs/reference/high-availability/cluster-design.asciidoc b/docs/reference/high-availability/cluster-design.asciidoc index 105c8b236b0b1..d187db83c43f9 100644 --- a/docs/reference/high-availability/cluster-design.asciidoc +++ b/docs/reference/high-availability/cluster-design.asciidoc @@ -87,7 +87,7 @@ the same thing, but it's not necessary to use this feature in such a small cluster. We recommend you set only one of your two nodes to be -<>. This means you can be certain which of your +<>. This means you can be certain which of your nodes is the elected master of the cluster. The cluster can tolerate the loss of the other master-ineligible node. If you set both nodes to master-eligible, two nodes are required for a master election. Since the election will fail if either @@ -164,12 +164,12 @@ cluster that is suitable for production deployments. [[high-availability-cluster-design-three-nodes]] ==== Three-node clusters -If you have three nodes, we recommend they all be <> and +If you have three nodes, we recommend they all be <> and every index that is not a <> should have at least one replica. Nodes are data nodes by default. You may prefer for some indices to have two replicas so that each node has a copy of each shard in those indices. You should also configure each node to be -<> so that any two of them can hold a master +<> so that any two of them can hold a master election without needing to communicate with the third node. Nodes are master-eligible by default. This cluster will be resilient to the loss of any single node. @@ -188,8 +188,8 @@ service provides such a load balancer. Once your cluster grows to more than three nodes, you can start to specialise these nodes according to their responsibilities, allowing you to scale their -resources independently as needed. You can have as many <>, <>, <>, etc. as needed to +resources independently as needed. You can have as many <>, <>, <>, etc. as needed to support your workload. As your cluster grows larger, we recommend using dedicated nodes for each role. This allows you to independently scale resources for each task. diff --git a/docs/reference/index-modules/allocation/data_tier_allocation.asciidoc b/docs/reference/index-modules/allocation/data_tier_allocation.asciidoc index d08af21007622..2d59e9be31cd4 100644 --- a/docs/reference/index-modules/allocation/data_tier_allocation.asciidoc +++ b/docs/reference/index-modules/allocation/data_tier_allocation.asciidoc @@ -13,7 +13,7 @@ This setting corresponds to the data node roles: * <> * <> -NOTE: The <> role is not a valid data tier and cannot be used +NOTE: The <> role is not a valid data tier and cannot be used with the `_tier_preference` setting. The frozen tier stores <> exclusively. diff --git a/docs/reference/index-modules/allocation/filtering.asciidoc b/docs/reference/index-modules/allocation/filtering.asciidoc index 042b30dc32229..5da8e254cb4f2 100644 --- a/docs/reference/index-modules/allocation/filtering.asciidoc +++ b/docs/reference/index-modules/allocation/filtering.asciidoc @@ -114,7 +114,7 @@ The index allocation settings support the following built-in attributes: NOTE: `_tier` filtering is based on <> roles. Only a subset of roles are <> roles, and the generic -<> will match any tier filtering. +<> will match any tier filtering. You can use wildcards when specifying attribute values, for example: diff --git a/docs/reference/modules/cluster/allocation_filtering.asciidoc b/docs/reference/modules/cluster/allocation_filtering.asciidoc index e6705dc10361b..d0d2652059048 100644 --- a/docs/reference/modules/cluster/allocation_filtering.asciidoc +++ b/docs/reference/modules/cluster/allocation_filtering.asciidoc @@ -59,9 +59,9 @@ The cluster allocation settings support the following built-in attributes: NOTE: `_tier` filtering is based on <> roles. Only a subset of roles are <> roles, and the generic -<> will match any tier filtering. +<> will match any tier filtering. a subset of roles that are <> roles, but the generic -<> will match any tier filtering. +<> will match any tier filtering. You can use wildcards when specifying attribute values, for example: diff --git a/docs/reference/modules/discovery/bootstrapping.asciidoc b/docs/reference/modules/discovery/bootstrapping.asciidoc index 5120c1d17e69b..a885f1633ea49 100644 --- a/docs/reference/modules/discovery/bootstrapping.asciidoc +++ b/docs/reference/modules/discovery/bootstrapping.asciidoc @@ -2,7 +2,7 @@ === Bootstrapping a cluster Starting an Elasticsearch cluster for the very first time requires the initial -set of <> to be explicitly defined on one or +set of <> to be explicitly defined on one or more of the master-eligible nodes in the cluster. This is known as _cluster bootstrapping_. This is only required the first time a cluster starts up. Freshly-started nodes that are joining a running cluster obtain this diff --git a/docs/reference/modules/discovery/voting.asciidoc b/docs/reference/modules/discovery/voting.asciidoc index 04cae9d02ab66..f4bd4756d8978 100644 --- a/docs/reference/modules/discovery/voting.asciidoc +++ b/docs/reference/modules/discovery/voting.asciidoc @@ -2,7 +2,7 @@ === Voting configurations Each {es} cluster has a _voting configuration_, which is the set of -<> whose responses are counted when making +<> whose responses are counted when making decisions such as electing a new master or committing a new cluster state. Decisions are made only after a majority (more than half) of the nodes in the voting configuration respond. diff --git a/docs/reference/modules/gateway.asciidoc b/docs/reference/modules/gateway.asciidoc index 12490a3d43216..4716effb43083 100644 --- a/docs/reference/modules/gateway.asciidoc +++ b/docs/reference/modules/gateway.asciidoc @@ -4,7 +4,7 @@ The local gateway stores the cluster state and shard data across full cluster restarts. -The following _static_ settings, which must be set on every <>, +The following _static_ settings, which must be set on every <>, control how long a freshly elected master should wait before it tries to recover the <> and the cluster's data. diff --git a/docs/reference/modules/indices/request_cache.asciidoc b/docs/reference/modules/indices/request_cache.asciidoc index 088adfcacfcbf..f6ad65245836f 100644 --- a/docs/reference/modules/indices/request_cache.asciidoc +++ b/docs/reference/modules/indices/request_cache.asciidoc @@ -17,9 +17,9 @@ To learn more about the shard request cache, see <>. [discrete] ==== Cache settings -indices.requests.cache.size:: +`indices.requests.cache.size`:: (<>) The maximum size of the cache, as a percentage of the heap. Default: `1%`. -indices.requests.cache.expire:: +`indices.requests.cache.expire`:: (<>) The TTL for cached results. Stale results are automatically invalidated when the index is refreshed, so you shouldn't need to use this setting. diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index 0e99a9cd90ab3..d6e49bdebaca6 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -25,6 +25,10 @@ include::{es-ref-dir}/setup/important-settings/node-name.asciidoc[] [[node-roles]] ==== Node role settings +[[ml-node]] +[[master-node]] +[[data-node]] +[[transform-node]] You define a node's roles by setting `node.roles` in `elasticsearch.yml`. If you set `node.roles`, the node is only assigned the roles you specify. If you don't diff --git a/docs/reference/modules/remote-clusters.asciidoc b/docs/reference/modules/remote-clusters.asciidoc index ca1c507aa4ed9..87078c0f1956f 100644 --- a/docs/reference/modules/remote-clusters.asciidoc +++ b/docs/reference/modules/remote-clusters.asciidoc @@ -80,7 +80,7 @@ The _gateway nodes_ selection depends on the following criteria: + * *version*: Remote nodes must be compatible with the cluster they are registered to. -* *role*: By default, any non-<> node can act as a +* *role*: By default, any non-<> node can act as a gateway node. Dedicated master nodes are never selected as gateway nodes. * *attributes*: You can define the gateway nodes for a cluster by setting <> to `true`. diff --git a/docs/reference/node-roles.asciidoc b/docs/reference/node-roles.asciidoc index e414dd7ffcdfa..e8c1d9143a38e 100644 --- a/docs/reference/node-roles.asciidoc +++ b/docs/reference/node-roles.asciidoc @@ -103,18 +103,18 @@ excess data that prevents a node from starting. The following is a list of the roles that a node can perform in a cluster. A node can have one or more roles. -* <> (`master`): A node that is eligible to be +* <> (`master`): A node that is eligible to be <>, which controls the cluster. -* <> (`data`, `data_content`, `data_hot`, `data_warm`, `data_cold`, `data_frozen`): A node that has one of several data roles. Data nodes hold data and perform data related operations such as CRUD, search, and aggregations. You might use multiple data roles in a cluster so you can implement <>. +* <> (`data`, `data_content`, `data_hot`, `data_warm`, `data_cold`, `data_frozen`): A node that has one of several data roles. Data nodes hold data and perform data related operations such as CRUD, search, and aggregations. You might use multiple data roles in a cluster so you can implement <>. * <> (`ingest`): Ingest nodes are able to apply an <> to a document in order to transform and enrich the document before indexing. With a heavy ingest load, it makes sense to use dedicated ingest nodes and to not include the `ingest` role from nodes that have the `master` or `data` roles. * <> (`remote_cluster_client`): A node that is eligible to act as a remote client. -* <> (`ml`): A node that can run {ml-features}. If you want to use {ml-features}, there must be at least one {ml} node in your cluster. For more information, see <> and {ml-docs}/index.html[Machine learning in the {stack}]. +* <> (`ml`): A node that can run {ml-features}. If you want to use {ml-features}, there must be at least one {ml} node in your cluster. For more information, see <> and {ml-docs}/index.html[Machine learning in the {stack}]. -* <> (`transform`): A node that can perform {transforms}. If you want to use {transforms}, there must be at least one {transform} node in your cluster. For more information, see <> and <>. +* <> (`transform`): A node that can perform {transforms}. If you want to use {transforms}, there must be at least one {transform} node in your cluster. For more information, see <> and <>. [NOTE] [[coordinating-node]] @@ -141,7 +141,7 @@ memory and CPU in order to deal with the gather phase. [discrete] -[[master-node]] +[[master-node-role]] ==== Master-eligible node The master node is responsible for lightweight cluster-wide actions such as @@ -244,7 +244,7 @@ therefore ensure that the storage and networking available to the nodes in your cluster are good enough to meet your performance goals. [discrete] -[[data-node]] +[[data-node-role]] ==== Data nodes Data nodes hold the shards that contain the documents you have indexed. Data @@ -401,7 +401,7 @@ node.roles: [ remote_cluster_client ] ---- [discrete] -[[ml-node]] +[[ml-node-role]] ==== [xpack]#Machine learning node# {ml-cap} nodes run jobs and handle {ml} API requests. For more information, see @@ -420,7 +420,7 @@ your {anomaly-jobs}, the `remote_cluster_client` role is also required on all master-eligible nodes. Otherwise, the {dfeed} cannot start. See <>. [discrete] -[[transform-node]] +[[transform-node-role]] ==== [xpack]#{transform-cap} node# {transform-cap} nodes run {transforms} and handle {transform} API requests. For diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index c3bf84fa600d2..8c894b2e7fdf1 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -1947,3 +1947,8 @@ Refer to <>. === Delete _knn_search API Refer to <>. + +[role="exclude",id="advanced-configuration"] +=== Advanced configuration + +Refer to <>. \ No newline at end of file diff --git a/docs/reference/search/multi-search-template-api.asciidoc b/docs/reference/search/multi-search-template-api.asciidoc index b1c9518b1f2bc..dec64a5794c60 100644 --- a/docs/reference/search/multi-search-template-api.asciidoc +++ b/docs/reference/search/multi-search-template-api.asciidoc @@ -79,7 +79,7 @@ cross-cluster search requests. Defaults to `true`. `max_concurrent_searches`:: (Optional, integer) Maximum number of concurrent searches the API can run. -Defaults to +max(1, (# of <> * +Defaults to +max(1, (# of <> * min(<>, 10)))+. `rest_total_hits_as_int`:: diff --git a/docs/reference/search/multi-search.asciidoc b/docs/reference/search/multi-search.asciidoc index 9cafa756f035e..b42150580531c 100644 --- a/docs/reference/search/multi-search.asciidoc +++ b/docs/reference/search/multi-search.asciidoc @@ -91,7 +91,7 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailabl `max_concurrent_searches`:: (Optional, integer) Maximum number of concurrent searches the multi search API can execute. Defaults -to +max(1, (# of <> * min(<>, 10)))+. +to +max(1, (# of <> * min(<>, 10)))+. `max_concurrent_shard_requests`:: + diff --git a/docs/reference/setup/add-nodes.asciidoc b/docs/reference/setup/add-nodes.asciidoc index ba749782c092f..941a3e6c40f79 100644 --- a/docs/reference/setup/add-nodes.asciidoc +++ b/docs/reference/setup/add-nodes.asciidoc @@ -48,7 +48,7 @@ For more information about discovery and shard allocation, refer to As nodes are added or removed Elasticsearch maintains an optimal level of fault tolerance by automatically updating the cluster's _voting configuration_, which -is the set of <> whose responses are counted +is the set of <> whose responses are counted when making decisions such as electing a new master or committing a new cluster state. diff --git a/docs/reference/setup/advanced-configuration.asciidoc b/docs/reference/setup/advanced-configuration.asciidoc index 555650ff447e3..4140246ce6dea 100644 --- a/docs/reference/setup/advanced-configuration.asciidoc +++ b/docs/reference/setup/advanced-configuration.asciidoc @@ -1,5 +1,5 @@ [[set-jvm-options]] -==== Set JVM options +=== Set JVM options If needed, you can override the default JVM options by adding custom options files (preferred) or setting the `ES_JAVA_OPTS` environment variable. @@ -21,7 +21,7 @@ is recommended in most circumstances. NOTE: Do not modify the root `jvm.options` file. Use files in `jvm.options.d/` instead. [[jvm-options-syntax]] -===== JVM options syntax +==== JVM options syntax A JVM options file contains a line-delimited list of JVM arguments. Arguments are preceded by a dash (`-`). @@ -63,7 +63,7 @@ and ignored. Lines that aren't commented out and aren't recognized as valid JVM arguments are rejected and {es} will fail to start. [[jvm-options-env]] -===== Use environment variables to set JVM options +==== Use environment variables to set JVM options In production, use JVM options files to override the default settings. In testing and development environments, @@ -153,10 +153,10 @@ NOTE: If you are running {es} as a Windows service, you can change the heap size using the service manager. See <>. [[heap-dump-path]] -include::important-settings/heap-dump-path.asciidoc[] +include::important-settings/heap-dump-path.asciidoc[leveloffset=-1] [[gc-logging]] -include::important-settings/gc-logging.asciidoc[] +include::important-settings/gc-logging.asciidoc[leveloffset=-1] [[error-file-path]] -include::important-settings/error-file.asciidoc[] \ No newline at end of file +include::important-settings/error-file.asciidoc[leveloffset=-1] \ No newline at end of file diff --git a/docs/reference/setup/important-settings.asciidoc b/docs/reference/setup/important-settings.asciidoc index f90d8a0503bb5..26f9c79cb6693 100644 --- a/docs/reference/setup/important-settings.asciidoc +++ b/docs/reference/setup/important-settings.asciidoc @@ -31,6 +31,8 @@ To learn about this setting, and how to migrate away from it, refer to <>. The snapshot's repository must be +<>. The snapshot's repository must be <> and available to the cluster. * The snapshot and cluster versions must be compatible. See diff --git a/docs/reference/snapshot-restore/take-snapshot.asciidoc b/docs/reference/snapshot-restore/take-snapshot.asciidoc index 711fcfe4cc484..1ae2258c7da89 100644 --- a/docs/reference/snapshot-restore/take-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/take-snapshot.asciidoc @@ -46,7 +46,7 @@ taking snapshots at different time intervals. include::register-repository.asciidoc[tag=kib-snapshot-prereqs] * You can only take a snapshot from a running cluster with an elected -<>. +<>. * A snapshot repository must be <> and available to the cluster. diff --git a/docs/reference/transform/setup.asciidoc b/docs/reference/transform/setup.asciidoc index dab357546d93e..3171086e43d61 100644 --- a/docs/reference/transform/setup.asciidoc +++ b/docs/reference/transform/setup.asciidoc @@ -11,7 +11,7 @@ To use {transforms}, you must have: -* at least one <>, +* at least one <>, * management features visible in the {kib} space, and * security privileges that: + diff --git a/docs/reference/upgrade/disable-shard-alloc.asciidoc b/docs/reference/upgrade/disable-shard-alloc.asciidoc index f69a673095257..4672ea65446b4 100644 --- a/docs/reference/upgrade/disable-shard-alloc.asciidoc +++ b/docs/reference/upgrade/disable-shard-alloc.asciidoc @@ -5,7 +5,7 @@ starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by <> of replicas before -shutting down <>: +shutting down <>: [source,console] -------------------------------------------------- From de2d65cf933514128fb4a521a4e643aeca1ec71c Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Tue, 31 Dec 2024 13:00:16 -0500 Subject: [PATCH 09/11] try again --- docs/reference/modules/node.asciidoc | 12 ++++-------- docs/reference/redirects.asciidoc | 5 ----- docs/reference/setup/advanced-configuration.asciidoc | 2 +- 3 files changed, 5 insertions(+), 14 deletions(-) diff --git a/docs/reference/modules/node.asciidoc b/docs/reference/modules/node.asciidoc index d6e49bdebaca6..e8dd995623a1d 100644 --- a/docs/reference/modules/node.asciidoc +++ b/docs/reference/modules/node.asciidoc @@ -25,26 +25,22 @@ include::{es-ref-dir}/setup/important-settings/node-name.asciidoc[] [[node-roles]] ==== Node role settings -[[ml-node]] -[[master-node]] -[[data-node]] -[[transform-node]] You define a node's roles by setting `node.roles` in `elasticsearch.yml`. If you set `node.roles`, the node is only assigned the roles you specify. If you don't set `node.roles`, the node is assigned the following roles: -* `master` -* `data` +* [[master-node]]`master` +* [[data-node]]`data` * `data_content` * `data_hot` * `data_warm` * `data_cold` * `data_frozen` * `ingest` -* `ml` +* [[ml-node]]`ml` * `remote_cluster_client` -* `transform` +* [[transform-node]]`transform` The following additional roles are available: diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index 8c894b2e7fdf1..c3bf84fa600d2 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -1947,8 +1947,3 @@ Refer to <>. === Delete _knn_search API Refer to <>. - -[role="exclude",id="advanced-configuration"] -=== Advanced configuration - -Refer to <>. \ No newline at end of file diff --git a/docs/reference/setup/advanced-configuration.asciidoc b/docs/reference/setup/advanced-configuration.asciidoc index 4140246ce6dea..16bdb8dd7f068 100644 --- a/docs/reference/setup/advanced-configuration.asciidoc +++ b/docs/reference/setup/advanced-configuration.asciidoc @@ -1,4 +1,4 @@ -[[set-jvm-options]] +[[advanced-configuration]] === Set JVM options If needed, you can override the default JVM options by adding custom options From ba8b5f83bb909263aab56ce56246e7887315b7ca Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Tue, 31 Dec 2024 13:15:22 -0500 Subject: [PATCH 10/11] another one --- docs/reference/setup/advanced-configuration.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/setup/advanced-configuration.asciidoc b/docs/reference/setup/advanced-configuration.asciidoc index 16bdb8dd7f068..73b210ea559b2 100644 --- a/docs/reference/setup/advanced-configuration.asciidoc +++ b/docs/reference/setup/advanced-configuration.asciidoc @@ -1,6 +1,7 @@ [[advanced-configuration]] === Set JVM options +[[set-jvm-options]] If needed, you can override the default JVM options by adding custom options files (preferred) or setting the `ES_JAVA_OPTS` environment variable. From 137d4556f54b5734416edf6a0ff56943a06fe144 Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Tue, 31 Dec 2024 14:15:59 -0500 Subject: [PATCH 11/11] fix nesting --- docs/reference/setup/logging-config.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/setup/logging-config.asciidoc b/docs/reference/setup/logging-config.asciidoc index 81a28e6c51335..04e9ba3f0bef9 100644 --- a/docs/reference/setup/logging-config.asciidoc +++ b/docs/reference/setup/logging-config.asciidoc @@ -1,5 +1,5 @@ [[logging]] -=== Logging +== Elasticsearch application logging You can use {es}'s application logs to monitor your cluster and diagnose issues. If you run {es} as a service, the default location of the logs varies based on @@ -304,6 +304,7 @@ The user ID is included in the `X-Opaque-ID` field in deprecation JSON logs. Deprecation logs can be indexed into `.logs-deprecation.elasticsearch-default` data stream `cluster.deprecation_indexing.enabled` setting is set to true. +[discrete] ==== Deprecation logs throttling :es-rate-limiting-filter-java-doc: {elasticsearch-javadoc}/org/elasticsearch/common/logging/RateLimitingFilter.html Deprecation logs are deduplicated based on a deprecated feature key