Merge branch 'main' into fix-upgrade-links

Author: jmikell821

deploy-manage/distributed-architecture.md

@@ -1,6 +1,9 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/_data_store_architecture.html
+applies_to:
+  stack:
+  serverless:
 ---
 
 # Distributed architecture [_data_store_architecture]
@@ -9,12 +12,15 @@ mapped_pages:
 
 The topics in this section provides information about the architecture of {{es}} and how it stores and retrieves data:
 
-* [Nodes and shards](distributed-architecture/clusters-nodes-shards.md): Learn about the basic building blocks of an {{es}} cluster, including nodes, shards, primaries, and replicas.
-* [Node roles](distributed-architecture/clusters-nodes-shards/node-roles.md): Learn about the different roles that nodes can have in an {{es}} cluster.
+::::{note}
+{{serverless-full}} scales with your workload and automates nodes, shards, and replicas for you. Some of the content in this section does not apply to you if you are using {{serverless-full}}. Instead, the information in this section will provide you information about how the platform works for you.
+::::
+
+* [Cluster, nodes, and shards](distributed-architecture/clusters-nodes-shards.md): Learn about the basic building blocks of an {{es}} cluster, including nodes, shards, primaries, and replicas.
+  * [Node roles](distributed-architecture/clusters-nodes-shards/node-roles.md): Learn about the different roles that nodes can have in an {{es}} cluster.
 * [Reading and writing documents](distributed-architecture/reading-and-writing-documents.md): Learn how {{es}} replicates read and write operations across shards and shard copies.
 * [Shard allocation, relocation, and recovery](distributed-architecture/shard-allocation-relocation-recovery.md): Learn how {{es}} allocates and balances shards across nodes.
-
-    * [Shard allocation awareness](distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md): Learn how to use custom node attributes to distribute shards across different racks or availability zones.
-
+  * [Shard allocation awareness](distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md): Learn how to use custom node attributes to distribute shards across different racks or availability zones.
+* [Disocvery and cluster formation](distributed-architecture/discovery-cluster-formation.md): Learn about the cluster formation process including voting, adding nodes and publishing the cluster state.
 * [Shard request cache](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md): Learn how {{es}} caches search requests to improve performance.
-
+* [Kibana task management](distributed-architecture/kibana-tasks-management.md): Learn how {{kib}} runs background tasks and distribute work across multiple {{kib}} instances to be persistent and scale with your deployment.

deploy-manage/distributed-architecture/clusters-nodes-shards.md

@@ -1,11 +1,14 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/nodes-shards.html
+applies_to:
+  stack:
+  serverless:
 ---
 
 # Clusters, nodes, and shards [nodes-shards]
 
-::::{note} 
+::::{note}
 Nodes and shards are what make {{es}} distributed and scalable. These concepts aren’t essential if you’re just getting started. How you [deploy {{es}}](../../get-started/deployment-options.md) in production determines what you need to know:
 
 * **Self-managed {{es}}**: You are responsible for setting up and managing nodes, clusters, shards, and replicas. This includes managing the underlying infrastructure, scaling, and ensuring high availability through failover and backup strategies.
@@ -14,24 +17,20 @@ Nodes and shards are what make {{es}} distributed and scalable. These concepts a
 
 ::::
 
-
 You can add servers (*nodes*) to a cluster to increase capacity, and {{es}} automatically distributes your data and query load across all of the available nodes.
 
 Elastic is able to distribute your data across nodes by subdividing an index into *shards*. Each index in {{es}} is a grouping of one or more physical shards, where each shard is a self-contained Lucene index containing a subset of the documents in the index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple nodes, {{es}} increases indexing and query capacity.
 
 There are two types of shards: *primaries* and *replicas*. Each document in an index belongs to one primary shard. A replica shard is a copy of a primary shard. Replicas maintain redundant copies of your data across the nodes in your cluster. This protects against hardware failure and increases capacity to serve read requests like searching or retrieving a document.
 
-::::{tip} 
+::::{tip}
 The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time, without interrupting indexing or query operations.
-
 ::::
 
-
 Shard copies in your cluster are automatically balanced across nodes to provide scale and high availability. All nodes are aware of all the other nodes in the cluster and can forward client requests to the appropriate node. This allows {{es}} to distribute indexing and query load across the cluster.
 
 If you’re exploring {{es}} for the first time or working in a development environment, then you can use a cluster with a single node and create indices with only one shard. However, in a production environment, you should build a cluster with multiple nodes and indices with multiple shards to increase performance and resilience.
 
 * To learn about optimizing the number and size of shards in your cluster, refer to [Size your shards](../production-guidance/optimize-performance/size-shards.md).
 * To learn about how read and write operations are replicated across shards and shard copies, refer to [Reading and writing documents](reading-and-writing-documents.md).
-* To adjust how shards are allocated and balanced across nodes, refer to [Shard allocation, relocation, and recovery](shard-allocation-relocation-recovery.md).
-
+* To adjust how shards are allocated and balanced across nodes, refer to [Shard allocation, relocation, and recovery](shard-allocation-relocation-recovery.md).

deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md

@@ -1,6 +1,9 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/node-roles-overview.html
+applies_to:
+  stack:
+  self:
 ---
 
 # Node roles [node-roles-overview]
@@ -9,7 +12,6 @@ Any time that you start an instance of {{es}}, you are starting a *node*. A coll
 
 Each node performs one or more roles. Roles control the behavior of the node in the cluster.
 
-
 ## Set node roles [set-node-roles]
 
 You define a node’s roles by setting `node.roles` in [`elasticsearch.yml`](../../deploy/self-managed/configure-elasticsearch.md). If you set `node.roles`, the node is only assigned the roles you specify. If you don’t set `node.roles`, the node is assigned the following roles:
@@ -30,10 +32,7 @@ You define a node’s roles by setting `node.roles` in [`elasticsearch.yml`](../
 If you set `node.roles`, ensure you specify every node role your cluster needs. Every cluster requires the following node roles:
 
 * `master`
-*
-
-    `data_content` and `data_hot`<br> OR<br> `data`
-
+* `data_content` and `data_hot`<br> OR<br> `data`
 
 Some {{stack}} features also require specific node roles:
 

deploy-manage/distributed-architecture/discovery-cluster-formation.md

@@ -1,8 +1,14 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html
+applies_to:
+  stack:
 ---
 
+::::{important}
+The information provided in this section is applicable to all deployment types. However, the configuration settings detailed here are only valid for self-managed {{es}} deployments. For {{ecloud}} and {{serverless-full}} deployments this seciton should only be used for general information.
+::::
+
 # Discovery and cluster formation [modules-discovery]
 
 The discovery and cluster formation processes are responsible for discovering nodes, electing a master, forming a cluster, and publishing the cluster state each time it changes.
@@ -33,9 +39,16 @@ The following processes and settings are part of discovery and cluster formation
 [Settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md)
 :   There are settings that enable users to influence the discovery, cluster formation, master election and fault detection processes.
 
+[Quorum-based decision making](discovery-cluster-formation/modules-discovery-quorums.md): How {{es}} uses a quorum-based voting mechanism to make decisions even if some nodes are unavailable.
 
+[Voting configurations](discovery-cluster-formation/modules-discovery-voting.md): How {{es}} automatically updates voting configurations as nodes leave and join a cluster.
 
+[Bootstrapping a cluster](discovery-cluster-formation/modules-discovery-bootstrap-cluster.md): Bootstrapping a cluster is required when an {{es}} cluster starts up for the very first time. In [development mode](../deploy/self-managed/bootstrap-checks.md#dev-vs-prod-mode), with no discovery settings configured, this is automatically performed by the nodes themselves. As this auto-bootstrapping is [inherently unsafe](discovery-cluster-formation/modules-discovery-quorums.md), running a node in [production mode](../deploy/self-managed/bootstrap-checks.md#dev-vs-prod-mode) requires bootstrapping to be [explicitly configured](discovery-cluster-formation/modules-discovery-bootstrap-cluster.md).
 
+[Adding and removing master-eligible nodes](../maintenance/add-and-remove-elasticsearch-nodes.md): It is recommended to have a small and fixed number of master-eligible nodes in a cluster, and to scale the cluster up and down by adding and removing master-ineligible nodes only. However there are situations in which it may be desirable to add or remove some master-eligible nodes to or from a cluster. This section describes the process for adding or removing master-eligible nodes, including the extra steps that need to be performed when removing more than half of the master-eligible nodes at the same time.
 
+[Publishing the cluster state](discovery-cluster-formation/cluster-state-overview.md#cluster-state-publishing): Cluster state publishing is the process by which the elected master node updates the cluster state on all the other nodes in the cluster.
 
+[Cluster fault detection](discovery-cluster-formation/cluster-fault-detection.md): {{es}} performs health checks to detect and remove faulty nodes.
 
+[Settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md): There are settings that enable users to influence the discovery, cluster formation, master election and fault detection processes.

deploy-manage/distributed-architecture/discovery-cluster-formation/cluster-fault-detection.md

@@ -1,6 +1,8 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-fault-detection.html
+applies_to:
+  stack:
 ---
 
 # Cluster fault detection [cluster-fault-detection]

deploy-manage/distributed-architecture/discovery-cluster-formation/cluster-state-overview.md

@@ -1,6 +1,8 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-state-overview.html
+applies_to:
+  stack:
 ---
 
 # Cluster state [cluster-state-overview]

deploy-manage/distributed-architecture/discovery-cluster-formation/discovery-hosts-providers.md

@@ -1,6 +1,8 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-hosts-providers.html
+applies_to:
+  stack:
 ---
 
 # Discovery [discovery-hosts-providers]
@@ -23,7 +25,6 @@ By default the cluster formation module offers two seed hosts providers to confi
 
 Each seed hosts provider yields the IP addresses or hostnames of the seed nodes. If it returns any hostnames then these are resolved to IP addresses using a DNS lookup. If a hostname resolves to multiple IP addresses then {{es}} tries to find a seed node at all of these addresses. If the hosts provider does not explicitly give the TCP port of the node by then, it will implicitly use the first port in the port range given by `transport.profiles.default.port`, or by `transport.port` if `transport.profiles.default.port` is not set. The number of concurrent lookups is controlled by `discovery.seed_resolver.max_concurrent_resolvers` which defaults to `10`, and the timeout for each lookup is controlled by `discovery.seed_resolver.timeout` which defaults to `5s`. Note that DNS lookups are subject to [JVM DNS caching](../../deploy/self-managed/networkaddress-cache-ttl.md).
 
-
 #### Settings-based seed hosts provider [settings-based-hosts-provider]
 
 The settings-based seed hosts provider uses a node setting to configure a static list of the addresses of the seed nodes. These addresses can be given as hostnames or IP addresses; hosts specified as hostnames are resolved to IP addresses during each round of discovery.
@@ -40,8 +41,6 @@ discovery.seed_hosts:
 1. The port will default to `transport.profiles.default.port` and fallback to `transport.port` if not specified.
 2. If a hostname resolves to multiple IP addresses, {{es}} will attempt to connect to every resolved address.
 
-
-
 #### File-based seed hosts provider [file-based-hosts-provider]
 
 The file-based seed hosts provider configures a list of hosts via an external file.  {{es}} reloads this file when it changes, so that the list of seed nodes can change dynamically without needing to restart each node. For example, this gives a convenient mechanism for an {{es}} instance that is run in a Docker container to be dynamically supplied with a list of IP addresses to connect to when those IP addresses may not be known at node startup.
@@ -72,19 +71,14 @@ Host names are allowed instead of IP addresses and are resolved by DNS as descri
 
 You can also add comments to this file. All comments must appear on their lines starting with `#` (i.e. comments cannot start in the middle of a line).
 
-
 #### EC2 hosts provider [ec2-hosts-provider]
 
 The [EC2 discovery plugin](elasticsearch://reference/elasticsearch-plugins/discovery-ec2.md) adds a hosts provider that uses the [AWS API](https://github.com/aws/aws-sdk-java) to find a list of seed nodes.
 
-
 #### Azure Classic hosts provider [azure-classic-hosts-provider]
 
 The [Azure Classic discovery plugin](elasticsearch://reference/elasticsearch-plugins/discovery-azure-classic.md) adds a hosts provider that uses the Azure Classic API find a list of seed nodes.
 
-
 #### Google Compute Engine hosts provider [gce-hosts-provider]
 
 The [GCE discovery plugin](elasticsearch://reference/elasticsearch-plugins/discovery-gce.md) adds a hosts provider that uses the GCE API find a list of seed nodes.
-
-

deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster.md

@@ -1,6 +1,8 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html
+applies_to:
+  stack:
 ---
 
 # Bootstrapping a cluster [modules-discovery-bootstrap-cluster]
@@ -23,7 +25,6 @@ If you leave `cluster.initial_master_nodes` in place once the cluster has formed
 
 ::::
 
-
 The simplest way to create a new cluster is for you to select one of your master-eligible nodes that will bootstrap itself into a single-node cluster, which all the other nodes will then join. This simple approach is not resilient to failures until the other master-eligible nodes have joined the cluster. For example, if you have a master-eligible node with [node name](../../deploy/self-managed/important-settings-configuration.md#node-name) `master-a` then configure it as follows (omitting `cluster.initial_master_nodes` from the configuration of all other nodes):
 
 ```yaml
@@ -43,7 +44,6 @@ cluster.initial_master_nodes:
 You must set `cluster.initial_master_nodes` to the same list of nodes on each node on which it is set in order to be sure that only a single cluster forms during bootstrapping. If `cluster.initial_master_nodes` varies across the nodes on which it is set then you may bootstrap multiple clusters. It is usually not possible to recover from this situation without losing data.
 ::::
 
-
 ::::{admonition} Node name formats must match
 :name: modules-discovery-bootstrap-cluster-fqdns
 
@@ -60,12 +60,10 @@ This message shows the node names `master-a.example.com` and `master-b.example.c
 
 ::::
 
-
 ## Choosing a cluster name [bootstrap-cluster-name]
 
 The [`cluster.name`](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-name) setting enables you to create multiple clusters which are separated from each other. Nodes verify that they agree on their cluster name when they first connect to each other, and Elasticsearch will only form a cluster from nodes that all have the same cluster name. The default value for the cluster name is `elasticsearch`, but it is recommended to change this to reflect the logical name of the cluster.
 
-
 ## Auto-bootstrapping in development mode [bootstrap-auto-bootstrap]
 
 By default each node will automatically bootstrap itself into a single-node cluster the first time it starts. If any of the following settings are configured then auto-bootstrapping will not take place:
@@ -98,6 +96,3 @@ If you intended to form a new multi-node cluster but instead bootstrapped a coll
 6. Remove `cluster.initial_master_nodes` from every node’s configuration.
 
 ::::
-
-
-

deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md

@@ -1,6 +1,8 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-quorums.html
+applies_to:
+  stack:
 ---
 
 # Quorum-based decision making [modules-discovery-quorums]
@@ -18,15 +20,12 @@ If you stop half or more of the nodes in the voting configuration at the same ti
 
 ::::
 
-
 After a master-eligible node has joined or left the cluster the elected master may issue a cluster-state update that adjusts the voting configuration to match, and this can take a short time to complete. It is important to wait for this adjustment to complete before removing more nodes from the cluster. See [Removing master-eligible nodes](../../maintenance/add-and-remove-elasticsearch-nodes.md#modules-discovery-removing-nodes) for more information.
 
-
 ## Master elections [_master_elections]
 
 Elasticsearch uses an election process to agree on an elected master node, both at startup and if the existing elected master fails. Any master-eligible node can start an election, and normally the first election that takes place will succeed. Elections only usually fail when two nodes both happen to start their elections at about the same time, so elections are scheduled randomly on each node to reduce the probability of this happening. Nodes will retry elections until a master is elected, backing off on failure, so that eventually an election will succeed (with arbitrarily high probability). The scheduling of master elections are controlled by the [master election settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md#master-election-settings).
 
-
 ## Cluster maintenance, rolling restarts and migrations [_cluster_maintenance_rolling_restarts_and_migrations]
 
 Many cluster maintenance tasks involve temporarily shutting down one or more nodes and then starting them back up again. By default {{es}} can remain available if one of its master-eligible nodes is taken offline, such as during a rolling upgrade. Furthermore, if multiple nodes are stopped and then started again then it will automatically recover, such as during a full cluster restart. There is no need to take any further action with the APIs described here in these cases, because the set of master nodes is not changing permanently.