|
1 | 1 | --- |
| 2 | +navigation_title: High availability |
2 | 3 | applies_to: |
3 | 4 | deployment: |
4 | 5 | ece: all |
5 | 6 | mapped_pages: |
6 | 7 | - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-ha.html |
7 | 8 | --- |
8 | 9 |
|
9 | | -# High availability [ece-ha] |
| 10 | +# High availability in ECE |
10 | 11 |
|
11 | | -Ensuring high availability in {{ece}} (ECE) requires careful planning and implementation across multiple areas, including availability zones, master nodes, replica shards, snapshot backups, and Zookeeper nodes. |
| 12 | +Ensuring high availability (HA) in {{ece}} (ECE) requires careful planning and implementation across multiple areas, including availability zones, master nodes, replica shards, snapshot backups, and Zookeeper nodes. |
12 | 13 |
|
13 | | -This section describes key considerations and best practices to prevent downtime and data loss at both the ECE platform level and within orchestrated deployments. |
14 | | - |
15 | | -## Availability zones [ece-ece-ha-1-az] |
16 | | - |
17 | | -Fault tolerance for ECE is based around the concept of *availability zones*. |
18 | | - |
19 | | -An availability zone contains resources available to an ECE installation that are isolated from other availability zones to safeguard against potential failure. |
20 | | - |
21 | | -Planning for a fault-tolerant installation with multiple availability zones means avoiding any single point of failure that could bring down ECE. |
22 | | - |
23 | | -The main difference between ECE installations that include two or three availability zones is that three availability zones enable ECE to create clusters with a *tiebreaker*. If you have only two availability zones in total in your installation, no tiebreaker is created. |
| 14 | +::::{note} |
| 15 | +This section focuses on ensuring high availability at the ECE platform level. For deployment-level considerations, including resiliency, scaling, and performance optimizations for running {{es}} and {{kib}}, refer to the general [production guidance](/deploy-manage/production-guidance.md). |
| 16 | +:::: |
24 | 17 |
|
25 | | -We recommend that for each deployment you use at least two availability zones for production and three for mission-critical systems. Using more than three availability zones for a deployment is not required nor supported. Availability zones are intended for high availability, not scalability. |
| 18 | +To maintain a minimum HA, you should deploy at least two ECE hosts for each role—**allocator, constructor, and proxy**—and at least three hosts for the **director** role, which runs ZooKeeper and requires quorum to operate reliably. |
26 | 19 |
|
27 | | -::::{warning} |
28 | | -{{es}} clusters that are set up to use only one availability zone are not [highly available](/deploy-manage/production-guidance/availability-and-resilience.md) and are at risk of data loss. To safeguard against data loss, you must use at least two {{ece}} availability zones. |
29 | | -:::: |
| 20 | +In addition, to improve resiliency at the availability zone level, it’s recommended to deploy ECE across three availability zones, with at least two allocators per zone and spare capacity to accommodate instance failover and workload redistribution in case of failures. |
30 | 21 |
|
31 | | -::::{warning} |
32 | | -Increasing the number of zones should not be used to add more resources. The concept of zones is meant for High Availability (2 zones) and Fault Tolerance (3 zones), but neither will work if the cluster relies on the resources from those zones to be operational. The recommendation is to scale up the resources within a single zone until the cluster can take the full load (add some buffer to be prepared for a peak of requests), then scale out by adding additional zones depending on your requirements: 2 zones for High Availability, 3 zones for Fault Tolerance. |
33 | | -:::: |
| 22 | +All Elastic-documented architectures recommend using three availability zones with ECE roles distributed across all zones. Refer to [deployment scenarios](./identify-deployment-scenario.md) for examples of small, medium, and large installations. |
34 | 23 |
|
| 24 | +Regardless of the resiliency level at the platform level, it’s important to also [configure your deployments for high availability](/deploy-manage/production-guidance/availability-and-resilience/resilience-in-ech.md). |
35 | 25 |
|
36 | | -## Master nodes [ece-ece-ha-2-master-nodes] |
| 26 | +## Availability zones [ece-ece-ha-1-az] |
37 | 27 |
|
38 | | -Tiebreakers are used in distributed clusters to avoid cases of [split brain](https://en.wikipedia.org/wiki/Split-brain_(computing)), where an {{es}} cluster splits into multiple, autonomous parts that continue to handle requests independently of each other, at the risk of affecting cluster consistency and data loss. A split-brain scenario is avoided by making sure that a minimum number of [master-eligible nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#master-node) must be present in order for any part of the cluster to elect a master node and accept user requests. To prevent multiple parts of a cluster from being eligible, there must be a [quorum-based majority](/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md) of `(n/2)+1` nodes, where `n` is the number of master-eligible nodes in the cluster. The minimum number of master nodes to reach quorum in a two-node cluster is the same as for a three-node cluster: two nodes must be available. |
| 28 | +Fault tolerance for ECE is based around the concept of *availability zones*. |
39 | 29 |
|
40 | | -When you create a cluster with nodes in two availability zones when a third zone is available, ECE can create a tiebreaker in the third availability zone to help establish quorum in case of loss of an availability zone. The extra tiebreaker node that helps to provide quorum does not have to be a full-fledged and expensive node, as it does not hold data. For example: By tagging allocators hosts in ECE, can you create a cluster with eight nodes each in zones `ece-1a` and `ece-1b`, for a total of 16 nodes, and one tiebreaker node in zone `ece-1c`. This cluster can lose any of the three availability zones whilst maintaining quorum, which means that the cluster can continue to process user requests, provided that there is sufficient capacity available when an availability zone goes down. |
| 30 | +An availability zone contains resources available to an ECE installation that are isolated from other availability zones to safeguard against potential failure. |
41 | 31 |
|
42 | | -By default, each node in an {{es}} cluster is a master-eligible node and a data node. In larger clusters, such as production clusters, it’s a good practice to split the roles, so that master nodes are not handling search or indexing work. When you create a cluster, you can specify to use dedicated [master-eligible nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#master-node), one per availability zone. |
| 32 | +Planning for a fault-tolerant installation with multiple availability zones means avoiding any single point of failure that could bring down ECE. |
43 | 33 |
|
44 | | -::::{warning} |
45 | | -Clusters that only have two or fewer master-eligible node are not [highly available](/deploy-manage/production-guidance/availability-and-resilience.md) and are at risk of data loss. You must have [at least three master-eligible nodes](/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md). |
| 34 | +::::{important} |
| 35 | +Adding more availability zones should not be used as a way to increase processing capacity and performance. The concept of zones is meant for high availability (2 zones) and fault tolerance (3 zones), but neither will work if your deployments rely on the resources from those zones to be operational. Refer to [scaling considerations](/deploy-manage/production-guidance/scaling-considerations.md#scaling-and-fault-tolerance) for more information. |
46 | 36 | :::: |
47 | 37 |
|
48 | | -## Replica shards [ece-ece-ha-3-replica-shards] |
| 38 | +The main difference between ECE installations that include two or three availability zones is that three availability zones enable ECE to create {{es}} clusters with a [voting-only tiebreaker](/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#voting-only-node) instance. If you have only two availability zones in your installation, no tiebreaker can be placed in a third zone, limiting the cluster’s ability to tolerate certain failures. |
49 | 39 |
|
50 | | -With multiple {{es}} nodes in multiple availability zones you have the recommended hardware, the next thing to consider is having the recommended index replication. Each index, with the exception of searchable snapshot indexes, should have one or more replicas. Use the index settings API to find any indices with no replica: |
| 40 | +## Tiebreaker master nodes |
51 | 41 |
|
52 | | -```sh |
53 | | -GET _all/_settings/index.number_of_replicas |
54 | | -``` |
| 42 | +A tiebreaker is a lightweight voting-only node used in distributed clusters to help avoid split-brain scenarios, where the cluster could incorrectly split into multiple autonomous parts during a network partition. |
55 | 43 |
|
56 | | -::::{warning} |
57 | | -Indices with no replica, except for [searchable snapshot indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), are not highly available. You should use replicas to mitigate against possible data loss. |
58 | | -:::: |
| 44 | +When you create a cluster with nodes in two availability zones when a third zone is available, ECE can create a tiebreaker in the third availability zone to help establish quorum in case of loss of an availability zone. The extra tiebreaker node that helps to provide quorum does not have to be a full-fledged and expensive node, as it does not hold data. For example: By [tagging allocators](./ece-configuring-ece-tag-allocators.md) hosts in ECE, can you create a cluster with eight nodes each in zones `ece-1a` and `ece-1b`, for a total of 16 nodes, and one tiebreaker node in zone `ece-1c`. This cluster can lose any of the three availability zones whilst maintaining quorum, which means that the cluster can continue to process user requests, provided that there is sufficient capacity available when an availability zone goes down. |
59 | 45 |
|
60 | | -Refer to [](../../reference-architectures.md) for information about {{es}} architectures. |
| 46 | +## Zookeeper nodes |
61 | 47 |
|
62 | | -## Snapshot backups [ece-ece-ha-4-snapshot] |
| 48 | +Make sure you have three Zookeepers—by default, on the Director host—for your ECE installation. Similar to three {{es}} master nodes can form a quorum, three Zookeepers can form the quorum for high availability purposes. |
63 | 49 |
|
64 | | -You should configure and use [{{es}} snapshots](/deploy-manage/tools/snapshot-and-restore.md). Snapshots provide a way to backup and restore your {{es}} indices. They can be used to copy indices for testing, to recover from failures or accidental deletions, or to migrate data to other deployments. We recommend configuring an [{{ece}}-level repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to apply across all deployments. See [Work with snapshots](../../tools/snapshot-and-restore.md) for more guidance. |
| 50 | +Backing up Zookeeper data directory is also recommended. Refer to [rebuilding a broken Zookeeper quorum](../../../troubleshoot/deployments/cloud-enterprise/rebuilding-broken-zookeeper-quorum.md) for more guidance. |
65 | 51 |
|
66 | | -## Furthermore considerations [ece-ece-ha-5-other] |
| 52 | +## External resources accessibility |
67 | 53 |
|
68 | | -* Make sure you have three Zookeepers - by default, on the Director host - for your ECE installation. Similar to three Elasticsearch master nodes can form a quorum, three Zookeepers can forum the quorum for high availability purposes. Backing up Zookeeper data directory is also recommended, read [this doc](../../../troubleshoot/deployments/cloud-enterprise/rebuilding-broken-zookeeper-quorum.md) for more guidance. |
| 54 | +If you’re using a [private Docker registry server](ece-install-offline-with-registry.md) or hosting any [custom bundles and plugins](../../../solutions/search/full-text/search-with-synonyms.md) on a web server, make sure these resources are accessible from all ECE allocators, so they can continue to be accessed in the event of a network partition or zone outage. |
69 | 55 |
|
70 | | -* Make sure that if you’re using a [private Docker registry server](ece-install-offline-with-registry.md) or are using any [custom bundles and plugins](../../../solutions/search/full-text/search-with-synonyms.md) hosted on a web server, that these are available to all ECE allocators, so that they can continue to be accessed in the event of a network partition or zone outage. |
| 56 | +## Other recommendations |
71 | 57 |
|
72 | | -* Don’t delete containers unless guided by Elastic Support or there’s public documentation explicitly describing this as required action. Otherwise, it can cause issues and you may lose access or functionality of your {{ece}} platform. See [Troubleshooting container engines](../../../troubleshoot/deployments/cloud-enterprise/troubleshooting-container-engines.md) for more information. |
| 58 | +Avoid deleting containers unless explicitly instructed by Elastic Support or official documentation. Doing so may lead to unexpected issues or loss of access to your {{ece}} platform. For more details, refer to [](/troubleshoot/deployments/cloud-enterprise/troubleshooting-container-engines.md). |
73 | 59 |
|
74 | | -If in doubt, please [contact support for help](../../../troubleshoot/deployments/cloud-enterprise/ask-for-help.md). |
| 60 | +If in doubt, please [contact support for help](/troubleshoot/index.md#contact-us). |
0 commit comments