You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
% start bringing https://www.elastic.co/guide/en/elasticsearch/reference/current/scalability.html here
10
+
% try to merge https://www.elastic.co/guide/en/cloud/current/ec-planning.html and https://www.elastic.co/guide/en/cloud/current/ec-best-practices-data.html
11
+
% mention all deployment types! what the user needs to be aware for orchestrated deployments.
8
12
13
+
Many teams rely on {{es}} to run their key services. To keep these services running, you can design your {{es}} deployment to keep {{es}} available, even in case of large-scale outages. To keep it running fast, you also can design your deployment to be responsive to production workloads.
14
+
15
+
{{es}} is built to be always available and to scale with your needs. It does this using a [distributed architecture](./distributed-architecture.md). By distributing your cluster, you can keep Elastic online and responsive to requests.
16
+
17
+
In case of failure, {{es}} offers tools for [cross-cluster replication](./tools/cross-cluster-replication.md) and [cluster snapshots](./tools/snapshot-and-restore.md) that can help you fall back or recover quickly. You can also use cross-cluster replication to serve requests based on the geographic location of your users and your resources.
18
+
19
+
% not very relevant
20
+
{{es}} also offers security and monitoring tools to help you keep your cluster highly available.
21
+
22
+
23
+
Recommendations out there:
24
+
- Use multiple nodes and shards
25
+
26
+
## Section overview
27
+
28
+
Running the {{stack}} in production requires careful planning to ensure resilience, performance, and scalability. This section outlines best practices and recommendations for optimizing {{es}} and {{kib}} in production environments.
29
+
30
+
* High availability (HA) and resilience
31
+
* Resilience in small clusters
32
+
* Resilienve in larger clusters
33
+
34
+
* Performance optimizations:
35
+
* Elasticsearch
36
+
* General recomendations
37
+
* Tune for indexing speed
38
+
* Tune for search speed
39
+
* Tune for approximate kNN search
40
+
* Tune for disk usage
41
+
* Size your shards
42
+
* Use {{es}} for time series data
43
+
* Kibana
44
+
* Kibana task manager scaling considerations
45
+
* Kibana alerting
46
+
47
+
* Scaling
48
+
49
+
For additional production-critical topics, refer to:
50
+
51
+
*[](./security.md)
52
+
53
+
*[](./tools.md)
54
+
55
+
*[](./monitor.md)
56
+
57
+
58
+
59
+
(Regardless if you are running a hosted or a self managed deployment, the content of this section allow you to understand and take key decisions when designing your clusters in the following areas:)
60
+
61
+
Cluster design tiene:
62
+
- design for resilience
63
+
- tune for xxx
64
+
- tune for xxx
65
+
66
+
67
+
68
+
## Deployment types
69
+
70
+
These concepts aren’t essential if you’re just getting started. How you [deploy {{es}}](/get-started/deployment-options.md) in production determines what you need to know:
71
+
72
+
***Self-managed {{es}}**: You are responsible for setting up and managing nodes, clusters, shards, and replicas. This includes managing the underlying infrastructure, scaling, and ensuring high availability through failover and backup strategies.
73
+
***Elastic Cloud**: Elastic can autoscale resources in response to workload changes. Choose from different deployment types to apply sensible defaults for your use case. A basic understanding of nodes, shards, and replicas is still important.
74
+
***Elastic Cloud Serverless**: You don’t need to worry about nodes, shards, or replicas. These resources are 100% automated on the serverless platform, which is designed to scale with your workload.
75
+
76
+
(add ECE and ECK)
77
+
78
+
79
+
% discarded text (from ECH best practices)
80
+
81
+
## HA and Resilience
82
+
83
+
{{es}} and {{kib}} provide mechanisms for HA and resilience
84
+
85
+
86
+
### Use multiple nodes and shards
87
+
88
+
### CCR for disaster recovery and geo-proximity
89
+
90
+
## Performance tuning [cluster-design]
91
+
92
+
{{es}} offers many options that allow you to configure your cluster to meet your organization’s goals, requirements, and restrictions. You can review the following guides to learn how to tune your cluster to meet your needs:
93
+
94
+
::::{note}
95
+
In orchestrated deployments some of the settings mentioned in this section are not applicable. Refer to each of the section headers to understand whether is applicable to your deployment type.
96
+
::::
97
+
98
+
*[Designing for resilience](availability-and-resilience.md)
99
+
*[Tune for indexing speed](optimize-performance/indexing-speed.md)
100
+
*[Tune for search speed](optimize-performance/search-speed.md)
101
+
*[Tune for disk usage](optimize-performance/disk-usage.md)
102
+
*[Tune for time series data](../../manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md)
103
+
104
+
Many {{es}} options come with different performance considerations and trade-offs. The best way to determine the optimal configuration for your use case is through [testing with your own data and queries](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing).
105
+
106
+
107
+
## Scaling
108
+
109
+
% from https://www.elastic.co/guide/en/cloud/current/ec-planning.html ?
9
110
This section provides some best practices for managing your data to help you set up a production environment that matches your workloads, policies, and deployment needs.
Copy file name to clipboardExpand all lines: deploy-manage/production-guidance/getting-ready-for-production-elasticsearch.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -56,6 +56,7 @@ As with any enterprise system, you need tools to secure, manage, and monitor you
56
56
57
57
58
58
## Cluster design [cluster-design]
59
+
% moved to landing page.
59
60
60
61
{{es}} offers many options that allow you to configure your cluster to meet your organization’s goals, requirements, and restrictions. You can review the following guides to learn how to tune your cluster to meet your needs:
@@ -22,7 +20,11 @@ While {{kib}} isn’t terribly resource intensive, we still recommend running {{
22
20
23
21
## Load balancing across multiple {{kib}} instances [load-balancing-kibana]
24
22
25
-
To serve multiple {{kib}} installations behind a load balancer, you must change the configuration. See [Configuring {{kib}}](../deploy/self-managed/configure.md) for details on each setting.
23
+
To serve multiple {{kib}} instances from the same deployment behind a load balancer, you must change the configuration. See [Configuring {{kib}}](../deploy/self-managed/configure.md) for details on each setting.
24
+
25
+
::::{note}
26
+
In orchestrated deployments such as {{ech}}, {{ece}}, and {{eck}}, the necessary configuration for multiple {{kib}} instances within the same deployment is automatically managed by the orchestrator. This process is transparent to the user, requiring no manual configuration.
27
+
::::
26
28
27
29
These settings must be unique across each {{kib}} instance:
To access multiple load-balanced {{kib}} clusters from the same browser, explicitly set `xpack.security.cookieName` to the same value in the {{kib}} configuration of each {{kib}} instance.
77
+
To access different load-balanced {{kib}} deployments from the same browser, explicitly set `xpack.security.cookieName` to the same value in the {{kib}} configuration of each {{kib}} instance.
78
+
79
+
To access different load-balanced {{kib}} instances within the same deployment from the same browser, explicitly set `xpack.security.cookieName` to the same value in the configuration of each instance.
80
+
81
+
Configure different value of `xpack.security.cookieName` in {{kib}} instances belonging to other deployments.
75
82
76
83
Each {{kib}} cluster must have a different value of `xpack.security.cookieName`.
{{ech}} supports a wide range of configurations. With such flexibility comes great freedom, but also the first rule of deployment planning: Your deployment needs to be matched to the workloads that you plan to run on your {{es}} clusters and {{kib}} instances. Specifically, this means two things:
35
+
36
+
*[Does your data need to be highly available?](../../../deploy-manage/production-guidance/plan-for-production-elastic-cloud.md#ec-ha)
37
+
*[Do you know when to scale?](../../../deploy-manage/production-guidance/plan-for-production-elastic-cloud.md#ec-workloads)
38
+
39
+
40
+
## Does your data need to be highly available? [ec-ha]
41
+
42
+
With {{ech}}, your deployment can be spread across as many as three separate availability zones, each hosted in its own, separate data center. Why this matters:
43
+
44
+
* Data centers can have issues with availability. Internet outages, earthquakes, floods, or other events could affect the availability of a single data center. With a single availability zone, you have a single point of failure that can bring down your deployment.
45
+
* Multiple availability zones help your deployment remain available. This includes your {{es}} cluster, provided that your cluster is sized so that it can sustain your workload on the remaining data centers and that your indices are configured to have at least one replica.
46
+
* Multiple availability zones enable you to perform changes to resize your deployment with zero downtime.
47
+
48
+
We recommend that you use at least two availability zones for production and three for mission-critical systems. Just one zone might be sufficient, if your {{es}} cluster is mainly used for testing or development and downtime is acceptable, but should never be used for production.
49
+
50
+
With multiple {{es}} nodes in multiple availability zones you have the recommended hardware, the next thing to consider is having the recommended index replication. Each index, with the exception of searchable snapshot indexes, should have one or more replicas. Use the index settings API to find any indices with no replica:
51
+
52
+
```sh
53
+
GET _all/_settings/index.number_of_replicas
54
+
```
55
+
56
+
Moreover, a high availability (HA) cluster requires at least three master-eligible nodes. For clusters that have fewer than six {{es}} nodes, any data node in the hot tier will also be a master-eligible node. You can achieve this by having hot nodes (serving as both data and master-eligible nodes) in three availability zones, or by having data nodes in two zones and a tiebreaker (will be automatically added if you choose two zones). For clusters that have six {{es}} nodes and beyond, dedicated master-eligible nodes are introduced. When your cluster grows, consider separating dedicated master-eligible nodes from dedicated data nodes. We recommend using at least 4GB RAM for dedicated master nodes.
57
+
58
+
The data in your {{es}} clusters is also backed up every 30 minutes, 4 hours, or 24 hours, depending on which snapshot interval you choose. These regular intervals provide an extra level of redundancy. We do support [snapshot and restore](../../../deploy-manage/tools/snapshot-and-restore.md), regardless of whether you use one, two, or three availability zones. However, with only a single availability zone and in the event of an outage, it might take a while for your cluster come back online. Using a single availability zone also leaves your cluster exposed to the risk of data loss, if the backups you need are not useable (failed or partial snapshots missing the indices to restore) or no longer available by the time that you realize that you might need the data (snapshots have a retention policy).
59
+
60
+
::::{warning}
61
+
Clusters that use only one availability zone are not highly available and are at risk of data loss. To safeguard against data loss, you must use at least two availability zones.
62
+
::::
63
+
64
+
65
+
::::{warning}
66
+
Indices with no replica, except for searchable snapshot indices, are not highly available. You should use replicas to mitigate against possible data loss.
67
+
::::
68
+
69
+
70
+
::::{warning}
71
+
Clusters that only have one master node are not highly available and are at risk of data loss. You must have three master-eligible nodes.
72
+
::::
73
+
74
+
75
+
76
+
## Do you know when to scale? [ec-workloads]
77
+
78
+
Knowing how to scale your deployment is critical, especially when unexpected workloads hits. Don’t forget to [check your performance metrics](../../../deploy-manage/monitor/monitoring-data/ec-saas-metrics-accessing.md) to make sure your deployments are healthy and can cope with your workloads.
79
+
80
+
Scaling with {{ech}} is easy:
81
+
82
+
* Turn on [deployment autoscaling](../../../deploy-manage/autoscaling.md) to let {{ecloud}} manage your deployments by adjusting their available resources automatically.
83
+
* Or, if you prefer manual control, log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body), select your deployment, select **Edit deployment** from the **Actions** dropdown, and either increase the number of zones or the size per zone.
84
+
85
+
::::{warning}
86
+
Increasing the number of zones should not be used to add more resources. The concept of zones is meant for High Availability (2 zones) and Fault Tolerance (3 zones), but neither will work if the cluster relies on the resources from those zones to be operational. The recommendation is to scale up the resources within a single zone until the cluster can take the full load (add some buffer to be prepared for a peak of requests), then scale out by adding additional zones depending on your requirements: 2 zones for High Availability, 3 zones for Fault Tolerance.
87
+
::::
88
+
89
+
90
+
Refer to [Sizing {{es}}: Scaling up and out](https://www.elastic.co/blog/found-sizing-elasticsearch) to identify which questions to ask yourself when determining which cluster size is the best fit for your {{es}} use case.
0 commit comments