Merge pull request #51475 from rsevilla87/etcd-improvements-main

Etcd recommendations improvements

Author: lpettyjo

modules/etcd-defrag.adoc

@@ -7,6 +7,14 @@
 [id="etcd-defrag_{context}"]
 = Defragmenting etcd data
 
+For large and dense clusters, etcd can suffer from poor performance if the keyspace grows too large and exceeds the space quota. Periodically maintain and defragment etcd to free up space in the data store. Monitor Prometheus for etcd metrics and defragment it when required; otherwise, etcd can raise a cluster-wide alarm that puts the cluster into a maintenance mode that accepts only key reads and deletes.
+
+.Monitor these key metrics:
+
+* `etcd_server_quota_backend_bytes`, which is the current quota limit
+* `etcd_mvcc_db_total_size_in_use_in_bytes`, which indicates the actual database usage after a history compaction
+* `etcd_debugging_mvcc_db_total_size_in_bytes`, which shows the database size, including free space waiting for defragmentation
+
 Defragment etcd data to reclaim disk space after events that cause disk fragmentation, such as etcd history compaction.
 
 History compaction is performed automatically every five minutes and leaves gaps in the back-end database. This fragmented space is available for use by etcd, but is not available to the host file system. You must defragment etcd to make this space available to the host file system.
@@ -58,6 +66,8 @@ A Prometheus alert indicates when you need to use manual defragmentation. The al
    * When etcd uses more than 50% of its available space for more than 10 minutes
    * When etcd is actively using less than 50% of its total database size for more than 10 minutes
 
+You can also determine whether defragmentation is needed by checking the etcd database size in MB that will be freed by defragmentation with the PromQL expression: `(etcd_mvcc_db_total_size_in_bytes - etcd_mvcc_db_total_size_in_use_in_bytes)/1024/1024`
+
 [WARNING]
 ====
 Defragmenting etcd is a blocking action. The etcd member will not respond until defragmentation is complete. For this reason, wait at least one minute between defragmentation actions on each of the pods to allow the cluster to recover.

modules/openshift-cluster-maximums-environment.adoc

@@ -15,16 +15,16 @@
 | r5.4xlarge
 | 16
 | 128
-| io1
-| 220 / 3000
+| gp3
+| 220
 | 3
 | us-west-2
 
 | Infra ^[2]^
 | m5.12xlarge
 | 48
 | 192
-| gp2
+| gp3
 | 100
 | 3
 | us-west-2
@@ -33,7 +33,7 @@
 | m5.4xlarge
 | 16
 | 64
-| gp2
+| gp3
 | 500 ^[4]^
 | 1
 | us-west-2
@@ -42,15 +42,15 @@
 | m5.2xlarge
 | 8
 | 32
-| gp2
+| gp3
 | 100
 | 3/25/250/500 ^[5]^
 | us-west-2
 
 |===
 [.small]
 --
-1. io1 disks with 3000 IOPS are used for control plane/etcd nodes as etcd is I/O intensive and latency sensitive.
+1. gp3 disks with a baseline performance of 3000 IOPS and 125 MiB per second are used for control plane/etcd nodes because etcd is latency sensitive. gp3 volumes do not use burst performance.
 2. Infra nodes are used to host Monitoring, Ingress, and Registry components to ensure they have enough resources to run at large scale.
 3. Workload node is dedicated to run performance and scalability workload generators.
 4. Larger disk size is used so that there is enough space to store the large amounts of data that is collected during the performance and scalability test run.
@@ -67,7 +67,7 @@
 | 16
 | 32
 | io1
-| 120 / 10 IOPS per GB
+| 120 / 10 IOPS per GiB
 | 3
 
 | Infra ^[2]^
@@ -126,7 +126,7 @@
 --
 1. Nodes are distributed between two logical control units (LCUs) to optimize disk I/O load of the control plane/etcd nodes as etcd is I/O intensive and latency sensitive. Etcd I/O demand should not interfere with other workloads.
 2. Four compute nodes are used for the tests running several iterations with 100/250/500 pods at the same time. First, idling pods were used to evaluate if pods can be instanced. Next, a network and CPU demanding client/server workload were used to evaluate the stability of the system under stress. Client and server pods were pairwise deployed and each pair was spread over two compute nodes.
-3. No separate workload node was used. The workload simulates a micro-service workload between two compute nodes.
+3. No separate workload node was used. The workload simulates a microservice workload between two compute nodes.
 4. Physical number of processors used is six Integrated Facilities for Linux (IFLs).
 5. Total physical memory used is 512 GiB.
---
+--

modules/recommended-etcd-practices.adoc

@@ -6,17 +6,14 @@
 [id="recommended-etcd-practices_{context}"]
 = Recommended etcd practices
 
-For large and dense clusters, etcd can suffer from poor performance if the keyspace grows too large and exceeds the space quota. Periodically maintain and defragment etcd to free up space in the data store. Monitor Prometheus for etcd metrics and defragment it when required; otherwise, etcd can raise a cluster-wide alarm that puts the cluster into a maintenance mode that accepts only key reads and deletes.
+Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance.
+Although etcd is not particularly I/O intensive, it requires a low latency block device for optimal performance and stability. Because etcd's consensus protocol depends on persistently storing metadata to a log (WAL), etcd is sensitive to disk-write latency. Slow disks and disk activity from other processes can cause long fsync latencies.
 
-.Monitor these key metrics:
+Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. High write latencies also lead to a OpenShift API slowness, which affects cluster performance. Because of these reasons, avoid colocating other workloads on the control-plane nodes.
 
-* `etcd_server_quota_backend_bytes`, which is the current quota limit
-* `etcd_mvcc_db_total_size_in_use_in_bytes`, which indicates the actual database usage after a history compaction
-* `etcd_debugging_mvcc_db_total_size_in_bytes`, which shows the database size, including free space waiting for defragmentation
+In terms of latency, run etcd on top of a block device that can write at least 50 IOPS of 8000 bytes long sequentially. That is, with a latency of 20ms, keep in mind that uses fdatasync to synchronize each write in the WAL. For heavy loaded clusters, sequential 500 IOPS of 8000 bytes (2 ms) are recommended. To measure those numbers, you can use a benchmarking tool, such as fio.
 
-For more information about defragmenting etcd, see the "Defragmenting etcd data" section.
-
-Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance. Slow disks and disk activity from other processes can cause long fsync latencies. Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. Run etcd on machines that are backed by SSD or NVMe disks with low latency and high throughput. Consider single-level cell (SLC) solid-state drives (SSDs), which provide 1 bit per memory cell, are durable and reliable, and are ideal for write-intensive workloads.
+To achieve such performance, run etcd on machines that are backed by SSD or NVMe disks with low latency and high throughput. Consider single-level cell (SLC) solid-state drives (SSDs), which provide 1 bit per memory cell, are durable and reliable, and are ideal for write-intensive workloads.
 
 The following hard disk features provide optimal etcd performance:
 
@@ -28,17 +25,14 @@ The following hard disk features provide optimal etcd performance:
 * RAID 0 technology for increased performance.
 * Dedicated etcd drives. Do not place log files or other heavy workloads on etcd drives. 
 
-Avoid NAS or SAN setups, and spinning drives. Always benchmark using utilities such as `fio`. Continuously monitor the cluster performance as it increases.
+Avoid NAS or SAN setups and spinning drives. Always benchmark by using utilities such as fio. Continuously monitor the cluster performance as it increases.
 
-IMPORTANT: Avoid using the Network File System (NFS) protocol.
+IMPORTANT: Avoid using the Network File System (NFS) protocol or other network based file systems.
 
 Some key metrics to monitor on a deployed {product-title} cluster are p99 of etcd disk write ahead log duration and the number of etcd leader changes. Use Prometheus to track these metrics.
 
-* The `etcd_disk_wal_fsync_duration_seconds_bucket` metric reports the etcd disk fsync duration.
-* The `etcd_server_leader_changes_seen_total` metric reports the leader changes.
-* To rule out a slow disk and confirm that the disk is reasonably fast, verify that the 99th percentile of the `etcd_disk_wal_fsync_duration_seconds_bucket` is less than 10 ms.
 
-To validate the hardware for etcd before or after you create the {product-title} cluster, you can use an I/O benchmarking tool called fio.
+To validate the hardware for etcd before or after you create the {product-title} cluster, you can use fio.
 
 .Prerequisites
 
@@ -64,7 +58,11 @@ $ sudo docker run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/openshift-scale
 ----
 --
 
-The output reports whether the disk is fast enough to host etcd by comparing the 99th percentile of the fsync metric captured from the run to see if it is less than 10 ms.
+The output reports whether the disk is fast enough to host etcd by comparing the 99th percentile of the fsync metric captured from the run to see if it is less than 20 ms. A few of the most important etcd metrics that might affected by I/O performance are as follow:
+
+- `etcd_disk_wal_fsync_duration_seconds_bucket` metric reports the etcd's WAL fsync duration.
+- `etcd_disk_backend_commit_duration_seconds_bucket`  metric reports the etcd backend commit latency duration.
+- `etcd_server_leader_changes_seen_total` metric reports the leader changes.
 
 Because etcd replicates the requests among all the members, its performance strongly depends on network input/output (I/O) latency. High network latencies result in etcd heartbeats taking longer than the election timeout, which results in leader elections that are disruptive to the cluster. A key metric to monitor on a deployed {product-title} cluster is the 99th percentile of etcd network peer latency on each etcd cluster member. Use Prometheus to track the metric.