Make etcd perf module changes for OSDOCS-4530

Author: maxwelldb

installing/installing_openstack/installing-openstack-installer-custom.adoc

@@ -15,6 +15,7 @@ In {product-title} version {product-version}, you can install a customized clust
 * You read the documentation on xref:../../installing/installing-preparing.adoc#installing-preparing[selecting a cluster installation method and preparing it for users].
 * You verified that {product-title} {product-version} is compatible with your {rh-openstack} version by using the xref:../../architecture/architecture-installation.adoc#supported-platforms-for-openshift-clusters_architecture-installation[Supported platforms for OpenShift clusters] section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
 * You have a storage service installed in {rh-openstack}, such as block storage (Cinder) or object storage (Swift). Object storage is the recommended storage technology for {product-title} registry cluster deployment. For more information, see xref:../../scalability_and_performance/optimization/optimizing-storage.adoc#optimizing-storage[Optimizing storage].
+* You understand performance and scalability practices for cluster scaling, control plane sizing, and etcd. For more information, see xref:../../scalability_and_performance/recommended-performance-scale-practices/recommended-control-plane-practices.adoc#recommended-host-practices[Recommended practices for scaling the cluster].
 * You have the metadata service enabled in {rh-openstack}.
 
 include::modules/installation-osp-default-deployment.adoc[leveloffset=+1]

installing/installing_openstack/installing-openstack-installer-kuryr.adoc

@@ -18,6 +18,7 @@ In {product-title} version {product-version}, you can install a customized clust
 * You read the documentation on xref:../../installing/installing-preparing.adoc#installing-preparing[selecting a cluster installation method and preparing it for users].
 * You verified that {product-title} {product-version} is compatible with your {rh-openstack} version by using the xref:../../architecture/architecture-installation.adoc#supported-platforms-for-openshift-clusters_architecture-installation[Supported platforms for OpenShift clusters] section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
 * You have a storage service installed in {rh-openstack}, such as block storage (Cinder) or object storage (Swift). Object storage is the recommended storage technology for {product-title} registry cluster deployment. For more information, see xref:../../scalability_and_performance/optimization/optimizing-storage.adoc#optimizing-storage[Optimizing storage].
+* You understand performance and scalability practices for cluster scaling, control plane sizing, and etcd. For more information, see xref:../../scalability_and_performance/recommended-performance-scale-practices/recommended-control-plane-practices.adoc#recommended-host-practices[Recommended practices for scaling the cluster].
 
 include::modules/installation-osp-about-kuryr.adoc[leveloffset=+1]
 include::modules/installation-osp-default-kuryr-deployment.adoc[leveloffset=+1]

installing/installing_openstack/installing-openstack-installer-restricted.adoc

@@ -20,6 +20,7 @@ In {product-title} {product-version}, you can install a cluster on
 ====
 Because the installation media is on the mirror host, you can use that computer to complete all installation steps.
 ====
+* You understand performance and scalability practices for cluster scaling, control plane sizing, and etcd. For more information, see xref:../../scalability_and_performance/recommended-performance-scale-practices/recommended-control-plane-practices.adoc#recommended-host-practices[Recommended practices for scaling the cluster].
 * You have the metadata service enabled in {rh-openstack}.
 
 include::modules/installation-about-restricted-network.adoc[leveloffset=+1]

installing/installing_openstack/installing-openstack-user-kuryr.adoc

@@ -20,6 +20,7 @@ Using your own infrastructure allows you to integrate your cluster with existing
 * You read the documentation on xref:../../installing/installing-preparing.adoc#installing-preparing[selecting a cluster installation method and preparing it for users].
 * You verified that {product-title} {product-version} is compatible with your {rh-openstack} version by using the xref:../../architecture/architecture-installation.adoc#supported-platforms-for-openshift-clusters_architecture-installation[Supported platforms for OpenShift clusters] section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
 * You have an {rh-openstack} account where you want to install {product-title}.
+* You understand performance and scalability practices for cluster scaling, control plane sizing, and etcd. For more information, see xref:../../scalability_and_performance/recommended-performance-scale-practices/recommended-control-plane-practices.adoc#recommended-host-practices[Recommended practices for scaling the cluster].
 * On the machine from which you run the installation program, you have:
 ** A single directory in which you can keep the files you create during the installation process
 ** Python 3

installing/installing_openstack/installing-openstack-user-sr-iov-kuryr.adoc

@@ -18,6 +18,7 @@ Using your own infrastructure allows you to integrate your cluster with existing
 * You verified that {product-title} {product-version} is compatible with your {rh-openstack} version by using the xref:../../architecture/architecture-installation.adoc#supported-platforms-for-openshift-clusters_architecture-installation[Supported platforms for OpenShift clusters] section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
 * Your network configuration does not rely on a provider network. Provider networks are not supported.
 * You have a {rh-openstack} account where you want to install {product-title}.
+* You understand performance and scalability practices for cluster scaling, control plane sizing, and etcd. For more information, see xref:../../scalability_and_performance/recommended-performance-scale-practices/recommended-control-plane-practices.adoc#recommended-host-practices[Recommended practices for scaling the cluster].
 * On the machine where you run the installation program, you have:
 ** A single directory in which you can keep the files you create during the installation process
 ** Python 3

installing/installing_openstack/installing-openstack-user-sr-iov.adoc

@@ -18,6 +18,7 @@ Using your own infrastructure allows you to integrate your cluster with existing
 * You verified that {product-title} {product-version} is compatible with your {rh-openstack} version by using the xref:../../architecture/architecture-installation.adoc#supported-platforms-for-openshift-clusters_architecture-installation[Supported platforms for OpenShift clusters] section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
 * Your network configuration does not rely on a provider network. Provider networks are not supported.
 * You have an {rh-openstack} account where you want to install {product-title}.
+* You understand performance and scalability practices for cluster scaling, control plane sizing, and etcd. For more information, see xref:../../scalability_and_performance/recommended-performance-scale-practices/recommended-control-plane-practices.adoc#recommended-host-practices[Recommended practices for scaling the cluster].
 * On the machine where you run the installation program, you have:
 ** A single directory in which you can keep the files you create during the installation process
 ** Python 3

installing/installing_openstack/installing-openstack-user.adoc

@@ -17,6 +17,7 @@ Using your own infrastructure allows you to integrate your cluster with existing
 * You read the documentation on xref:../../installing/installing-preparing.adoc#installing-preparing[selecting a cluster installation method and preparing it for users].
 * You verified that {product-title} {product-version} is compatible with your {rh-openstack} version by using the xref:../../architecture/architecture-installation.adoc#supported-platforms-for-openshift-clusters_architecture-installation[Supported platforms for OpenShift clusters] section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
 * You have an {rh-openstack} account where you want to install {product-title}.
+* You understand performance and scalability practices for cluster scaling, control plane sizing, and etcd. For more information, see xref:../../scalability_and_performance/recommended-performance-scale-practices/recommended-control-plane-practices.adoc#recommended-host-practices[Recommended practices for scaling the cluster].
 * On the machine from which you run the installation program, you have:
 ** A single directory in which you can keep the files you create during the installation process
 ** Python 3

modules/recommended-etcd-practices.adoc

@@ -9,12 +9,17 @@
 Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance.
 Although etcd is not particularly I/O intensive, it requires a low latency block device for optimal performance and stability. Because etcd's consensus protocol depends on persistently storing metadata to a log (WAL), etcd is sensitive to disk-write latency. Slow disks and disk activity from other processes can cause long fsync latencies.
 
-Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. High write latencies also lead to an OpenShift API slowness, which affects cluster performance. Because of these reasons, avoid colocating other workloads on the control-plane nodes.
+Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. High write latencies also lead to an OpenShift API slowness, which affects cluster performance. Because of these reasons, avoid colocating other workloads on the control-plane nodes that are I/O sensitive or intensive and share the same underlying I/O infrastructure. 
 
 In terms of latency, run etcd on top of a block device that can write at least 50 IOPS of 8000 bytes long sequentially. That is, with a latency of 10ms, keep in mind that uses fdatasync to synchronize each write in the WAL. For heavy loaded clusters, sequential 500 IOPS of 8000 bytes (2 ms) are recommended. To measure those numbers, you can use a benchmarking tool, such as fio.
 
 To achieve such performance, run etcd on machines that are backed by SSD or NVMe disks with low latency and high throughput. Consider single-level cell (SLC) solid-state drives (SSDs), which provide 1 bit per memory cell, are durable and reliable, and are ideal for write-intensive workloads.
 
+[NOTE]
+====
+The load on etcd arises from static factors, such as the number of nodes and pods, and dynamic factors, including changes in endpoints due to pod autoscaling, pod restarts, job executions, and other workload-related events. To accurately size your etcd setup, you must analyze the specific requirements of your workload. Consider the number of nodes, pods, and other relevant factors that impact the load on etcd.
+====
+
 The following hard disk features provide optimal etcd performance:
 
 * Low latency to support fast read operation.
@@ -25,7 +30,12 @@ The following hard disk features provide optimal etcd performance:
 * RAID 0 technology for increased performance.
 * Dedicated etcd drives. Do not place log files or other heavy workloads on etcd drives.
 
-Avoid NAS or SAN setups and spinning drives. Always benchmark by using utilities such as fio. Continuously monitor the cluster performance as it increases.
+[NOTE]
+====
+Avoid NAS or SAN setups and spinning drives. Ceph Rados Block Device (RBD) and other types of network-attached storage can result in unpredictable network latency. To provide fast storage to etcd nodes at scale, use PCI passthrough to pass NVM devices directly to the nodes. 
+====
+
+Always benchmark by using utilities such as fio. You can use such utilities to continuously monitor the cluster performance as it increases. 
 
 [NOTE]
 ====