updates per acrolinx suggestions

Author: lesage-oded

articles/operator-nexus/howto-run-instance-readiness-testing.md

@@ -1,14 +1,3 @@
----
-title: "Azure Operator Nexus: How to run Instance Readiness Testing"
-description: Learn how to run instance readiness testing.
-author: lesage-oded
-ms.author: odedlesage
-ms.service: azure-operator-nexus
-ms.topic: how-to
-ms.date: 02/29/2024
-ms.custom: template-how-to
----
-
 # Azure Operator Nexus Instance Readiness Test (IRT)
 
 The Instance Readiness Test (IRT) framework is an optional/add-on tool for the Nexus platform. It enables operators to verify the successful deployment and readiness of the Azure Operator Nexus instance for workload deployment. This verification applies to both initial deployment and subsequent upgrades of the Nexus. It runs a series of tests and provides the test results as an html report.
@@ -27,10 +16,25 @@ The Instance Readiness Test (IRT) framework is an optional/add-on tool for the N
 ## Tests executed with IRT
 - Validate that l3 domains in the fabric subscription and resource group exist after all tests on the resources under test are done.
 - Validate that there are l3 networks created in the testing resource group after all tests on the resources under test are done.
+- Validate that ApiserverAuditRequestsRejectedTotal metric data is present within the last 10 minutes.
+  Every average metric should be greater than 0.
+- Validate that ContainerMemoryUsageBytes metric data is present within the last 10 minutes.
+  Every average metric should be greater than 0.
+- Validate that CorednsDnsRequestsTotal metric data is present within the last 10 minutes.
+  Every average metric should be greater than 0.
+- Validate that EtcdServerIsLeader metric data is present within the last 10 minutes. Every count metric should be greater than 0.
 - Validate that FelixClusterNumHosts metric data is present within the last 10 minutes.
   Every average metric should be greater than 0.
+- Validate that IdracPowerOn metric data is present within the last 10 minutes. Every count metric should be greater than 0.
+- Validate that KubeDaemonsetStatusCurrentNumberScheduled metric data is present within the last 10 minutes. Every average metric should be greater than 0.
+- Validate that KubeletRunningPods metric data is present within the last 10 minutes.
+  Every average metric should be greater than 0.
+- Validate that KubevirtInfo metric data is present within the last 10 minutes.
+  Every average metric should be greater than 0.
 - Validate that NodeOsInfo metric data for a baremetal machine is present within the last 10 minutes.
   Every count metric should be greater than 0.
+- Validate that TyphaConnectionsAccepted metric data is present within the last 10 minutes.
+  Every average metric should be greater than 0.
 - Test the transmission of IPv4 TCP data between two virtual machines using iPerf3 and affinity settings in the ARM template.
   The test ensures that the data throughput exceeds 60 Mbps.
 - Test the transmission of IPv6 TCP data between two virtual machines using iPerf3 and affinity settings in the ARM template.
@@ -64,17 +68,17 @@ The Instance Readiness Test (IRT) framework is an optional/add-on tool for the N
   Stderr should be empty and no packet loss should be observed.
 - Test IPv6 ping between a NAKS cluster pod and a VM with jumbo frames enabled.
   Stderr should be empty and no packet loss should be observed.
-- Validate PVC has been created successfully.
-- Validate PV has been created successfully.
-- Test creating  a PVC with volumeMode Block and accessMode RWO.
-- Validate that all the nexus-shared and nexus-volume volumes that were added are mounted in sts 0
+- Validate PersistentVolumeClaim is created successfully.
+- Validate PersistentVolume is created successfully.
+- Test creating a PVC with volumeMode Block and accessMode RWO.
+- Validate that all the nexus-shared and nexus-volume volumes that were added are mounted in sts 0.
 - Validate that all the nexus-shared and nexus-volume volumes that were added are mounted in sts 1.
 - Validate that nfs storage mounted on sts 0 is writable.
 - Validate that nfs storage file written to sts 0 can be read.
 - Validate that shared nfs storage mounted on sts 0 is writable.
 - Validate that shared nfs file written to sts 0 can be read.
 - Validate that nfs storage mounted on sts 1 is writable.
-- Validate that shared file written to sts 0 can be read in sts 1
+- Validate that shared file written to sts 0 can be read in sts 1.
 - Validate that shared nfs storage mounted on sts 1 is writable.
 - Validate that shared file written to sts 0 and sts 1 can be read from sts 1.
 - Validate that shared file written to in sts 0 and sts 1 can be read from sts 0.
@@ -106,19 +110,19 @@ For access to the nexus-samples GitHub repository
 
 ## Environment Requirements
 
-- A Linux environment (Ubuntu suggested) capable of calling Azure APIs
-- Support for other Linux distros e.g. RedHat, Mariner, etc. depends on being able to install the necessary tooling. See [Install Dependencies](#install-dependencies) section.
+- A Linux environment (Ubuntu suggested) capable of calling Azure APIs.
+- Support for other Linux distros for example, RedHat, Mariner, etc. depends on being able to install the necessary tooling. See [Install Dependencies](#install-dependencies) section.
   - Any machine that has the required packages installed should be able to use the scripts.
-- Knowledge of networks to use for the test
+- Knowledge of networks to use for the test.
   * Networks to use for the test are specified in a "networks-blueprint.yml" file, see [Input Configuration](#input-configuration).
-- A way to download the IRT release package e.g. curl, wget, etc
-- The ability to create a service principal with the correct roles
-- The ability to read secrets from the KeyVault, see [Service Principal] (#create-service-principal-and-security-group) section for more details
-- The ability to create security groups in your Active Directory tenant
+- A way to download the IRT release package for example, curl, wget, etc.
+- The ability to create a service principal with the correct roles.
+- The ability to read secrets from the KeyVault, see [Service Principal] (#create-service-principal-and-security-group) section for more details.
+- The ability to create security groups in your Active Directory tenant.
 
 ## Input Configuration
 
-Start by building your input file. The IRT tarball provides `irt-input.example.yml` as an example. Follow the [instructions](#download-irt) to download the tarball. Please note that these values **will not work for your instances**. You need to manually change them and rename the file to `irt-input.yml`. We provide the example input file as a stub to help you configure new input files. The example outlines overridable values and their usage. The **[One Time Setup](#one-time-setup)** assists you in setting input values by writing key/value pairs to the config file as they execute.
+Start by building your input file. The IRT tarball provides `irt-input.example.yml` as an example. Download the tarball by following the [instructions](#download-irt). Note that these values **will not work for your instances**. You need to manually change them and rename the file to `irt-input.yml`. We provide the example input file as a stub to help you configure new input files. The example outlines overridable values and their usage. The **[One Time Setup](#one-time-setup)** assists you in setting input values by writing key/value pairs to the config file as they execute.
 
 You can provide the network information in a `networks-blueprint.yml` file, similar to the `networks-blueprint.example.yml` that we provide, or append it to the `irt-input.yml` file. The `networks-blueprint.example.yml` defines the schema for IRT. The test creates the networks, so provide network details that aren't in use. Currently, IRT has the following network requirements:
 
@@ -131,11 +135,11 @@ You can provide the network information in a `networks-blueprint.yml` file, simi
 ## One Time Setup
 
 ### Download IRT
-IRT is distributed via tarball from the release section of the [nexus-samples](https://aka.ms/nexus-irt) GitHub repo
+IRT is distributed via tarball from the release section of the [nexus-samples](https://aka.ms/nexus-irt) GitHub repo.
 1. Find the release package marked with 'Latest'. Download it, extract it, and navigate to the `irt` directory.
-1. Extract the tarball to the local file system: `mkdir -p irt && tar xf nexus-irt.tar.gz --directory ./irt`
-1. Switch to the new directory `cd irt`
-1. See RELEASE-CHANGELOG.md for any notable updates or changes
+1. Extract the tarball to the local file system: `mkdir -p irt && tar xf nexus-irt.tar.gz --directory ./irt`.
+1. Switch to the new directory `cd irt`.
+1. See RELEASE-CHANGELOG.md for any notable updates or changes.
 
 ### Install Dependencies
 There are multiple dependencies expected to be available during execution. Review this list;
@@ -185,7 +189,7 @@ The supplemental script, `create-service-principal.sh` creates a service princip
 
 Additionally, the script creates the necessary security group, and adds the service principal to the security group. If the security group exists, it adds the service principal to the existing security group.
 
-Executing `create-service-principal.sh` requires the input yaml to have the following properties, all of them can be overridden by the corresponding environment variables:
+Executing `create-service-principal.sh` requires the input yaml to have the following values. All values can be overridden by setting the corresponding environment variables:
 ```yml
 SERVICE_PRINCIPAL:
   NAME: "<name>" # env: SERVICE_PRINCIPAL_NAME
@@ -198,7 +202,7 @@ SERVICE_PRINCIPAL:
 * `SERVICE_PRINCIPAL.AAD_GROUP_NAME` - The name of the security group.
 * `SERVICE_PRINCIPAL.SUBSCRIPTION` - The subscription of the service principal.
 * `SERVICE_PRINCIPAL.KV_NAME` - The KeyVault to store the service principal password.
-* `SERVICE_PRINCIPAL.KV_ID` - The KeyVault secret where the service principal password is actually stored.
+* `SERVICE_PRINCIPAL.KV_ID` - The KeyVault secret where the service principal password is stored.
 
 > **_NOTE:_** Please ensure that you have already created a KeyVault (KV_NAME) and/or a Secret (KV_ID) with a dummy value prior to executing `create-service-principal.sh`.
 > The `az login` user (person executing IRT) should also be granted access to this KeyVault so secrets can be pulled at runtime.
@@ -227,7 +231,7 @@ KV_ID: "<provided-key-valut-secret>" # If SP already exists please fill it in to
 <details>
 <summary>Expand to see details for using a custom role </summary>
 
-If you have an existing service principal and would like the convenience of only having to assign one role for IRT execution, you can follow the steps below.
+If you have an existing service principal and would like the convenience of only having to assign one role for IRT execution, you can follow the directions in this section.
 
 ##### Prerequisites
 
@@ -236,9 +240,9 @@ If you have an existing service principal and would like the convenience of only
 
 ##### Steps
 
-1. Prepare Your Environment
+1. Prepare Your Environment:
    - Open a Bash Shell:
-   - You can use any terminal that supports Bash.
+   - You can use any terminal that supports Bash
 
 1. Sign in to Azure:
    - Execute the following command to sign in to your Azure account:
@@ -270,9 +274,9 @@ If you have an existing service principal and would like the convenience of only
         --parameters roleName="$roleName"
     ```
 
-1. Assign Role to Application Service Principal used for testing
+1. Assign Role to Application Service Principal used for testing:
 
-   Weather created via the all-in-one setup or using your own, assign the newly created role to your identity, this single role provides all the necessary authorizations to run Instance Readiness Testing.
+   Weather created via the all-in-one setup or using your own, assign the newly created role to your identity. This single role provides all the necessary authorizations to run Instance Readiness Testing.
 
    ```bash
     # The Application ID of your Service Principal for your application
@@ -297,7 +301,7 @@ If you have an existing service principal and would like the convenience of only
 <details>
 <summary>Expand to see how to create l3 isolation. </summary>
 
-The testing framework does not create, destroy, or manipulate isolation domains. Therefore, existing isolation domains can be used for execution. Each isolation domain requires at least one external network. The supplemental script, `create-l3-isolation-domains.sh`. Internal networks e.g. L3, trunked, etc. are created, manipulated, and destroyed through the course of testing.
+The testing framework does't create, destroy, or manipulate isolation domains. Therefore, existing isolation domains can be used for execution. Each isolation domain requires at least one external network. The supplemental script, `create-l3-isolation-domains.sh`. Internal networks for example, L3, trunked, etc. are created, manipulated, and destroyed through the course of testing.
 
 Executing `create-l3-isolation-domains.sh` requires one **parameter**, a path to a file containing the networks requirements. You can choose either the standalone network-blueprint.yml or the input.yml based on your workflow, either can contain the information needed.
 
@@ -324,11 +328,10 @@ Executing `create-l3-isolation-domains.sh` requires one **parameter**, a path to
 
 ## How to Read the IRT Summary Results
 
-The IRT summary page is a html page that can be downloaded after the
-execution of the IRT and can be viewed from any browser.
+The IRT summary page is an html page that is generated after the
+execution of IRT and can be viewed from any browser.
 
-IRT Summary Page comprises three major sections, which drills further to
-provide more details.
+IRT Summary Page comprises three major sections, which expand to provide more details.
 
 - Test Results
 
@@ -345,47 +348,25 @@ executed, different prerequisite test commands, so totals may not always be the
 
 ![irt summary header success](./media/irt/irt-header-success.png)
 
-In case of any failures in the tests, the values represent accordingly.
+If there is any failures in the tests, the values represent accordingly.
 
 ![irt summary header failure](./media/irt/irt-header-failure.png)
 
 ### Test Results
 
 The Test Results section provides all the tests (assertions) that IRT
 executes. The Asserters section expands to view the list of tests
-(assertions) that are run and available. Each asserter can be further
-expanded that loads an accordion pane which provides more details of the
-asserter, including the description of the test and any thresholds to be
-measured and asserted against.
-
-IRT asserters include the tests related to the following:
-
-- Connectivity between resources -- NS (nslookup) and EW
-    (pings/iperfs)
-
-- Dual Stack support
-
-- Iperf tests on IPv4 and IPv6
-
-- MTUs 1500 and 9000
-
-- DPDK throughputs measuring using PMDs
-
-- Nexus Kubernetes cluster status
-
-- Volume tests
-
-  - PVCs in block and access mode
-
-  - PVS write and read tests
+(assertions) that are run and available. Each asserter can be further expanded
+that loads an accordion pane, which provides more details of the asserter,
+including the description of the test and any thresholds to be measured and asserted against.
 
 ### Display of Test Results
 
 Display of Test results section with all successful tests:
 
 ![test results success](./media/irt/irt-test-success.png)
 
-In case of any failures, the asserts are highlighted in red.
+If there are any failures, the assertions are highlighted in red.
 
 ![test results failure](./media/irt/irt-test-failure.png)
 
@@ -397,11 +378,11 @@ description of it under standard log.
 An example of an Asserter:
 
 *Asserters \[It\] res-test-dpdk-naks-84f5b - network: \'l3network-704\'
-(**PMD) average of Rx-pps \[17668558\] should be greater than 8000000*
+(**PMD) average of Rx-pps \[17668558\] should be greater than 8000000*.
 
 The above example of an assert reads as the Rx (receive)-pps (packets
-per seconds) for l3network-704 is 17668558 which is greater than the
-expected 8000000
+per seconds) for l3network-704 is 17668558, which is greater than the
+expected 8000000.
 
 ![irt success details](./media/irt/irt-detail-success.png)
 
@@ -418,49 +399,47 @@ for debugging purposes. It consists of four test suites and every suite
 consists of the suite relevant tests that expand to provide details.
 Failures of any specific tests are highlighted in red.
 
-- Setup Suite - A common suite to deploy Nexus Resources as defined in
-    the Arm Template required for the framework and tests.
+- Setup Suite:
+    - Sets up test execution and deploys Nexus Resources as defined in
+    the ARM Template required for the framework and tests.
 
-- Inject Suite -- Injects the required environment variables and test
+- Inject Suite:
+    - Injects the required environment variables and test
     data to support the testing of NAKS resources
 
-- Collect Suite -- The collect suite collects the data published by
+- Collect Suite:
+    - The collect suite collects the data published by
     the Setup Suite.
 
-- Cleanup Suite -- Deletes the Nexus resources created for the tests
+- Cleanup Suite:
+    - Deletes the Nexus resources created for the tests
     after the data is collected.
 
 ![irt debug section](./media/irt/irt-debug-section.png)
 
 ## Extras Section
 
-This is an information only section that provides additional details of
-the Nexus platform. There are no assertions/tests that represent this
+This section is an informational only, that provides additional details about
+the Nexus instance. There are no assertions/tests that represent this
 section. It helps operators to check the status of underlying cluster
 resources and tenant resources running on the cluster after IRT is
 executed.
 
-Extras section consists of results displayed by running two different
-text files separately.
+The Extras section consists of results displayed by running two different
+script files separately.
 
-- Platform Validation Results -- Displays the Nexus under cloud
-    deployed resources details and their current statuses, including
-    Cluster Manager details and its extensions, Fabric related details,
-    Nexus cluster and its extensions, BareMetal Machines, Arc related
-    and Storage appliances.
+- Platform Validation Results:
+   - Displays the Nexus under cloud deployed resources details and their current statuses, including Cluster Manager details and its extensions, Fabric related details, Nexus cluster and its extensions, BareMetal Machines, Arc related, and Storage appliances.
 
-- Tenant workloads Validation Results -- Displays the Nexus tenant
-    resources details and their current statuses running on the Nexus
-    cluster, including displaying of L2 and L3 Isolation Domains, Cloud
-    Service networks, default cni networks, L2 and L3 networks, trunked
-    networks, available list of VMs and Nexus Kubernetes clusters.
+- Tenant workloads Validation Results:
+    - Displays the Nexus tenant resources details and their current statuses running on the Nexus cluster, including displaying of L2 and L3 Isolation Domains, Cloud Service networks, default cni networks, L2 and L3 networks, trunked networks, available list of VMs…
 
 ## Troubleshooting
 
 Asserters and debug sections with failures are effective troubleshooting
 methods to address failures and technical problems.
 
-If you still have questions, please [contact
+If you still have questions, [contact
 support](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade).
 For more information about Support plans, see [Azure Support
 plans](https://azure.microsoft.com/support/plans/response/)