Merge branch 'patch-44' of https://github.com/MikeRayMSFT/azure-docs-pr into 202304230-update-automated-integration-testing

Author: Mike Ray (Microsoft)

articles/azure-arc/data/automated-integration-testing.md

@@ -424,6 +424,58 @@ At a high-level, the launcher performs the following sequence of steps:
 12. Attempt to use the SAS token `LOGS_STORAGE_ACCOUNT_SAS` provided to create a new Storage Account container named based on `LOGS_STORAGE_CONTAINER`, in the **pre-existing** Storage Account `LOGS_STORAGE_ACCOUNT`. If Storage Account container already exists, use it. Upload all local test results and logs to this storage container as a tarball (see below).
 13. Exit.
 
+## Tests performed per test suite
+
+There are approximately **375** unique integration tests available, across **27** test suites - each testing a separate functionality.
+
+| Suite # | Test suite name                 | Description of test                                                                                                                                    |
+| ------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| 1       | ad-connector                   | Tests the deployment and update of an Active Directory Connector (AD Connector).                                                                       |
+| 2       | billing                        | Testing various Business Critical license types are reflected in resource table in controller, used for Billing upload.                                |
+| 3       | ci-billing                     | Similar as `billing`, but with more CPU/Memory permutations.                                                                                           |
+| 4       | ci-sqlinstance                 | Long running tests for multi-replica creation, updates, GP -> BC Update, Backup validation and SQL Server Agent.                                       |
+| 5       | controldb                      | Tests Control database - SA secret check, system login verification, audit creation, and sanity checks for SQL build version.                          |
+| 6       | dc-export                      | Indirect Mode billing and usage upload.                                                                                                                |
+| 7       | direct-crud                    | Creates a SQL instance using ARM calls, validates in both Kubernetes and ARM.                                                                          |
+| 8       | direct-fog                     | Creates multiple SQL instances and creates a Failover Group between them using ARM calls.                                                              |
+| 9       | direct-hydration               | Creates SQL Instance with Kubernetes API, validates presence in ARM.                                                                                   |
+| 10      | direct-upload                  | Validates billing upload in Direct Mode                                                                                                                |
+| 11      | kube-rbac                      | Ensures Kubernetes Service Account permissions for Arc Data Services matches least-privilege expectations.                                             |
+| 12      | nonroot                        | Ensures containers run as non-root user                                                                                                                |
+| 13      | postgres                       | Various Postgres creation, scaling, backup/restore tests.                                                                                              |
+| 14      | release-sanitychecks           | Sanity checks for month-to-month releases, such as SQL Server Build versions.                                                                          |
+| 15      | sqlinstance                    | Shorter version of `ci-sqlinstance`, for fast validations.                                                                                             |
+| 16      | sqlinstance-ad                 | Tests creation of SQL Instances with Active Directory Connector.                                                                                       |
+| 17      | sqlinstance-credentialrotation | Tests automated Credential Rotation for both General Purpose and Business Critical.                                                                    |
+| 18      | sqlinstance-ha                 | Various High Availability Stress tests, including pod reboots, forced failovers and suspensions.                                                       |
+| 19      | sqlinstance-tde                | Various Transparent Data Encryption tests.                                                                                                             |
+| 20      | telemetry-elasticsearch        | Validates Log ingestion into Elasticsearch.                                                                                                            |
+| 21      | telemetry-grafana              | Validates Grafana is reachable.                                                                                                                        |
+| 22      | telemetry-influxdb             | Validates Metric ingestion into InfluxDB.                                                                                                              |
+| 23      | telemetry-kafka                | Various tests for Kafka using SSL, single/multi-broker setup.                                                                                          |
+| 24      | telemetry-monitorstack         | Tests Monitoring components, such as Fluentbit and Collectd are functional.                                                                            |
+| 25      | telemetry-telemetryrouter      | Tests Open Telemetry.                                                                                                                                  |
+| 26      | telemetry-webhook              | Tests Data Services Webhooks with valid and invalid calls.                                                                                             |
+| 27      | upgrade-arcdata                | Upgrades a full suite of SQL Instances (GP, BC 2 replica, BC 3 replica, with Active Directory) and upgrades from last month's release to latest build. |
+
+As an example, for `sqlinstance-ha`, the following tests are performed:
+
+- `test_critical_configmaps_present`: Ensures the ConfigMaps and relevant fields are present for a SQL Instance.
+- `test_suspended_system_dbs_auto_heal_by_orchestrator`: Ensures if master and msdb is suspended by any means (in this case, user), Orchestrator maintenance reconcile auto-heals it.
+- `test_suspended_user_db_does_not_auto_heal_by_orchestrator`: Ensures if a User Database is deliberately suspended by user, Orchestrator maintenance reconcile does not auto-heal it.
+- `test_delete_active_orchestrator_twice_and_delete_primary_pod`: Deletes orchestrator pod multiple times, followed by the primary replica, and verifies all replicas are synchronized. Failover time expectations for 2 replica are relaxed.
+- `test_delete_primary_pod`: Deletes primary replica and verifies all replicas are synchronized. Failover time expectations for 2 replica are relaxed.
+- `test_delete_primary_and_orchestrator_pod`: Deletes primary replica and orchestrator pod and verifies all replicas are synchronized.
+- `test_delete_primary_and_controller`: Deletes primary replica and data controller pod and verifies primary endpoint is accessible and the new primary replica is synchronized. Failover time expectations for 2 replica are relaxed.
+- `test_delete_one_secondary_pod`: Deletes secondary replica and data controller pod and verifies all replicas are synchronized.
+- `test_delete_two_secondaries_pods`: Deletes secondary replicas and data controller pod and verifies all replicas are synchronized.
+- `test_delete_controller_orchestrator_secondary_replica_pods`:
+- `test_failaway`: Forces AG failover away from current primary, ensures the new primary is not the same as the old primary. Verifies all replicas are synchronized.
+- `test_update_while_rebooting_all_non_primary_replicas`: Tests Controller-driven updates are resilient with retries despite various turbulent circumstances.
+
+> [!NOTE]
+> Certain tests may require specific hardware, such as privileged Access to Domain Controllers for `ad` tests for Account and DNS entry creation - which may not be available in all environments looking to use the `arc-ci-launcher`.
+
 ## Examining Test Results
 
 A sample storage container and file uploaded by the launcher: