Merge pull request #235362 from MikeRayMSFT/202304230-update-automated-integration-testing

202304230 update automated integration testing

Author: prmerger-automator[bot]

articles/azure-arc/data/automated-integration-testing.md

@@ -424,6 +424,58 @@ At a high-level, the launcher performs the following sequence of steps:
 12. Attempt to use the SAS token `LOGS_STORAGE_ACCOUNT_SAS` provided to create a new Storage Account container named based on `LOGS_STORAGE_CONTAINER`, in the **pre-existing** Storage Account `LOGS_STORAGE_ACCOUNT`. If Storage Account container already exists, use it. Upload all local test results and logs to this storage container as a tarball (see below).
 13. Exit.
 
+## Tests performed per test suite
+
+There are approximately **375** unique integration tests available, across **27** test suites - each testing a separate functionality.
+
+| Suite # | Test suite name                 | Description of test                                                                                                                                    |
+| ------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| 1       | `ad-connector`                   | Tests the deployment and update of an Active Directory Connector (AD Connector).                                                                       |
+| 2       | `billing`                        | Testing various Business Critical license types are reflected in resource table in controller, used for Billing upload.                                |
+| 3       | `ci-billing`                     | Similar as `billing`, but with more CPU/Memory permutations.                                                                                           |
+| 4       | `ci-sqlinstance`                 | Long running tests for multi-replica creation, updates, GP -> BC Update, Backup validation and SQL Server Agent.                                       |
+| 5       | `controldb`                      | Tests Control database - SA secret check, system login verification, audit creation, and sanity checks for SQL build version.                          |
+| 6       | `dc-export`                      | Indirect Mode billing and usage upload.                                                                                                                |
+| 7       | `direct-crud`                    | Creates a SQL instance using ARM calls, validates in both Kubernetes and ARM.                                                                          |
+| 8       | `direct-fog`                     | Creates multiple SQL instances and creates a Failover Group between them using ARM calls.                                                              |
+| 9       | `direct-hydration`               | Creates SQL Instance with Kubernetes API, validates presence in ARM.                                                                                   |
+| 10      | `direct-upload`                  | Validates billing upload in Direct Mode                                                                                                                |
+| 11      | `kube-rbac`                      | Ensures Kubernetes Service Account permissions for Arc Data Services matches least-privilege expectations.                                             |
+| 12      | `nonroot`                        | Ensures containers run as non-root user                                                                                                                |
+| 13      | `postgres`                       | Completes various Postgres creation, scaling, backup/restore tests.                                                                                              |
+| 14      | `release-sanitychecks`           | Sanity checks for month-to-month releases, such as SQL Server Build versions.                                                                          |
+| 15      | `sqlinstance`                    | Shorter version of `ci-sqlinstance`, for fast validations.                                                                                             |
+| 16      | `sqlinstance-ad`                 | Tests creation of SQL Instances with Active Directory Connector.                                                                                       |
+| 17      | `sqlinstance-credentialrotation` | Tests automated Credential Rotation for both General Purpose and Business Critical.                                                                    |
+| 18      | `sqlinstance-ha`                 | Various High Availability Stress tests, including pod reboots, forced failovers and suspensions.                                                       |
+| 19      | `sqlinstance-tde`                | Various Transparent Data Encryption tests.                                                                                                             |
+| 20      | `telemetry-elasticsearch`        | Validates Log ingestion into Elasticsearch.                                                                                                            |
+| 21      | `telemetry-grafana`              | Validates Grafana is reachable.                                                                                                                        |
+| 22      | `telemetry-influxdb`             | Validates Metric ingestion into InfluxDB.                                                                                                              |
+| 23      | `telemetry-kafka`                | Various tests for Kafka using SSL, single/multi-broker setup.                                                                                          |
+| 24      | `telemetry-monitorstack`         | Tests Monitoring components, such as `Fluentbit` and `Collectd` are functional.                                                                            |
+| 25      | `telemetry-telemetryrouter`      | Tests Open Telemetry.                                                                                                                                  |
+| 26      | `telemetry-webhook`              | Tests Data Services Webhooks with valid and invalid calls.                                                                                             |
+| 27      | `upgrade-arcdata`                | Upgrades a full suite of SQL Instances (GP, BC 2 replica, BC 3 replica, with Active Directory) and upgrades from last month's release to latest build. |
+
+As an example, for `sqlinstance-ha`, the following tests are performed:
+
+- `test_critical_configmaps_present`: Ensures the ConfigMaps and relevant fields are present for a SQL Instance.
+- `test_suspended_system_dbs_auto_heal_by_orchestrator`: Ensures if `master` and `msdb` are suspended by any means (in this case, user). Orchestrator maintenance reconcile auto-heals it.
+- `test_suspended_user_db_does_not_auto_heal_by_orchestrator`: Ensures if a User Database is deliberately suspended by user, Orchestrator maintenance reconcile does not auto-heal it.
+- `test_delete_active_orchestrator_twice_and_delete_primary_pod`: Deletes orchestrator pod multiple times, followed by the primary replica, and verifies all replicas are synchronized. Failover time expectations for 2 replica are relaxed.
+- `test_delete_primary_pod`: Deletes primary replica and verifies all replicas are synchronized. Failover time expectations for 2 replica are relaxed.
+- `test_delete_primary_and_orchestrator_pod`: Deletes primary replica and orchestrator pod and verifies all replicas are synchronized.
+- `test_delete_primary_and_controller`: Deletes primary replica and data controller pod and verifies primary endpoint is accessible and the new primary replica is synchronized. Failover time expectations for 2 replica are relaxed.
+- `test_delete_one_secondary_pod`: Deletes secondary replica and data controller pod and verifies all replicas are synchronized.
+- `test_delete_two_secondaries_pods`: Deletes secondary replicas and data controller pod and verifies all replicas are synchronized.
+- `test_delete_controller_orchestrator_secondary_replica_pods`:
+- `test_failaway`: Forces AG failover away from current primary, ensures the new primary is not the same as the old primary. Verifies all replicas are synchronized.
+- `test_update_while_rebooting_all_non_primary_replicas`: Tests Controller-driven updates are resilient with retries despite various turbulent circumstances.
+
+> [!NOTE]
+> Certain tests may require specific hardware, such as privileged Access to Domain Controllers for `ad` tests for Account and DNS entry creation - which may not be available in all environments looking to use the `arc-ci-launcher`.
+
 ## Examining Test Results
 
 A sample storage container and file uploaded by the launcher: