Update documentation and improve configuration check reporting; enhance error handling in validation logic

Author: devanshjainms

README.md

@@ -11,48 +11,42 @@
 
 ## 🔍 Overview
 
-The SAP Testing Automation Framework is an open-source orchestration tool designed to validate SAP deployments on Microsoft Azure. It enables you to assess system configurations against SAP on Azure best practices and guidelines. Additionally, the framework facilitates automation for various testing scenarios, including High Availability (HA) functional testing.
+The SAP Testing Automation Framework is an open-source orchestration tool designed to validate SAP deployments on Microsoft Azure. It enables you to assess system configurations against SAP on Azure best practices and guidelines, and facilitates automation for various testing scenarios.
 
 > **NOTE**: This repository is currently in public preview and is intended for testing and feedback purposes. As this is an early release, it is not yet production-ready, and breaking changes can be introduced at any time.
 
 ![SAP Testing Automation Framework](./docs/images/sap-testing-automation-framework.png)
 
-## Supported Configuration Matrix
-
-The following SAP components are supported in a two-node Pacemaker cluster running on SUSE Linux Enterprise Server (SLES) or Red Hat Enterprise Linux (RHEL):
-
-- **SAP HANA Scale-Up**
-- **SAP Central Services**
-
-For additional information on supported configuration patterns, such as cluster types (Azure Fence Agent or SBD) and storage options (Azure Files or Azure NetApp Files) in this automated testing framework, refer to [supported high availability configuration](./docs/HIGH_AVAILABILITY.md).
-
 ## 📊 Key Features
 
 - **High Availability Testing** - Thorough validation of the SAP HANA scale-up and SAP Central Services failover mechanism in a two node pacemaker cluster, ensuring the system operates correctly across various test cases.
   - **Configuration Validation** - Ensures that SAP HANA scale-up and SAP Central Services configurations comply with SAP on Azure best practices and guidelines.
   - **Functional Testing** - Executes test scenarios on the high availability setup to identify potential issues, whether during a new system deployment or before implementing cluster changes in a production environment.
+- **Configuration Checks** - Validates OS parameters, database settings, Azure resources, and storage configurations against SAP and Azure best practices for supported databases. Performs comprehensive validation including kernel parameters, filesystem mounts, VM sizing, and network setup to ensure compliance with recommended guidelines.
 - **Detailed Reporting** - Generates comprehensive reports, highlighting configuration mismatch or deviations from recommended best practices. Includes failover test outcomes, any failures encountered, and logs with insights to aid in troubleshooting identified issues.
 
 ## 🏆 Purpose
 
 Testing is crucial for keeping SAP systems running smoothly, especially for critical business operations. This framework helps by addressing key challenges:
 
-- **Preventing Risks** - It simulates system failures like node crashes, network issues, and storage failures to check if recovery mechanisms work properly, helping to catch problems before they affect real operations.
-- **Meeting Compliance Requirements** - Many businesses need to prove their SAP systems are reliable. This framework provides detailed reports and logs that help with audits and ensure compliance with internal and regulatory standards.
-- **Ensuring Quality** -  The framework runs automated tests to verify whether the failover behavior of SAP components functions as expected on Azure across various test scenarios. It also ensures that the cluster and resource configurations are set up correctly, helping to maintain system reliability.
-- **Automating Testing**: Manually testing high availability (HA) setups is slow and error-prone. This framework automates the process—from setup to reporting—saving time and ensuring more accurate and consistent results.
+- **Preventing Risks** - Identifies configuration issues and validates system behavior before problems affect production operations. It simulates system failures like node crashes, network issues, and storage failures to check if recovery mechanisms work properly, helping to catch potential issues early.
+- **Meeting Compliance Requirements** - Provides detailed reports and logs that help with audits and ensure compliance with internal and regulatory standards.
+- **Ensuring Quality** - The framework runs automated tests to verify whether the failover behavior of SAP components functions as expected on Azure across various test scenarios. It also ensures that the cluster and resource configurations are set up correctly, helping to maintain system reliability.
+- **Automating Testing** - Automates validation processes from configuration checks to reporting, saving time and ensuring consistent results.
 
 ## 🚦 Get Started
 
 There are two primary ways to get started with the SAP Testing Automated Framework. Choose the path that best fits your current environment and objectives:
 
-### Option 1: [Integration with SAP Deployment Automation Framework (SDAF)](./docs/SDAF_INTEGRATION.md)
+### Option 1: Integration with SAP Deployment Automation Framework (SDAF)
 
 If you already have [SDAF](https://github.com/Azure/sap-automation) environment set up, integrating the SAP Testing Automation Framework is a natural extension that allows you to leverage existing deployment pipelines and configurations.
 
-### Option 2: [Getting Started with High Availability Testing (Standalone)](./docs/HIGH_AVAILABILITY.md)
+### Option 2: Getting Started with High Availability Testing (Standalone)
 
 For users focused solely on validating SAP functionality and configurations, the standalone approach offers a streamlined process to test critical SAP components without the complexity of full deployment integration.
+ - For High Availability testing details, see the [High Availability documentation](./docs/HIGH_AVAILABILITY.md).
+ - For Configuration Checks and Testing details, see the [Configuration Checks documentation](./docs/CONFIGURATION_CHECKS.md).
 
 ## 🏗️ Architecture and Components
 
@@ -68,7 +62,8 @@ For support and questions, please:
 ## 📚 Additional Resources
 
 - [Azure SAP Documentation](https://docs.microsoft.com/azure/sap)
-- [SAP on Azure: High Availability Guide](https://docs.microsoft.com/azure/sap/workloads/sap-high-availability-guide-start)
+- [Configuration Checks Guide](./docs/CONFIGURATION_CHECKS.md)
+- [High Availability Testing Guide](./docs/HIGH_AVAILABILITY.md)
 
 ## 🤝 Contributing
 

docs/CONFIGURATION_CHECKS.md

@@ -58,8 +58,16 @@ Follow the steps (2.1 - 2.2) in [Setup Guide for SAP Testing Automation Framewor
 
 > **Note**: High Availability (HA) configuration checks and functional tests are currently supported only for SAP HANA databases. For IBM DB2 databases, only non-HA configuration checks are available.
 
+### 3. Required Access and Permissions
 
-### 3. Test Execution
+Ensure that the managed identity or service principal used by the controller virtual machine has the necessary permissions to access Azure resources and SAP systems for configuration validation.
+1. "Reader" role to the user-assigned managed identity on the resource group containing the SAP VMs and the Azure Load Balancer.
+1. "Reader" role to the user-assigned managed identity on the resource group containing the Azure NetApp Files account (if using Azure NetApp Files as shared storage).
+1. "Reader" role to the user-assigned managed identity on the resource group containing the storage account (if using Azure File Share as shared storage).
+1. "Reader" role to the user-assigned managed identity on the resource group containing the managed disks (if using Azure Managed Disks for SAP HANA data and log volumes).
+1. "Reader" role to the user-assigned managed identity on the resource group containing the shared disks (if using Azure Shared Disks for SBD devices).
+
+### 4. Test Execution
 
 To execute the script, run following command:
 
@@ -83,7 +91,7 @@ To execute the script, run following command:
 ./scripts/sap_automation_qa.sh --extra-vars='{"configuration_test_type":"ApplicationInstances"}'
 ```
 
-### 4. Viewing Test Results
+### 5. Viewing Test Results
 
 After the test execution completes, a detailed HTML report is generated that summarizes the PASS/FAIL status of each test case and includes detailed execution logs for every step of the automation run.
 
@@ -101,12 +109,11 @@ After the test execution completes, a detailed HTML report is generated that sum
    The report file is named using the following format:
 
    ```
-   HA_{SAP_TIER}_{DATABASE_TYPE}_{OS_DISTRO_NAME}_{INVOCATION_ID}.html
+   CONFIG_{SAP_SID}_{DATABASE_TYPE}_{INVOCATION_ID}.html
    ```
 
-   - `SAP_TIER`: The SAP tier tested (e.g., DB, SCS)
+   - `SAP_SID`: The SAP system ID (e.g., HN1, NWP)
    - `DATABASE_TYPE`: The database type (e.g., HANA)
-   - `OS_DISTRO_NAME`: The operating system distribution (e.g., SLES15SP4)
    - `INVOCATION_ID`: A unique identifier (Group invocation ID) for the test run which is logged at the end of test execution. Find example screenshot below:
 
       ![Test Execution Completion Screenshot](./images/execution_screenshot.png)

docs/SETUP.MD

@@ -44,7 +44,7 @@ For the framework to access the properties of the Azure Load Balancer in a high
 **Permissions required for Configuration Checks:**
 1. "Reader" role to the user-assigned managed identity on the resource group containing the SAP VMs and the Azure Load Balancer.
 1. "Reader" role to the user-assigned managed identity on the resource group containing the Azure NetApp Files account (if using Azure NetApp Files as shared storage).
-1. "Storage Account Reader" role to the user-assigned managed identity on the resource group containing the storage account (if using Azure File Share as shared storage).
+1. "Reader" role to the user-assigned managed identity on the resource group containing the storage account (if using Azure File Share as shared storage).
 1. "Reader" role to the user-assigned managed identity on the resource group containing the managed disks (if using Azure Managed Disks for SAP HANA data and log volumes).
 1. "Reader" role to the user-assigned managed identity on the resource group containing the shared disks (if using Azure Shared Disks for SBD devices).
 

src/modules/configuration_check_module.py

@@ -547,6 +547,9 @@ def validate_min_list(self, check: Check, collected_data: str) -> Dict[str, Any]
             }
         except Exception as ex:
             self.log(logging.ERROR, f"Error while validating min list {ex}")
+            return {
+                "status": TestStatus.ERROR.value,
+            }
 
     def validate_vm_support(self, check: Check, collected_data: str) -> Dict[str, Any]:
         """

src/playbook_00_configuration_checks.yml

@@ -438,3 +438,8 @@
       ansible.builtin.include_tasks:   "./roles/misc/tasks/render-html-report.yml"
       vars:
         html_template_name:            "./templates/config_checks_report.html"
+        report_file_name:              "CONFIG_{{ sap_sid | upper }}_{{ platform | upper }}_{{ test_group_invocation_id }}"
+
+    - name:                             "Debug the file name of the report generated"
+      ansible.builtin.debug:
+        msg:                            "Report file CONFIG_{{ sap_sid | upper }}_{{ platform | upper }}_{{ test_group_invocation_id }} generated."

src/roles/misc/tasks/render-html-report.yml

@@ -16,7 +16,7 @@
     - name:                             "Read the log file and create a HTML report"
       render_html_report:
         test_group_invocation_id:       "{{ test_group_invocation_id }}"
-        test_group_name:                "{{ test_group_name }}_{{ ansible_os_family | upper }}"
+        test_group_name:                "{{ report_file_name | default(test_group_name + '_' + ansible_os_family | upper) }}"
         report_template:                "{{ html_report_template }}"
         workspace_directory:            "{{ _workspace_directory }}"
         test_case_results:              "{{ all_results | default([]) }}"