Skip to content

Commit f4561b5

Browse files
Updated parameters for global ini parameters and added unit tests (#138)
1 parent 18e9339 commit f4561b5

20 files changed

+3191
-151
lines changed

README.md

Lines changed: 12 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -11,48 +11,42 @@
1111

1212
## 🔍 Overview
1313

14-
The SAP Testing Automation Framework is an open-source orchestration tool designed to validate SAP deployments on Microsoft Azure. It enables you to assess system configurations against SAP on Azure best practices and guidelines. Additionally, the framework facilitates automation for various testing scenarios, including High Availability (HA) functional testing.
14+
The SAP Testing Automation Framework is an open-source orchestration tool designed to validate SAP deployments on Microsoft Azure. It enables you to assess system configurations against SAP on Azure best practices and guidelines, and facilitates automation for various testing scenarios.
1515

1616
> **NOTE**: This repository is currently in public preview and is intended for testing and feedback purposes. As this is an early release, it is not yet production-ready, and breaking changes can be introduced at any time.
1717
1818
![SAP Testing Automation Framework](./docs/images/sap-testing-automation-framework.png)
1919

20-
## Supported Configuration Matrix
21-
22-
The following SAP components are supported in a two-node Pacemaker cluster running on SUSE Linux Enterprise Server (SLES) or Red Hat Enterprise Linux (RHEL):
23-
24-
- **SAP HANA Scale-Up**
25-
- **SAP Central Services**
26-
27-
For additional information on supported configuration patterns, such as cluster types (Azure Fence Agent or SBD) and storage options (Azure Files or Azure NetApp Files) in this automated testing framework, refer to [supported high availability configuration](./docs/HIGH_AVAILABILITY.md).
28-
2920
## 📊 Key Features
3021

3122
- **High Availability Testing** - Thorough validation of the SAP HANA scale-up and SAP Central Services failover mechanism in a two node pacemaker cluster, ensuring the system operates correctly across various test cases.
3223
- **Configuration Validation** - Ensures that SAP HANA scale-up and SAP Central Services configurations comply with SAP on Azure best practices and guidelines.
3324
- **Functional Testing** - Executes test scenarios on the high availability setup to identify potential issues, whether during a new system deployment or before implementing cluster changes in a production environment.
25+
- **Configuration Checks** - Validates OS parameters, database settings, Azure resources, and storage configurations against SAP and Azure best practices for supported databases. Performs comprehensive validation including kernel parameters, filesystem mounts, VM sizing, and network setup to ensure compliance with recommended guidelines.
3426
- **Detailed Reporting** - Generates comprehensive reports, highlighting configuration mismatch or deviations from recommended best practices. Includes failover test outcomes, any failures encountered, and logs with insights to aid in troubleshooting identified issues.
3527

3628
## 🏆 Purpose
3729

3830
Testing is crucial for keeping SAP systems running smoothly, especially for critical business operations. This framework helps by addressing key challenges:
3931

40-
- **Preventing Risks** - It simulates system failures like node crashes, network issues, and storage failures to check if recovery mechanisms work properly, helping to catch problems before they affect real operations.
41-
- **Meeting Compliance Requirements** - Many businesses need to prove their SAP systems are reliable. This framework provides detailed reports and logs that help with audits and ensure compliance with internal and regulatory standards.
42-
- **Ensuring Quality** - The framework runs automated tests to verify whether the failover behavior of SAP components functions as expected on Azure across various test scenarios. It also ensures that the cluster and resource configurations are set up correctly, helping to maintain system reliability.
43-
- **Automating Testing**: Manually testing high availability (HA) setups is slow and error-prone. This framework automates the process—from setup to reportingsaving time and ensuring more accurate and consistent results.
32+
- **Preventing Risks** - Identifies configuration issues and validates system behavior before problems affect production operations. It simulates system failures like node crashes, network issues, and storage failures to check if recovery mechanisms work properly, helping to catch potential issues early.
33+
- **Meeting Compliance Requirements** - Provides detailed reports and logs that help with audits and ensure compliance with internal and regulatory standards.
34+
- **Ensuring Quality** - The framework runs automated tests to verify whether the failover behavior of SAP components functions as expected on Azure across various test scenarios. It also ensures that the cluster and resource configurations are set up correctly, helping to maintain system reliability.
35+
- **Automating Testing** - Automates validation processes from configuration checks to reporting, saving time and ensuring consistent results.
4436

4537
## 🚦 Get Started
4638

4739
There are two primary ways to get started with the SAP Testing Automated Framework. Choose the path that best fits your current environment and objectives:
4840

49-
### Option 1: [Integration with SAP Deployment Automation Framework (SDAF)](./docs/SDAF_INTEGRATION.md)
41+
### Option 1: Integration with SAP Deployment Automation Framework (SDAF)
5042

5143
If you already have [SDAF](https://github.com/Azure/sap-automation) environment set up, integrating the SAP Testing Automation Framework is a natural extension that allows you to leverage existing deployment pipelines and configurations.
5244

53-
### Option 2: [Getting Started with High Availability Testing (Standalone)](./docs/HIGH_AVAILABILITY.md)
45+
### Option 2: Getting Started with High Availability Testing (Standalone)
5446

5547
For users focused solely on validating SAP functionality and configurations, the standalone approach offers a streamlined process to test critical SAP components without the complexity of full deployment integration.
48+
- For High Availability testing details, see the [High Availability documentation](./docs/HIGH_AVAILABILITY.md).
49+
- For Configuration Checks and Testing details, see the [Configuration Checks documentation](./docs/CONFIGURATION_CHECKS.md).
5650

5751
## 🏗️ Architecture and Components
5852

@@ -68,7 +62,8 @@ For support and questions, please:
6862
## 📚 Additional Resources
6963

7064
- [Azure SAP Documentation](https://docs.microsoft.com/azure/sap)
71-
- [SAP on Azure: High Availability Guide](https://docs.microsoft.com/azure/sap/workloads/sap-high-availability-guide-start)
65+
- [Configuration Checks Guide](./docs/CONFIGURATION_CHECKS.md)
66+
- [High Availability Testing Guide](./docs/HIGH_AVAILABILITY.md)
7267

7368
## 🤝 Contributing
7469

docs/CONFIGURATION_CHECKS.md

Lines changed: 35 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -29,17 +29,17 @@ Configuration validation serves as a critical quality gate in the SAP deployment
2929
- Storage account redundancy settings
3030
- Disk caching policies
3131

32-
**SAP HANA Configuration**
33-
- Memory allocation
34-
- System replication parameters
32+
**SAP Database Configuration**
33+
- SAP HANA: Memory allocation, system replication parameters
34+
- IBM DB2: Hardware requirements, system language, OS tuning parameters
3535

36-
**Pacemaker Cluster**
36+
**Pacemaker Cluster (HANA only)**
3737
- Resource agent versions and parameters
3838
- Fencing (STONITH) configuration
3939
- Resource constraints and colocation rules
4040
- Cluster communication settings
4141

42-
**SAP HA Resources**
42+
**SAP HA Resources (HANA only)**
4343
- Virtual hostname configuration
4444
- File system mount options
4545
- Service startup ordering
@@ -56,8 +56,33 @@ Update the `TEST_TYPE` parameter in [`vars.yaml`](./../vars.yaml) file to `Confi
5656

5757
Follow the steps (2.1 - 2.2) in [Setup Guide for SAP Testing Automation Framework](./SETUP.MD#2-system-configuration) to configure your system details.
5858

59+
> **Note**: High Availability (HA) configuration checks and functional tests are currently supported only for SAP HANA databases. For IBM DB2 databases, only non-HA configuration checks are available.
5960
60-
### 3. Test Execution
61+
### 3. Required Access and Permissions
62+
63+
Ensure that the managed identity or service principal used by the controller virtual machine has the necessary permissions to access Azure resources and SAP systems for configuration validation.
64+
1. "Reader" role to the user-assigned managed identity on the resource group containing the SAP VMs and the Azure Load Balancer.
65+
1. "Reader" role to the user-assigned managed identity on the resource group containing the Azure NetApp Files account (if using Azure NetApp Files as shared storage).
66+
1. "Reader" role to the user-assigned managed identity on the resource group containing the storage account (if using Azure File Share as shared storage).
67+
1. "Reader" role to the user-assigned managed identity on the resource group containing the managed disks (if using Azure Managed Disks for SAP HANA data and log volumes).
68+
1. "Reader" role to the user-assigned managed identity on the resource group containing the shared disks (if using Azure Shared Disks for SBD devices).
69+
70+
### 4. Azure Login (required)
71+
72+
Ensure that you are logged into Azure CLI on the controller VM with the appropriate subscription context:
73+
74+
```bash
75+
# Login to Azure using System Assigned Managed Identity
76+
az login --identity
77+
78+
# Login to Azure using User Assigned Managed Identity
79+
az login --identity -u <client-id-of-user-assigned-managed-identity>
80+
81+
# Set the desired subscription context
82+
az account set --subscription <subscription-id>
83+
```
84+
85+
### 5. Test Execution
6186

6287
To execute the script, run following command:
6388

@@ -71,7 +96,7 @@ To execute the script, run following command:
7196
# Run checks with verbose logging
7297
./scripts/sap_automation_qa.sh -vv
7398

74-
# Run only Database (HANA) configuration checks
99+
# Run only Database configuration checks (supports both HANA and DB2)
75100
./scripts/sap_automation_qa.sh --extra-vars='{"configuration_test_type":"Database"}'
76101

77102
# Run only ASCS/ERS configuration checks
@@ -81,7 +106,7 @@ To execute the script, run following command:
81106
./scripts/sap_automation_qa.sh --extra-vars='{"configuration_test_type":"ApplicationInstances"}'
82107
```
83108

84-
### 4. Viewing Test Results
109+
### 6. Viewing Test Results
85110

86111
After the test execution completes, a detailed HTML report is generated that summarizes the PASS/FAIL status of each test case and includes detailed execution logs for every step of the automation run.
87112

@@ -99,12 +124,11 @@ After the test execution completes, a detailed HTML report is generated that sum
99124
The report file is named using the following format:
100125

101126
```
102-
HA_{SAP_TIER}_{DATABASE_TYPE}_{OS_DISTRO_NAME}_{INVOCATION_ID}.html
127+
CONFIG_{SAP_SID}_{DATABASE_TYPE}_{INVOCATION_ID}.html
103128
```
104129

105-
- `SAP_TIER`: The SAP tier tested (e.g., DB, SCS)
130+
- `SAP_SID`: The SAP system ID (e.g., HN1, NWP)
106131
- `DATABASE_TYPE`: The database type (e.g., HANA)
107-
- `OS_DISTRO_NAME`: The operating system distribution (e.g., SLES15SP4)
108132
- `INVOCATION_ID`: A unique identifier (Group invocation ID) for the test run which is logged at the end of test execution. Find example screenshot below:
109133

110134
![Test Execution Completion Screenshot](./images/execution_screenshot.png)

docs/SETUP.MD

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ For the framework to access the properties of the Azure Load Balancer in a high
4444
**Permissions required for Configuration Checks:**
4545
1. "Reader" role to the user-assigned managed identity on the resource group containing the SAP VMs and the Azure Load Balancer.
4646
1. "Reader" role to the user-assigned managed identity on the resource group containing the Azure NetApp Files account (if using Azure NetApp Files as shared storage).
47-
1. "Storage Account Reader" role to the user-assigned managed identity on the resource group containing the storage account (if using Azure File Share as shared storage).
47+
1. "Reader" role to the user-assigned managed identity on the resource group containing the storage account (if using Azure File Share as shared storage).
4848
1. "Reader" role to the user-assigned managed identity on the resource group containing the managed disks (if using Azure Managed Disks for SAP HANA data and log volumes).
4949
1. "Reader" role to the user-assigned managed identity on the resource group containing the shared disks (if using Azure Shared Disks for SBD devices).
5050

src/module_utils/collector.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,10 @@ def collect(self, check, context) -> str:
128128
if not re.match(r"^[a-zA-Z0-9_-]+$", user):
129129
self.parent.log(logging.ERROR, f"Invalid user parameter: {user}")
130130
return f"ERROR: Invalid user parameter: {user}"
131+
132+
if user == "db2sid":
133+
user = f"db2{context.get('database_sid', '').lower()}"
134+
131135
command = f"sudo -u {shlex.quote(user)} {command}"
132136

133137
return self.parent.execute_command_subprocess(

src/modules/configuration_check_module.py

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
import time
1010
import json
1111
import re
12+
import sys
1213
from typing import Optional, Dict, Any, List, Type
1314
from datetime import datetime
1415
from concurrent.futures import ThreadPoolExecutor
@@ -91,6 +92,7 @@ def _init_validator_registry(self) -> Dict[str, Any]:
9192
"string": self.validate_string,
9293
"range": self.validate_numeric_range,
9394
"list": self.validate_list,
95+
"min_list": self.validate_min_list,
9496
"check_support": self.validate_vm_support,
9597
"properties": self.validate_properties,
9698
}
@@ -497,6 +499,58 @@ def validate_list(self, check: Check, collected_data: str) -> Dict[str, Any]:
497499
),
498500
}
499501

502+
def validate_min_list(self, check: Check, collected_data: str) -> Dict[str, Any]:
503+
"""
504+
Validate that each value in a space-separated list meets or exceeds minimum values.
505+
Used for kernel parameters like kernel.sem where actual values must be >= minimum required.
506+
507+
:param check: Check definition containing min_values and separator in validator_args
508+
:type check: Check
509+
:param collected_data: Space-separated string of values from system
510+
:type collected_data: str
511+
:return: Validation result dictionary
512+
:rtype: Dict[str, Any]
513+
"""
514+
min_values = check.validator_args.get("min_values", [])
515+
separator = check.validator_args.get("separator", " ")
516+
try:
517+
518+
if not isinstance(min_values, list):
519+
return {
520+
"status": TestStatus.ERROR.value,
521+
}
522+
523+
collected_values = (
524+
str(collected_data).strip().split(separator) if collected_data else []
525+
)
526+
collected_values = [val.strip() for val in collected_values if val.strip()]
527+
if len(collected_values) != len(min_values):
528+
return {
529+
"status": self._create_validation_result(check.severity, False),
530+
}
531+
all_valid = True
532+
for actual, minimum in zip(collected_values, min_values):
533+
try:
534+
actual_int = int(actual)
535+
minimum_int = int(minimum)
536+
if actual_int > sys.maxsize or minimum_int > sys.maxsize:
537+
continue
538+
if actual_int < minimum_int:
539+
all_valid = False
540+
break
541+
except (ValueError, OverflowError):
542+
all_valid = False
543+
break
544+
545+
return {
546+
"status": self._create_validation_result(check.severity, all_valid),
547+
}
548+
except Exception as ex:
549+
self.log(logging.ERROR, f"Error while validating min list {ex}")
550+
return {
551+
"status": TestStatus.ERROR.value,
552+
}
553+
500554
def validate_vm_support(self, check: Check, collected_data: str) -> Dict[str, Any]:
501555
"""
502556
Validates if a VM SKU is supported for the given role and database type
@@ -609,6 +663,11 @@ def create_result(
609663
valid_list = check.validator_args.get("valid_list", [])
610664
if isinstance(valid_list, list) and valid_list:
611665
expected_value = ", ".join(str(v) for v in valid_list)
666+
elif check.validator_type == "min_list":
667+
min_values = check.validator_args.get("min_values", [])
668+
separator = check.validator_args.get("separator", " ")
669+
if isinstance(min_values, list) and min_values:
670+
expected_value = f"Min: {separator.join(str(v) for v in min_values)}"
612671
elif check.validator_type == "properties":
613672
props = check.validator_args.get("properties", [])
614673
if isinstance(props, list) and props:
@@ -875,7 +934,7 @@ def run(self):
875934
context["hostname"] = custom_hostname
876935

877936
self.set_context(context)
878-
if self.context.get("check_type", {}).get("file_name") == "hana":
937+
if self.context.get("check_type", {}).get("file_name") in ["hana", "db2"]:
879938
temp_context = FileSystemCollector(parent=self).collect(
880939
check=None, context=self.context
881940
)

src/modules/get_azure_lb.py

Lines changed: 41 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -233,31 +233,66 @@ def get_load_balancers_details(self) -> None:
233233
for inbound_rule in inbound_rules
234234
if "privateIpAddress" in inbound_rule
235235
)
236+
237+
self.log(logging.INFO, f"Looking for load balancers with IPs: {load_balancer_ips}")
238+
236239
found_load_balancer = None
237240

241+
def get_private_ip_from_config(config):
242+
"""
243+
Extract private IP from frontend config, handling different key variations.
244+
Azure SDK might return different structures based on authentication context.
245+
"""
246+
private_ip = (
247+
config.get("private_ip_address")
248+
or config.get("privateIpAddress")
249+
)
250+
return private_ip
251+
238252
found_load_balancer = next(
239253
(
240254
lb
241255
for lb in load_balancers
242-
for frontend_ip_config in lb["frontend_ip_configurations"]
243-
if frontend_ip_config["private_ip_address"] in load_balancer_ips
256+
for frontend_ip_config in lb.get("frontend_ip_configurations", [])
257+
if get_private_ip_from_config(frontend_ip_config) in load_balancer_ips
244258
),
245259
None,
246260
)
261+
262+
if not found_load_balancer and load_balancers:
263+
available_ips = []
264+
self.log(
265+
logging.WARNING, f"No matching load balancer found for IPs: {load_balancer_ips}"
266+
)
267+
for lb in load_balancers:
268+
lb_name = lb.get("name", "unknown")
269+
for config in lb.get("frontend_ip_configurations", []):
270+
private_ip = get_private_ip_from_config(config)
271+
if private_ip:
272+
available_ips.append(f"{lb_name}:{private_ip}")
273+
else:
274+
self.log(
275+
logging.DEBUG,
276+
f"Frontend config structure for {lb_name}: {list(config.keys())}",
277+
)
278+
self.log(logging.WARNING, f"Available load balancers and private IPs: {available_ips}")
247279
parameters = []
248280

249281
def check_parameters(entity, parameters_dict, entity_type):
250282
for key, value_object in parameters_dict.items():
283+
entity_value = entity.get(key, "N/A")
284+
expected_value = value_object.get("value", "")
285+
251286
parameters.append(
252287
Parameters(
253288
category=entity_type,
254-
id=entity["name"],
289+
id=entity.get("name", "unknown"),
255290
name=key,
256-
value=str(entity[key]),
257-
expected_value=str(value_object.get("value", "")),
291+
value=str(entity_value),
292+
expected_value=str(expected_value),
258293
status=(
259294
TestStatus.SUCCESS.value
260-
if entity[key] == value_object.get("value", "")
295+
if entity_value == expected_value
261296
else TestStatus.ERROR.value
262297
),
263298
).to_dict()

0 commit comments

Comments
 (0)