Skip to content

Commit 18c57e9

Browse files
Enhance SCS cluster status tests with additional resource attributes (#76)
Co-authored-by: Dhruv Aggarwal <[email protected]>
1 parent 4198651 commit 18c57e9

File tree

81 files changed

+3991
-591
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+3991
-591
lines changed

.github/workflows/codeql.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,14 +46,14 @@ jobs:
4646
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
4747

4848
- name: Initialize CodeQL
49-
uses: github/codeql-action/init@45775bd8235c68ba998cffa5171334d58593da47 # v3.28.15
49+
uses: github/codeql-action/init@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16
5050
with:
5151
languages: ${{ matrix.language }}
5252

5353
- name: Autobuild
54-
uses: github/codeql-action/autobuild@45775bd8235c68ba998cffa5171334d58593da47 # v3.28.15
54+
uses: github/codeql-action/autobuild@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16
5555

5656
- name: Perform CodeQL Analysis
57-
uses: github/codeql-action/analyze@45775bd8235c68ba998cffa5171334d58593da47 # v3.28.15
57+
uses: github/codeql-action/analyze@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16
5858
with:
5959
category: "/language:${{matrix.language}}"

.github/workflows/ossf-scoreboard.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,6 @@ jobs:
5050
retention-days: 5
5151

5252
- name: "Upload to code-scanning"
53-
uses: github/codeql-action/upload-sarif@45775bd8235c68ba998cffa5171334d58593da47 # v3.28.15
53+
uses: github/codeql-action/upload-sarif@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16
5454
with:
5555
sarif_file: results.sarif

.github/workflows/trivy.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ jobs:
3636
output: report-fs.sarif
3737

3838
- name: Upload Trivy report (fs) GitHub Security
39-
uses: github/codeql-action/upload-sarif@45775bd8235c68ba998cffa5171334d58593da47 # v3.28.15
39+
uses: github/codeql-action/upload-sarif@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16
4040
with:
4141
sarif_file: report-fs.sarif
4242
category: 'fs'

WORKSPACES/SYSTEM/DEV-WEEU-SAP01-X00/sap-parameters.yaml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,16 @@ database_cluster_type: AFA
2727
# Storage Profile #
2828
#############################################################################
2929
NFS_provider: AFS
30+
31+
#############################################################################
32+
# Key Vault Parameters (optional) #
33+
#############################################################################
34+
key_vault_id: /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.KeyVault/vaults/<key-vault-name>
35+
secret_id: https://<key-vault-name>.vault.azure.net/secrets/<secret-name>/<id>
36+
37+
#############################################################################
38+
# MSI Client ID #
39+
#############################################################################
40+
# The MSI Client ID is used to authenticate to Azure services
41+
# and is required if the management server uses user assigned managed identity
42+
user_assigned_identity_client_id: "00000000-0000-0000-0000-000000000000"

docs/HIGH_AVAILABILITY.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,13 @@ platform: "HANA"
255255
# - ANF (for Azure NetApp Files)
256256
# - AFS (for Azure File Share)
257257
NFS_provider: "ANF" # or "AFS"
258+
259+
# If you're using a user-assigned managed identity (as explained in "Azure RBAC" section above):
260+
# - Enter the client ID of that identity here
261+
# - You can find this ID in Azure Portal → Managed Identities → Your Identity → Properties → Client ID
262+
# If you're using system-assigned managed identity instead:
263+
# - Leave this blank or set to empty string ""
264+
user_assigned_identity_client_id: "000000-00000-00000-00000-000000"
258265
```
259266

260267
2.2.3. Credential Files

docs/SCS_HIGH_AVAILABILITY.md

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,14 @@
55
| Test Case | Type | Description | More Info |
66
|------------------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|
77
| HA Parameters Validation | Configuration | The HA parameter validation test validates HA configuration including Corosync settings, Pacemaker resources, SBD device configuration, and SCS system replication setup. | [ha-config.yml](../src/roles/ha_scs/tasks/ha-config.yml) |
8-
| Resource Migration | Failover | The Resource Migration test validates planned failover scenarios by controlling resource movement between SCS nodes, ensuring proper role changes and data synchronization. | [ascs-migration.yml](../src/roles/ha_scs/tasks/ascs-migration.yml) |
9-
| ASCS Node Crash | Network | The ASCS Node Crash test simulates cluster behavior when the ASCS node crashes. It simulates an ASCS node failure by forcefully terminating the process, then verifies automatic failover to the ERS node, monitors system replication status, and confirms service recovery without data loss. | [ascs-node-crash.yml](../src/roles/ha_scs/tasks/ascs-node-crash.yml) |
8+
| Azure Load Balancer | Configuration | The Azure LB configuration test validates Azure Load Balancer setup including health probe configuration, backend pool settings, load balancing rules, and frontend IP configuration. | [azure-lb.yml](../src/roles/ha_scs/tasks/azure-lb.yml) |
9+
| SAPControl Config Validation | Configuration | The SAPControl Config Validation test runs multiple sapcontrol commands to validate the SCS configuration. It executes commands like HAGetFailoverConfig, HACheckFailoverConfig, and HACheckConfig, capturing their outputs and statuses to ensure proper configuration and functionality. | [sapcontrol-config.yml](../src/roles/ha_scs/tasks/sapcontrol-config.yml) |
10+
| Resource Migration | Failover | The Resource Migration test validates planned failover scenarios by controlling resource movement between SCS nodes, ensuring proper role changes. | [ascs-migration.yml](../src/roles/ha_scs/tasks/ascs-migration.yml) |
11+
| ASCS Node Crash | Failover | The ASCS Node Crash test simulates cluster behavior when the ASCS node crashes. It simulates an ASCS node failure by forcefully terminating the process, then verifies automatic failover to the ERS node, monitors system replication status, and confirms service recovery. | [ascs-node-crash.yml](../src/roles/ha_scs/tasks/ascs-node-crash.yml) |
12+
| Block Network Communication | Network | The Block Network test validates cluster behavior during network partition scenarios by implementing iptables rules to block communication between ASCS and ERS nodes. It verifies split-brain prevention mechanisms, validates proper failover execution when nodes become isolated, and ensures cluster stability after network connectivity is restored. | [block-network.yml](../src/roles/ha_scs/tasks/block-network.yml) |
13+
| Kill Message Server Process | Process | The Message Server Process Kill test simulates failure of the message server process on the ASCS node by forcefully terminating it using the kill -9 signal. It verifies proper cluster reaction, automatic failover to the ERS node, and ensures service continuity after the process failure. | [kill-message-server.yml](../src/roles/ha_scs/tasks/kill-message-server.yml) |
14+
| Kill Enqueue Server Process | Process | The Enqueue Server Process Kill test simulates failure of the enqueue server process on the ASCS node by forcefully terminating it using the kill -9 signal. It validates proper cluster behavior, automatic failover execution. | [kill-enqueue-server.yml](../src/roles/ha_scs/tasks/kill-enqueue-server.yml) |
15+
| Kill Enqueue Replication Server Process | Process | The Enqueue Replication Server Process Kill test simulates failure of the replication server process on the ERS node by forcefully terminating it using the kill -9 signal. This test handles both ENSA1 and ENSA2 architectures. It validates the automatic restart of the process. | [kill-enqueue-replication.yml](../src/roles/ha_scs/tasks/kill-enqueue-replication.yml) |
16+
| Kill sapstartsrv Process for ASCS | Process | The sapstartsrv Process Kill test simulates failure of the SAP Start Service for the ASCS instance by forcefully terminating it using the kill -9 signal. It validates proper cluster reaction, automatic failover to the ERS node, and verifies service restoration after the process failure. | [kill-sapstartsrv.yml](../src/roles/ha_scs/tasks/kill-sapstartsrv.yml) |
17+
| Manual Restart of ASCS Instance | Control | The Manual Restart test validates cluster behavior when the ASCS instance is manually stopped using sapcontrol. It verifies proper cluster reaction to a controlled instance shutdown, ensures automatic failover to the ERS node, and confirms service continuity throughout the operation. | [manual-restart.yml](../src/roles/ha_scs/tasks/manual-restart.yml) |
18+
| HAFailoverToNode Test | Control | The HAFailoverToNode test validates SAP's built-in high availability functionality by using the sapcontrol command to trigger a controlled failover. It executes 'HAFailoverToNode' as the SAP administrator user, which initiates a clean migration of the ASCS instance to another node. | [ha-failover-to-node.yml](../src/roles/ha_scs/tasks/ha-failover-to-node.yml) |

docs/SDAF_INTEGRATION.md

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,13 +60,41 @@ This guide will help you set up your existing SAP Deployment Automation Framewor
6060
| **SAP Functional Tests Type** | Test category to run | Yes | DatabaseHighAvailability |
6161
| **Telemetry Data Destination** | Where to send test data | No | AzureLogAnalytics |
6262

63-
**For AzureLogAnalytics destination** (required parameters):
63+
**Providing Telemetry Data Destination Parameters**
64+
65+
To configure the Telemetry Data Destination for the SAP Testing Automation Framework, you need to specify the required parameters in the Extra Parameters input field. This allows the pipeline to send telemetry data to the desired destination, such as Azure Log Analytics or Azure Data Explorer.
66+
67+
**How to Specify Telemetry Parameters**
68+
69+
Use the following format in the Extra Parameters field:
70+
71+
```bash
72+
--extra-vars "laws_workspace_id=<workspace-id>,laws_shared_key=<shared-key>,telemetry_table_name=<table-name>"
73+
```
74+
75+
Telemetry Data Destination Options
76+
1. **Azure Log Analytics**
77+
If you are using Azure Log Analytics as the telemetry destination, the following parameters are required:
6478
- `laws_workspace_id`: Log Analytics Workspace ID
6579
- `laws_shared_key`: Log Analytics Shared Key
6680
- `telemetry_table_name`: Name of the table in Log Analytics
6781

68-
**For AzureDataExplorer destination** (required parameters):
82+
```bash
83+
--extra-vars "laws_workspace_id=12345678-1234-1234-1234-123456789abc,laws_shared_key=**********,telemetry_table_name=SAPTelemetry"
84+
```
85+
86+
2. **Azure Data Explorer**
87+
If you are using Azure Data Explorer (ADX) as the telemetry destination, the following parameters are required:
6988
- `adx_cluster_fqdn`: Azure Data Explorer Cluster FQDN
7089
- `adx_database_name`: Azure Data Explorer Database Name
7190
- `adx_client_id`: Azure Data Explorer Client ID
7291
- `telemetry_table_name`: Name of the table in ADX database
92+
93+
```bash
94+
--extra-vars "adx_cluster_fqdn=myadxcluster.kusto.windows.net,adx_database_name=SAPTelemetryDB,adx_client_id=12345678-1234-1234-1234-123456789abc,telemetry_table_name=SAPTelemetry"
95+
```
96+
97+
98+
99+
100+

docs/pseudocode/block-network.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,4 +81,34 @@ FUNCTION BlockNetworkTest():
8181
8282
RETURN "TEST_PASSED"
8383
END FUNCTION
84+
```
85+
86+
## ASCS Block Network Test Case
87+
88+
This test case is a specific instance of blocking network communication, focusing on ASCS-specific scenarios.
89+
90+
### Pre-requisites
91+
92+
- Functioning ASCS/ERS cluster
93+
- Two active nodes (ASCS and ERS)
94+
- Cluster services running
95+
- iptables service accessible
96+
- STONITH configuration (stonith-enabled=true)
97+
98+
### Additional Steps for ASCS Block Network
99+
100+
- Validate ASCS-specific failover behavior.
101+
- Ensure proper role changes for ASCS and ERS nodes.
102+
103+
### Pseudocode Extension
104+
105+
```pseudocode
106+
FUNCTION ASCSBlockNetworkTest():
107+
// Reuse BlockNetworkTest pseudocode
108+
CALL BlockNetworkTest()
109+
110+
// Additional ASCS-specific validations
111+
validate_ascs_failover_behavior()
112+
ensure_ascs_role_changes()
113+
END FUNCTION
84114
```
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
<!-- filepath: /home/devanshjain/SDAF/sap-automation-qa/docs/pseudocode/ha-failover-to-node.md -->
2+
# HAFailoverToNode Test Case
3+
4+
## Prerequisites
5+
6+
- Functioning SCS cluster
7+
- Two active nodes (ASCS and ERS)
8+
- Cluster services running
9+
- Proper resource configuration
10+
11+
## Validation
12+
13+
- Verify failover to the ERS node
14+
- Check cluster stability
15+
- Validate proper role changes
16+
17+
## Pseudocode
18+
19+
```pseudocode
20+
FUNCTION HAFailoverToNodeTest():
21+
// Setup Phase
22+
EXECUTE TestSetup()
23+
EXECUTE PreValidations()
24+
25+
IF pre_validations_status != "PASSED" THEN
26+
RETURN "Test Prerequisites Failed"
27+
28+
// Main Test Execution
29+
TRY:
30+
IF current_node == ascs_node THEN
31+
record_start_time()
32+
33+
// Execute Failover Command
34+
success = execute_failover_command(ers_node)
35+
IF NOT success THEN
36+
THROW "Failed to execute failover command"
37+
38+
// Validate Cluster Status
39+
cluster_status = validate_cluster_status()
40+
IF cluster_status.ascs_node != ers_node OR cluster_status.ers_node != ascs_node THEN
41+
THROW "Cluster status validation failed after failover"
42+
43+
// Cleanup Constraints
44+
success = remove_location_constraints()
45+
IF NOT success THEN
46+
THROW "Failed to remove location constraints"
47+
48+
// Cleanup Resources
49+
success = cleanup_cluster_resources()
50+
IF NOT success THEN
51+
THROW "Failed to cleanup cluster resources"
52+
53+
record_end_time()
54+
generate_test_report()
55+
END IF
56+
57+
EXECUTE PostValidations()
58+
59+
CATCH any_error:
60+
LOG "Error occurred: " + any_error
61+
EXECUTE RescueOperations()
62+
EXECUTE CleanupOperations()
63+
RETURN "TEST_FAILED"
64+
FINALLY:
65+
EXECUTE EnsureClusterHealth()
66+
67+
RETURN "TEST_PASSED"
68+
END FUNCTION
69+
```
Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
<!-- filepath: /home/devanshjain/SDAF/sap-automation-qa/docs/pseudocode/kill-message-server.md -->
2+
# Kill Message Server Process Test Case
3+
4+
## Prerequisites
5+
6+
- Functioning SCS cluster
7+
- Two active nodes (ASCS and ERS)
8+
- Cluster services running
9+
- Proper resource configuration
10+
11+
## Validation
12+
13+
- Verify failover to the ERS node
14+
- Check cluster stability
15+
- Validate proper role changes
16+
17+
## Pseudocode
18+
19+
```pseudocode
20+
FUNCTION KillMessageServerTest():
21+
// Setup Phase
22+
EXECUTE TestSetup()
23+
EXECUTE PreValidations()
24+
25+
IF pre_validations_status != "PASSED" THEN
26+
RETURN "Test Prerequisites Failed"
27+
28+
// Main Test Execution
29+
TRY:
30+
IF current_node == ascs_node THEN
31+
record_start_time()
32+
33+
// Check ENSA Version
34+
ensa_version = check_ensa_version()
35+
36+
// Kill Message Server Process
37+
success = kill_message_server_process()
38+
IF NOT success THEN
39+
THROW "Failed to kill message server process"
40+
41+
// Validate ASCS Node Stopped
42+
cluster_status = validate_cluster_status()
43+
IF cluster_status.ascs_node != "" THEN
44+
THROW "ASCS node did not stop as expected"
45+
46+
// Validate Failover to ERS Node
47+
cluster_status = validate_cluster_status()
48+
IF cluster_status.ascs_node != ers_node OR cluster_status.ers_node != ascs_node THEN
49+
THROW "Failover validation failed"
50+
51+
// Cleanup Resources
52+
success = cleanup_cluster_resources()
53+
IF NOT success THEN
54+
THROW "Failed to cleanup cluster resources"
55+
56+
record_end_time()
57+
generate_test_report()
58+
END IF
59+
60+
EXECUTE PostValidations()
61+
62+
CATCH any_error:
63+
LOG "Error occurred: " + any_error
64+
EXECUTE RescueOperations()
65+
EXECUTE CleanupOperations()
66+
RETURN "TEST_FAILED"
67+
FINALLY:
68+
EXECUTE EnsureClusterHealth()
69+
70+
RETURN "TEST_PASSED"
71+
END FUNCTION
72+
```
73+
74+
## Kill Enqueue, Enqueue Replication, and sapstartsrv Processes
75+
76+
These test cases are specific instances of killing processes, focusing on enqueue, enqueue replication, and sapstartsrv processes.
77+
78+
### Additional Steps for Each Process
79+
80+
- Validate process-specific failover behavior.
81+
- Ensure proper role changes for ASCS and ERS nodes.
82+
83+
### Pseudocode Extension
84+
85+
```pseudocode
86+
FUNCTION KillEnqueueProcessTest():
87+
// Reuse KillMessageServerTest pseudocode
88+
CALL KillMessageServerTest()
89+
90+
// Additional enqueue-specific validations
91+
validate_enqueue_failover_behavior()
92+
ensure_enqueue_role_changes()
93+
END FUNCTION
94+
95+
FUNCTION KillEnqueueReplicationProcessTest():
96+
// Reuse KillMessageServerTest pseudocode
97+
CALL KillMessageServerTest()
98+
99+
// Additional enqueue replication-specific validations
100+
validate_enqueue_replication_failover_behavior()
101+
ensure_enqueue_replication_role_changes()
102+
END FUNCTION
103+
104+
FUNCTION KillSapstartsrvProcessTest():
105+
// Reuse KillMessageServerTest pseudocode
106+
CALL KillMessageServerTest()
107+
108+
// Additional sapstartsrv-specific validations
109+
validate_sapstartsrv_failover_behavior()
110+
ensure_sapstartsrv_role_changes()
111+
END FUNCTION
112+
```

0 commit comments

Comments
 (0)