feat: gp alerts #651

Deezzir · 2025-08-27T02:07:00Z

Issue

The current setup lacks the alerts for Group Replication

Solution

The PR introduces a new alert group for Group Replication. Additionally, it restructures the alerts and creates a CI job to lint and unit test them.

Improves on: #631

codecov · 2025-08-27T02:08:16Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@29437d3). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #651   +/-   ##
=======================================
  Coverage        ?   50.95%           
=======================================
  Files           ?       13           
  Lines           ?     1841           
  Branches        ?      315           
=======================================
  Hits            ?      938           
  Misses          ?      826           
  Partials        ?       77

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull Request Overview

This PR introduces comprehensive MySQL Group Replication alert rules for Prometheus monitoring. It restructures the existing alert system and adds CI testing for alert validation.

Adds new alert group specifically for MySQL Group Replication scenarios (offline members, primary issues, conflicts)
Restructures existing general MySQL alerts with improved naming and organization
Implements CI pipeline for alert rule validation and unit testing

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/prometheus_alert_rules/replication_rules.yaml	New alert rules for MySQL Group Replication monitoring
src/prometheus_alert_rules/general_rules.yaml	Refactored general MySQL alerts with improved organization
tests/alerts/test_replication_alerts.yaml	Unit tests for Group Replication alert rules
tests/alerts/test_general_alerts.yaml	Unit tests for general MySQL alert rules
.github/workflows/ci.yaml	Added CI job for alert rule validation and testing

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-28T12:41:45Z

src/prometheus_alert_rules/replication_rules.yaml

+          summary: MySQL cluster member {{ $labels.instance }} is offline
+          description: |
+            The MySQL member is marked offline in the cluster, although the process might still be running.
+            If this is unexptected, please check the logs.


There is a spelling error in 'unexptected'. It should be 'unexpected'.

Suggested change

If this is unexptected, please check the logs.

If this is unexpected, please check the logs.

Copilot · 2025-08-28T12:41:46Z

src/prometheus_alert_rules/replication_rules.yaml

+        labels:
+          severity: critical
+        annotations:
+          summary: MySQL cluster reports no primariy


There is a spelling error in 'primariy'. It should be 'primary'.

Suggested change

summary: MySQL cluster reports no primariy

summary: MySQL cluster reports no primary

Copilot · 2025-08-28T12:41:46Z

src/prometheus_alert_rules/replication_rules.yaml

+        annotations:
+          summary: MySQL cluster reports more than one primary.
+          description: |
+            MySQL reports more than one primary. This is can indicate a split-brain situation.


There is a grammatical error in 'This is can indicate'. It should be 'This can indicate'.

Suggested change

MySQL reports more than one primary. This is can indicate a split-brain situation.

MySQL reports more than one primary. This can indicate a split-brain situation.

Copilot · 2025-08-28T12:41:46Z

tests/alerts/test_replication_alerts.yaml

+              summary: MySQL cluster member db1 is offline
+              description: |
+                The MySQL member is marked offline in the cluster, although the process might still be running.
+                If this is unexptected, please check the logs.


There is a spelling error in 'unexptected'. It should be 'unexpected'.

Suggested change

If this is unexptected, please check the logs.

If this is unexpected, please check the logs.

Copilot · 2025-08-28T12:41:46Z

tests/alerts/test_replication_alerts.yaml

+              severity: critical
+              member_role: PRIMARY
+            exp_annotations:
+              summary: MySQL cluster reports no primariy


There is a spelling error in 'primariy'. It should be 'primary'.

Suggested change

summary: MySQL cluster reports no primariy

summary: MySQL cluster reports no primary

Copilot · 2025-08-28T12:41:47Z

tests/alerts/test_replication_alerts.yaml

+            exp_annotations:
+              summary: MySQL cluster reports more than one primary.
+              description: |
+                MySQL reports more than one primary. This is can indicate a split-brain situation.


There is a grammatical error in 'This is can indicate'. It should be 'This can indicate'.

Suggested change

MySQL reports more than one primary. This is can indicate a split-brain situation.

MySQL reports more than one primary. This can indicate a split-brain situation.

Copilot · 2025-08-28T12:41:47Z

src/prometheus_alert_rules/general_rules.yaml

      - alert: MySQLDown
-        expr: "mysql_up == 0"
+        expr: mysql_up == 0
        for: 0m


[nitpick] Setting 'for: 0m' makes the alert fire immediately without any grace period. Consider if a brief delay (like 1m or 2m) would be more appropriate to avoid false positives from temporary network issues.

carlcsaposs-canonical · 2025-08-28T14:39:47Z

.github/workflows/ci.yaml

@@ -42,6 +42,19 @@ jobs:
      - name: Upload Coverage to Codecov
        uses: codecov/codecov-action@v5

+  alert-test:
+    name: Test Prometheus Alert Rules
+    runs-on: ubuntu-latest


Suggested change

runs-on: ubuntu-latest

runs-on: ubuntu-latest

timeout-minutes: 5

paulomach

Did not manage to trigger MySQLClusterNoPrimary although I did simulate it.

paulomach · 2025-08-29T19:18:01Z

src/prometheus_alert_rules/replication_rules.yaml

+          max_over_time(
+            count(mysql_perf_schema_replication_group_member_info{member_state="ONLINE"} == 1)[6h:]
+          )
+        for: 15m


This could be faster to trigger.

Suggested change

for: 15m

for: 5m

paulomach · 2025-08-29T19:24:05Z

src/prometheus_alert_rules/replication_rules.yaml

+            If this is unexptected, please check the logs.
+            LABELS = {{ $labels }}.
+
+      - alert: MySQLClusterNoPrimary


@Deezzir
For whatever reason, this is not firing properly when cluster looses its quorum.
The test this on k8s, you can do a kubectl delete pod -n <model> mysql-k8s-0 mysql-k8s-1 on the three node cluster where the primary is include in the delete command.

Also not firing when all three units are offline

Deezzir · 2025-08-29T19:37:26Z

Did not manage to trigger MySQLClusterNoPrimary although I did simulate it.

Do you mind sharing the metrics from the exporter at the time of simulation?

paulomach · 2025-08-29T19:54:44Z

Did not manage to trigger MySQLClusterNoPrimary although I did simulate it.

Do you mind sharing the metrics from the exporter at the time of simulation?

Here:
https://pastebin.canonical.com/p/X7rjZJSThk/

This is queried from the not killed unit, there are about 7 or 8 outputs in sequence, separated by #####

paulomach · 2025-08-29T20:48:23Z

src/prometheus_alert_rules/replication_rules.yaml

+            LABELS = {{ $labels }}.
+
+      - alert: MySQLClusterNoPrimary
+        expr: absent(mysql_perf_schema_replication_group_member_info{member_role="PRIMARY"})


We need to test for an online primary

Suggested change

expr: absent(mysql_perf_schema_replication_group_member_info{member_role="PRIMARY"})

expr: absent(mysql_perf_schema_replication_group_member_info{member_role="PRIMARY",member_state="ONLINE"})

feat: gp alerts

0f84173

Deezzir force-pushed the DPE-7859-my-sql-add-basic-gr-metrics-to-charms branch from 56c4f9d to 0f84173 Compare August 27, 2025 02:07

fix: update checkout action version

bbfd74d

paulomach mentioned this pull request Aug 28, 2025

Add basic cluster health alerts #631

Closed

1 task

paulomach requested review from Copilot and paulomach August 28, 2025 12:39

paulomach assigned carlcsaposs-canonical and unassigned carlcsaposs-canonical Aug 28, 2025

paulomach requested review from carlcsaposs-canonical and sinclert-canonical August 28, 2025 12:40

Copilot AI reviewed Aug 28, 2025

View reviewed changes

carlcsaposs-canonical approved these changes Aug 28, 2025

View reviewed changes

canonical deleted a comment from Copilot AI Aug 29, 2025

paulomach reviewed Aug 29, 2025

View reviewed changes

fix: spelling & rules

56718d0

	If this is unexptected, please check the logs.
	If this is unexpected, please check the logs.

	summary: MySQL cluster reports no primariy
	summary: MySQL cluster reports no primary

	MySQL reports more than one primary. This is can indicate a split-brain situation.
	MySQL reports more than one primary. This can indicate a split-brain situation.

	runs-on: ubuntu-latest
	runs-on: ubuntu-latest
	timeout-minutes: 5

	expr: absent(mysql_perf_schema_replication_group_member_info{member_role="PRIMARY"})
	expr: absent(mysql_perf_schema_replication_group_member_info{member_role="PRIMARY",member_state="ONLINE"})

feat: gp alerts #651

Are you sure you want to change the base?

feat: gp alerts #651

Uh oh!

Conversation

Deezzir commented Aug 27, 2025

Issue

Solution

Uh oh!

codecov bot commented Aug 27, 2025

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

carlcsaposs-canonical Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

paulomach left a comment

Choose a reason for hiding this comment

Uh oh!

paulomach Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

paulomach Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

paulomach Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Deezzir commented Aug 29, 2025

Uh oh!

paulomach commented Aug 29, 2025

Uh oh!

paulomach Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!