Improvement on administrator alerts #6613
Description
ISSUE TYPE
- Improvement Request
COMPONENT NAME
Alerts
CLOUDSTACK VERSION
4.16.1 and onwards
CONFIGURATION
Use the default values for the configurations:
- capacity.check.period : 300000
- alert.purge.delay : 0
- alert.purge.interval : 86400
- Define notification thresholds for cluster capacity: (for example cluster.storage.capacity.notificationthreshold : 0.75 )
OS / ENVIRONMENT
N/A
SUMMARY
The first time a cluster threshold is exceeded CloudStack notifies the administrator by sending an email and creates a record on the alert table. If the alert.purge.delay value is set to 0 (default value) then this record is never deleted, so the next time the threshold is exceeded the administrator does not get notified. CloudStack queries the database and searches for records on the alert table for the same alert type and cluster, and in case it finds an existing records it does not send the notification email.
Example:
DB:
mysql> select * from alert where type = 3 and cluster_id = 1 and archived = false;
+-------+--------------------------------------+------+------------+--------+----------------+------------------------------------------------------------------------------------------------------------------+------------+---------------------+---------------------+----------+----------+-------------------------+---------------------------------------------------------------------------------------+
| id | uuid | type | cluster_id | pod_id | data_center_id | subject | sent_count | created | last_sent | resolved | archived | name | content |
+-------+--------------------------------------+------+------------+--------+----------------+------------------------------------------------------------------------------------------------------------------+------------+---------------------+---------------------+----------+----------+-------------------------+---------------------------------------------------------------------------------------+
| 28623 | 885be044-4507-4a18-b71a-44af87ada1cf | 3 | 1 | 1 | 1 | System Alert: Remaining unallocated Storage is low in cluster BNZ-CL-001 pod BNZ-POD-01 of availability zone BNZ | 1 | 2021-09-05 02:41:48 | 2021-09-05 02:41:48 | NULL | 0 | ALERT.STORAGE.ALLOCATED | Unallocated storage space is low, total: 20971496 MB, allocated: 16332262 MB (77.88%) |
+-------+--------------------------------------+------+------------+--------+----------------+------------------------------------------------------------------------------------------------------------------+------------+---------------------+---------------------+----------+----------+-------------------------+---------------------------------------------------------------------------------------+
1 row in set (1.26 sec)
Logs:
2022-08-03 00:01:53,297 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-166624fc) (logid:8e4ca7a5) System Alert: Remaining unallocated Storage is low in cluster BNZ-CL-001 pod BNZ-POD-01 of availability zone BNZ
2022-08-03 00:01:53,298 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-166624fc) (logid:8e4ca7a5) Unallocated storage space is low, total: 62914488 MB, allocated: 50750692 MB (80.67%)
2022-08-03 00:01:53,298 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-166624fc) (logid:8e4ca7a5) alertType=[3] dataCenterId=[1] podId=[1] clusterId=[1] message=[System Alert: Remaining unallocated Storage is low in cluster BNZ-CL-001 pod BNZ-POD-01 of availability zone BNZ].
2022-08-03 00:01:54,255 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-166624fc) (logid:8e4ca7a5) Have already sent: 1 emails for alert type '3' -- skipping send email