Turn alarms on and off on a schedule

[AnthonySteele]

It is frequently requested that we add the feature that the alerting group or an item in it contains schedule when the alarm is active, or stop alerting during "quiet hours".

A run would have to look at this schedule and decide to Disable or re-enable relevant Cloudwatch alarms.

Discussion of pros and cons goes here.

I'm not confident that turning alarms off is a good idea. Would prefer to make an alarm that works 24/7 (i.e. goes of for the right reasons but no false alarms). We have looked into several over-sensitive alarms and usually we identify a current, new or requested AWS feature that we need to fix them - e.g.

• When alerting on lambdas , use p90 rather than max of execution time,
• When we alert on several consecutive alerting periods over a limit, we may want to regard "no data" as "zero" not "same as previous point".

[tomhaigh]

Another option would be to send these alarms via a lambda (and a secondary sns topic). At the moment we have this:

Cloudwatch -> Alarm SNS topic -> Target (e.g. Pagerduty URL)

For alarms which don't need to be active all the time we could have this:

Cloudwatch -> New intermediate SNS topic -> Lambda -> Alarm SNS topic -> Target (e.g. Pagerduty URL)

The lambda would need to know about the alarm schedule and drop any messages during the times the alarm is scheduled to be off. At other times it would just pass it on to the usual SNS topic that we use to notify the Email/URL targets.

One way that the lambda could read the alarm schedule is that AWS watchman could publish it to an s3 bucket, and the lambda could read it from there - would just need to decide on a filename format that could be read out of the SNS message when the alarm goes off.

The benefit of this approach is that alarms do not need to be modified on a schedule, which has risk (e.g. if the alarm reactivation fails). Obviously the lambda would have other risks though,