Proposal: Embedded TSDB

[siavashs]

Proposal

Background

Alertmanager has multiple internal storages:

• Alerts stores
• Alerts provider
• Markers
• Silences
• ...

There are multiple API endpoints which expose the above (raw or mutated):

• /alerts
• /alerts/groups
• /silences
• /metrics
• ...

The above APIs have the following issues:

• process intensive
• causing contention, blocking Alertmanager from doing its job
• only show the current state (no history)
  • in case of /alerts/groups we predict which alerts will be suppressed with which silences in future!

Proposal

Prometheus already provides the ALERTS timeseries which provides both current and historic information about the status of alerts.
My proposal is to embed a TSDB into Alertmanager so it can also provide an ALERTS timeseries with a compatible Prometheus API.
Alertmanager will have at least one extra label value for alertstate which would be suppressed or muted, plus more extra labels like muted_why and muted_by, etc.

This has multiple benefits:

• it provides both current and historic status of all alerts
  • it removes the need to hammer Alertmanager API to capture and record such data
  • ALERTS timeseries from Prometheus and Alertmanager can be joined in a PromQL query for analysis
• TSDB has already a familiar interface for users of Prometheus which is PromQL
• This can be extended to more things like silences, notifications, etc.
• each cluster member can expose its own ALERTS timeseries including a member label which would allow to detect inconsistencies across the cluster

There is an existing example of TSDB embedding in Thanos Ruler which exposes a similar ALERTS timeseries data through its embedded TSDB.

We could optionally:

• support remote_write
• write the TSDB to disk, which would allow Thanos Sidecar to upload this data to longterm storage

Risks

The only risks I see so far is high cardinality of ALERTS, specially in setups with 100s or 1000s of Prometheus instances, the cardinality on ALERTS timeseries can be high.
Alertmanager can simply be protected from high cardinality by applying limits per alertname (similar to how Prometheus does it per scrape job, etc.)

[Spaceman1701]

One thing we do is export an alerts metrics from alertmanager that looks like ALERTS but has receivers, group_ids, and muted state.

We never thought about embedding TSDB, but it's an interesting idea...

[siavashs]

#4816 is a prototype implementation which currently records:

• ALERTS
• ALERTS_MARKED: better potential naming FINGERPRINTS_MARKED
• ALERTS_INHIBITED: same as above
• ALERTS_SILENCED same as above

We can also add the following later:

• INHIBIT_RULES
• SILENCES
• AGGREGATION_GROUPS
• RECEIVERS
• NOTIFICATIONS_SENT