Alert without annotation "description" doesn't send to discord without throwing errors

[thanatchakromsang]

I found this problem while setting up alertmanager-discord in my cluster and tested that It doesn't send to discord and silent leading to investigation consider following Aleart rules

groups:
- name: customized.container.rules
  rules:
  - alert: containerRestarted
    annotations:
      runbook_url: ''
      summary: "More than 2 restarts in a pod {{ $labels.pod }}"
    expr: delta(kube_pod_container_status_restarts_total[5m]) > 2
    for: 5m
    labels:
      severity: critical

It does trigger discord webhook with annotation description not empty though

groups:
- name: customized.container.rules
  rules:
  - alert: containerRestarted
    annotations:
      runbook_url: ''
      summary: "More than 2 restarts in a pod {{ $labels.pod }}"
      description: "{{ $labels.container }} restarted (current value: {{ $value }}s) times in pod {{ $labels.namespace }}/{{ $labels.pod }}"
    expr: delta(kube_pod_container_status_restarts_total[5m]) > 2
    for: 5m
    labels:
      severity: critical

It doesn't have description then when the rules triggered alertmanager firing to webhooks but It does nothing

consider this example alert output from alertmanager to alertmanager-discord. This example doesn't send to discord because It lack annotation description but It does send after alert rules to send description though

{
  "receiver": "webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "Test",
        "dc": "eu-west-1",
        "instance": "localhost:9090",
        "job": "prometheus24"
      },
      "annotations": {
        "summary": "hello",
+       "description": "test"
      },
      "startsAt": "2018-08-03T09:52:26.739266876+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://simon-laptop:9090/graph?g0.expr=go_memstats_alloc_bytes+%3E+0\u0026g0.tab=1"                                                                                  
    }
  ],
  "groupLabels": {
    "alertname": "Test",
    "job": "prometheus24"
  },
  "commonLabels": {
    "alertname": "Test",
    "dc": "eu-west-1",
    "instance": "localhost:9090",
    "job": "prometheus24"
  },
  "commonAnnotations": {
    "summary": "hello",
    "description": "some description"
  },
  "externalURL": "http://simon-laptop:9093",
  "version": "4",
  "groupKey": "{}:{alertname=\"Test\", job=\"prometheus24\"}"
}

My question is how should we fix this? Should we just thrown an error when no description or send empty string to discord instead?

Or should we just Update README.md to have a required annotation for alertmanager-discord?

[benjojo]

Hmm, how does the stock alertmanager handle this edge case for Slack?

[thanatchakromsang]

From https://prometheus.io/docs/alerting/latest/notification_examples/ alertmanager seems to use templating to handle how to send commonAnnotations such as summary and description through slack

Example.
Alert

groups:
- name: Instances
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    # Prometheus templates apply here in the annotation and label fields of the alert.
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'

Receivers

- name: 'team-x'
  slack_configs:
  - channel: '#alerts'
    # Alertmanager templates apply here.
    text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}" 

So I think It's a matter of using description and summary in the template or not

And by limitation of webhook_configs we can only send webhook alert with stricts JSON format following

{
  "version": "4",
  "groupKey": <string>,              // key identifying the group of alerts (e.g. to deduplicate)
  "truncatedAlerts": <int>,          // how many alerts have been truncated due to "max_alerts"
  "status": "<resolved|firing>",
  "receiver": <string>,
  "groupLabels": <object>,
  "commonLabels": <object>,
  "commonAnnotations": <object>,
  "externalURL": <string>,           // backlink to the Alertmanager.
  "alerts": [
    {
      "status": "<resolved|firing>",
      "labels": <object>,
      "annotations": <object>,
      "startsAt": "<rfc3339>",
      "endsAt": "<rfc3339>",
      "generatorURL": <string>       // identifies the entity that caused the alert
    },
    ...
  ]
}

So I don't think we can do anything more except create a precaution in README.md or handle description gracefully

[Smiche]

I don't even get any throws using the docker hub image. Had to read this issue and add the description field to get the hook to work. Should probably include it in the readme + log or just have an empty string in place when description is missing?

[oleksandr-semak]

Yeah faced with the same issue as it no error ether in alertmanager/prometheus or webhook just alert not displayed