How to correctly modify AlertManager config file

[dnaranjor]

In my cluster, I have leveraged the AlertmanagerConfig custom resource https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#alertmanagerconfig, which in my understanding, help us change the currently deployed Alertmanager config files at will.

So, to configure my Sendgrid account to deal with desired email integration (a.k.a. Alertmanager sending e-mail alert notifications), I have created a file named AlertManagerConfigmap.yaml with the following:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: 'email'
email_configs:
- to: '[email protected]'
from: '[email protected]'
smarthost: smtp.sendgrid.net:587
auth_username: 'apikey'
auth_password: 'XXXXXXXXX'
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 10s
receiver: 'email'

Hoping this configuration will add to the existing one after applying it to the cluster.

Now, when I try to apply it, I got this error:

$ kubectl apply -f monitoring-alertmanager-configmap.yaml

error: error validating "monitoring-alertmanager-configmap.yaml": error validating data: [ValidationError(AlertmanagerConfig.metadata): unknown field "route" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(AlertmanagerConfig.metadata): unknown field "spec" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(AlertmanagerConfig): missing required field "spec" in com.coreos.monitoring.v1alpha1.AlertmanagerConfig]; if you choose to ignore these errors, turn validation off with --validate=false

Someone could guide me in this?

[dnaranjor]

Nobody has seen this request so far... Wondering if @paulfantom could take a look at this?

[paulfantom]

If you want to modify the whole file and do not use the tenancy model associated with AlertmanagerConfig, then just modify the alertmanager Secret as described in https://github.com/prometheus-operator/kube-prometheus#alertmanager-configuration. Example manifest for that file is in https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/alertmanager-secret.yaml

If you wish to use the tenancy model, then you need to follow AlertmanagerConfigSpec.

However considering your config, it looks like you need to modify the Secret instead of using AlertmanagerConfig.

[dnaranjor]

Thanks for the guidance, @paulfantom ...

I'm adding the Sendgrid receiver and route rules into the https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/alertmanager-secret.yaml file, but I'm wondering... Where should I add the spec.receivers data required to find the Sendgrid credentials?

[dnaranjor]

Well, following @paulfantom suggestion, I've pulled and decrypted the cluster Alertmanager secret using:

kubectl get secret alertmanager-main -o go-template='{{ index .data "alertmanager.yaml" }}' | base64 -d

Then, using the pulled config file, I have modify it to include my Sendgrid information, and the final result is this:

apiVersion: v1
kind: Secret
metadata:
labels:
alertmanager: main
app.kubernetes.io/component: alert-router
app.kubernetes.io/name: alertmanager
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.22.2
name: alertmanager-main
namespace: monitoring
stringData:
alertmanager.yaml: |-
"global":
"resolve_timeout": "5m"
"receivers":
- "name": "email"
"email_configs":
- "to": "[email protected]"
"from": "[email protected]"
"smarthost": "smtp.sendgrid.net:587"
"auth_username": "apikey"
"auth_password": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
"inhibit_rules":
- "equal":
- "namespace"
- "alertname"
"source_match":
"severity": "critical"
"target_match_re":
"severity": "warning|info"
- "equal":
- "namespace"
- "alertname"
"source_match":
"severity": "warning"
"target_match_re":
"severity": "info"
"receivers":
- "name": "Default"
- "name": "Watchdog"
- "name": "Critical"
- "name": "email"
"route":
"group_by":
- "namespace"
"group_interval": "5m"
"group_wait": "30s"
"receiver": "Default"
"repeat_interval": "12h"
"routes":
- "match":
"alertname": "Watchdog"
"receiver": "Watchdog"
- "match":
"severity": "critical"
"receiver": "Critical"
- "match":
"severity": "email"
"receiver": "email"
type: Opaque

Finally, applied this new secret into the cluster, doing a:

kubectl appy -f modified_config_file.yaml -n monitoring

And checked changes were performed correctly.

To wrap it up, I have restarted all alertmanager running Pods, but when I check in Alertmanager GUI if the changes are there, it remains the same, no changes at all... :-(

[paulfantom]

The Secret is picked up by prometheus-operator, which runs validations and creates another Secret (the *-generated one) which is then used by alertmanager pods. Check if both secrets are similar and if there is anything meaningful in prometheus-operator logs.

[dnaranjor]

Figured what the actual problem was using Alertmanager log output: a misplaced semicolon, preventing the manifest to be correctly processed.

Now fixed, and ConfigMap correctly ingested into all Alertmanager nodes.

[ahmad-mari]

@dnaranjor

i copied the exact file posted above and didnt work, can i have an updated one pls with gmail smtp?