RocketChat notifier - support for incoming webhook

[jrrdev]

Proposal

Currently, the RocketChat notifier works directly with the RocketChat REST API and is not compatible with an Incoming Webhook configuration on the RocketChat side.

The issue with this setup is that it requires the use of a personal API key tied to a specific account.
This can limit usage because non-administrator users can only create an API key on their personal account. In a company setting, this becomes a problem if the employee leaves the organization.

As it stands, the payload sent is identical whether using the REST API or an Incoming Webhook.
The problem is that the notifier appends /api/v1/chat.postMessage to the configured URL, which makes it impossible to use in Webhook mode. See :

alertmanager/notify/rocketchat/rocketchat.go

Line 221 in e542c68

url := n.conf.APIURL.JoinPath("api/v1/chat.postMessage").String()

It would be desirable to have a configuration option to enable an Incoming Webhook mode.
In this case:

• Do not alter the configured URL
• Make the fields token, token_file, token_id, token_id_file optional

[TheMeier]

@jrrdev you can use the native webhook notifier with rocketchat and a simple script on the servers side. See https://github.com/puzzle/prometheus-rocket-chat. The authentication is then ussually done in the path of webhook URL.

This is a well tested solution which we did use for years before I added the API notifier.

[jrrdev]

I was indeed aware of this solution, but it was ruled out for several reasons.

We are currently migrating from Slack to RocketChat and we already have AlertManager templates to handle our various messages. It will be a little bit cumbersome to adapt into a RocketChat server-side WebHook script.
Moreover, such a script would sit outside the codebase of our alerting stack, which could make maintenance more difficult.

We also have some concerns that if something goes wrong with the RocketChat server-side script, it may be more difficult to debug. And also that on unexpected conditions, this may prevent AlertManager from knowing that the notification wasn’t delivered correctly and so, not trigger our alert on alertmanager_notifications_failed_total (which is routed to another delivery system).

In general, we want to have as few points of failure as possible in our alerting system and keep it simple and stupid.

I’m willing to work on a PR to improve the RocketChat notifier and, for example, add a mode field to rocketchat_config, which could be set to either rest_api (default value) or incoming_webhook.
Does that sound like a good idea to you?

[TheMeier]

The issue with this setup is that it requires the use of a personal API key tied to a specific account.

The usual way to go about that is to create a ServiceAccount (aka bot user) with at static API token and then invite the bot for each channel it should have access to.