Activate OEC alarms, send message to Mattermost

[pcrespov]

As a developer I think that a useful alarm would be something like the following:

• I would like to have a repository of issues for each OEC we detect (only the unique part)
• each time unique part of the OEC is created a new issue is added
• when an error with the same unique OEC part presents itself it comments on the issue with a traceback reference and the deployment where this was generated

This would be super helpful for tracking down issues and figuring out which of them occurred recently and which of them occurred the most.

[GitHK]

@pcrespov a comment on my above edit (feel free to change)

Here, for sure I want the issues to only be generated from the unique part of the OEC, when you comment on it you can include the entire OEC

[mrnicegyu11]

@pcrespov I try to address this in ITISFoundation/osparc-ops-environments#1005

[pcrespov]

I think the changes we did are good enough. I do not think we need to have issues ... having the info in graylog we can retrieve the data i needed.thx

[pcrespov]

I would like the alarms to have more information. Thi sis the current message

The message in the title (or somewhere else) should have more info:

• A permanent link to the error in graylog
• Optionally the OEC (e.g. then I can distinguish whether this is new or was already notified)
• Optionally any of the log_XX fields that we are extrating

I wonder if you can use the "event fields" for that.

[pcrespov]

@mrnicegyu11 Graylog (master) alos shows hundreds of errors that seems related to the notification setup