Idea - Event source / outputs : message queues

[xkilian]

Since we are dealing with events. Integration with message queues seems a pretty natural things. For example receive events from AMQP 0.9.1 (RabbitMQ), STOMP (ActiveMQ) and/or ZeroMQ.

The same applies when sending notifications, these could be sent to a message queue which would be responsible in transforming it into the appropriate medium (email, rss, pagerduty, sms, etc.).

Cheers,

xkilian

[Dieterbe]

I'm thinking this seems like a good scope to integrate with logstash. i.e. you could do all that because logstash supports the input plugins, and it can emit to elasticsearch.

but:

• in anthracite, there can be different schemas with different validation rules. all the events that show up in anthracite should be in a schema supported by anthracite, and validated.
• one can run multiple ES servers/indices/... to separate the logs from logstash from the events in anthracite. though it may be better to have them in the same place (though anthracite events are more "important", and should be stored more resiliently), but the ability to have them together and use tools like kibana on both types of events at the same time seems very useful. also, having log events from logstash and adding tags to them that make them into anthracite events (for example outage keys) (without having to make a new event in anthracite) seems useful. that said, the events that show up in the anthracite event UI should only be "real events". we could probably do this by adding a field that marks the ES documents as such. (i.e. every time an event gets added through anthracite, or through logstash but with the purpose of having it show up in anthracite, it would have an extra field marking it as such. but that would still leave the issue of validation, and some concerns about having a small amount of critical events among a vast multitude of somewhat-volatile log events)

I wonder what @jordansissel thinks about this

[jordansissel]

Sounds good to me! I think I'm missing some context, but overall this sounds like the right strategy in terms of not having to duplicate efforts.

As for having kibana make this data visible, I think having anthracite events sent to the 'logstash' index as well as the anthracite index separately is fine (duplicating those evnets in both places). Kibana may be able to search multiple indexes, so this duplication may not matter (as long as the schemas are roughly the same on both indexes)

As for giving anthracite data more resilience you can tell elasticsearch to store more replicas of this data. For disaster recovery, with logstash, you could ship anthracite events off to an archival source (like a file on disk, or a secondary service). There's an S3 output coming soon that can help that as well - all depends on what your resiliency/DR strategies are.