Retries in notification-publisher can cause duplicate notifications to be sent

[nscuro]

notification-publisher consumes Notifications from dtrack.notification.* topics. Notification is the unit of work for it.

It then checks which NotificationRules are applicable to the Notification at hand. Multiple NotificationRules can apply to one Notification.

Each NotificationRule defines a NotificationPublisher that shall be used to deliver the Notification.

This means that, to process one Notification, notification-publisher has to potentially perform N calls to external services, each of which may fail (due to transient, or non-transient errors).

Since switching from Kafka Streams to Confluent's Parallel Consumer (#586), processing failures can be retried.

---

Assuming a Notification is covered by two NotificationRules, both referring to WebhookPublisher.

When processing the Notification, publishing for the first NotificationRule succeeds, but the second one experiences a timeout. If we were to retry the processing of the Notification, the first NotificationRule will receive duplicates until publishing succeeds for the second rule.

[nscuro]

Not pursuing this anymore as a consequence of #1619. The way notifications are being published will become more granular, completely voiding this issue.