Some strategies to build reliable webhook push server?

[javadevmtl]

Hi, I know this a very general question but just trying to gather some thoughts/ideas...

I'm building a webhook push server...

1. Filter phase

• Messages coming on Kafka.
• I filter them out and write them into separate topic (100 partitions) keyed by domain.
• That consumer has a DLQ and if there's any failure the events get DLQ for possible retry later.

2. Push phase

• This is where things get tricky.
• At least once delivery is ok.
• Ordering doesn't really matter.
• The consumer group id is configured with 100 consumer threads 1 thread per partition. So like this each domain get's it's own consumer.
• Read a batch of events of the partition and send them to the webhook one by one.
  • So what happens here out of the batch of events 2-3 fails, do we nack the whole batch and retry it again or do we ack the whole batch and emit individual failed events into separate topic for later retry?
  • Maybe marked the failed message in some sort of cache nack the whole batch and DLQ it. When it comes back for retry skip all the acked messages and only reprocessthat where successfull.
  • Should we forgo the whole reading as batch and just process each event one by one? This make Dlq retry easier. But less performant.

Any other crazy ideas? :)