Quarkus kafka streams: create multiple kafka streams application inside the same quarkus application connected to different kafka clusters

[vinstola]

Good morning,
I'm Vincenzo and I'm new in this forum.
I need you help for this topic.
By requirements, we must have an unique quarkus application that needs to define more kafka streams applications, each of them connected to a different kafka cluster (similar to this problem: https://stackoverflow.com/questions/45847690/how-to-connect-to-multiple-clusters-in-a-single-kafka-streams-application).
The question is: how can I setup this in Quarkus kafka stream application using the quarkus-kafka-streams / quarkus-smallrye-reactive-messaging-kafka libraries (view example https://quarkus.io/guides/kafka-streams)?

Viewing the libraries source code, it seems impossible.

Can you help me?
Regards,
Vincenzo

[quarkus-bot[bot]]

/cc @alesj (kafka,kafka-streams), @cescoffier (kafka), @gunnarmorling (kafka-streams), @ozangunalp (kafka,kafka-streams), @rquinio (kafka-streams)

[rquinio1A]

This is not possible with quarkus-kafka-streams, as it only manages a single KafkaStreams instance, which is limited to consuming and producing to the same Kafka cluster (limitation of KafkaStreams library).
It seems possible with quarkus-smallrye-reactive-messaging-kafka library: smallrye/smallrye-reactive-messaging#897

Another solution would be to split the concerns:

• replicate the needed topics between Kafka clusters using a solution like MirrorMaker2 - it should never fail
• only work within one cluster for a given microservice (and deal with failures/poisonous messages in business logic)

[vinstola]

@rquinio1A Thanks for the answer. I imagined it but I hoped that in quarkus-kafka-streams library roadmap there were the adding of this feature (manage more kafkaStreams instance instead of 1).

We also evaluated the topics replication but we are analyzing also other solutions.