Dynamic configuration for Reactive Messaging Kafka

[nithinbandaru1]

Hi,

Can you guys please point me to a blog or reference document to configure Reactive Message Kafka dynamically.

Basically, topic names will come from HTTP request and then I would like to create reactive messaging queue and connect it with Kafka topic using Kafka connector and also clear dispose this setup on other HTTP request.

It can be both Topic reading and also top writing.

In reference documents I can see that we can make static configurations using applicaiton.properties.

Is there any API I can use in Quarkus to configure this setup dynamically. I am expecting now they would be part of startup code in kafka reactive extension. Just wondering if it is publicly accessible.

I just feel that there are lot of use cases to for this type of setup.

[quarkus-bot[bot]]

/cc @alesj (kafka), @cescoffier (kafka,reactive-messaging), @ozangunalp (kafka,reactive-messaging)

[cescoffier]

For the writing part, you can use Kafka Outgoing Metadata - https://smallrye.io/smallrye-reactive-messaging/latest/kafka/writing-kafka-records/#outbound-metadata.

Unfortunately, for the reading part, you will need to use the low-level Kafka client (see https://github.com/quarkusio/quarkus-quickstarts/tree/main/kafka-bare-quickstart for an example)

[nithinbandaru1]

I just came across this today.

https://stackoverflow.com/questions/76663303/creating-kafka-consumers-dynamically-without-incoming-annotation

Just adding for reference.

One scenario I can see is:

• Let say there is some clients connected to cluster-based server application using web sockets or tcp or something similar.
• And we need to subscribe to Kafka topics and partitions based on connected client like their identifier.
• As server application is part of cluster, client can disconnect from one instance and connect to different instance.
• Now ideally server application should decide which topics and partitions need to be consumed and which need to be stopped from consuming.
• This totally makes sense to me for cloud based scalable applications.

I think this is because Quarkus want to use declarative approach mostly like using app settings file. But few app settings can be changed dynamically. So, similarly it should be possible to dynamically change/add Kafka settings, also I guess there is no build time code generations for this.

Hope an interface for such use case will be introduced in future. For now, I will try using bare Kafka client.

[cescoffier]

The interface exists already: use the regular Kafka client for this.

See https://github.com/quarkusio/quarkus-quickstarts/tree/main/kafka-bare-quickstart as an example.