[Question] dynamically registering new topics

[dams]

Hi, I haven't found in the doc and at a first glance of the code if it was possible and how easy, to register to new topics dynamically, without having to stop the app, change the config file and restart. I'd like to be able to instruct an application to now consume an additional given topic. Also, be able to stop consuming a topic as well.

Thanks for your help

[objectuser]

Kaffe doesn't support dynamic configuration. However, there was some discussions around using a library called Weave for that. I've not used it myself, but we did have a PR on it some time ago.

[dams]

thanks. I'm going to hack kaffe to make it dynamic, as I have a pool of boxes that I need to consume various topics at different time. I'm going to use Swarm for the distributed aspect of it. I think it'll be easy-ish to make kaffe dynamic. I'll let you know here. Feel free to close

[dams]

see #58

[davidsantoso]

@dams thanks for the pings in the dynamic topic subscribe/unsubscribe PRs. Sorry for the delay, we've been busy here at Spreedly the last few months and are just starting to look back at continuing development on Kaffe.

We were chatting about this internally and thought we'd get a little more details. We're curious about use cases in which systems are only occasionally consuming a topic. Could be that this is a domain specific situation, of if there are other factors that make this a more common scenario and we're just not experiencing it within our system.

I do think it's worth nothing that this particular behavior could also be managed on the message handler side of things by ignoring messages from a particular topic if certain criteria are true (such as the time of day, etc). Of course the criteria would need to be static and maybe your particular case won't allow it to be handled in your message handler code?

[dams]

It’s not a domain specific situation. To me it is mandatory that à Kafka client allows this. The official clients in other languages do provide this.

My need is very common and simple: provide a feature ( here an aggregation over time ) of data transported over Kafka, * as a service *. In the current world, things are service oriented, as a service, with zero downtime.

My solution will run 24/7 with hot code reloading. It’ll run for the next years without being restarted. So it needs to be able to adapt to the change in flow if data, namely new topics, new partitions, etc.

Your workaround works only if all the topics exist at start time, which is not the case.

This feature is mandatory imho to be able to provide a strong kafka client.

Also, other PRs by other authors are important too, like the one that allows to have more than one consumer group

[davidsantoso]

That makes sense. I think since we're operating slightly differently at the moment so it wasn't something that had quite caught our attention. But I can imagine sometime in the future we'd make use of dynamic topics. Just not right now. I'll give your PR another look next week and see if any changes need to happen before merging.

Thanks!