Consumer behavior when topic doesn't exist on broker

[cmessick]

Would it be possible to emulate the behavior of confluent-kafka when starting a consumer for a topic that doesn't yet exist on the broker, specifically it would be helpful if a warning was emitted without crashing the consumer instead of an error.

The reason I ask is I'm working on setting up a consumer for a topic on GCN Kafka, and GCN Kafka uses the workflow model that topics are created when a producer publishes to it. We can possibly ask the people responsible for the producer to send something (anything) to the topic to create it, but it'd be nice to not have to do that.

[cnweaver]

This seems reasonable to me. I'll have to see if it's possible for the client to detect whether the broker has that topic auto-creation feature turned on, since if we can do that 1) we should definitely not emit an error if we see that active, and 2) we should perhaps not even emit a warning if we see it (although something like an INFO level message that a new topic will be started could still be helpful).

EDIT: I partially misread the first time, and I now see that this is about the consumer, rather than the producer. That may be more complicated, since I'm not sure how subscribing to a topic which does not currently exist, but may come into being at any time, works. Presumably the broker issues an updated partition assignment when the topic is created for any active consumers which had previously requested it? I think I'll need to double-check that our code will handle that sensibly, but in general your request still seems desirable, and hopefully doable.

[cmessick]

That would be awesome.