Partition Consumer API

[Sal Sorrentino (ssorren)]

I'm currently writing a Kafka lib using the Go wrapper for this project. Part of this includes writing a state store, similar to the official Kafka Streams API. I have the need to consume a a single compacted partition from "beginning" to EOF on bootstrap of the state store. This can be done with the current API, but you are forced into a consumer group approach. In this use case, the consumer never has the need to commit offsets and the partition will never be re-balanced. Right now I'm creating a unique "group.id" per host and just flipping auto-commit to false and manually doing an incremental assign for the desired topic/partition. This seems a bit wasteful as we are creating consumer group metadata that is not needed. It would be helpful to have a PartitionConsumer that would not carry the extra weight of a ConsumerGroup.