Parallel Consumer with KStream StateStore

[milansanjeev]

I am using the parallel consumer in one of our projects and found it reduces leg effectively and process record in less time.
https://github.com/confluentinc/parallel-consumer/commits?author=astubbs

We also had a discussion around this
#326

I have one question can we parallel consumer with StateStore as we want to aggregate data in state store and due to the null processor context I am unable to store data in the state store, Do you have any working examples of it?

I am using KStream in which aggreagte data in stateStore and running punctuator after every minute to store data in DB.
I have 168 partitions in the stream so want to use a parallel consumer.

@astubbs

[astubbs]

so you are applying aggregate functions in ks, data ending in a state store, and you want to process all the state store data using pc, and write it to a db?
sure- have a look at the KS section in the docs. But basically, just output the state store to a topic, and then use pc to read from the topic.
https://github.com/confluentinc/parallel-consumer#streams-usage-code

Have a look at the streams example project in the repo.

    builder.<String, String>stream(inputTopic)
            .mapValues((key, value) -> {
                log.info("Streams preprocessing key: {} value: {}", key, value);
                return String.valueOf(value.length());
            })
            .groupByKey()
            .aggregate(() -> init(),(key, value, aggregate) -> aggregate(key,value,aggregate))
            .toStream()
            .to(outputTopicName);