View and manage consumer groups and offsets

[rhauch]

Users

• Development manager tasked with ensuring a team is successful in using streaming data
• Developer on a team tasked with building new features using streaming data
• Developer at an org who manages a streaming platform while building new features that use streaming data
• Developer tasked with building pipelines for use in business analytics
• Other: [User type] tasked with [primary goal in the realm of streaming data]

Problems

When developing streaming application, a developer may:

1. want to check whether the consumers in their application are properly forming groups
2. want to check whether the consumers in their application are committing offsets and/or making progress
3. want to modify committed offsets so so that when new consumers join the group (e.g., when an application is restarted) the consumers will begin consuming from the desired offset(s).

Solutions

When developing streaming application, a developer may:

1. want to view the consumer groups and information about the consumers in the group (e.g., consumer IDs, consumer lags, partition assignments, etc.)
2. view the committed offsets for each/specific consumers in a group
3. reset/change/remove the committed offset(s) of a consumer group on particular topic partition(s)

Goals

1. View and manage consumer groups and committed offsets
2. Provide guardrails to prevent the user from accidentally changing group or offset information

Non-Goals

No response

Background

Kafka consumers can form consumer groups to coordinate the responsibilities of consuming messages from multiple topic partitions. When consumers join a group, the consumer group coordinator assigns topic partitions to specific consumers. As each consumer in the group consume messages, the consumers record their progress by periodically committing the offsets of the consumed messages. Should a consumer leave the group, the consumer group coordinator may reassign that failed consumer's topic partition assignments to other consumers, which read the committed offsets for the newly-assigned topic partitions to know where to begin consuming.

References:

• https://docs.confluent.io/platform/current/clients/consumer.html
• https://developer.confluent.io/learn-more/kafka-on-the-go/consumer-groups
• https://docs.confluent.io/kafka/operations-tools/manage-consumer-groups.html
• https://docs.confluent.io/platform/current/control-center/consumers.html
• https://docs.confluent.io/cloud/current/client-apps/consumer.html
• https://docs.confluent.io/confluent-cli/current/command-reference/kafka/consumer/group/confluent_kafka_consumer_group_list.html
• https://docs.confluent.io/cloud/current/api.html#tag/Consumer-Group-(v3)

Proposal

Add support to allow developers to view and manage consumer groups, consumers, committed offsets, consumer lag, and other information.

[RichardKaufmannOG]

I would also love to see such a feature in the VSCode plugin. I have used it in another IDE, and it was very handy there.