KAFKA-19977: Documentation section on the streams rebalance protocol #21170

lucasbru · 2025-12-18T09:52:41Z

Add new developer guide page documenting the broker-driven Streams
Rebalance Protocol, including features, configuration, administration,
and architecture. Update navigation links across developer guide pages
to integrate the new section.

Some of the pagination links didn't look right. Not sure if it matters
since we anyway are replacing this representation

Reviewers: Matthias J. Sax [email protected]

Copilot

Pull request overview

This PR adds comprehensive documentation for the Streams Rebalance Protocol (KIP-1071), a broker-driven rebalancing system for Kafka Streams applications introduced in Apache Kafka 4.2. The documentation covers features, configuration, administration, and architecture details.

Key changes:

New developer guide page documenting the Streams Rebalance Protocol with detailed sections on configuration, administration, architecture, and migration
Updated pagination links across multiple developer guide pages to properly integrate the new documentation page into the navigation flow

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
docs/streams/developer-guide/streams-rebalance-protocol.html	New comprehensive documentation page covering the Streams Rebalance Protocol, including overview, configuration options, administration tools, architecture details, and migration guidance
docs/streams/developer-guide/processor-api.html	Updated pagination to point "Next" link to dsl-topology-naming instead of datatypes, maintaining proper navigation flow
docs/streams/developer-guide/dsl-topology-naming.html	Added pagination block to connect to processor-api (previous) and datatypes (next)
docs/streams/developer-guide/datatypes.html	Updated pagination to point "Previous" link to dsl-topology-naming instead of processor-api
docs/streams/developer-guide/app-reset-tool.html	Updated pagination to point "Next" link to streams-rebalance-protocol instead of upgrade-guide
docs/streams/developer-guide/kafka-streams-group-sh.html	Added pagination block to connect to streams-rebalance-protocol (previous) and upgrade-guide (next)
docs/streams/developer-guide/index.html	Added link to the new streams-rebalance-protocol page in the table of contents
docs/documentation/streams/developer-guide/streams-rebalance-protocol.html	Redirect file that includes the main streams-rebalance-protocol.html to support version-agnostic documentation URLs

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-18T09:56:04Z

docs/streams/developer-guide/streams-rebalance-protocol.html

+
+        <ul class="simple">
+          <li>Check that all source topics exist and resolve source topic regular expressions (checking that each resolves to at least one topic).</li>
+          <li>Check that "copartition groups" are satisfied - that is, all source topics that are supposed to be copartitioned are indeed copartitioned.</li>


The term should be "co-partitioned" (with hyphen) to be consistent with the rest of the Kafka Streams documentation. The codebase consistently uses "co-partitioned" throughout other documentation files.

Copilot · 2025-12-18T09:56:04Z

docs/streams/developer-guide/streams-rebalance-protocol.html

+        <h3><a class="toc-backref" href="#id13">Topology Configuration and Validation</a><a class="headerlink" href="#topology-configuration" title="Permalink to this headline"></a></h3>
+        <p>To assign tasks among streams clients, the group coordinator uses topology metadata that is initialized when a member joins the group and persisted in the consumer offsets topic.</p>
+
+        <p>Whenever a member joins the streams group, the first heartbeat request contains metadata of the topology. The metadata describes the topology as a set of subtopologies, each identified by a unique string identifier and containing metadata relevant for creation of internal topics and assignment.</p>


The term should be "sub-topologies" (with hyphen) to be consistent with the rest of the Kafka Streams documentation. While "subtopology" (without hyphen) is used in code contexts like method names, in documentation text describing the topology structure, the hyphenated form "sub-topology/sub-topologies" is consistently used (see docs/streams/tutorial.html:504-505, docs/streams/developer-guide/dsl-topology-naming.html).

lucasbru · 2025-12-18T12:31:43Z

localhost_8080_dev_documentation_streams_developer-guide_streams-rebalance-protocol html

mumrah · 2025-12-22T01:15:39Z

Hey @lucasbru, we just merged the HTML to Markdown changes. You'll need to re-do these changes with the new system. Sorry for the inconvenience!

lucasbru · 2026-01-05T14:08:43Z

Updated to markdown, @mjsax .

localhost_1313_43_streams_developer-guide_streams-rebalance-protocol_

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/streams/developer-guide/streams-rebalance-protocol.md

mjsax · 2026-01-09T20:10:19Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+
+* **Interactive Query Support**: IQ operations are compatible with the new streams protocol.
+
+* **New Admin RPC**: The StreamsGroupDescribe RPC provides streams-specific metadata separate from consumer group information, with corresponding access via the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html).


Suggested change

* **New Admin RPC**: The StreamsGroupDescribe RPC provides streams-specific metadata separate from consumer group information, with corresponding access via the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html).

* **New Admin RPC**: The StreamsGroupDescribe RPC provides streams-specific metadata separate from consumer group information, with corresponding access via the [`Admin`](/{version}/javadoc/org/apache/kafka/clients/admin/Admin.html).

mjsax · 2026-01-09T20:11:01Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+
+* **New Admin RPC**: The StreamsGroupDescribe RPC provides streams-specific metadata separate from consumer group information, with corresponding access via the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html).
+
+* **CLI Integration**: You can list, describe, and delete streams groups via the [kafka-streams-groups.sh](kafka-streams-group-sh.md) script.


Suggested change

* **CLI Integration**: You can list, describe, and delete streams groups via the [kafka-streams-groups.sh](kafka-streams-group-sh.md) script.

* **CLI Integration**: You can list, describe, and delete streams groups via the [bin/kafka-streams-groups.sh](kafka-streams-group-sh.md) script.

mjsax · 2026-01-09T20:13:15Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+
+* **CLI Integration**: You can list, describe, and delete streams groups via the [kafka-streams-groups.sh](kafka-streams-group-sh.md) script.
+
+* **Offline Migration**: After shutting down all members and waiting for their `session.timeout.ms` to expire, a classic group can be converted to a streams group and a streams group can be converted to a classic group. The only thing that will be preserved broker-side are the committed offsets of the application.


Why do we need to wait for session.timeout.ms? I guess it hold if we don't send a leave group request, what is the default, but it also possible to send leave group request explicitly now https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Refactor+Kafka+Streams+CloseOptions+to+Fluent+API+Style.

If you think we want to keep the instructions simple, I am also ok to leave it as is. Just wanted to ask.

The only thing that will be preserved broker-side are the committed offsets of the application.

Could this be misleading with regard to internal topics?

mjsax · 2026-01-09T20:15:35Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+
+* **Faster, More Stable Rebalances**: Reduces rebalance duration and impact by removing the global synchronization point. This minimizes application downtime during membership changes or failures.
+
+* **Better Observability**: Provides dedicated metrics and admin interfaces that separate streams from consumer groups, leading to clearer troubleshooting with broker-side observability. See the [streams groups metrics]({{< relref "/43/operations/monitoring#group-coordinator-monitoring" >}}) documentation for details.


Suggested change

* **Better Observability**: Provides dedicated metrics and admin interfaces that separate streams from consumer groups, leading to clearer troubleshooting with broker-side observability. See the [streams groups metrics]({{< relref "/43/operations/monitoring#group-coordinator-monitoring" >}}) documentation for details.

* **Better Observability**: Provides dedicated metrics and admin interfaces that separate streams from consumer groups, leading to clearer troubleshooting with broker-side observability. See the [streams groups metrics]({{< relref "/{version}/operations/monitoring#group-coordinator-monitoring" >}}) documentation for details.

mjsax · 2026-01-09T20:16:43Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+
+### Broker Configuration
+
+The protocol is enabled by default on new Apache Kafka 4.2 clusters. To enable the feature on existing clusters or to explicitly control it:


Suggested change

The protocol is enabled by default on new Apache Kafka 4.2 clusters. To enable the feature on existing clusters or to explicitly control it:

The protocol is enabled by default on new Apache Kafka 4.2 clusters. To enable the feature on existing clusters (after upgrading to 4.2) or to explicitly control it:

mjsax · 2026-01-09T20:27:36Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+
+### Admin API
+
+Use the "streams groups" methods of the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html) interface to manage streams groups programmatically. These APIs are mostly backed by the same implementations as the consumer group API.


Suggested change

Use the "streams groups" methods of the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html) interface to manage streams groups programmatically. These APIs are mostly backed by the same implementations as the consumer group API.

Use the "streams groups" methods of the [`Admin`](/{version}/javadoc/org/apache/kafka/clients/admin/Admin.html) interface to manage streams groups programmatically. These APIs are mostly backed by the same implementations as the consumer group API.

mjsax · 2026-01-09T20:28:43Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+
+### kafka-streams-groups.sh
+
+A new tool called [kafka-streams-groups.sh](kafka-streams-group-sh.md) is added for working with streams groups. It replaces `kafka-streams-application-reset` for streams groups and can be used to list, describe, and delete streams groups. See the [kafka-streams-groups.sh documentation](kafka-streams-group-sh.md) for detailed usage information.


Suggested change

A new tool called [kafka-streams-groups.sh](kafka-streams-group-sh.md) is added for working with streams groups. It replaces `kafka-streams-application-reset` for streams groups and can be used to list, describe, and delete streams groups. See the [kafka-streams-groups.sh documentation](kafka-streams-group-sh.md) for detailed usage information.

A new tool called [bin/kafka-streams-groups.sh](kafka-streams-group-sh.md) is added for working with streams groups. It replaces `bin/kafka-streams-application-reset.sh` for streams groups and can be used to list, describe, and delete streams groups. See the [bin/kafka-streams-groups.sh documentation](kafka-streams-group-sh.md) for detailed usage information.

mjsax · 2026-01-09T20:32:07Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+
+## Monitoring and Metrics
+
+The existing group metrics are extended to differentiate between streams groups and consumer groups and account for streams group states. For complete details, see the [streams groups metrics]({{< relref "/43/operations/monitoring#group-coordinator-monitoring" >}}) documentation.


Suggested change

The existing group metrics are extended to differentiate between streams groups and consumer groups and account for streams group states. For complete details, see the [streams groups metrics]({{< relref "/43/operations/monitoring#group-coordinator-monitoring" >}}) documentation.

The existing group metrics are extended to differentiate between streams groups and consumer groups and account for streams group states. For complete details, see the [streams groups metrics]({{< relref "/{version}/operations/monitoring#group-coordinator-monitoring" >}}) documentation.

mjsax · 2026-01-09T20:32:40Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+Currently, only offline migration is supported. To migrate a Kafka Streams application from the classic protocol to the streams rebalance protocol:
+
+1. Shut down all application instances.
+2. Wait for the `session.timeout.ms` to expire so the group becomes empty.


Similar to above.

mjsax · 2026-01-09T20:33:37Z

docs/streams/developer-guide/streams-rebalance-protocol.md

+3. Update the application configuration to set `group.protocol=streams`.
+4. Restart the application instances.
+
+The only broker-side data that will be preserved are the committed offsets of the application. All other group metadata will be recreated when the application starts with the new protocol.


As above. Misleading with regard to internal topics?

Add new developer guide page documenting the broker-driven Streams Rebalance Protocol, including features, configuration, administration, and architecture. Update navigation links across developer guide pages to integrate the new section.

Co-authored-by: Copilot <[email protected]>

fixes fixes

lucasbru · 2026-01-13T12:57:06Z

@mjsax all addressed

lucasbru · 2026-01-13T12:58:16Z

localhost_1313_43_streams_developer-guide_streams-rebalance-protocol_ (1)

mjsax

Overall LGTM. A few minor follow ups.

Also, we should add something to the top-level "notable changes" section in docs/getting-started/upgrade.md

mjsax · 2026-01-14T18:35:16Z

docs/streams/developer-guide/kafka-streams-group-sh.md

 type: docs
 description: 
-weight: 14
+weight: 15


For my own education: what is this and why does it get changed?

It's used to order the pages in TOCs and sidebars. I wanted the streams group tool appear below the streams rebalance protocol page.

docs/streams/developer-guide/streams-rebalance-protocol.md

Co-authored-by: Matthias J. Sax <[email protected]>

mjsax

Thanks for the PR. LGTM.

lucasbru · 2026-01-19T09:18:49Z

cherry-picked to 4.2

…21170) Add new developer guide page documenting the broker-driven Streams Rebalance Protocol, including features, configuration, administration, and architecture. Update navigation links across developer guide pages to integrate the new section. Some of the pagination links didn't look right. Not sure if it matters since we anyway are replacing this representation Reviewers: Matthias J. Sax <[email protected]>

lucasbru requested review from Copilot and mjsax December 18, 2025 09:52

github-actions bot added the docs label Dec 18, 2025

Copilot started reviewing on behalf of lucasbru December 18, 2025 09:53 View session

Copilot AI reviewed Dec 18, 2025

View reviewed changes

lucasbru force-pushed the streams_docs branch from 12c8315 to 08260e7 Compare January 5, 2026 13:56

lucasbru requested a review from Copilot January 5, 2026 15:00

Copilot started reviewing on behalf of lucasbru January 5, 2026 15:01 View session

Copilot AI reviewed Jan 5, 2026

View reviewed changes

mjsax added the kip Requires or implements a KIP label Jan 9, 2026

mjsax reviewed Jan 9, 2026

View reviewed changes

lucasbru and others added 3 commits January 13, 2026 13:37

Apply suggestions from code review

63eea04

Co-authored-by: Copilot <[email protected]>

address comments and update migration guide

cc2bcd5

fixes fixes

lucasbru force-pushed the streams_docs branch from 21b1af3 to cc2bcd5 Compare January 13, 2026 12:55

mjsax reviewed Jan 15, 2026

View reviewed changes

lucasbru and others added 2 commits January 15, 2026 11:31

Apply suggestions from code review

e11f8fb

Co-authored-by: Matthias J. Sax <[email protected]>

upgrade guide kafka

7add6c2

mjsax approved these changes Jan 16, 2026

View reviewed changes

mjsax added the streams label Jan 16, 2026

lucasbru merged commit c62942d into apache:trunk Jan 19, 2026
24 checks passed


		* Interactive Query Support: IQ operations are compatible with the new streams protocol.

		* New Admin RPC: The StreamsGroupDescribe RPC provides streams-specific metadata separate from consumer group information, with corresponding access via the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html).


		* New Admin RPC: The StreamsGroupDescribe RPC provides streams-specific metadata separate from consumer group information, with corresponding access via the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html).

		* CLI Integration: You can list, describe, and delete streams groups via the [kafka-streams-groups.sh](kafka-streams-group-sh.md) script.


		* CLI Integration: You can list, describe, and delete streams groups via the [kafka-streams-groups.sh](kafka-streams-group-sh.md) script.

		* Offline Migration: After shutting down all members and waiting for their `session.timeout.ms` to expire, a classic group can be converted to a streams group and a streams group can be converted to a classic group. The only thing that will be preserved broker-side are the committed offsets of the application.


		* Faster, More Stable Rebalances: Reduces rebalance duration and impact by removing the global synchronization point. This minimizes application downtime during membership changes or failures.

		* Better Observability: Provides dedicated metrics and admin interfaces that separate streams from consumer groups, leading to clearer troubleshooting with broker-side observability. See the [streams groups metrics]({{< relref "/43/operations/monitoring#group-coordinator-monitoring" >}}) documentation for details.


		### Broker Configuration

		The protocol is enabled by default on new Apache Kafka 4.2 clusters. To enable the feature on existing clusters or to explicitly control it:


		### Admin API

		Use the "streams groups" methods of the [`Admin`](/43/javadoc/org/apache/kafka/clients/admin/Admin.html) interface to manage streams groups programmatically. These APIs are mostly backed by the same implementations as the consumer group API.


		### kafka-streams-groups.sh

		A new tool called [kafka-streams-groups.sh](kafka-streams-group-sh.md) is added for working with streams groups. It replaces `kafka-streams-application-reset` for streams groups and can be used to list, describe, and delete streams groups. See the [kafka-streams-groups.sh documentation](kafka-streams-group-sh.md) for detailed usage information.

	A new tool called [kafka-streams-groups.sh](kafka-streams-group-sh.md) is added for working with streams groups. It replaces `kafka-streams-application-reset` for streams groups and can be used to list, describe, and delete streams groups. See the [kafka-streams-groups.sh documentation](kafka-streams-group-sh.md) for detailed usage information.
	A new tool called [bin/kafka-streams-groups.sh](kafka-streams-group-sh.md) is added for working with streams groups. It replaces `bin/kafka-streams-application-reset.sh` for streams groups and can be used to list, describe, and delete streams groups. See the [bin/kafka-streams-groups.sh documentation](kafka-streams-group-sh.md) for detailed usage information.


		## Monitoring and Metrics

		The existing group metrics are extended to differentiate between streams groups and consumer groups and account for streams group states. For complete details, see the [streams groups metrics]({{< relref "/43/operations/monitoring#group-coordinator-monitoring" >}}) documentation.

KAFKA-19977: Documentation section on the streams rebalance protocol #21170

KAFKA-19977: Documentation section on the streams rebalance protocol #21170

Conversation

lucasbru commented Dec 18, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

lucasbru commented Dec 18, 2025

Uh oh!

mumrah commented Dec 22, 2025

Uh oh!

lucasbru commented Jan 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucasbru commented Jan 13, 2026

Uh oh!

lucasbru commented Jan 13, 2026

Uh oh!

mjsax left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mjsax left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lucasbru commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

lucasbru commented Dec 18, 2025 •

edited by github-actions bot

Loading