KAFKA-19977: Documentation section on the streams rebalance protocol (#21170)

Add new developer guide page documenting the broker-driven Streams
Rebalance Protocol, including features, configuration, administration,
and architecture. Update navigation links across developer guide pages
to integrate the new section.

Some of the pagination links didn't look right. Not sure if it matters
since we anyway are replacing this representation

Reviewers: Matthias J. Sax <matthias@confluent.io>

Author: lucasbru

docs/documentation/streams/developer-guide/streams-rebalance-protocol.md

@@ -0,0 +1,26 @@
+---
+title: "Documentation Redirect"
+robots: "noindex"
+_build:
+  list: false
+---
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements.  See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+
+{{< doc-redirect >}}

docs/getting-started/upgrade.md

@@ -34,6 +34,7 @@ type: docs
 
   * The `--max-partition-memory-bytes` option in `kafka-console-producer` is deprecated and will be removed in Kafka 5.0. Please use `--batch-size` instead. 
   * Queues for Kafka ([KIP-932](https://cwiki.apache.org/confluence/x/4hA0Dw)) is production-ready in Apache Kafka 4.2. This feature introduces a new kind of group called share groups, as an alternative to consumer groups. Consumers in a share group cooperatively consume records from topics, without assigning each partition to just one consumer. Share groups also introduce per-record acknowledgement and counting of delivery attempts. Use share groups in cases where records are processed one at a time, rather than as part of an ordered stream. 
+  * The Streams Rebalance Protocol ([KIP-1071](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol)) is now production-ready for its core feature set. This broker-driven rebalancing system designed specifically for Kafka Streams applications provides faster, more stable rebalances and better observability. For more information about the supported feature set, usage, and migration, please refer to the [Streams developer guide](/{version}/documentation/streams/developer-guide/streams-rebalance-protocol.html).
   * The `org.apache.kafka.common.header.internals.RecordHeader` class has been updated to be read thread-safe. See [KIP-1205](https://cwiki.apache.org/confluence/x/nYmhFg) for details. In other words, each individual `Header` object within a `ConsumerRecord`'s `headers` can now be safely read from multiple threads concurrently. 
   * The `org.apache.kafka.disallowed.login.modules` config was deprecated. Please use the `org.apache.kafka.allowed.login.modules` instead. 
   * The `remote.log.manager.thread.pool.size` config was deprecated. Please use the `remote.log.manager.follower.thread.pool.size` instead. 
@@ -84,8 +85,7 @@ For further details, please refer to [KIP-1120](https://cwiki.apache.org/conflue
   * Deprecated `org.apache.kafka.streams.KafkaStreams$CloseOptions` and its related methods, such as `KafkaStreams#close(org.apache.kafka.streams.KafkaStreams$CloseOptions)`. As a replacement, please use `org.apache.kafka.streams.CloseOptions` and `KafkaStreams#close(org.apache.kafka.streams.CloseOptions)`. For further details, please refer to [KIP-1153](https://cwiki.apache.org/confluence/x/QAq9F). 
   * A new implementation of `ConnectorClientConfigOverridePolicy`, `AllowlistConnectorClientConfigOverridePolicy`, has been added. This enables specifying the configurations that connectors can override via `connector.client.config.override.allowlist`. From Kafka 5.0.0, this will be the default [connector.client.config.override.policy](documentation/#connectconfigs_connector.client.config.override.policy) policy. The `PrincipalConnectorClientConfigOverridePolicy` policy is now deprecated and will be removed in Kafka 5.0.0. For further details, please refer to [KIP-1188](https://cwiki.apache.org/confluence/x/2IkvFg). 
   * It is now possible to specify the start time for a Kafka Streams punctuation, instead of relying on the non-deterministic time when you register it. For further details, please refer to [KIP-1146](https://cwiki.apache.org/confluence/x/9QqWF). 
-  * Added an optional `--node-id` flag to the `FeatureCommand` command. It specifies the node to describe. If not provided, an arbitrary node is used. 
-
+  * Added an optional `--node-id` flag to the `FeatureCommand` command. It specifies the node to describe. If not provided, an arbitrary node is used.
 
 
 ## Upgrading to 4.1.0

docs/streams/developer-guide/kafka-streams-group-sh.md

@@ -2,7 +2,7 @@
 title: Kafka Streams Groups Tool
 type: docs
 description: 
-weight: 14
+weight: 15
 tags: ['kafka', 'docs']
 aliases: 
 keywords: