patch sticky rebalance protocol to mitigate endless loop issue (#493)

* patch rebalance

* c

* readme

* backport of missing variable fix

Author: mensfeld

CHANGELOG.md

@@ -2,7 +2,9 @@
 
 ## 0.18.0 (Unreleased)
 - [Enhancement] Update `librdkafka` to `2.5.0`
+- [Patch] Patch cooperative-sticky assignments in librdkafka.
 - [Fix] Mitigate a case where FFI would not restart the background events callback dispatcher in forks
+- [Fix] Fix unused variable reference in producer (lucasmvnascimento)
 
 ## 0.17.0 (2024-08-03)
 - [Feature] Add `#seek_by` to be able to seek for a message by topic, partition and offset (zinahia)

README.md

@@ -163,12 +163,12 @@ bundle exec rake produce_messages
 
 | rdkafka-ruby | librdkafka | patches |
 |-|-|-|
-| 0.18.0 (Unreleased) | 2.5.0 (2024-06-10) | no |
-| 0.17.0 (2024-08-03) | 2.4.0 (2024-05-07) | no |
-| 0.16.0 (2024-06-13) | 2.3.0 (2023-10-25) | no |
-| 0.15.0 (2023-12-03) | 2.3.0 (2023-10-25) | no |
-| 0.14.0 (2023-11-21) | 2.2.0 (2023-07-12) | no |
-| 0.13.0 (2023-07-24) | 2.0.2 (2023-01-20) | no |
-| 0.12.0 (2022-06-17) | 1.9.0 (2022-06-16) | no |
-| 0.11.0 (2021-11-17) | 1.8.2 (2021-10-18) | no |
-| 0.10.0 (2021-09-07) | 1.5.0 (2020-07-20) | no |
+| 0.18.0 (Unreleased) | 2.5.0 (2024-06-10) | yes |
+| 0.17.0 (2024-08-03) | 2.4.0 (2024-05-07) | no  |
+| 0.16.0 (2024-06-13) | 2.3.0 (2023-10-25) | no  |
+| 0.15.0 (2023-12-03) | 2.3.0 (2023-10-25) | no  |
+| 0.14.0 (2023-11-21) | 2.2.0 (2023-07-12) | no  |
+| 0.13.0 (2023-07-24) | 2.0.2 (2023-01-20) | no  |
+| 0.12.0 (2022-06-17) | 1.9.0 (2022-06-16) | no  |
+| 0.11.0 (2021-11-17) | 1.8.2 (2021-10-18) | no  |
+| 0.10.0 (2021-09-07) | 1.5.0 (2020-07-20) | no  |

dist/patches/rdkafka_sticky_assignor.c.patch

@@ -0,0 +1,26 @@
+# This patch is released under the 2-clause BSD license, same as librdkafka
+# Fixes: https://github.com/confluentinc/librdkafka/issues/4783
+#
+--- librdkafka_2.5.0/src/rdkafka_sticky_assignor.c	2024-07-08 09:47:43.000000000 +0200
++++ librdkafka_2.5.0/src/rdkafka_sticky_assignor.c	2024-07-30 09:44:38.529759640 +0200
+@@ -769,7 +769,7 @@
+         const rd_kafka_topic_partition_list_t *partitions;
+         const char *consumer;
+         const rd_map_elem_t *elem;
+-        int i;
++        int i, j;
+ 
+         /* The assignment is balanced if minimum and maximum numbers of
+          * partitions assigned to consumers differ by at most one. */
+@@ -836,9 +836,9 @@
+ 
+                 /* Otherwise make sure it can't get any more partitions */
+ 
+-                for (i = 0; i < potentialTopicPartitions->cnt; i++) {
++                for (j = 0; j < potentialTopicPartitions->cnt; j++) {
+                         const rd_kafka_topic_partition_t *partition =
+-                            &potentialTopicPartitions->elems[i];
++                            &potentialTopicPartitions->elems[j];
+                         const char *otherConsumer;
+                         int otherConsumerPartitionCount;
+ 

lib/rdkafka/producer.rb

@@ -298,7 +298,7 @@ def produce(
         partitioner_name = @topics_configs.dig(topic, topic_config_hash, :partitioner) || @partitioner_name
 
         # If the topic is not present, set to -1
-        partition = Rdkafka::Bindings.partitioner(partition_key, partition_count, @partitioner_name) if partition_count.positive?
+        partition = Rdkafka::Bindings.partitioner(partition_key, partition_count, partitioner_name) if partition_count.positive?
       end
 
       # If partition is nil, use -1 to let librdafka set the partition randomly or