Skip to content

Cooperative rebalancing consumer hangs on disconnect, rebalanceProtocol() returns NONE #1144

@kathrynkodama

Description

@kathrynkodama

Hi there, I believe this is a bug in the node-rdkafka library. When we use cooperative rebalancing, on shutdown some consumers return the rebalance protocol as NONE rather than COOPERATIVE.

Environment Information

  • OS [e.g. Mac, Arch, Windows 10]:
  • Node Version [e.g. 8.2.1]: 22.22.0
  • NPM Version [e.g. 5.4.2]: 10.9.4
  • C++ Toolchain [e.g. Visual Studio, llvm, g++]:
  • node-rdkafka version [e.g. 2.3.3]: 3.6.0

Steps to Reproduce

  1. Create a KafkaConsumer with partition.assignment.strategy: 'cooperative-sticky' and rebalance_cb: true
  2. Connect the consumer and wait for it to join a consumer group and receive partition assignments
  3. Call consumer.disconnect() (or stream.destroy() on a KafkaConsumerStream)
  4. Observe that the final revoke rebalance callback fires, but rebalanceProtocol() returns 'NONE' instead of 'COOPERATIVE'
  5. It looks like the built-in handler calls unassign() instead of incrementalUnassign(), which throws inside librdkafka. The error is silently swallowed because isConnected() is also false at this point. disconnect() never completes.

node-rdkafka Configuration Settings

'partition.assignment.strategy': 'cooperative-sticky',
rebalance_cb: true,

Additional context

We observe this happening intermittently, when we have multiple consumers with a large number of partitions. When rebalancing happens, only some consumers return 'NONE' instead of 'COOPERATIVE'.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions