As an API, I would actually prefer to have a new method that provides a Node.js stream per partition, which would be able to independently have its own backpressure mechanism. That would provide a better DX.

As an API, I would actually prefer to have a new method that provides a Node.js stream per partition, which would be able to independently have its own backpressure mechanism. That would provide a better DX.

I think that depends on the usecase, we actually prefer to have pause/resume because if we want to consume messages in some past datetime range, we don't want to continue consuming from partitions that already satisfied the end date, it just adds unnecessary overhead. So, it's not only useful for backpressure control, and we only have to deal with 1 stream.

Just to clarify, I don't find separate streams a bad DX, but rather difficult to implement, at least for me.

I also agree with @mcollina. Given we have followed a streams only API for consumers, I don't want to have additional method on the consumer itself.

But, @xxzefgh I see your use case. I would add the logic to the MessagesStream, in particular, add the following methods:

• pauseTopicPartitions
• resumeTopicPartitions
• getPausedTopicPartitions

Then you modify #fetch to skipped paused partitions.
This has the advantage to avoid any checks about joining the group and so forth.

Note that if a #fetch call has no partitions after filtering out all paused one then you should immediately pause the stream and bail out of the function.

This also implies that a call to resumeTopicPartions will also have to resume the stream if it was previously paused.

Also, use just this.emit('pausedTopicPartitions') and this.emit('resumedTopicPartitions') where appropriate.

What do you think? Are you willing to implement this?

@ShogunPanda Sure, I can get it done. Would you be willing to accept seek API's too in another PR?

Amazing!
About seek, what is the intended usage?