Update number of partitions of super streams and update consumers and producers seemlessly

[aeb-dev]

I believe clients already use AMQP to get information related to super streams. Maybe, the same AMQP connection can be used to update the clients

[acogoluegnes]

Which stream client library are you using?

Super streams are not supported out of the box with AMQP.

No stream library supports partition changes automatically AFAIK, the applications must be restarted to discover the new topology.

Apart from scaling out, super streams are meant to maintain the message ordering within a partition (e.g. having all the messages for a given key in the same partition with appropriate routing). Changing the number of partitions breaks this guarantee.

Would you mind telling us more about your use case, the need for super streams, and the need to change the number of partitions?

[aeb-dev]

I am using the official one.

AFAIK, when super stream connection is to be made, the client first queries the direct exchange to get stream partitions with AMQP. After that, it creates connections to each stream.

No stream library supports partition changes automatically AFAIK, the applications must be restarted to discover the new topology.

Yes, the clients does not support it. I created the discussion here because I am not sure if this requires a server side change

Apart from scaling out, super streams are meant to maintain the message ordering within a partition (e.g. having all the messages for a given key in the same partition with appropriate routing). Changing the number of partitions breaks this guarantee.

We are aware of this, it is a price to be paid. I forgot to mention it but this feature is supported on kafka and they also don't guarantee order after number of partitions changed

Would you mind telling us more about your use case, the need for super streams, and the need to change the number of partitions?

We have x number of services that consume from x number of partitions. When the services start to lag behind the stream we want to increase the number of services. In order to do that we need to increase the number of partitions.

[aeb-dev]

AFAIK, when super stream connection is to be made, the client first queries the direct exchange to get stream partitions with AMQP. After that, it creates connections to each stream.

My assumption was wrong. But I believe the feature would still be useful

[acogoluegnes]

I answered some of the questions in another discussion: rabbitmq/rabbitmq-stream-dotnet-client#297 (comment).

Adding or removing a partition to/from a super stream is just a matter of changing the AMQP 0.9.1 topology. Then the applications must be gracefully restarted to take the changes into account.

We can imagine an asynchronous task in the client libraries to detect topology changes every now and then (e.g. every minute) and change the producers and consumers accordingly.

[Gsantomaggio]

related to rabbitmq/rabbitmq-stream-dotnet-client#185