Are Client-side Partitioned Consumer Groups equivilent to Pulsar Key-Shared subscriptions?

[CoenraadS]

Hello

Currently evaluating Pulsar vs NATS.

One requirement we have is to consume messages with the same key in order

E.g. in Pulsar:

Topic: Users
Message A (key 1)
Message B (key 2)
Message C (key 1)
Message D (key 2)

I can then listen to the 'Users' topic, and using 'Key-Shared' subscriptions, I can scale my consumers up and down independent of partitions/keys. Messages are guaranteed to be consumed in order for the same key (assuming we never NACK).

In NATS world, I would use subjects like

Users.1
Users.2

Then listen via 'Users.*'

I was trying to understand https://nats.io/blog/orbit-partitioned-consumer-groups/

Does this provide the same functionality? Regardless of the amount of consumers, which may scale up or down, messages will be consumed in order per subject?

Furthermore, the blog mentions Orbit.go, is this supported for C# libraries as well?

[jnmoyne]

At a fundamental level, JetStream consumers deliver messages strictly in order in two different ways:

• Ordered consumers: but they are ephemeral only (unless you persist your 'offset' to for example a KV bucket), do not support acking/nacking and can not be shared by multiple client application instances.
• Consumers with explicit acknowledgment and a 'max acks pending' value of 1: but this limit the consumption of messages to 'one at a time' regardless of the number of clients application instances trying to consume, and therefore limits severely the throughput of the consumption which can not be scaled horizontally.

Because of the throughput impact of setting max acks pending to 1, the Partitioned Consumer Groups library in Orbit was created and it does indeed provide the same kind of functionality as you can get out of Kafka's consumer groups (or it's Pulsar equivalent): messages are delivered in strict 'per key' order (and in NATS's case, including acking and nacking of the messages individually with automated re-deliveries) and yet you can parallelize through consistency hashing and partitioning the consumption of the messages, with in the case of the 'elastic' functionality of the library of being able to add/remove client applications administratively at any time (up to a pre-configured maximum number). The 'key' is any or all of the tokens of the subject the messages are published on (in your example it would be the second token of the subject that you partition on, or all of it since your first token never changes).

At this point indeed it is only in Go, but is not Go specific and could be ported to other languages.