Need Some General Info Regarding Client Design & partitioning

[YashasAnand]

1. I currently have a single connection per application set up on my clients. Thus all Produce & consume happen on this same connection. Does having a Producer per subject helpful in anyway or is it an anti-pattern?

2. What is the recommended replication factor for streams in a 3 node nats k8s cluster? In kafka we have min.isr & is set to 2, it says minimum 2 replicas should ack a request to be successful, does this mean i need to have replication of 3 to achive min.isr of 2 in NATS? I observe replication overhead is quite high at high rps rates. Considering we have multiple streams is it recommended to replicate all streams with replication of 3?

3. As @jnmoyne states in Event Synchronization with NATS nats.go#970,
   "If you have a very high message rate you want to capture in a stream and you are reaching the limit of performance a single stream can handle and you want to scale that stream then you would create a stream per partition (and one durable per partition stream)."

We have a use case where we dont require strict ordering of messages, my 1st question is , lets say if i have 5 partitioned streams & 1 durable consumer per stream , how should the producing application produce to this stream, do we have any example blog on this? Is there some stream mapping functionality similar to subject transforms?
My 2nd question is as per @Jarema in #6100, having many streams can cause RAFT issues especially at higher replication factors. So can anyone tell me what kind of trade offs am i looking at here for implemnting stream per partition?

4. Also in Kafka a partition can be understood as a physical data stucture of a logical topic which holds data & pulled from consumers. In NATS this concept im trying to understand in little more depth, can anyone pls help with clear picture of partitions & how partitions scale?

Is a stream equivalent to partition in kafka?

[RepoBreather]

I too want to know more on Question 3 & 4.

@jnmoyne @Jarema @ripienaar can you please help with some info on this, after reading this https://docs.nats.io/nats-concepts/subject_mapping it is quite unclear how nats handels partitioning and how it scales.

[jnmoyne]

At a high level:

You want to distribute messages in a stream between multiple instances of a consuming application: you don't need to use partitioning just have all the instances use the same durable consumer.

You want to have 'strict ordering' of the consumption of the messages in the stream: you have to set max acks pending to 1 in the consumer. That however has a significant impact on the throughput as you effectively only consume one message at a time.

You want to have strict ordering and need to distribute the consumption over multiple instances of the consuming application: you insert a partition number in the subject of the message (as part of the stream configuration) and then create a durable consumer (with max acks pending 1) per partition (using a subject filter for that particular partition number).

Separately, (and this is the case being discussed in this particular question) your inbound traffic flow throughput is higher than the max write throughput you get out of a single stream in your deployment: you need to distribute that inbound traffic over more than one stream, in which case you insert the partition number using a Core NATS (account level) subject transformation and then you create one stream per partition (using a subject filter for that particular partition number).

[jnmoyne]

You can indeed use the {{partition()}} function in a subject mapping transform to insert a partition number token in the subject of the message being published. In this case you can use Core NATS subject transformation (which is part of the NATS account's settings) to have the NATS server(s) automatically insert that token in the subject as the message is being published, and before that message is captured (by the stream listening on that particular partition number).

In your example some applications publish messages on foo.<id> and you have a subject mapping defined in your account's settings with source: foo.* and destination foo.{{partition(5,1)}}.{{wildcard(1)}} and then create 5 streams, one that captures messages on subject foo.0.*, one on foo.1.*, etc... all the way up to foo.4.*.

In your case since you do not need strict ordering, on the consumers that you create on each stream you can have a max acks pending value higher than 1 (which will allow much higher throughput per consumer).

As mentioned the Core NATS mapping are defined at the account level, meaning that if you use static security you would define them in the server's configuration file. If you are using de-centralized operator JWT-based security then the mapping(s) would be part of the account's JWT.

For an example of how to define Core NATS subject mappings in a server config based security model you can take a look at the examples in this blog post (not using subject transform to specifically insert a partition number token in the subject, but still using subject transformation).

https://www.synadia.com/blog/multi-cluster-consistency-models