Produce from anywhere, consume from the same availability zone

[huttotw]

I have an R3 cluster running on GKE in GCP. Since inter-zone network costs are high, how can I design a cluster (or super-cluster) in a way where my Jetstream consumers are always reading from a nats-server in the same zone as my consuming pod?

What will happen if the pod hosting my consumer dies, and a new pod starts consuming from a different zone, will the consumer leader eventually be moved to that new zone? Is there a way I can force that?

[derekcollison]

If you know which cluster the consumer is in apriori, I would create a DNS entry that is used by consuming apps that only resolves to the servers in that cluster.

If this information is dynamic, use cluster info to determine which cluster the consumer is in.

[huttotw]

@derekcollison currently our cluster spans multiple availability zones, are you suggesting that we should have 1 cluster per availability zone, and create a super cluster from those clusters?

[derekcollison]

If your cluster spans multiple AZs, the leader could move between AZs on leader elections. Reading from consumers is always done from the leader, so you would need to know where that is and move the consumer apps into that AZ and only connect to the consumer leader.

[huttotw]

To add a layer of complexity here, our consumers are dynamically created based on partitions (each partition is mapped to a single consumer). We may have ~100,000 consumers at any given time. Today, when we discover a new partition, the pod that a partition gets assigned to (via a token ring) is responsible for creating (or updating) the Jetstream consumer on start up.

It seems like when we create a consumer, it's possible that the leader will be in a different AZ than the pod that just created the consumer, causing us to move the app almost immediately before consuming any data.

I suppose this means that we should create consumers out-of-band and not from within the consuming application. That way we can know where the leader is prior to starting the consuming app.

Does this sound right? Would this be easier using a super cluster? Leaf nodes?

[derekcollison]

Sounds like you should do a consult with Synadia on your app tier architecture so we can better understand what you are trying to achieve.

[huttotw]

Okay, will do, thanks for the help!