Architecture question: Number of channels

[pabermod]

-Hi, I'm developing a software than will receive messages from different sources and send message to other different sources:

For example it can receive messages from 4000 origins and send messages to other different 4000 destinations.

I need messages from different sources to be processed independently so they don't slowdown processing of other sources in any way. Same for sending messages.

As opening a different Connection for each source/destination is a bad choice, i'm creating the following design:

• 1 Connection for receiving messages.
  • 1 Channel for each source with 1 queue and 1 consumer
• 1 connection for sending messages
  • 1 channel for each destination

With this approach I need to change the rabbitmq config to increase the max channel number.

Is there a better approach??

Thanks in advance

[lukebakken]

How about one connection for every N origins? N could be 100, 250, etc etc

[pabermod]

But using one channel for 4000 consumers will also reduce throughput right?

[michaelklishin]

Possibly. I'd mostly expect this to be visible on the consumer application side with some libraries.
Assuming there are more connections than CPU cores, 1 channel with 4000 consumers won't be all that different from 20 channels with 200 consumers each.

I don't see any need to use 4000 consumers on a channel.

[pabermod]

So, what is the Best approach?

• 1 connection with 4000 channels and 1 consumer each.
• 16 connections with 250 channels each and 1 consumer per channel
• 8 connections with 125 channels each and 4 consumer each

[lukebakken]

There's a decent chance you can model each of those with PerfTest - https://perftest.rabbitmq.com/

If not, you can write an app and benchmark it yourself.

We don't know anything about your environment or workload, so we can't really say.

Be sure to monitor your cluster so you can watch memory usage, CPU, etc as you perform your benchmark.

[pabermod]

Thanks, I will try it