Distributed Performance Issue

[JoeStagner]

I have a python application that reads data from the web and writes the retrieved text to RabbitMQ.

I have a workgroup of 6 8 core Dell Optiplex boxes running Ubuntu 22.04.

If I run 6 instances of the app on ONE box, everything runs as expected.

However - if I run on instance on each machine writing to RabbitMQ slows to a crawl (though one machine is writing faster than all the others.)

Can anyone suggest why the performance is so much worse when distributed to separate machines ?

System monitor reports CPU, RAM and Network consumptions are all under 50%

[mkuratczyk]

Where is the RMQ cluster in all of this? If I understand correctly, you're comparing 6 app instances on 1 server vs 6 app instances, one per server. But is RabbitMQ running on all of these servers? One of them? On completely different servers?

[JoeStagner]

Thanks - RabbitMQ is a single node running on a dedicated server.

[mkuratczyk]

In that case, nothing comes to mind really. How slow is a crawl? What's the difference?

Some stuff you can try:

1. rabbitmq-diagnostics observer, especially sorted by mailbox size mq[ENTER] after it starts
2. Reproduce with https://perftest.rabbitmq.com/ to rule out client-side issues and allow easy experimentation with different settings
3. capture what the server is doing: https://www.rabbitmq.com/blog/2022/05/31/flame-graphs

[JoeStagner]

Thanks so much - I'll give it a try