Why is paho-mqtt-c slower than the Python variant?

[harsszegi]

Hi,
I have a strange issue. I am using paho-mqtt-cpp over -c to publish/subscribe.
I am running on Ubuntu 23.10, over TLS.
I am sending through a 2.5mbyte file 10x, in 10 different threads.

On python to do the same it takes ~80ms per image, while on C++ it takes 3x more.
If I use 1.3.12 AND leave “poll” enabled instead of “select” it will slow further down the transmit 3x, ending up being 8-9x slower than the Python implementation.

I have replaced the server/subscriber code with Python as well, didn’t help much, so it appears the publish code itself is that much slower.

Between python and C++ I have the same connection, QOS, etc setup.

Any idea what is going on?
Thanks,

[jumoog]

https://github.com/eclipse/paho.mqtt.cpp

[harsszegi]

yeah, I get that, but as that is a kind of a thin wrapper around paho-mqtt-c, I doubt the performance degradation comes from there. Furthermore I can clearly see a performance variance if I switch 1.3.12 with 1.3.9 underneath paho-mqtt-cpp

[harsszegi]

I did some tracing, and the issue appears to come from the SSL write itself:

20231012 072626.761 Partial write: incomplete write of 2735688 bytes on SSL socket 4
20231012 072628.375 SSL continueWrite: partial write now complete for socket 4

So for some reason for paho-mqtt-c it takes over 1second to send over 2.5Mbytes over SSL on the wire

[harsszegi]

Funny thing though that even without SSL the transmission speed is slow:

20231012 075036.852 Partial write: 110048 bytes of 2735688 actually written on socket 4
20231012 075038.357 ContinueWrite: partial write now complete for socket 4

[harsszegi]

It appears that after the first partial write (regardless of TCP or SSL), there is a fixed 1second wait before the transfer continues. Does it ring a bell for anybody?

[harsszegi]

Ok the issue is that the Async client locks way too opportunistic, it locks its semaphore before quite large sleeps making it impossible for other threads to do their job. I reduced the sleeps here and there, now is performs better

[icraggs]

Also persistence. Have you set MQTTCLIENT_PERSISTENCE_NONE ? The Pyhon client doesn't have any, as far as I'm aware.

The most optimal solution might be to signal the send_sem/send_cond in the MQTTAsync_writeContinue and MQTTAsync_writeComplete callback functions.