Support for QNX (performance issues)

[fschmitz925]

Hi,

we use the mqtt client lib on QNX (via the C++ async_client class). Compared to builds for Windows/Linux, it turned out that on QNX pushing is extremely slow (with payloads typically between 500 KB and 1 MB).

Using MQTT_C_CLIENT_TRACE, we figured out that the problem are partial writes, so we increased the system's net.inet.tcp.sendspace from 22528 to 180224, which is about the max value we should be using.

This helped somewhat, but the push performance was still quite low. So we looked into the source code and identified four possible changes, which have effects independently, and together dramatically improve push performance:

- In : 3rdParty\paho.mqtt.cpp\src\externals\paho-mqtt-c\src\MQTTAsyncUtils.c		   

	- In : thread_return_type WINAPI MQTTAsync_sendThread(void* n)
		Remove (line 1814) 
				timeout = 1000; // 1 second for follow on waits
		to leave the to at 10 [ms]

	- In : thread_return_type WINAPI MQTTAsync_receiveThread(void* n)
		Remove (line 2035) 
				timeout = 1000L;
		to leave the to at 10 [ms]

	- In : static MQTTPacket* MQTTAsync_cycle(SOCKET* sock, unsigned long timeout, int* rc)
		Replace line (3023) 
			MQTTAsync_sleep(100L);
		with 
			MQTTAsync_sleep(5L);

- In : 3rdParty\paho.mqtt.cpp\src\externals\paho-mqtt-c\src\Socket.h

		#define USE_SELECT

While this works fine for us (no excessive CPU load or threading or stability issues observed so far), we are a hesitant to change the 3rd party code, for fear of losing compatibility with future releases, and, to be honest, because we don't fully understand all implications / possible consequences of these changes. And, of course, the legalese involved in changing 3rd party code ...

So we would greatly appreciate if someone with a deeper understanding of the matter could review these changes, and if a future release could take them somehow into account, e.g. by using #defines for the literal to/sleep values, so we may use cmake add_compile_definitions to inject working values - If still applicable, this is, because I notice that in other threads the sleeps/timeouts are already being discussed.

Thanks a lot and best regards,
Frank Schmitz
( PROMESS Montage- und Prüfsysteme GmbH, Berlin, Germany )

[speedy-h]

Hello,

I noticed the same performance issue when sending packets on a FreeBSD system and narrowed it down to partial writes and the same 1s delay you've highlighted.

I think the root cause is after a partial write, if the socket becomes available for more data, nothing is notified. We instead wait for the full timeout to happen before running the MQTTAsync_processCommand in the MQTTAsync_sendThread loop. Based on this I don't think reducing the timeout is the best solution, but rather send a trigger to the send thread when there's a pending write that could continue.

I managed to get this to work by making the send_cond variable visible in the Socket.c file. It get triggered in theSocket_getReadySocket after the select function call if a partial write happened. I introduced a new event in the Sockets structure to help with this.

I did this on version 1.3.9 and it looks like it could work for master.
If the current maintainers think this is allong the right track to get this performance improvement I can try to get my change working on master and open a PR.

[fschmitz925]

Sounds promising, thx

[harmjonker]

Question, did you build the Paho library for QNX 8? I am also looking into that but have no idea how to build it for QNX 8. Are you able to share the makefile ?

[fschmitz925]

We build the Paho lib (cpp+c) for QNX 7.1 with cmake. I could share some cmake file fragments, together with some comments of a colleague of mine, but not sure if this would help you for QNX 8 (?)

[fpagliughi]

I'm really interested in getting to the bottom of these performance issues.

I have a number of systems in production that use RPC over MQTT and are highly reactive to event messages. So my motivation is perhaps driven by a need for low latency rather than overall bandwidth, but these are tied together, so fixing one would help the other.

And some general cleanup of the code base would probably be warranted.

I started with a bottom-up approach, looking at the synchronization mechanisms in the library. For example, both semaphores and condition variables are just used as simple thread event signals, so can be consolidated, pushing platform conditional compilation down into Thread.c, cleaning up the rest of the code. But this is perhaps more of a medium-to-long-term solution.

Looking at this discussion, there's clearly more that we could do sooner to boost performance.

I think the root cause is after a partial write, if the socket becomes available for more data, nothing is notified.

Yes. This. Threads shouldn't be stuck in a wait() or sleep() if something needs to be done. Reducing the timeouts in a polling loop would help, but I agree that waking a thread immediately when there's work to be done is the most proper solution.

If you folks have code to share, I would love to see it. Perhaps we can start a branch here, or maybe work in a fork temporarily to iterate on a solution? I have a personal fork up that we could use: https://github.com/fpagliughi/paho.mqtt.c