Flask-Socketio to MQTT bridge

[ramanraja]

I have to write an IoT server that receives (a) MQTT messages from IoT devices and (b) socket messages from a mobile app. My project is essentially a socketIO to MQTT bridge. I use Paho-MQTT Python library.
It has a few house keeping web routines like logging in and registering users, for which Flask will be ideal.
My question is: Since my MQTT library may be synchronous and Flask-socketIO being async, are they compatible? Can I spawn a background thread in which the MQTT server can run?
I am open to using eventlet, gevent, gunicorn, nginx or anything else you suggest for the web server part.

[miguelgrinberg]

Unfortunately I do not have the answer to your question. If your MQTT library can be used under gevent or eventlet, then it is likely to work well alongside Socket.IO. Worst case you can always use the threading mode, which will allow you to use a regular thread for MQTT.

[ramanraja]

Thank you very much.
I don't think the MQTT library is compatible with async operations. (I did try it with gevent, but the whole code freezes at erratic intervals). Now I will try threading mode as you suggested.
I have another architecture in mind: Let the server be Flask-socketIO with gevent, but it will not explicitly spawn any threads. Instead the server will communicate with the bridge through socketIO. The bridge will be a separate process using your python-socketio library in the synchronous mode, so it can safely host the MQTT thread. Does this sound like a good design?

[mhanuel26]

@ramanraja

I think you can create a thread under socketio and use a queue to send message received from socketio into the thread that will process mqtt messages, and I believe you can use emit outside context to send messages from the mqtt thread.

[miguelgrinberg]

It's not as easy as you paint it. If you use eventlet and gevent, which are single-threaded frameworks, then you can only use Socket.IO from the eventlet/gevent thread. Communication between the eventlet/gevent thread and other threads is possible, but tricky to implement. Now, if you use the threading async mode, then this is all fine.

[ramanraja]

I made a curious observation about flask-socketio server freezing: I mixed a blocking thread with gevent based socketio server, which was a big mistake, as you have pointed out. To reproduce the bug, I put these two lines in my other thread:

while True:
   pass

As expected, this froze the entire socketio application.
From another machine, I called
curl http://myserver.com:5000/
This call just froze. It never returned.
I pointed a browser to the above URL. The browser tab also froze. It never timed out.
If the server is not sending any response, should the clients not just time out?

[miguelgrinberg]

What you are calling a "blocking thread" must have been a "blocking greenlet" instead. A thread that blocks in a loop would not have prevented gevent from running. A greenlet would. If this does not make any sense to you, I suggest you read about asynchronous frameworks and greenlets in particular to understand how this all works.

Wether a connection times out or waits forever is a matter of how the web server and the web client configure their sockets. It has nothing to do with Socket.IO, so I cannot answer.

[ramanraja]

OK, this helps. Thanks