Spawning python processes that create socketio clients from the server (Help with architecture)

[adriallopgirones]

Hello all,

I'm working on a project that needs to implement quite a difficult architecture. I would really appreciate comments or some direction on how to do this in the best way.

General Idea:

• I have several HW pieces that run on their own python process and use a serial connection (Think of them as sensors)
• I also have some data processing programs that run on their own python processes
• These HW and programs generate data that need to be displayed on a REACT front end
• Users can interact with the REACT front end and changes need to propagate back to HW (i.e increase temp by 2 degrees)
• The idea is to deploy all this in a light computer (raspberry PI type of deal), so the front-end can be accessed from the outside.

My general approach:

• A flask socketio server acts as the middle man between the REACT GUI and Python processes
• The REACT GUI runs independently from flask, it creates a socketio client and tells the flask server to start the HW python processes
• Flask socketio creates the HW python processes using the multiprocessing libray
• python processes create a socketio client using python-socketio and continuously emit data to the server, and the server emits to the REACT client.
• Eventually a user interacts with the GUI. React client emits action -> Flask server receives it and sends it to Python processes.

Questions:

• Does this architecture make sense?
• Will those processes created with multiprocessing play nice with flask? Should I totally separate them from flask?
• Should the python processes communicate in a different way to the server (not being socketio clients maybe? shared memory?)
• How difficult will be to deploy something like this?
• I'm not using any type of queue for now, should I?
• Are the python processes really running independently from the flask server?

[miguelgrinberg]

Does this architecture make sense?

In general yes, this seems like a good solution to me.

Will those processes created with multiprocessing play nice with flask? Should I totally separate them from flask?

This is the one aspect of your solution that I feel is wrong. The Flask server should not own those other Python processes. The server may need to be upgraded and restarted, and that is no reason for those other processes to die. Also your Flask server may need to run multiple workers, which creates a complication if the server is the parent for those other processes (like which of the N workers is the parent?).

Should the python processes communicate in a different way to the server (not being socketio clients maybe? shared memory?)

Socket.IO clients is a simple way to communicate, and you already have the server so it is convenient.

How difficult will be to deploy something like this?

The deployment shouldn't be too difficult. You can use systemd to manage your processes (and drop the multiprocessing idea).

I'm not using any type of queue for now, should I?

Not from the Socket.IO side. The message queue is used when you have auxiliary processes that emit to clients on behalf of the server. You don't have that, your auxiliary processes are clients, they can emit and also receive so you have a more robust solution that replaces the message queue with WebSocket.

Are the python processes really running independently from the flask server?

They should be completely independent in my opinion.

[adriallopgirones]

Thank you so much for your reply and for your work!

The thing is that an external user will be the one that starts these processes from the GUI (Clicking start), that request is going to the Flask Server, so how would I start a process completely independent from Flask?

Thanks

[miguelgrinberg]

If you have 1000 external users all connected at the same time, how is your server going to handle starting thousands of individual processes, each serving one client? I think you need to rethink your solution so that it can scale with your users.

[adriallopgirones]

This is meant to be an internal tool for a product, this product has several HW pieces (sensors), so it'll be one process for each sensor, and you can assume only one user will be interacting with the GUI.

[miguelgrinberg]

Well, I gave you the advice that you asked. If what I suggested does not work for you then you can do something else. I wouldn't change a design based on the assumption that there will be just one user.