Can Message queues be used as a external service/using any endpoint ?

[arun6894]

I could understand below the use case of the message queues in the framework.

Clients communicate directly with the server, and the server communicates directly with clients. The message queue is used internally when you have more than one server, as a way to coordinate operations such as broadcasts.

To my understanding, For eg., if we can use KafkaManager to emit externally to the client by producing the message to Kafka internally and internally consumes the message and emit it to the client. (Correct me if I'm wrong here, as I have checked external emit is producing to the channel and with my custom python file which is already consuming the same channel could read the message)

external_sio.emit('server' : {'check': 'test'})

My requirement :

1. I have a topic in Kafka and I want to produce data for a topic to emit. But not from KafkaManager external emit. How can I make the KafkaManager external emit understand this message from another python service? From the above code snippet Is there a way I can replace the {'check': 'test'} with any of the data from another python service without accessing its object using any endpoint/? Please suggest to me if that can be achieved through fast API/flask integration or if not by python-socketio. (Currently, I'm producing the data from other python services and running a consumer logic in different thread to access the producer message. Just want to use the framework feature for this )

2. Point 1 covers emitting from different python services by using kafkaManager external emit to the client directly. During the client emits to the socket server I don't think we have a feature to produce the emit data to a consumer of Kafka topic. I have already gone through the document and there is no option for this. Just want to reconfirm ?. Currently, I have used an external code to produce it for a channel. I hope this is fine as we don't have this feature to produce it alone.

Also, I just want to ask one last question.

Today I have been stress testing the server with gevent async_mode .Due to multiple clients with CPU load. I have divided it into three different machines and ran the client on 3 processors on a single server. I see the server is going to a hung state when I started a background task after certain client connections. It is going to the hung state during emitting and it's closing the client connections? It works when I ran the client on 1 processor.

When I have changed the async_mode to eventlet. It works. Please give your ideas on this.

[miguelgrinberg]

No, I don't think you understood. The bottom line is that there is nothing that this package does with the message queue that can help your application. You provide a queue, and this package uses it exclusively for internal purposes. There is absolutely nothing of use for you in the use of the queue.

If you need a queue for your application, you can use one, even the same that you provide to this package, as long as you use a different channel for your needs.

[arun6894]

@miguelgrinberg Could you comment on the question related to the gevent server being in the hung state when running with multiple client processes when it's not the same with eventlet which works? When running gevent with a single client it is fine.

[arun6894]

After removing the client manager of kafka-python. It is working with multiple client processes. Even with the Redis Message queue, it is not working with multiple client processes. Need your thoughts on this.

[miguelgrinberg]

If you are going to use gevent or eventlet, you need to read their documentation and understand how concurrency works under these frameworks, and how to avoid blocking. Both frameworks offer a monkey patching feature that sometimes can help, so you may want to look into that as well.

This is all general information that is unrelated to this package. If you prefer to work with a simpler concurrency model, then uninstall gevent and eventlet and use the threading async mode offered by this library, which is based on regular threads.