Socket IO - Connection Error (When I make more then 100 concurrent connections)

[ticatwolves]

I am currently testing my flask socket io server which is deployed on ec2 using gunicorn, gevent and nginx

Here are the configurations for the same

gunicorn --threads 1 --worker-class geventwebsocket.gunicorn.workers.GeventWebSocketWorker -w 1 application:app -b 0.0.0.0:5000 --worker-connections 8000

Nginx config is this

worker_processes auto;
events { worker_connections 8192; }
http {
gzip on;
gzip_proxied any;
gzip_types text/plain application/json;
gzip_min_length 1000;
server_tokens off;
log_format main1 '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"'
'$upstream_response_time $request_time ';
upstream socketio_nodes {
ip_hash;
server 0.0.0.0:5000;
}
server {
listen 80;
server_name _;
location /health {
access_log off;
return 200 "healthy\n";
}
location / {
proxy_pass http://0.0.0.0:5000;
}
location /socket.io {
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass http://socketio_nodes/socket.io;
}
}
}

I am using gevent for handling async requests -> application.py

from __future__ import unicode_literals
import random, string
from gevent import monkey
monkey.patch_all()
from flask import Flask
from flask_socketio import SocketIO
app = Flask(__name__)
secret_key = ''.join(random.choices(string.ascii_uppercase + string.digits, k=8))
socket_app = SocketIO(
    app,
    cors_allowed_origins="*",
    message_queue="redis://",
    async_handlers=True,
    async_mode="gevent",
    cookie={"iosession": secret_key},
)

Here are some events that are written in the code on which i am testing
connect(): -> checking user authorization
punchEvent(): -> call psql db and do some action -> basically adding data into aws sqs

Now client side script

import socketio
ec2_ip = "" # this is public ip of ec2 instance
token = ""
sio = socketio.Client(reconnection_attempts=3)
sio.connect(
    f'ws://ec2_ip', headers={"Authorization": f"Bearer {token}"},
    namespaces=["/"],
    wait_timeout=60,
    transports="websocket",
)
payload = {} # here comes the payload
sio.emit(event="punchEvent", data=payload, callback=lambda *n, **o: print(n, o))

Now I have a AWS test lambda function with 1000 concurrency attached to a SQS. Now inserting 800 objects into queue and when lambda function triggers them then it was able to successfully able to create x connections but rest gives Connection Error
I also tried to increase max open file limit.

The above scripts works fine when i make single connection at a time in a loop.

I thought nginx is the bottleneck but i tried the same on 5000 port without nginx but the issue was same

Please suggest anything So i can hit at least 5k concurrent requests in parallel

[bluthen]

100 seems a little low, I can't remember what limits I hit in my load testing from over a year ago, I was testing my application and not just connection. I was thinking it was more like 200 for me.

Anyway I always hit some other issue after increasing max open file limit. I got egg on my face suggesting to just up the max files limit, after testing I realized that was never going to work.

I do load balancing with session affinity using multiple smaller machines instead of a large one. I however do not need to communicate between all the connections. I can either smartly group them with the load balancer (So small group of connections that need to talk to each other all go to the same instance) or they just do their own things and don't communicate with each other.

If you need the communication, Flask-SocketIO can utilize something like redis.
https://flask-socketio.readthedocs.io/en/latest/deployment.html#using-multiple-workers

[ticatwolves]

Hi @bluthen

I tried with 4 different ec2 running flask socket IO and one server of nginx passing the load to those servers.

This time i test for 100 concurrent connections from which 68 were success and rest failed

While in nginx access logs i see incomming traffic of all 100 requests.

CLIENT_IP - - [22/Feb/2023:08:31:45 +0000] "GET /socket.io/?transport=websocket&EIO=4&t=1677054292.2940793 HTTP/1.1" 101 56235 "-" "-"
CLIENT_IP - - [22/Feb/2023:08:27:49 +0000] "GET /socket.io/?transport=websocket&EIO=4&t=1677054464.634056 HTTP/1.1" 499 0 "-" "-"

One thing to check here is sometimes nginx is returning 499 instead of 101 - is that a problem ??

[bluthen]

It looks like your gunicorn timeout might need to be increased. 499 I think can be caused from gunicorn closing the connection after a timeout of there being no response from the far end server.

I imagine it is unacceptable for responses to take so long though.

Keep in mind an EC2 machine with a vCPU is a lot slower than a real processor. My 2018, 6 core laptop, is able to handle much more requests than 6 vCPU of EC2. I am not familiar with how a vCPU is measured or divided in EC2, but my experience shows I have to think about their performance by comparing them to themselves not actual computers I have. The bottleneck might not be the EC2 processing but stuff like context switching, or memory io.

Sorry I can't remember the stats from the benchmarks I did on EC2. I caused a major downage in my org, because I didn't do proper load testing on EC2 beforehand, I think it cost us a lot of money from having to pay fines for downtime as stipulated in the contract as well as negative press. We just solved it quickly by increasing the instances behind the load balancer to over 25 instances each with 2vCPUs.

My laptop handled more than I thought we needed so I thought an equivalent number of vCPUs would be fine in EC2 and didn't test. That high load system has been phased out for other reasons, currently socketio is used at my org only with smaller load use cases.

Also we didn't use nginx but used aws network load balancer instead.

Good luck!

[miguelgrinberg]

Flask-SocketIO does not have any limits. I suggest you review your system logs to help you determine what's causing these failures.

[miguelgrinberg]

Gunicorn is exiting unexpectedly. Why is that? Did you check your Gunicorn log?

[ticatwolves]

I have fixed this issue, but still getting connection error on client side out of 100 now only 40 was success

[miguelgrinberg]

Why is your log showing failures then? Or are you showing me old logs? In that case, can you check your current logs?

[ticatwolves]

Yes those were old logs these are new logs. I don't think they are related to my problem

Feb 22 12:17:01 ip-172-31-26-36 CRON[14732]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 22 12:29:15 ip-172-31-26-36 systemd[1]: Started Session 62 of User ubuntu.

[miguelgrinberg]

You are going to need to look better for any logs that show clues about this. Log files have dates and times, so you should be able to tell if an error happened while your application was running or not, you do not need to guess or even think about this. In addition to the sytstem log that you are looking at, nginx has a logfile, your Gunicorn probably has one too (and if it doesn't you should fix that). I really don't have any other advice to give you.