Flask socketio server crashes after a few days, ran out of ideas to fix it

[ScottasM]

for the past few months i've been running a socketio server for my multiplayer game.

it seems to work alright, only after a day or two to crash with these errors :

[2024-06-18 14:55:02 +0300] [250979] [INFO] Handling signal: int
[2024-06-18 14:55:02 +0300] [250981] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/lib/python3.8/ssl.py", line 1019, in _create
    self.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/geventlet.py", line 143, in init_process
    super().init_process()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/base.py", line 142, in init_process
    self.run()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/geventlet.py", line 182, in run
    a.wait()
  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 225, in wait
    return self._exit_event.wait()
  File "/usr/local/lib/python3.8/dist-packages/eventlet/event.py", line 131, in wait
    current.throw(*self._exc)
  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 265, in main
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/geventlet.py", line 83, in _eventlet_serve
    conn, addr = sock.accept()
  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 228, in accept
    self._trampoline(fd, read=True, timeout=self.gettimeout(), timeout_exc=_timeout_exc)
  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 206, in _trampoline
    return trampoline(fd, read=read, write=write, timeout=timeout,
  File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/__init__.py", line 157, in trampoline
    return hub.switch()
  File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/geventlet.py", line 102, in _eventlet_stop
    client.wait()
  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 225, in wait
    return self._exit_event.wait()
  File "/usr/local/lib/python3.8/dist-packages/eventlet/event.py", line 131, in wait
    current.throw(*self._exc)
  File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 265, in main
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/geventlet.py", line 156, in handle
    client = ssl_wrap_socket(client, self.cfg)
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/sock.py", line 229, in ssl_wrap_socket
    return ssl_context(conf).wrap_socket(sock,
  File "/usr/local/lib/python3.8/dist-packages/eventlet/green/ssl.py", line 436, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/usr/local/lib/python3.8/dist-packages/eventlet/green/ssl.py", line 65, in __new__
    ret = _original_sslsocket._create(
  File "/usr/lib/python3.8/ssl.py", line 1031, in _create
    notconn_pre_handshake_data = self.recv(1)
  File "/usr/lib/python3.8/ssl.py", line 1257, in recv
    return super().recv(buflen, flags)
ConnectionResetError: [Errno 104] Connection reset by peer

Here is the server code, most likely it crashes on disconnection:

https://pastebin.com/tSCS1d6a

I'm running it with gunicorn, but the same used to hapen when i ran it like a normal flask app. This is on ubuntu.

once the process is stopped and restarted, it works again fine, only to repeat after a few days again.

I've tried running the whole thing with gunicorn using this command : gunicorn --workers 1 --worker-class eventlet --bind 0.0.0.0:5000 --certfile=/etc/ssl/chained.pem --keyfile=/etc/ssl/private/private-unencrypted.key --error-logfile ./error.txt --access-logfile ./access.txt RequestManager:app

I've tried logging when the crash happens, seems to be on disconnection, but really not sure, cant pin point exactly.

Tried a few other things, basically ran out of ideas, nothing seems to fix it.

[miguelgrinberg]

This is likely an issue with eventlet's implementation of SSL. Unfortunately there isn't much that can be done since eventlet is not well maintained anymore. Maybe try switching to gevent?

[ScottasM]

I will try this, and see if there are any crashes in the upcoming week. Thank you.