Some thought about performance data

[qiulang]

It is very hard to find the performance data about socket.io, e.g. the peak number of connection, the memory usage, etc. The only article I find, which published later than 2021 is this one, https://ably.com/topic/scaling-socketio

The articles published earlier than 2021 are also very limited, actually I can only find four articles, 2 from µWebSockets.js in 2019 & 2020, claiming their performance were better, Moving from Socket.IO to µWebSockets.js and 100k secure WebSockets with Raspberry Pi 4, one from a guy called Ferit Özcan (but this article is not necessarily about the performance data), one from nginx in 2014 but it focused on raw ws connection I believe.

Of course we all know it is very hard to do performance test and what parameters we benchmark against is a tricky business. But still, I believe some ballpark numbers should serve many people.

For me I have done 2 type of performance tests,

1. my test team uses jmeter to test up to 5000 concurrent connections. Due to the nature of our product, 5000 connections is enough for us, so we didn't go higher.
2. I use https://a.testable.io/ to do the test. The reason I uses testable (this is NOT a sales pitch for them lol) I can choose ws clients from different locations, e.g. I chose 3 different locations from NA and one location from Asia. I believe this would be more closer to real life scenario than I just run a test script from my local machine(which I do have too). Doing this kind of test caused money, to quote from their technical support words after I did my test, "I see you ran a 20,000 user test successfully today too, great! Less than $20 for a test of this size is by far the best pricing out there :)."

So I would like to make some suggestion,

1. Can anyone share their performance data ?
2. If @darrachequesne wants to organize some kind of performance test, I would love to contribute, e.g. I can connect from China and other can connect from other locations so the test will be more real (just like what I use testable).

So is this a doable idea ?

[darrachequesne]

Hi! Thanks for opening this, that's indeed an important subject and the documentation is clearly lacking 👍

First, regarding the links that you provided, please note that:

• Ably is a company that provides hosted WebSocket solutions, so they might see Socket.IO as a direct competitor and I think readers should take their marketing material with a grain of salt
• µWebSockets.js is a WebSocket server implementation, which means that it will obviously have a lower overhead than Socket.IO, which provides HTTP long-polling fallback. And Socket.IO can use µWebSockets.js under the hood, replacing both the Node.js native HTTP server and the default WebSocket server implementation: https://socket.io/docs/v4/server-initialization/#with-uwebsocketsjs

The difference in memory consumption can be seen here: https://socket.io/docs/v4/memory-usage/

We already have some documentation about:

• load testing: https://socket.io/docs/v4/load-testing/
• performance tuning: https://socket.io/docs/v4/performance-tuning/

That being said, we could indeed create some performance tests, for different values of the following parameters:

• number of clients
• percentage of clients with a WebSocket connection
• distance from the servers (~ latency)
• number of servers
• adapter (Redis, MongoDB, ...)
• number of 1-to-1 packets per second (client to server)
• number of 1-to-1 packets per second (server to client)
• number of 1-to-n packets per second (server to client)

Output metrics:

• resources (memory, CPU) used by the servers
• latency of the packets (1-to-1 and 1-to-n)

[qiulang]

Hi, regarding the memory consumption shown in https://socket.io/docs/v4/memory-usage/ I have a question, as socket.io client can only connect to socket.io server, it will not be able to connect to a WebSocket server so how does a plain WebSocket server based on the ws package work there?

Regarding Ably's document I know Ably clearly sees socket.io as a direct competitor but https://ably.com/topic/scaling-socketio mentions show some interesting data (whether those data are reliable is another story), e.g.

SockJS could not handle more than 7,000 concurrent clients ... however, up to about 5,000 concurrent clients, it performed significantly better than plain WebSockets and Socket.IO

About the memory usage we can clearly compare its data with what https://socket.io/docs/v4/memory-usage/ said (almost 1.8G for 10,000 connections)

201 MB were needed to sustain 10,000 concurrent clients connected over vanilla WebSockets. As for Socket.IO? Almost 2 GB.

For me this 2GB claim seems not agree with my data because in my pm2 setting I have max_memory_restart: '1G' and I remembered I did test up to 10,000 concurrent clients and pm2 didn't to restart my nodejs server. But because 10,000 concurrent connection is far more than enough for me so I didn't test that number thoroughly so if it helps I can test that again.