socket.io connect_error timeout on localhost

[tehnaz]

Hello,

We have a NodeJS server and ReactJS client communicating via socket.io.
This setup has been working for years. We are suddenly experiencing the following connect_error (without any changes in code):

We are using transport 'polling' that is later upgraded to 'websocket' after the handshake, but in this case the polling requests go into pending state and then timeout.
The issue is only observed on some machines while on the others it is still working fine on localhost.
The same code is also working fine on the QA, Stage and Prod deployment environments.

Furthermore, we have a multi-node setup, therefore, we use the following packages to run our server on multiple CPU cores - cluster, @socket.io/sticky and socket.io-redis (now @socket.io/redis-adapter).
The React app runs on port 3000. The NodeJS app includes an HTTP server running on port 8090 and the websocket server running on port 8092.

Current versions:
OS: Windows 11
NodeJS: 14.16
socket.io: 3.1.2
@socket.io/sticky: 1.0.1
@socket.io/redis: 6.0.1
redis: 3.0.2

Since @socket.io/redis is now renamed to @socket.io/redis-adapter, we upgraded it from 6.0.1 to 8.2.1 and also migrated from socket.io 3.1.2 and 4.6.1. We have also tried upgrading the other related packages but we still see the same error.
Disabling the antivirus and firewall on the windows machine hasn't helped either.
However, we have observed that if we use transport 'websocket' instead of 'polling', the error goes away.

Based on Chrome network tab, network port monitoring and debug logs of socket.io, we have observed that when long polling starts, 1-2 polling requests are reaching the server, while the others are not.

Browser:

Port traffic:

socket.io logs:
logs.txt

Code Snippets:
Client:

import { io } from 'socket.io-client';
socket = io(payload,
  {
    transportOptions: {
      polling: {
        extraHeaders: {
          'Authorization': 'Bearer ${payload.token}'
        }
      }
    }
  });
socket.on('connect', () => {
...
});
socket.on('connect_error', () => {
...
});
socket.on('message', () => {
...
});
socket.on('disconnect', () => {
...
});

Server:
Master:

const cluster = require('cluster');
const http = require('http');
const { setupMaster } = require('@socket.io/sticky');

const httpServer = http.createServer();
setupMaster(httpServer, {
    loadBalancingMethod: 'least-connection',
});
httpServer.listen(8092);

cluster.setupMaster({
    ...
});

for (let i = 0; i < numberOfCores; i += 1) {
    cluster.fork();
}
cluster.on('exit', (worker) => {
    cluster.fork();
});

Worker:

const { setupWorker } = require('@socket.io/sticky');
const socketIo = require('socket.io');
const redis = require('redis');
const { createAdapter } = require('@socket.io/redis-adapter');

this.io = socketIo(http.createServer() {
    pingInterval,
    pingTimeout,
    cors: {
        origin: [https://localhost:3000],
        methods: ['GET', 'POST', 'OPTIONS'],
        allowedHeaders: ['Authorization'],
        credentials: true,
        preflightContinue: false,
        optionsSuccessStatus: 204,
    },
    cookie: {
        name: 'io',
        httpOnly: false,
    },
    allowEIO3: true,
});

setupWorker(this.io);

const redisClient = redis.createClient(credential);
const subClient = redisClient.duplicate();
const  adapter = createAdapter(redisClient, subClient);
this.io.adapter(adapter)

What could be the issue with polling + websocket?
Kindly assist.

Thanks
Tehnaz

[darrachequesne]

Hi! I see you are using HTTPS (origin: [https://localhost:3000]), could the error be linked to a self-signed certificate?

we have observed that when long polling starts, 1-2 polling requests are reaching the server, while the others are not.
The issue is only observed on some machines while on the others it is still working fine on localhost.

That sounds indeed like a firewall issue or something else getting in the way.

[tehnaz]

Hi @darrachequesne

We were finally able to understand the cause of the issue.
We were using socket.io/sticky version 1.0.1 but due to a mistake, our package-lock.json had version 1.0.3
The developers who didn't update their node modules after this change were able to connect the sockets while the others encountered the issue.
Therefore, we are now able to establish the socket connection successfully using socket.io/sticky 1.0.1 with socket.io v3 and v4 both.

However, while dedugging the same, we found out that our client side code that uses extraHeaders is not working with socket.io/sticky 1.0.3 and that was causing the issue:

That is, this doesn't work:

socket = io(payload,
  {
    transportOptions: {
      polling: {
        extraHeaders: {
          'Authorization': 'Bearer ${payload.token}'
        }
      }
    }
  });

but this does:

{
    transports: ['polling', 'websocket'],
    auth: { token: `${payload.token}` },
 }

We also tried with socket.io/sticky 1.0.2 and observed the same results as 1.0.3.
Only 1.0.1 is working properly.
I checked the release notes and couldn't find anything that could break this.

Why is transportOptions with extraHeaders not working with socket.io/sticky 1.0.2 and 1.0.3?
Kindly assist. Thanks.

Regards,
Tehnaz

[darrachequesne]

Thanks for digging into this 👍

This commit might be at fault then (included in @socket.io/[email protected]), but the CI only fails for Windows + Node.js 10: https://github.com/socketio/socket.io-sticky/actions/runs/5207695992

Regarding the transportOptions value, I don't know what could go wrong, maybe your CORS configuration is not valid? Do you use cors.allowedHeaders?

Reference: https://socket.io/docs/v4/server-options/#cors

[tehnaz]

The CI test result seems similar to our usecase but we are on Node 14.

Yes, we do use allowedHeaders:

this.io = socketIo(http.createServer() {
    pingInterval,
    pingTimeout,
    cors: {
        origin: [https://localhost:3000],
        methods: ['GET', 'POST', 'OPTIONS'],
        allowedHeaders: ['Authorization'],
        credentials: true,
        preflightContinue: false,
        optionsSuccessStatus: 204,
    },
    cookie: {
        name: 'io',
        httpOnly: false,
    },
    allowEIO3: true,
});

[tehnaz]

Hi @darrachequesne

Do you see any issues with the above config?
Can we confirm this is a bug, should I report it on the socket.io-sticky page?

Regards,
Tehnaz

[darrachequesne]

OK, I was able to reproduce, the error indeed seems to come from the sticky package. I'll keep you updated.