nodejs process hang

[ericzhucode]

Version

14.18.2 / 16.16.0

Platform

Linux 3.10.0-864.14.1.5.h428.eulerosv2r7.x86_64

Subsystem

No response

What steps will reproduce the bug?

use nodejs to start a http server based on express framework
after 1 week to 1 month, the process hangs
here's the core file of the process which has the issue

All machines which has hang issue generate the same content of core file and all stuck at write() at /lib64/libpthread.so.0

At this point, js code is not working at all, all request to nodejs server will not receive any response

I'm not quite sure which information to collect in order to troubleshoot this issue. Could your team suggest?
Really appreciated your help!

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior?

No response

What do you see instead?

Additional information

No response

[bnoordhuis]

Can you strace -yy -p <pid> the hanging process? That should hopefully tell the kind of file descriptor it's blocking on.

[ericzhucode]

Can you strace -yy -p <pid> the hanging process? That should hopefully tell the kind of file descriptor it's blocking on.

Thank you for your suggestion.
I have already tried this command and the output is indeed different from other working processes.
It just output the following one line and no other info at all, seems stucking here

Other working process will ouput continously and the status will change to epoll_wait like

Although I can tell that this issue happens when extension installed s but I still can't know the details of this issue. Is there any other command line I can use to debug?

[bnoordhuis]

Okay, so I can kind of tell you what's going on. The stuck process is trying to log to stdout, probably using console.log.

stdout is a UNIX socket and seemingly its pipe is full (probably because the other end isn't reading) because the write gets stuck.

By the way, if you add -s 512 to strace, you can see the full message it's trying to write.

[ericzhucode]

Okay, so I can kind of tell you what's going on. The stuck process is trying to log to stdout, probably using console.log.

stdout is a UNIX socket and seemingly its pipe is full (probably because the other end isn't reading) because the write gets stuck.

By the way, if you add -s 512 to strace, you can see the full message it's trying to write.

Thank you for your brilliant suggestion and currently we have found that the log did stop at extension installed xxx which strace -s 512 -yy -p <pid> command shows.
Can I ask more if there's any way for us to know why UNIX socket pipe is full and stucks? Or is there any way that we can do to escape this issue? Since we find when console.log stucks, the payload is not high and not so much log was being written at the same time.
Is it possible that this issue is related to linux system? Or is this an issue that can be handled by modifing our js code or nodejs version?
Thank you!

[bnoordhuis]

To get answers to your questions you'll have to find out who owns the other end of the UNIX socket - maybe a process manager like pm2? - and why it isn't reading. All signs so far point to it not being an issue with node itself.

edit: I'll convert this to a discussion in the mean time.

[ericzhucode]

To get answers to your questions you'll have to find out who owns the other end of the UNIX socket - maybe a process manager like pm2? - and why it isn't reading. All signs so far point to it not being an issue with node itself.

edit: I'll convert this to a discussion in the mean time.

Really appreciated your suggestions so far. Your analysis is accurate and helpful!