Child_process.exec returns empty stdout when many child processes are created

[bunglegrind]

Version

19.7.0 (WSL),
16.19.1 (Debian)
18.14.2 (Windows 11)

Platform

Linux MSI 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 GNU/Linux;
Linux Server 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux
Microsoft Windows NT 10.0.22621.0 x64

Subsystem

Child process

What steps will reproduce the bug?

run the following snippet passing a large number (> 10000), as argument

node index.js 20000

index.js:

import cp from "node:child_process";

const throttle = Number(process.argv[2]);
//it doesn't  matter the exact command, but the time it requires to be completed. Even ls it is fine, possibly executed on a directory with many files
const cmd = `file -b a.pdf`;
// const cmd = "echo a.pdf";//for windows
const timeout = 100;

let i = 0;
while (i < throttle) {
    setTimeout(() => {
        const subp = cp.exec(
            cmd,
            {timeout: 1000 * timeout},
            function (error, stdout, stderr) {
                if (error) {
                    throw error;
                }
                if (stderr) {
                    throw stderr;
                }
                if (!stdout) {
                    throw `stdout is empty!`;
                }
            }, 0);
        subp.on("error", function (error) {
            throw error;
        });
    });
    i += 1;
};

How often does it reproduce? Is there a required condition?

When the number of processess spawned is very large, it may happen. The exact threshold depends on what the machine is currenty executing at the time, the time needed by the child process to terminate its work, etc.

What is the expected behavior?

any kind of error/exception, signaling possibly too many spawned, etc.

What do you see instead?

stdout is empty

Additional information

No response

[bunglegrind]

in windows I can reproduce the issue even with node index.js 3000

[bnoordhuis]

I've converted this to a discussion because I see no reason to believe this is a bug in node rather than a problem with the program you're trying to execute. I'll convert it back if you can conclusively demonstrate the issue lies with node.

[bunglegrind]

you mean the file command? Or the while loop? I've tried to rearrange my code in order to find clues. This is my latest version, still happening (empty string exception):

import cp from "node:child_process";
import fs from "node:fs";

const throttle = Number(process.argv[2]);
//const cmd = `ls -l && echo hi`;  //WSL2
const cmd = `dir && echo hi`;  //Windows 11
//const cmd = `sha256sum a.pdf && echo hi`;  //Linux Debian, a.pdf is a large file
const timeout = 100;

const output = [];

let i = 0;
while (i < throttle) {
    (function exec(i) {
        const subp = cp.exec(
            cmd,
            {timeout: 1000 * timeout},
            function (error, stdout, stderr) {
                if (error) {//it happens only when throttle is larger than a fixed value (see below)
                    error.i = i;
                    throw error;
                }
                if (stderr) {//never happens
                    throw stderr;
                }
                if (stdout === "") {
                    fs.writeFileSync("out.txt", output.join("\n"));
                    throw `stdout ${i} is empty string!`;
                }
                if (stdout === undefined) {//never happens
                    throw `stdout ${i} is undefined!`;
                }
                output.push(`process ${i} ended.`);
            }
        );
        subp.on("error", function (error) {//never happens
            error.i = i;
            throw error;
        });
        subp.on("disconnect", function () {//never happens
            throw `Process ${i} disconnected.`;
        });
        if (i % 1000 === 0) console.log(`Process ${i} spawned`);
    }(i));
    i += 1;
};

for windows 11 throttle is set to 10000, WSL2 is 20000, Debian is 8000.
If I increase the throttle, an error occurs (which I suppose is the desired behavior) errno: -11, EAGAIN

Do you have any suggestions?

[bnoordhuis]

you mean the file command?

Yes, as running dir && echo hi eventually fails with EAGAIN (i.e., Windows telling you "too many processes.")

[bunglegrind]

I'm still investigating the issue but I found that the issue does not appear if I call spawn instead of exec.
And btw, I was able to reproduce the bug even in replit:
https://replit.com/@bunglegrind/child-processes

with the command line node index.js 700

[bunglegrind]

Well, I think I've discovered what happens.
The issue (because it's a node issue, or at least of its documentation) regards what happens when timeout expires:
Using exec (and execFile) process is terminated but no clue is provided to the client code regarding the expiration. No error codes, or specific events ("timeout" or "error") are raised.
Using spawn, well, timeout simply does not work. Processes are executed till they finish.

In short, imho this discussion should go back in the issue section

[bnoordhuis]

when a timeout expires, client code should be notified

That sounds reasonable. You can open a pull request for that if you want.

using spawn [..] the script keeps running

The spawn() documentation mentions it (albeit maybe obliquely) but cp.spawn("foo && bar", {shell:true, timeout:42}) runs this:

/bin/sh -c foo && bar # read as: "/bin/sh -c foo" AND "bar"

The signal terminates /bin/sh but not bar. exec() starts it the way you want:

/bin/sh -c "foo && bar"

[bunglegrind]

when a timeout expires, client code should be notified

That sounds reasonable. You can open a pull request for that if you want.

Well, I can try. Never looked at how to build node or how tests are written.

using spawn [..] the script keeps running

The spawn() documentation mentions it (albeit maybe obliquely) but cp.spawn("foo && bar", {shell:true, timeout:42}) runs this:

/bin/sh -c foo && bar # read as: "/bin/sh -c foo" AND "bar"

The signal terminates /bin/sh but not bar. exec() starts it the way you want:

/bin/sh -c "foo && bar"

Ok, I can see, but spawn still not closes even if I set cmd just to sleep 100. Because the "close" event is not emitted (but the "exit" event is emitted). Maybe it's the expected behavior? I'm not sure...anyway, thanks for your suggestions. I'll try to provide a PR in the next days.

[bnoordhuis]

Yes, it's expected. 'close' is emitted only after stdio closes but since that's also inherited by the other process, said process keeps it alive.

[bunglegrind]

what is weird is that sometimes (depending on the actual command) an error is raised (with SIGTERM as description), sometimes not; or when using exec I don't see any exit events from the child process. BTW, I think nodejs/node#43704 is connected to this discussion.
This PR nodejs/node#42959 was a first attempt to unify spawn and exec timeout, but for some reasons, it was closed

[bnoordhuis]

I think #43704 is connected to this discussion.

Good point. I've left a comment and closed it out.

[bunglegrind]

Ok, after having performed another bazillion of tests, I'm going to write the "accepted" aswer.

When exec (or execFile) is invoked with a timeout, three different main cases may occur (there are some variations, though):

1. Process is exited and the socket is closed before timeout expires, that is, the process close event is emitted (happy path)
2. Process is still running when the timeout expires
3. Process is exited but the socket is still open when the timeout expires, that is, process exit event is emitted, but not the process close event.

The problem here is that , from the point of view of the callback, there's no difference in those three cases, which is actually a bug imho.
you must check some other child process attributes before assuming that the callback stdout argument is reliable. For instance process.killed and process.stdout.closed.

And I couldn't find any mentions on the docs.

I don't think anymore that a "timeout" event is the correct behavior, I'm prone to consider a not empty callback error argument as the proper behaviour in 2. and 3. Otherwise, checking the aforementioned attributes in the callback body should do as a workaround

@bnoordhuis, what is you opinion?

Cheers

[bnoordhuis]

I expect most programmers don't consider (2) and (3) to be materially different. Their programs will wait for the 'exit' event and inspect the code and signal arguments, to see if the child terminated normally or not.

If you don't know what the subprocess you're spawning is really doing, then yeah, you'll need to handle all contingencies. Node gives you all the tools, you just need to be mindful to use them correctly.

I don't think node is doing anything wrong or that the documentation is incomplete. The worst you can say is it doesn't include a primer on the Unix or Windows process model but I don't think it's our job to educate people on that.

[bunglegrind]

Well, if the docs are saying that exec & co. are a handful of synchronous and asynchronous alternatives but I still must attach listeners (or inspect attributes) to the child process and its context, what's the point in having a callback at all. I wouldn't call it "alternative". IMHO, the callback arguments should have all the relevant information about the process, which is not true.

Anyway, thanks again for your suggestions. I 'm closing the discussion and mark it as solved.