keeping node alive while workers are working

[bakkot]

Version of emscripten/emsdk:

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.73 (ac676d5e437525d15df5fd46bc2c208ec6d376a3)
clang version 20.0.0git (https:/github.com/llvm/llvm-project 1d810ece2b2c8fab77720493864257f0ea3336a9)

Failing command line in full:

emcc  --pre-js pre.js -pthread -s MODULARIZE=1 -s EXPORTED_FUNCTIONS="_spawn" main.cc && node run.mjs

---

I have a library which is intended to be driven from JS (i.e., no main loop in C, and using -s MODULARIZE=1).

I'm trying to spawn a thread, do some work, and then get the result back in the main thread. I have the following three bits of code, with the first two being the inputs to emcc and the third being the driver:

// pre.js
let spawnTimeout;
let resolve;

Module.async_sum = () => {
  _spawn();
  return new Promise((res) => {
    resolve = res;
  });
};

// main.cc
#include <thread>
#include <stdio.h>
#include <emscripten.h>

extern "C" void spawn() {
  printf("spawning\n");
  std::thread t([] {
    MAIN_THREAD_ASYNC_EM_ASM({
      clearTimeout(spawnTimeout);
    });

    printf("spawned\n");

    double v = 1.0;
    for (int i = 1; i < 1e8; ++i) {
      v += 1.0 / i;
    }
    printf("sum: %f\n", v);
    MAIN_THREAD_ASYNC_EM_ASM({
      resolve($0);
    }, v);
  });
  t.detach();
  EM_ASM({
    spawnTimeout = setTimeout(() => {}, 5000);
  });
}

// run.mjs
import { default as init } from './a.out.js';

let x = await init();

console.log(await x.async_sum());

Running node run.mjs prints

spawning
spawned
Warning: Detected unsettled top-level await at file:///Users/kevin/code/emscripten-thread-test/run.mjs:5
console.log(await x.async_sum());

and then exits, without printing the sum or giving me the result.

I was really hoping it would instead do the work and give me the result. That worked prior to #19073 (or to be more precise, it worked in version 3.1.15; I'm assuming that PR is the relevant change). I assume this is because node exits when the main thread has no tasks queued and the workers are all unref'd.

If I comment out the "clearTimeout" line it works, but then of course I need to wait for the timeout. I can move the clearTimeout later, but then if the worker crashes or is killed (as sometimes happens) it will again need to wait for the timeout.

Is there some way I can get it to stay alive? Maybe keep the workers .ref'd while they're actually doing work?

[kripken]

ccing people from that PR: @kleisauke @sbc100

[kleisauke]

Ah, this is indeed due to PR #19073. Calling the internal _emscripten_thread_set_strongref() function would prevent Node.js from exiting.

How about exposing this function in Emscripten's public API? That way, you could do:

--- a/main.cc
+++ b/main.cc
@@ -2,14 +2,11 @@
 #include <thread>
 #include <stdio.h>
 #include <emscripten.h>
+#include <emscripten/threading.h>
 
 extern "C" void spawn() {
     printf("spawning\n");
     std::thread t([] {
-      MAIN_THREAD_ASYNC_EM_ASM({
-        clearTimeout(spawnTimeout);
-      });
-
       printf("spawned\n");
 
       double v = 1.0;
@@ -21,8 +18,6 @@ extern "C" void spawn() {
         resolve($0);
       }, v);
     });
+    emscripten_thread_set_strongref(t.native_handle());
     t.detach();
-    EM_ASM({
-      spawnTimeout = setTimeout(() => {}, 5000);
-    });
 }

I just opened PR #23152 for this.

[bakkot]

Nice! Is there an easy way to try the PR locally? I'd be happy to test it for my use case if so.

[kleisauke]

@bakkot To test that PR with the tip-of-tree compilers, you could use:

$ cd emsdk/
$ ./emsdk install tot emscripten-main-64bit --override-repository emscripten-main-64bit@https://github.com/kleisauke/emscripten/tree/node-strong-ref-helper-public
$ ./emsdk activate tot emscripten-main-64bit
$ source ./emsdk_env.sh

# Verify commit
$ git -C emscripten/main rev-parse HEAD
cfe6160f56273902cf9e52f875ce4083d967c35a

# Invalidate cache
$ emcc --clear-cache

Alternatively, you can test that private function using:

--- a/main.cc
+++ b/main.cc
@@ -3,13 +3,11 @@
 #include <stdio.h>
 #include <emscripten.h>
 
+extern "C" void _emscripten_thread_set_strongref(pthread_t thread);
+
 extern "C" void spawn() {
     printf("spawning\n");
     std::thread t([] {
-      MAIN_THREAD_ASYNC_EM_ASM({
-        clearTimeout(spawnTimeout);
-      });
-
       printf("spawned\n");
 
       double v = 1.0;
@@ -21,8 +19,6 @@ extern "C" void spawn() {
         resolve($0);
       }, v);
     });
+    _emscripten_thread_set_strongref(t.native_handle());
     t.detach();
-    EM_ASM({
-      spawnTimeout = setTimeout(() => {}, 5000);
-    });
 }

However, this causes Node.js to hang indefinitely due the && PROXY_TO_PTHREAD guard here:

emscripten/src/library_pthread.js

Line 191 in 1092ec3

#if ENVIRONMENT_MAY_BE_NODE && PROXY_TO_PTHREAD

(i.e. it doesn't .unref() the worker when it's finished)

To fix this, you can temporarily remove that conditional in $EMSDK/upstream/emscripten/src/library_pthread.js (as done in the referenced PR).

[sbc100]

I would hope we could make fix this without exposing a custom API like that, but maybe there is no other option.

Also I wonder if that API should be in the form a pthread_attr_xxx call? (Like we have for emscripten_pthread_attr_settransferredcanvases)?

[sbc100]

Also, it makes me wonder what the corresponding technique would be in the native world. If you return from main with pthreads running does you app always exit? Is there any way to prevent that and keep it alive for async signals from its worker threads?

[sbc100]

Also, can you explain a little more why don't want to use the emscripten_set_timeout / emscripten_clear_timeout approach?

It seems that you could store the result of emscripten_set_timeout globally and then call emscripten_clear_timeout after you call resolve? What is wrong with this approach?

You say "then if the worker crashes or is killed (as sometimes happens) it will again need to wait for the timeout." couldn't you clear the timeout in those cases? Also if there is a possiblity that the thread dies or is killed wouldn't you want a settimeout to fire at some point so you can report that error (e.g. "thread unresponsive!")

[sbc100]

It seems fairly common to use setTimeout / setInterval to keep a node process alive longer than it normally would be: https://stackoverflow.com/questions/23622051/how-to-forcibly-keep-a-node-js-process-from-terminating

[bakkot]

Also, can you explain a little more why don't want to use the emscripten_set_timeout / emscripten_clear_timeout approach?

It seems that you could store the result of emscripten_set_timeout globally and then call emscripten_clear_timeout after you call resolve? What is wrong with this approach?

Because then the timer won't get cleared if the thread dies before resolving. And in my application the timer needs to be extremely long, because the work done in the thread may take a very long time (hours).

You say "then if the worker crashes or is killed (as sometimes happens) it will again need to wait for the timeout." couldn't you clear the timeout in those cases?

If there's an easy way to do that, sure, that would work too. I didn't see one offhand.

Also if there is a possiblity that the thread dies or is killed wouldn't you want a settimeout to fire at some point so you can report that error (e.g. "thread unresponsive!")

I have a have a process.on('exit', handler) which prints an error when the main thread exits while something is waiting on work from a worker thread. A timeout isn't suitable for this because the timer would need to be unreasonably long.

[sbc100]

Also, can you explain a little more why don't want to use the emscripten_set_timeout / emscripten_clear_timeout approach?
It seems that you could store the result of emscripten_set_timeout globally and then call emscripten_clear_timeout after you call resolve? What is wrong with this approach?

Because then the timer won't get cleared if the thread dies before resolving. And in my application the timer needs to be extremely long, because the work done in the thread may take a very long time (hours).

But that setInterval can go on a long a you want.. forever if you want.

How does your application detect the thread "dying"?

You say "then if the worker crashes or is killed (as sometimes happens) it will again need to wait for the timeout." couldn't you clear the timeout in those cases?

If there's an easy way to do that, sure, that would work too. I didn't see one offhand.

Also if there is a possiblity that the thread dies or is killed wouldn't you want a settimeout to fire at some point so you can report that error (e.g. "thread unresponsive!")

I have a have a process.on('exit', handler) which prints an error when the main thread exits while something is waiting on work from a worker thread. A timeout isn't suitable for this because the timer would need to be unreasonably long.

If you have a setInterval or setTimeout going on the main thread then the process.on('exit', handler) should not be called.

You would do something like this:

var intervalID = setInterval(() => {
  if (workIsDone) {
    clearInterval(intervalId);
  }
}, 100);

You would also write the same thing in C/C++ using emscripten_set_interval/emscripten_clear_interval.

[bakkot]

But that setInterval can go on a long a you want.. forever if you want.

I'm fine with having a long timer. I just don't want to have a long timer if it's possible that no work is actually happening, which can happen if the worker thread exits without clearing the timer.

How does your application detect the thread "dying"?

The on('exit') handler, like I said.

If you have a setInterval or setTimeout going on the main thread then the process.on('exit', handler) should not be called.

... Right, which is part of why I'm trying to avoid a setInterval or setTimeout. If I have a timer going then the main thread will be kept alive. That is exactly what I want to not happen when the worker dies.

---

I am trying to find a solution that does not require me to have a long-running setInterval or setTimeout so that I do not run into the possibility of the main thread being kept alive while no work is happening. With the above PR, that works great: while the worker is actually doing work it is kept alive, and when it stops the main thread exits cleanly. If the worker dies then the main thread will still exit and will trigger my process.on('exit') handler so I know it crashed and can report the error. Trying to do the same thing with timers would require me to have some way of detecting the worker being killed other than the exit handler, and I don't know of a straightforward way to do that.

Also, timers are a bit of a hack here. Workers which are doing work you care about are supposed to be strongly ref'd in order to keep the main thread from exiting.

[sbc100]

Also, timers are a bit of a hack here. Workers which are doing work you care about are supposed to be strongly ref'd in order to keep the main thread from exiting.

While this is true for node Workers its not true for pthreads and POSIX. When then main thread exits here the semantics are that it brings the whole application down, including any threads that might be doing work. So we have conflicting requirements.

Perhaps using -sWASM_WORKERS would be a way around this since they don't have this semantics and we don't use unref there.