proc_exit and traps do not stop thread executing blocking instructions

[eloparco]

With #1889, proc_exit and traps are properly propagated to all the threads in the process. Once a thread traps or calls proc_exit, all the other threads are stopped.

This is not true if the thread is executing a blocking operation (e.g. sleep, atomic.wait). In that case, the blocking operation is not interrupted by the exception propagation process.

Partial discussion here: #1869 (comment)

[eloparco]

@wenyongh @xujuntwt95329 @yamt after the initial discussion in #1869 (comment), any suggestion on the possible implementation of such a mechanism?
Initially I was thinking of using sigsetjmp/siglongjmp (to resume from here), but then I would need to keep track of sigjmp_bufs for each thread and also I don't know how portable it is for other non-linux platforms.
Otherwise, I'm not sure on what we can do in the signal handlers to stop the threads gracefully (ideally we'd like move to the next instruction after sleep/atomic_wait so that the exception flag can be read to stop the execution).

[loganek]

I guess another approach would be to make all operations non-blocking in the host, but I'm not sure how much effort would that require (e.g. for WASI calls). Also, it might be problematic for user-defined native methods, so I think signals might be the way to go.

// edit
We just had a discussion with @eloparco about the approach, he'll update the ticket with the notes.

[eloparco]

Two approaches:

• make all operations non-blocking in the runtime
  [-] additional work every time a new blocking function is introduced
  [-] it doesn't work with user-defined functions
  [+] it integrates well with current exception handling mechanism
  progress:
  • fixed terminating stale threads on trap/proc_exit #1929 (atomic.wait)
  • modified poll_oneoff to make it interruptible #1951 (poll_oneoff)
• through signal handling: when propagating the exception to other threads, a signal is sent to those threads to stop their execution and return to the runtime; it requires the usage of setjmp and longjmp, similar to what is done with HW bound checks.
  [-] portability problem because of setjmp/longjmp (e.g. not available on Windows)
  [+] no additional work required if new blocking functions are introduced
  [+] works well with user-defined functions
  progress:
  • feat: interrupt blocking instructions using signals #1946
  • Refactor interrupting blocking threads #1948
  • TODO: resource deallocation after exception
  • Implement async termination of blocking thread #2516

[yamt]

it wouldn't work with user-defined functions

in either approaches, i guess user-defined functions need to be adapted anyway.
eg. resource cleanup on termination.

maybe pthread cancellation is another alternative to consider for posix-like platforms?

[eloparco]

in either approaches, i guess user-defined functions need to be adapted anyway.
eg. resource cleanup on termination.

Why is that? If we use signals, the thread running the user-defined function would be stopped.

maybe pthread cancellation is another alternative to consider for posix-like platforms?

But then we wouldn't return to the runtime after stopping a (possibly blocking) instruction. It may even be fine for spawn threads, but what if it's the main thread?

[loganek]

I wonder what's the expected behavior of proc_exit? From what I see in the implementations, it just kills the process and doesn't allow for any resource cleanup (unlike C exit, which does a bit more before exiting, e.g. calling atexit callbacks). If that's the case, I don't think we have to worry about resource cleanup?

[yamt]

in either approaches, i guess user-defined functions need to be adapted anyway.
eg. resource cleanup on termination.

Why is that? If we use signals, the thread running the user-defined function would be stopped.

for example, if a user-defined function uses malloc(), it somehow needs to catch the signal to free() it.

maybe pthread cancellation is another alternative to consider for posix-like platforms?

But then we wouldn't return to the runtime after stopping a (possibly blocking) instruction. It may even be fine for spawn threads, but what if it's the main thread?

sure. probably you're right it's even trickier than signals to use for this purpose.

[eloparco]

for example, if a user-defined function uses malloc(), it somehow needs to catch the signal to free() it.

how would you address that? not sure if there's something we can do once the user-defined function is interrupted

[yamt]

for example, if a user-defined function uses malloc(), it somehow needs to catch the signal to free() it.

how would you address that? not sure if there's something we can do once the user-defined function is interrupted

for example, we can make the user-defined functions deal with interruption by themselves.

[loganek]

for example, if a user-defined function uses malloc(), it somehow needs to catch the signal to free() it.

how would you address that? not sure if there's something we can do once the user-defined function is interrupted

for example, we can make the user-defined functions deal with interruption by themselves.

I wonder if we actually have to provide that capability; whereas C allows registering callback for exit() (using atexit), I don't think there's equivalent for traps in either posix or C. If really needed, we can provide atexit-like functionality for both traps and proc_exit, for native user-defined functions, but I don't know how critical it is to have it right now.

[yamt]

for example, if a user-defined function uses malloc(), it somehow needs to catch the signal to free() it.

how would you address that? not sure if there's something we can do once the user-defined function is interrupted

for example, we can make the user-defined functions deal with interruption by themselves.

I wonder if we actually have to provide that capability; whereas C allows registering callback for exit() (using atexit), I don't think there's equivalent for traps in either posix or C. If really needed, we can provide atexit-like functionality for both traps and proc_exit, for native user-defined functions, but I don't know how critical it is to have it right now.

for embedders like iwasm command, maybe it's enough to call host exit() and let the host os terminate threads.

on the other hand, wamr, as a library, should provide a way to clean up associated resources.
termination of wasm instances doesn't necessarily involve termination of the host process.

[loganek]

for example, if a user-defined function uses malloc(), it somehow needs to catch the signal to free() it.

how would you address that? not sure if there's something we can do once the user-defined function is interrupted

for example, we can make the user-defined functions deal with interruption by themselves.

I wonder if we actually have to provide that capability; whereas C allows registering callback for exit() (using atexit), I don't think there's equivalent for traps in either posix or C. If really needed, we can provide atexit-like functionality for both traps and proc_exit, for native user-defined functions, but I don't know how critical it is to have it right now.

for embedders like iwasm command, maybe it's enough to call host exit() and let the host os terminate threads.

on the other hand, wamr, as a library, should provide a way to clean up associated resources. termination of wasm instances doesn't necessarily involve termination of the host process.

I didn't mean to call exit() and close the host. What I meant is that maybe we don't have to worry about user-defined functions, and we can use signals and interruptions. Embedders, if needed, can handle resource cleanup within the process after the runtime finalizes.

[yamt]

for example, if a user-defined function uses malloc(), it somehow needs to catch the signal to free() it.

how would you address that? not sure if there's something we can do once the user-defined function is interrupted

for example, we can make the user-defined functions deal with interruption by themselves.

I wonder if we actually have to provide that capability; whereas C allows registering callback for exit() (using atexit), I don't think there's equivalent for traps in either posix or C. If really needed, we can provide atexit-like functionality for both traps and proc_exit, for native user-defined functions, but I don't know how critical it is to have it right now.

for embedders like iwasm command, maybe it's enough to call host exit() and let the host os terminate threads.
on the other hand, wamr, as a library, should provide a way to clean up associated resources. termination of wasm instances doesn't necessarily involve termination of the host process.

I didn't mean to call exit() and close the host. What I meant is that maybe we don't have to worry about user-defined functions, and we can use signals and interruptions. Embedders, if needed, can handle resource cleanup within the process after the runtime finalizes.

it's certainly possible to code host functions that way.

however, it isn't how we code host functions right now. (including the ones within wamr tree like wasi)
so, most of host functions need to be adapted anyway.
also, my honest impression is that the approach is too difficult for average programmers around me.

in other words, i prefer "if a host function doesn't implement async termination properly, it can't be async terminated (eg. possibly block forever)" over "if a host function doesn't implement async termination properly, it might cause problems like resource leak on async termination."