`SIGINT` is not handled properly when waiting on OpenBSD

[omegametabroccolo]

For example:

$ hurl.srv 127.0.0.1:8080
^C
# nothing happens

hurl.srv only terminates after responding to a final request, say hurl http://localhost:8080/get. pkilling it does kill it instantly too.

Here's what I could gather with gdb:

$ egdb hurl.srv
# [...]
(gdb) run 127.0.0.1:8080
Starting program: /home/user/.opam/miou/bin/hurl.srv 127.0.0.1:8080
^C[New thread 460857 of process 5725]
[New thread 431086 of process 5725]
[New thread 184835 of process 5725]

Thread 2 received signal SIGINT, Interrupt.
[Switching to thread 460857 of process 5725]
futex () at /tmp/-:2
warning: 2      /tmp/-: No such file or directory

(gdb) info threads 
  Id   Target Id                     Frame 
  1    thread 338321 of process 5725 _thread_sys_select () at /tmp/-:2
* 2    thread 460857 of process 5725 futex () at /tmp/-:2
  3    thread 431086 of process 5725 futex () at /tmp/-:2
  4    thread 184835 of process 5725 futex () at /tmp/-:2

(gdb) thread apply 1 bt

Thread 1 (thread 338321 of process 5725):
#0  _thread_sys_select () at /tmp/-:2
#1  0xc4987c91079e9fd8 in ?? ()
#2  0x000004def4d99ce2 in _libc_select_cancel (nfds=8, readfds=0x78483c5bd4a0, writefds=0x78483c5bd520, exceptfds=0x4def4d8fb2b <_thread_sys_select+27>, timeout=0x0) at /usr/src/lib/libc/sys/w_select.c:28
#3  0x000004dc1c6cc29d in caml_unix_select (readfds=5356133755072, writefds=1, exceptfds=1, timeout=<optimized out>) at select_unix.c:91
#4  <signal handler called>
#5  0x000004dc1c5874ab in camlMiou_unix.select_1434 ()
#6  0x000004dc1c5931de in camlMiou.unblock_awaits_with_system_events_2039 ()
#7  0x000004dc1c593583 in camlMiou.run_2061 ()
#8  0x000004dc1c598ec5 in camlMiou.run_inner_5845 ()
#9  0x000004dc1c5ed488 in camlCmdliner_term.fun_662 ()
#10 0x000004dc1c5f19f9 in camlCmdliner_eval.run_parser_589 ()
#11 0x000004dc1c5f2bed in camlCmdliner_eval.eval_value_inner_1728 ()
#12 0x000004dc1c5f33d0 in camlCmdliner_eval.eval_1479 ()
#13 0x000004dc1c43f99e in camlDune__exe__Srv.entry ()
#14 0x000004dc1c437487 in caml_startup.code_begin ()
#15 <signal handler called>
#16 0x000004dc1c7121cc in caml_startup_common (argv=0x78483c5bd7c8, pooling=<optimized out>) at runtime/startup_nat.c:127
#17 0x000004dc1c71227d in caml_startup_exn (argv=0x8) at runtime/startup_nat.c:134
#18 caml_startup (argv=0x8) at runtime/startup_nat.c:139
#19 caml_main (argv=0x8) at runtime/startup_nat.c:146
#20 0x000004dc1c6f4f90 in main (argc=<optimized out>, argv=0x78483c5bd4a0) at runtime/main.c:37

Setting a breakpoint on _thread_sys_select shows that it's called multiple times during startup, not switching the current thread, but the last call (the one actually waiting on requests) does the switch to another thread (Id 2 in the gdb trace). I get the same behavior with select (what Miou_unix is currently based on),poll/ppoll and my WIP kqueue implementation.

The OpenBSD pthreads(3) man page mentions:

Signals handlers are normally run on the stack of the currently executing thread.

I've tried doing the same thing on another machine under Linux, and when breaking during select the current thread is still the one running select (the gdb trace shows __pselect6), not a futex thread.

I'm not sure if that's a problem on our side or on OpenBSD's, I'm not even sure to understand everything going on here, but here it is.

[omegametabroccolo]

Removing the SIGINT handler from hurl.srv makes it behave like on Linux, dying instantly instead of waiting for a last request. So my problem has to do with hurl, not miou per se.

[omegametabroccolo]

Reopening this since there is now a small example that only involves Miou / Miou_unix and shows a difference of behavior between Linux and OpenBSD:

let rec loop () =
  print_endline "LOOP";
  Miou_unix.sleep 2.;
  loop ()
;;

let () =
  Miou_unix.run
  @@ fun () ->
  let fn _sigint =
    print_endline "CAUGHT SIGINT";
    flush_all ()
  in
  ignore @@ Miou.sys_signal Sys.sigint (Sys.Signal_handle fn);
  let domains = Miou.Domain.available () in
  let prm = Miou.async loop in
  ignore @@ Miou.parallel loop (List.init domains (Fun.const ()));
  Miou.await_exn prm

On Linux, every ^C is handled as soon as it's sent:

LOOP                               
LOOP           
LOOP           
LOOP           
LOOP           
LOOP           
LOOP                                                                                         
LOOP           
^CCAUGHT SIGINT
^CCAUGHT SIGINT
^CCAUGHT SIGINT
^CCAUGHT SIGINT
LOOP
LOOP
LOOP
LOOP
LOOP
LOOP
LOOP
LOOP
^CCAUGHT SIGINT
^CCAUGHT SIGINT
^CCAUGHT SIGINT                                                                              
^CCAUGHT SIGINT
^CCAUGHT SIGINT                               
^CCAUGHT SIGINT                               
^CCAUGHT SIGINT                               
^CCAUGHT SIGINT                               
LOOP
[...]

On OpenBSD, all SIGINTs are handled at the end of each loop:

LOOP                                
LOOP
LOOP
LOOP
^C^C^C^C^CLOOP
LOOP
CAUGHT SIGINT
CAUGHT SIGINT
CAUGHT SIGINT
CAUGHT SIGINT
CAUGHT SIGINT
LOOP
LOOP
^C^C^CLOOP
CAUGHT SIGINT
CAUGHT SIGINT
CAUGHT SIGINT
LOOP
LOOP
LOOP
^C^C^C^CLOOP
LOOP
CAUGHT SIGINT
CAUGHT SIGINT
CAUGHT SIGINT
CAUGHT SIGINT
LOOP
[...]

[dinosaure]

Can you try with #75 to see if Miou behaves differently on OpenBSD with poll(2)?

[omegametabroccolo]

I'm getting the same behaviors with poll(2) (both Linux and OpenBSD) and the WIP kqueue(2) implementation.

[omegametabroccolo]

In the kernel both select(2) and poll(2)/ppoll(2) are based on kqueue(2) internals (see /sys/kern/sys_generic.c), maybe that's relevant.

[omegametabroccolo]

I think I've narrowed it down to something between unblock_awaits_with_system_events and synchronize_dom0_tasks.

Running with this diff to miou.ml:

--- ./lib/miou.ml  Tue Sep 23 22:28:23 2025
+++ ../miou.test/lib/miou.ml       Tue Sep 23 22:37:15 2025
@@ -775,11 +775,15 @@
       add_into_domain domain (Domain_cancel (prm, bt))
 
   let synchronize_dom0_tasks pool =
+    Logs.debug (fun m -> m "Calling synchronize_dom0_tasks");
     transfer_dom0_tasks pool; transfer_dom0_signals pool
 
   let list_empty = []
 
   let unblock_awaits_with_system_events pool (domain : domain) =
+    Logs.debug (fun m ->
+        m "Calling unblock_awaits_with_system_events on dom %d"
+          (Stdlib.Domain.self () :> int));
     if (Stdlib.Domain.self () :> int) = 0 then synchronize_dom0_tasks pool;
     let block = Heapq.size domain.tasks = 0 in
     let cancelled =

On Linux, unblock_awaits_with_system_events is called from dom0 when the program receives a SIGINT, so it calls synchronize_dom0_tasks:

[...]
sigint: [7][DEBUG][unix] select(9.998188)
^Csigint: [0][DEBUG] [0] got a signal -6
sigint: [0][DEBUG][unix] [0] interrupted by the system
CAUGHT SIGINT
sigint: [0][DEBUG] [0] handles signal (-6) [0:10](23) and its continuation <resolved>
sigint: [0][DEBUG] Calling unblock_awaits_with_system_events on dom 0
sigint: [0][DEBUG] Calling synchronize_dom0_tasks
sigint: [0][DEBUG][unix] [0] [readers:2w, writers:2w, sleepers:1344w, revert:262w]
sigint: [0][DEBUG][unix] select(8.562773)

But on OpenBSD, it consistently calls unblock_awaits_with_system_events from another domain (on my 4-core machine):

[...]
sigint: [0][DEBUG][unix] select(9.999069)
^Csigint: [3][DEBUG] [3] got a signal -6
sigint: [3][DEBUG][unix] [3] interrupted by the system
sigint: [3][DEBUG] Calling unblock_awaits_with_system_events on dom 3
sigint: [3][DEBUG][unix] [3] [readers:2w, writers:2w, sleepers:688w, revert:262w]
sigint: [3][DEBUG][unix] select(8.753501)
sigint: [2][DEBUG] Calling unblock_awaits_with_system_events on dom 2
sigint: [2][DEBUG][unix] [2] [readers:2w, writers:2w, sleepers:679w, revert:262w]
sigint: [2][DEBUG][unix] select(0.000000)
sigint: [2][DEBUG] Calling unblock_awaits_with_system_events on dom 2
sigint: [2][DEBUG][unix] [2] [readers:2w, writers:2w, sleepers:706w, revert:262w]
sigint: [2][DEBUG][unix] select(9.999817)
sigint: [1][DEBUG] Calling unblock_awaits_with_system_events on dom 1
sigint: [1][DEBUG][unix] [1] [readers:2w, writers:2w, sleepers:690w, revert:262w]
sigint: [1][DEBUG][unix] select(0.000000)
sigint: [0][DEBUG] Calling unblock_awaits_with_system_events on dom 0
sigint: [1][DEBUG] Calling unblock_awaits_with_system_events on dom 1
CAUGHT SIGINT
sigint: [1][DEBUG][unix] [1] [readers:2w, writers:2w, sleepers:745w, revert:262w]
sigint: [1][DEBUG][unix] select(9.999775)
sigint: [0][DEBUG][unix] [0] [readers:2w, writers:2w, sleepers:745w, revert:262w]
sigint: [0][DEBUG][unix] select(0.000000)
sigint: [0][DEBUG] Calling unblock_awaits_with_system_events on dom 0
sigint: [0][DEBUG][unix] [0] [readers:2w, writers:2w, sleepers:737w, revert:262w]
sigint: [0][DEBUG][unix] select(0.000000)
sigint: [0][DEBUG] [0] handles signal (-6) [0:6](23) and its continuation <resolved>
sigint: [0][DEBUG] Calling unblock_awaits_with_system_events on dom 0
sigint: [0][DEBUG][unix] [0] [readers:2w, writers:2w, sleepers:703w, revert:262w]
sigint: [0][DEBUG][unix] select(9.999462)
sigint: [3][DEBUG] Calling unblock_awaits_with_system_events on dom 3
sigint: [3][DEBUG][unix] [3] [readers:2w, writers:2w, sleepers:694w, revert:262w]
sigint: [3][DEBUG][unix] select(0.000000)
sigint: [3][DEBUG] Calling unblock_awaits_with_system_events on dom 3
sigint: [3][DEBUG][unix] [3] [readers:2w, writers:2w, sleepers:708w, revert:262w]
sigint: [3][DEBUG][unix] select(9.999840)

Hence synchronize_dom0_tasks isn't called, transfer_dom0_signals isn't called, SIGINT isn't handled.

Logging Domain.self () from the signal handler shows that it's running on dom0 though.

[dinosaure]

#78 partially solves the initial issue: on OpenBSD, a domain can catch the signal where, on Linux, the signal is caught by the dom0. However, a situation exists when all domains are blocked to select(-1) and when we try to signal the program. It seems that even the OCaml signal handler is not called. Some details are explained here with codes.