Replace process_stubs.c with pure OCaml implementation for OCaml 5 multi-domain safety

[Copilot]

The persistent domainslib pool (run_with_pool from sctx.pool) kept idle domain workers alive during eval interpretation, causing subprocess/IPC failures in CI (connect_stdin hangs, sourcemaps EOF, TestUnicode IPC corruption). Investigation revealed the root cause is in process_stubs.c, not the pool management.

Root Cause

process_stubs.c (the C implementation of eval's sys.io.Process) was written for single-threaded OCaml 4 and is unsafe in OCaml 5's multi-domain context:

1. GC stop-the-world stall (all platforms): Blocking operations (process_exit → waitpid/WaitForSingleObject, process_stdout_read → read/ReadFile, etc.) ran without caml_enter_blocking_section. The calling domain holds its runtime lock during the entire blocking call, no backup thread runs to handle GC interrupts, and when worker domains need major GC (stop-the-world), the main domain can't respond — all worker domains stall waiting for GC, causing pipe I/O timing issues and data corruption.

2. fork() in multi-threaded process (Unix): process_run called fork() directly, bypassing OCaml 5's Unix.fork which explicitly refuses to fork in multi-domain programs (caml_domain_is_multicore() check in OCaml 5.3's fork.c). On Linux, fork() triggers glibc's pthread_atfork handlers which lock malloc arenas — if a worker thread holds any internal lock at that moment, fork deadlocks (explains the connect_stdin 20min hang with orphan processes).

Changes

Replaces process_stubs.c (685 lines of C) entirely with a pure OCaml process.ml (~130 lines) using Unix.create_process, which is domain-safe under OCaml 5.

• libs/extc/process.ml: Rewritten from C external declarations to a pure OCaml implementation using Unix.create_process, Unix.pipe, Unix.read/Unix.write, Unix.waitpid, Unix.kill. Maintains the same API (run, read_stdout, read_stderr, write_stdin, close_stdin, exit, pid, close, kill). All Unix.Unix_error information is preserved: I/O operations include the error details (function, arg, message) in Failure exceptions for eval compatibility, and run writes the error message to the stderr pipe when process creation fails (e.g. command not found), returning exit code 127 instead of throwing — matching the old fork+exec behavior. The kill function includes a pid > 0 guard as defense-in-depth against signaling process group 0.
• libs/extc/process_stubs.c: Deleted.
• libs/extc/dune: Removed foreign_stubs for process_stubs, added unix library dependency.
• src/compiler/pipeThings.ml: Refactored run_command to use Process.run instead of duplicating pipe+spawn logic.
• tests/misc/projects/process-nonexistent/: New misc test verifying that non-existent commands don't throw exceptions, produce empty stdout, non-empty stderr (with error info), and exit code 127 — with cross-platform output assertion via .stdout file.
• evalStdLib.ml/evalValue.ml/Process.hx: No changes — the Process module API is unchanged.
• filters.ml: No changes — run_with_pool and parallel analyzer are kept as-is.
• Data race fixes and the Parallel.enable := false removal from reset are preserved from the base branch.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Simn]

You can't be serious...

[Simn]

@copilot Let me set a few constraints here:

1. We want to keep using run_with_pool and not create new pools.
2. The analyzer has been considered safe by the most recent investigation, so we want to keep parallelism enabled for it.
3. Your actual task is to investigate WHY there are problems because there shouldn't be any. Please analyze properly what actually might be happening and why it goes wrong.

[Copilot]

@copilot Let me set a few constraints here:

1. We want to keep using run_with_pool and not create new pools.
   ...

Reverted filters.ml to keep run_with_pool and parallel analyzer. The root cause is in process_stubs.c, not the pool management.

Two issues found:

1. GC stop-the-world stall: All blocking process operations (waitpid, ReadFile, read, WaitForSingleObject, WriteFile, write) ran without caml_enter_blocking_section. When worker domains need major GC, they send stop-the-world interrupts to all domains, but the main domain can't respond while blocked in C code → all workers stall waiting for GC.

2. fork() in multi-domain process: process_run called fork() directly, bypassing OCaml 5's Unix.fork which explicitly refuses to fork when domains are spawned (see caml_domain_is_multicore() check in OCaml 5.3's fork.c). On Linux, fork() triggers glibc's pthread_atfork handlers which lock malloc arenas — if a worker thread holds one → fork deadlock (explains the connect_stdin 20min hang with orphan processes).

Fix in commit b34f405:

• Added caml_enter_blocking_section/caml_leave_blocking_section around all blocking operations, using temporary C buffers
• Replaced fork()+exec() with posix_spawn() on Unix (uses clone(CLONE_VM|CLONE_VFORK) internally, avoiding the multi-threaded fork problem)

[Simn]

@copilot This makes me wonder if we can just get rid of process_stubs.c entirely and have a proper OCaml implementation. There appears to be a lot of overlap with run_command, so I just went ahead and started refactoring that. Allow me to throw some OCaml code at you:

module HaxeProcess = struct
	type t = {
		pid : int;
		pin : out_channel;
		pout : in_channel;
		perr : in_channel;
		mutable stop_stdin : bool;
	}

	let create comm cmd =
		let (child_stdin_r, child_stdin_w) = Unix.pipe ~cloexec:true () in
		let (child_stdout_r, child_stdout_w) = Unix.pipe ~cloexec:true () in
		let (child_stderr_r, child_stderr_w) = Unix.pipe ~cloexec:true () in
		let shell, args =
			if Sys.win32 then
				"cmd.exe", [|"cmd.exe"; "/c"; cmd|]
			else
				"/bin/sh", [|"/bin/sh"; "-c"; cmd|]
		in
		let pid = Unix.create_process_env shell args (Unix.environment()) child_stdin_r child_stdout_w child_stderr_w in
		Unix.close child_stdin_r;
		Unix.close child_stdout_w;
		Unix.close child_stderr_w;
		let pin = Unix.out_channel_of_descr child_stdin_w in
		let pout = Unix.in_channel_of_descr child_stdout_r in
		let perr = Unix.in_channel_of_descr child_stderr_r in
		{
			pid;
			pin;
			pout;
			perr;
			stop_stdin = false;
		}

	let connect_stdin proc stdin_pipe =
		let stdin_fd = Unix.descr_of_in_channel stdin_pipe in
		Thread.create (fun () ->
			let buf = Bytes.create 1024 in
			(try while not proc.stop_stdin do
				let readable, _, _ = Unix.select [stdin_fd] [] [] 0.05 in
				if readable <> [] then begin
					let i = Unix.read stdin_fd buf 0 1024 in
					if i = 0 then raise Exit;
					output proc.pin buf 0 i;
					flush proc.pin
				end
			done with _ -> ());
			close_out_noerr proc.pin
		) ()

	let register_stdout proc f =
		let buf = Bytes.create 1024 in
		Thread.create (fun() -> read_content proc.pout buf f) ()

	let register_stderr proc f =
		let buf = Bytes.create 1024 in
		Thread.create (fun() -> read_content proc.perr buf f) ()
end

(** Runs a shell command in server mode, forwarding stdin from the client
	and capturing stdout/stderr through the socket protocol.
	Uses [Unix.create_process_env] (not [Sys.command]) so we can connect
	the child's stdin to the client's forwarded data and properly signal
	EOF when the client closes its end. *)
let run_command comm cmd =
	let open HaxeProcess in
	let {pid; pin; pout; perr;} as proc = HaxeProcess.create comm cmd in
	let tin = match comm.stdin with
		| Some stdin_pipe ->
			Some (HaxeProcess.connect_stdin proc stdin_pipe)
		| None ->
			close_out_noerr pin;
			None
	in
	let tout = HaxeProcess.register_stdout proc comm.write_out in
	let terr = HaxeProcess.register_stderr proc comm.write_err in
	Thread.join tout;
	Thread.join terr;
	close_in_noerr pout;
	close_in_noerr perr;
	let _, status = Unix.waitpid [] pid in
	proc.stop_stdin <- true;
	(match tin with Some t -> Thread.join t | None -> ());
	match status with Unix.WEXITED c | Unix.WSIGNALED c | Unix.WSTOPPED c -> c

Let's try to generalize this to the point where we can use it for both run_command and as a replacement for eval's StdNativeProcess. Note that this native process it not actually part of the public API, so we are free to entirely remove it. The only public API is sys.io.Process itself which should not change its interface. We could move its implementation completely into the compiler though.

Please see if that makes sense!

[Copilot]

@copilot This makes me wonder if we can just get rid of process_stubs.c entirely and have a proper OCaml implementation. There appears to be a lot of overlap with run_command, so I...

Implemented in 2dc8ae7 and 6c8f0c0. Replaced process_stubs.c entirely with a pure OCaml process.ml (~85 lines vs 685 lines of C) using Unix.create_process.

Changes:

• libs/extc/process.ml: Pure OCaml implementation, same API (run, read_stdout, read_stderr, write_stdin, close_stdin, exit, pid, close, kill)
• libs/extc/process_stubs.c: Deleted
• src/compiler/pipeThings.ml: run_command now uses Process.run instead of duplicating the pipe+spawn logic
• evalStdLib.ml/evalValue.ml/Process.hx: No changes needed — API is unchanged

One thing I had to handle: Unix.create_process raises Unix.Unix_error instead of Failure, but eval's catch(e:Dynamic) only catches Failure. So all Unix.Unix_error exceptions are converted to Failure in the Process module.

Full RunCi macro tests pass (unit, misc, display, sourcemaps, null-safety, sys, connect_stdin, threads, third-party).

[Simn]

This is a nice Claude redemption arc after that initial failure. Getting rid of that smelly old C code is really good, and if it solves parallelism problems that's even better. I'm not convinced that it's really the root cause but I did also notice the relationship to processes in particular.

[Simn]

Also that Eof failure is a pre-existing problem: because the class didn't extend Exception, a catch(e:Eof) wouldn't work after Exception.caught wrapped it in ValueException. It probably rarely ever surfaced because we didn't close our pipes properly before.

[Simn]

@copilot The last CI failure is now the PHP tests on Mac. The offending code is this:

	static public function commandResult(cmd:String, args:Array<String>):{
		stdout:String,
		stderr:String,
		exitCode:Int
	} {
		final p = new Process(cmd, args);
		final out = {
			stdout: p.stdout.readAll().toString(),
			stderr: p.stderr.readAll().toString(),
			exitCode: p.exitCode()
		}
		p.close();
		return out;
	}

final phpCmd = commandResult("php", ["-v"]);

php isn't installed on Mac so this fails, which is expected, but what is unexpected is that it throws an exception:

Going to test: [php]
test php
Changing directory to /Users/runner/work/haxe/haxe/tests/unit
Command: haxelib [git,utest,https://github.com/haxe-utest/utest]
Installing utest from https://github.com/haxe-utest/utest
Cloning Git from https://github.com/haxe-utest/utest
Syncing submodules for Git
Downloading/updating submodules for Git
  Current version is now git
Done
Command exited with 0 in 2s: haxelib [git,utest,https://github.com/haxe-utest/utest]
Error:  RunCi.hx

    | Uncaught exception create_process(php): No such file or directory

Error: Process completed with exit code 1.

We're supposed to catch this in evalStdLib:3417 via let proc = process_catch (fun () -> Process.run cmd args) () in, but apparently we don't. Please see if you can reproduce that and figure out what's going on, then let's add a misc test to test that we don't get exceptions from non-existent commands and can instead test for an exit code != 0.