Process handling improvements (ProcessBuilder)

[dmlloyd]

On chat I mentioned looking through and auditing usages of ProcessBuilder and Runtime.exec(*) after fixing a recent hang which involved subprocess launching. Long story short, these APIs are misused in various ways almost 100% of the time, some of which may cause (and probably have already caused) hard-to-diagnose hangs. It looks like we have around 80 usages just in the Quarkus codebase itself, not counting Quarkiverse extensions.

I started gathering up initial requirements for a foolproof (or at least "fool-resistant") process API that we could enforce the usage of; things like this:

• Able to capture process error output so that exceptions can provide some kind of context (we'd nearly always want this AFAICT)
• Ability to log error output in cases where there is error output but the process was "successful"
• Handles exit codes in a customizable way (for example, non-zero or specific exit codes throw an exception, certain exit codes are treated as process output, etc.)
• Designed to prevent deadlocking (separate threads for input, output, error, and timeout handling)
• Robust cleanup (make it impossible to "leak" processes, have soft/hard timeouts for process cleanup, recursive subprocess cleanup, cleanup-on-exit)
• Customizable input/output handling (produce/consume strings, discard output, empty input, custom stream or reader/writer handlers; make simple usages easy, and make complex usages also easy)
• Reasonable defaults (discard output, empty input, reasonable timeouts, process is expected to do some work and then immediately exit by default)
• Support "run command and get result" use case with minimal overhead
• Support "run this long-running process while I do other stuff" use case
• Protects against memory-bombing (e.g. get list of strings but each string is 2GiB and there are 150,000 of them)
• Make it easy to launch our own java
• Allow a means for detection and protection against unexpectedly hanging processes (e.g. a trivial process unexpectedly stops accepting input because of a system issue) while also supporting slow processes (e.g. a process accepts a line of input and does a lot of thinking before accepting more input)
• Support pipelining of processes (possibly in the future)

I've looked at a few process exec alternatives (including zt-exec) but they tend to all fall short at the "make it impossible to leak" stage. So, in typical David Lloyd fashion I started prototyping an alternative.

I'm thinking of a fluent process builder API, which could be used something like this (just a few examples):

// run "date" command and get result as string
ProcessResult<String> dateResult = ProcessBuilder.newBuilder()
                                       .command("date")
                                       .output() // configure process output handling
                                       .asString(8192) // 8KiB or less, else throw
                                       .run();
// throw an exception
ProcessBuilder.newBuilder().command("false").run();

// cat some strings
ProcessResult<List<String>> catResult = ProcessBuilder.newBuilder()
                                             .command("cat")
                                             .input() // configure process input handling
                                             .fromStrings("Foo", "Bar", "Baz")
                                             .output() // configure process output handling
                                             .asStringList(16, 8192) // 16 strings max, else throw
                                             .run();

// execute a long-running process
CountDownLatch latch = new CountDownLatch(1);
publishToOtherThreads(latch);
ProcessBuilder.newBuilder().command("some-kinda-daemon")
              .whileRunning(handle -> {
                  System.out.printf("Subprocess initiated with PID %d%n", handle.pid());
                  // keep process alive until it's OK to exit
                  latch.await();
                  System.out.printf("Subprocess with PID %d will now be asked to exit%n", handle.pid());
              }).run(); // blocks until subprocess exits, throws exception if exit is abnormal

// pipeline - a really slow way to sort & unique strings
ProcessResult<List<String>> result = ProcessBuilder.newBuilder()
                   .command("sort")
                   .input()
                   .fromStrings("Jeff", "Jimmy", "Jimi", "George", "Yngwie", "Alex", "Steve", "Jeff")
                   .output()
                   .pipeline()
                   .command("uniq")
                   .output()
                   .asStringList(16, 8192)
                   .run();

I plan to create this API, plus a few process handling utilities, in smallrye-common as smallrye-common-process. If it is successful, once it is released, I would start migrating all existing usages of ProcessBuilder and Runtime#exec to use the new APi. I would also recommend that we then forbid usages of the ProcessBuilder and Runtime#exec APIs as a matter of policy.

Let me know if you think this is a good (or not-so-good) idea/plan, or if you have any feedback on the requirements I've listed above (particularly the blocking philosophy). In particular, please let me know if there's a requirement that you think I have missed. I did kick around the idea of fancier environment and command argument editing. But since those features are relatively tangential to the core need, I didn't pursue that immediately (but perhaps someone has some specific opinions/ideas to share).

Concurrently to the initial discussion, I'll start the work itself by pushing up a draft PR to stub out the API, to make sure it makes sense and give people a chance to complain about it before I implement it. Comments on detailed aspects of the API itself will be best directed there at that time.

[dmlloyd]

The draft API may be found here: smallrye/smallrye-common#418

[geoand]

Nice!

One small complaint I have is that the asString and friends name(s) is misleading (at least to me)

[dmlloyd]

Naming is always the hardest part. I'm still shuffling things around to see if the implementation can actually work, but feel free to give some suggestions if you have ideas.

[geoand]

I'm notoriously terrible at naming, but if I have any non-horrible ideas, I'll throw them out there

[dmlloyd]

The API first draft is done with some naming adjustments (among many other improvements). See: smallrye/smallrye-common#418

[cescoffier]

Awesome, looks great! Definitely something useful that would reduce the Quarkus code base (and fix several bugs)

[starksm64]

I took a look at the proposed API based on the work I had to do creating an arquillian appclient container launcher for the jakartaee tck. It covers everything I would need to use this.

…

On Tue, Jun 10, 2025 at 6:57 AM Clement Escoffier ***@***.***> wrote: Awesome, looks great! Definitely something useful that would reduce the Quarkus code base (and fix several bugs) — Reply to this email directly, view it on GitHub <#48223 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACRDMQ7YJFEAEFBY5Z37ZD3CZXVXAVCNFSM6AAAAAB6SWAPT6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNBRG4YDGMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[maxandersen]

You are hitting a lot of the same points made me go do https://github.com/jbangdev/jbang-jash (which borrows its fluent api from a now old/defunt fluentprocess api).

I like your output/consuming setup - to define some explicit limits. I assume you can also make them unbounded?

I'll definitely give it a go and happy to make jbang-jash use it instead (or completely replace it), as my main need for jash is the easy shell execution and piping.

Is shell execution something you would be interested in enabling in smallrye-common-process - on first glance I think it would be hard to make it smooth as a layer on top but curious what you think here?

[dmlloyd]

You can process output without bounds, but if you slurp it into a string or list of strings, then you must define limits.

As we briefly discussed earlier, I do want to allow execution of .cmd and .bat files (possibly in the future). We can work together on figuring this out if you want. I just put a "TODO" in the code for it. :-)

[dmlloyd]

The next steps are as follows:

• Review documentation (anyone) Process API docs and a small behavior update smallrye/smallrye-common#426
• Async execution Add async process start smallrye/smallrye-common#427
• Local Quarkus full test suite run
• Move PR Conversion to SmallRye Common Process #48354 to non-draft for CI
• Release SmallRye Common 2.13.0
• Final rebase of Conversion to SmallRye Common Process #48354
• Final merge of Conversion to SmallRye Common Process #48354

Deferred:

• Windows batch/ps1 support Add batch/powershell launching support smallrye/smallrye-common#428

[dmlloyd]

One thing that is missing that I would like to add is asynchronous process start. The question is, which API option?

1. Uni<O> result = pb.runAsync();
2. Future<O> result = pb.runAsync();

For option 1, we would need to add a dependency to Mutiny (obviously) which could become circular if we are not careful. Option 2 is obviously less powerful but maybe it's good enough. WDYT?

[geoand]

I think that 2 is fine as it is easy to adapt CompletableFuture to anything else