Pre-proposal for a WG: Multi-modal dev services

[holly-cummins]

I don't think this is ready for a working group, not least because #49 is still going, but I want to get a few thoughts down in a place where we can add to them.

The problem

For users in restricted environments, such as banks, most dev services just don't work. A container runtime isn't allowed, which means any dev service that uses containers can't run. The only dev services that can work are ones which don't rely on containers, such as H2 and Amazon Lambda. This is a real shame, because dev services are fantastic.

We offer some workarounds to missing container runtimes, such as using Testcontainers cloud instead of a local container runtime, but no generalised solution.

The inverse problem that also turns out to be a problem

We also occasionally get requests to allow dev services that don't run in containers to run in containers. For example, sometimes it's desirable to have the AWS Lambda dev service run in a container so that it can access the docker shared network

Ideas for solutions

At the moment, every dev service has to be either "container" or "not-container," each of which has drawbacks. What if a dev service could provide multiple implementations, and the Quarkus runtime would select the most suitable one for the environment? For example, an extension's processor could return a prioritised list, with attached conditions for each option (isContainerRuntimeAvailable(), etc).

Here are are some possible implementations:

• Java (internal or external process). Some services, such as H2 and Lambda, run this way at the moment. It's performant and robust but only a few services have Java implementations.
• Container. Obviously, this is great when it's available. It's easy to customise the image, it's robust, it's portable.
• Compose script. Some services do better with a compose script rather than a container so that configuration can be richer. At the moment we have some support for prioritised selecting between those two.
• Shell script (external process). This is tricky to get right, and non-portable, but it could be a fallback for the most popular services, such as postgresql. A service's availability would have to be declared per operating system and architecture, rather than as a global statement.
• WASM. @andreaTP has done some really cool things using the Chicory WASM runtime to run the SQLite database as an embedded process in Java. WASM provides a secure sandbox, and Chicory allows that sandbox to be embedded in Java. This model could be extended (with work!) to build WASM versions of other services. For example, there are WASM implementations of postgresql and MySQL. There is a performance penalty for running in a WASM sandbox, but that's probably acceptable if it's the difference between "working' and "not-working".

It's worth noting that we also have a related problem, "should this dev service start a new container or re-use a compose container or or re-use an external service?" We've solved this on an ad-hoc basis per service by doing discovery and chaining, but that implementation could fit neatly into a the "prioritised list of suppliers + metadata" solution.

cc @cescoffier @ozangunalp

[andreaTP]

Great write up @holly-cummins !

Regarding WebAssembly (WASM), yes, it’s definitely feasible, though the complexity and trade‑offs depend on the service in question. Take running PostgreSQL, for example: porting it to WASI is ongoing and involves some effort .

For completeness, it is possible to run containers via WASM using projects like container2wasm.
However, this approach is experimental. On my machine, booting an Ubuntu container takes around 30 seconds, potentially acceptable for CI pipelines but too slow for local development (as of today).

Happy to help and support this great initiative.

[turing85]

We are in the exact scenario described. We do not have a container runtime available. We occasionally use MQ Series. For quarkus tests, our current workaround to start an in-memory artemis and replace the mq connection factory with an artemis connection factory. It would be really nice to have the possibility to start a non-containerized mq series.

[Sanne]

Thanks to @andreaTP we now have a prototype running PostgreSQL v17 (with limitations..) via WASM/Chicory:

• https://github.com/roastedroot/pglite4j

Would love to see this as a "fallback mode" for the PosgreSQL devservices based on containers.

• we'd probably want a flag which turns the fallback mode off, in case limitations are unacceptable
• as it's highly experimental I'd suggest to default this to off initially, then reasses

@holly-cummins your analysis is spot-on - there's multiple modes we'll probably want to support eventually and it might make sense to design the mode-choice and related configuration properly so to have such services configured and behave in a consistent way, but at the same time I think it would be fine to allow rapid feedback for just pglite4j w/o necessarily overthinking all options. We could progress on both in parallel w/o strict ordering.

[andreaTP]

Thanks for the excellent write-up Sanne!
For the records, those are the changes to use pglite4j in all the Postgres backed quickstarts.

[holly-cummins]

I agree, @Sanne, at the moment we have centralised fallback from docker to podman, and we could extend that principle to docker → podman → container-free. That's non-trivial, though. Before making the investment, it makes sense to do a more localised fallback just in the postgres dev service. Then we can assess if it helps people, and also watch to see if a container-free capability becomes possible for any other services. I'm thinking container-free OracleDB and DB2 is only a remote possibility. :) Container-free messaging might be more likely. But until we have more than one, a centralised capability wouldn't add much.

[Sanne]

right, let's focus on the postgresql experiment for now then

[cescoffier]

Hey @holly-cummins, thanks for the great write-up! I really love the idea and the direction this is heading.

I do have a slight concern regarding the "shell script" fallback, though. Shell scripts can be notoriously tricky to get right, especially when trying to tailor and maintain them for every possible OS and architecture combination. The maintenance burden there could get pretty heavy.

Other than that, I completely agree with the rest of the proposal! It actually got me wondering: would it be possible to orchestrate WASM modules directly from a Compose file? (Would love your thoughts on this, @ozangunalp!).

Also, if we go the WASM route, how would we handle dynamic port allocation and network discoverability? We are already struggling a bit with network discoverability as it is, so I'm curious about how we'd go about retrieving those ports and wiring things up once they are assigned to a module at runtime (and how when running container IT it will connect to a WASM app).

[andreaTP]

would it be possible to orchestrate WASM modules directly from a Compose file?

Looking forward to hear more about this! At best of my understanding this would require Wasi P2/P3 support and a lot of tooling support.

Also, if we go the WASM route, how would we handle dynamic port allocation

For clarity, in pglite4j the socket interface is opened on the Java side (and is not even strictly "required").

[ozangunalp]

would it be possible to orchestrate WASM modules directly from a Compose file?

I looked at this yesterday, with podman it is possible. One can build OCI images from wasm modules and run them using any containerd supported wasm runtime and by setting the platform as wasi/wasm. (I think podman defaults to wasmedge as runtime (io.containerd.wasmedge.v1) ).

However, for Docker Desktop, the feature did not pass the beta stage: https://docs.docker.com/desktop/features/wasm/
Here it says that it is deprecated and will be removed in a future release.

For our Compose Dev Services, in theory, if a wasm service is defined with a matching image name and an exposed port, it'll be wired to the application. I need to test this though.

[andreaTP]

I'm curious to better understand what would be the problem we are solving by running Wasm this way.

• we still need a native runtime, how wasmedge is better than docker or podman in this context?
• we inherit all the wasm/wasi limitations, including single threaded execution
• it's very hard to get things to compile in a way that's usable(e.g. exposes a meaningful functionality)

As demonstrated by industry(e.g. Docker Desktop stepping back) Wasm value proposition shines when embedded in another language/runtime, most projects attempting to run Wasm as "standalone executables" or as an alternative to containers failed ~3-4 years ago.