Implement --target-exec

[toots]

This PR implements runing a target binary from the host following the proposal at: #13052

• Add --target-exec <toolchain>=<wrapper>
• Add per-target support in dune-workspace:

(context
 (default
   (targets
     native
     ((name test_toolchain) (target_exec wrapper_with_args.sh --arg1 --arg2)))))

• Wire it exactly where the binary is currently replaced with a native binary when cross-compiling.
• At the moment the wrapper is expected to be an external binary not produced by the project. This seems quite reasonable.
• Add DUNE_TARGET_EXEC and DUNE_CROSS_TARGET to specify --target-exec (resp. -x) through environment variables
• Add a new runexec action that bypasses the target exec wrapper when needed.

This is intended for cross-compilation only at the moment so the wrapper is only added in those cases.

Implementation details

Most of the implementation complexity is at the wrapper substitution level and mostly because it changes the relationship between program and dependency but also because it ties up the program and arguments together.

To help with that, the signature of map_exe is changed to include arguments and return dependency, program and arguments. The rest of the downstream code is adapted accordingly while trying to keep the APIs as similar as possible.

Example

Here's a PR with a complex build system that was lifted to do fully transparent cross-build: savonet/ocaml-posix#24

All you have to do it run:

dune build -x windows --target-exec windows=wine posix-socket

and it cross-builds for windows using the exact same build setup as for native compilation.

Future work

The long-term goal is to lift this to opam and allow transparent cross-building using the same dune-based native package.

Something like:

x-dune-cross-enabled: true
x-dune-cross-needs-target-exec: true

And use the above environment variables to do a cross-build.

This should allow cross-building opam repository such as https://github.com/ocaml-cross/opam-cross-windows/ to drop all crossbuild enabled packages and directly use the native opam packages.

[toots]

I did more testing and runexec will be needed for e.g. getting pkg-config options. Just added it.

[Alizter]

Q: I haven't had a good look at this yet, but why don't you just modify how the run action works in the cross context rather than introducing a new action?

[toots]

Q: I haven't had a good look at this yet, but why don't you just modify how the run action works in the cross context rather than introducing a new action?

That's what it's doing indeed, see: https://github.com/ocaml/dune/pull/13112/files#diff-bb90136ac4be87c54face4d7dd2a55b65ce978a4497d19d53eb51350bb613c50R744

However, in cross-compilation context, there is still a need to ensure that some run actions are always done in the host context, typically calls to pkg-config.

The pattern follows the existing lib/libexec, lib-private/libexec-private and path/exec patterns so it should feel pretty natural and make sense for people used to these considerations.

[rgrinberg]

Is -x supported via the workspace file? If so, it seems like this should be configurable there as well.

I think it's ok to accept this, but we need to communicate to users that this is experimental and unstable and that if we find a better way to do this, it is likely that the old way will be removed.

[toots]

@rgrinberg happy to add it to dune-workspace as part of the targets support. I was waiting for feedback before doing so.

Can you expand on the unstable? I do think that the way dune is doing it right now is really good and headed toward the right direction.

[toots]

@rgrinberg Just pushed support for dune-workspace. It wasn't immediately clear how or if we wanted to support variables there so for now it's only parsing strings.

[toots]

@Alizter @rgrinberg I'm not sure why the Nix tests are failing. Is that something I should take care of?

[Alizter]

Those aren't nix-specific but failures in the test-suite. Could you promote the changes in dune runtest test/blackbox-tests/test-cases/pkg. It seems some hashes have changed. You can run the specific tests you find in the logs too if thats easier.

[Alizter]

Let me know if you need assistance, I can push the test changes here for you.

[toots]

@Alizter should be good now!

[rgrinberg]

The name runexec doesn't seem particularly great to me. I might change it to run_host before we release this.

Anyway, the PR is good to go. Just needs to be bumped to 3.22 as 3.21 is already being released.

[toots]

The name runexec doesn't seem particularly great to me. I might change it to run_host before we release this.

Anyway, the PR is good to go. Just needs to be bumped to 3.22 as 3.21 is already being released.

I have no strong opinion on the naming but I wanted to point out that exec is already used to designate part of the variables that explicitly refer to the host (see: https://dune.readthedocs.io/en/latest/concepts/variables.html)

But I agree with you that exec is not clear.

Should we do run_host (or perhaps host_run to be more natural) and then consider deprecating the existing exec patterns in favor of e.g. %{host_lib:foo:prog.exe} ?

[rgrinberg]

I agree that we need a consistent treatment for all of this. This is why I'm not particularly keen on making all of this stable.