[new release] dune (18 packages) (3.24.0)

[shonfeder]

Fast, portable, and opinionated build system

• Project page: https://github.com/ocaml/dune
• Documentation: https://dune.readthedocs.io/

CHANGES:

Fixed

• Fix promotion failure when a target changes from a directory to a file
  between builds, causing "Is a directory" errors.
  (fix(promote): clean up stale staging path before rename in register_intermediate dune#14371, fixes Moving a file as a previously existing directory dune#5647, fixes Clean needed when turning a directory into a file dune#6575, @Alizter)

• Make dune build @doc pick up an odoc installed via dune tools install odoc even without DUNE_CONFIG__LOCK_DEV_TOOL=enabled, mirroring how
  dune fmt consumes a locked ocamlformat (fix: check lockdir even if lock_dev_tools is disabled when invoking odoc dune#14426, fixes dune build @doc fails to find dune tools install odoc installed odoc dune#14235, @mt-caret)

• Reject (package ...) inside a named dependency binding
  ((deps (:name (package foo)))). Previously this was silently accepted but
  %{name} would resolve to an empty path list. (fix: reject (package ...) inside named dependency bindings dune#14499, @Alizter)

• Fix incorrect dependency in the .cmxs build for libraries with
  mode-dependent foreign stubs: the rule depended on the byte stubs archive
  instead of the native one, so parallel builds could fail to find
  -l<lib>_stubs_native. (fix: .cmxs build racing native stubs archive dune#14500, fixes Possible race condition observed on FreeBSD CI dune#12964, @Alizter)

• Fix dune build failing with "No rule found" when lock_dir paths in
  dune-workspace contain a subdirectory (e.g. (path sub/dune.lock)).
  (fix(pkg): lock subdirectories dune#14524, fixes lock_dir with subdirectory path is broken dune#14523, @Alizter)

• Artifact substitution repairs executable bit if it's not set correctly in the workspace
  (artifact-substitution: compare executable bits when replacing files dune#14556, @rgrinberg)

• Ignore EINVAL when accepting sockets on MacOS (Handle RPC socket preparation failures dune#14612, fixes dune forwarding rpc occasionally gets EINVAL backtrace dune#12660, @rgrinberg)

• Validate profile names. Profile names must be non-empty and can only contain
  letters, digits, _ and -. The name _ is reserved as a wildcard in
  (env ...) stanzas. (Validate profile names. dune#14657, fixes Potential confusion around profile names and _ in the env stanza dune#14630, @rlepigre)

Added

• Add %{pkg:<package>:<section>:<path>} pform for resolving package install
  files. Works with workspace packages, lock-file packages, and installed
  packages. (feat: %{pkg:foo:lib:file} pforms for package install layouts dune#14200, fixes Support %{pkg:foo:lib} variables more generally dune#14193, fixes Library variable expansion needs a library installed to work dune#3378, @Alizter)

• Enable the relocatable compiler by default for package management (feat(pkg): use the relocatable compiler dune#14357,
  fixes Always use the relocatable compiler over the older OCaml ones. dune#14012, @Alizter)

• Add changed source files to the build-start trace event for watch-mode
  restarts, showing the paths that triggered the rebuild (feature: source files changed event dune#14396, @rgrinberg)

• Add dune's rusage information in start/finish build trace events (trace: include dune self rusage snapshots dune#14402, @rgrinberg)

• Add a status message for RPC clients that manage to connect (Show RPC connection status in forwarded builds dune#14424, @rgrinberg)

• Allow blang expressions in the runtest_alias field in the cram stanza (cram: allow boolean expressions in runtest_alias dune#14425, @rgrinberg)

• Extend the stat based cache to cache the contents of directories and not just
  source files (feature: cache directory reads dune#14469, @rgrinberg).

• Show the number of connected RPC clients in the watch mode status line
  (Show RPC client count in status line dune#14489, @rgrinberg)

• Promote directory targets from experimental to generally available in 3.24
  (Make directory targets available in Dune 3.24 dune#14579, @rgrinberg)

Changed

• Use / as directory separator when appending local paths to external paths,
  making path construction consistent across platforms. (path: appending local paths to external paths dune#14278, @Alizter)

• On Windows, normalise process paths to \ before passing them to
  CreateProcessW so that programs which scan argv[0] (notably cmd.exe)
  do not misparse mixed separators. (path: appending local paths to external paths dune#14278, @Alizter)

• %{bin:NAME} now resolves to the build artifact path rather than
  the install staging path. Rules with %{bin:NAME} deps additionally
  get a per-rule .binaries directory prepended to the action's
  PATH, containing correctly-named symlinks for each declared bin
  pform dep. (feat: bin-layout for %{bin:...} deps  dune#14432, fixes dune build %{bin:foo} returns an error dune#3324, @Alizter)

• Replace the Rocq language field (stdlib ...) with the presence-only field
  (no_corelib) in Rocq language 0.14. (Rocq (no_corelib) flag dune#14452, fixes Rocq (stdlib ...) field naming changes dune#14358, @Durbatuluk1701)

• Throttle sandbox creation to 250 concurrent sandboxes. (fix: Throttle live sandboxes dune#14464, @rgrinberg)

• The package solver now uses cached revision store lookups for improved
  performance. (pkg: enable and version the rev_store cache dune#14494, fixes Version and enable the lmdb cache dune#12637, @Alizter, @rgrinberg)

• Remove the deprecated (lang coq) Coq Build Language. Use the
  Rocq Build Language ((using rocq <version>)) instead. Projects that
  still declare (using coq <version>) now get an error pointing them
  at Rocq. (chore: remove (lang coq) Coq Build Language dune#14525, fixes [coq] Deprecation and removal of (lang coq) for Dune 3.24 dune#12788, @Alizter)

[shonfeder]

Prior two rounds of pre-releases:

• [new release] dune (18 packages) (3.24.0~alpha0) #29969
• [new release] dune (18 packages) (3.24.0~alpha2) #29996

All issues and regressions have been fixed, to the best of our knowledge, and this PR now presents our minor version release.

Prior triaging of CI errors can be found on the above draft PRs. I will perform another round of analysis here once CI has completed, and we share the results, either recommending merge or detailing the plans for any other needed fixes.

[olafhering]

This breaks cppo (test suite), camomile and lablgtk3. Sorry for not testing -rc. Maybe the CI will recognize this as well?

[   13s] --- test/missing_endscope.ref
[   13s] +++ test/missing_endscope.err
[   13s] @@ -1,2 +1,2 @@
[   13s] -Error: File "missing_endscope.cppo", line 3, characters 0-6
[   13s] +Error: File "./missing_endscope.cppo", line 3, characters 0-6
[   13s]  Error: This #scope is never closed: perhaps #endscope is missing

[   29s] File "src/charmaps/dune.inc", lines 804-807, characters 0-111:
[   29s] 804 | (rule
[   29s] 805 |  (targets EBCDIC-PT.mar)
[   29s] 806 |  (deps    EBCDIC-PT)
[   29s] 807 |  (action  (run ../tools/camomilecharmap.exe -d . %{deps})))
[   29s] Error: Rule failed to generate the following targets:
[   29s] - src/charmaps/EBCDIC-PT.mar

[   41s] File "src/dune-prop.sexp", lines 6-8, characters 0-100:
[   41s] 6 | (rule
[   41s] 7 |  (targets gtkActionProps.ml ogtkActionProps.ml)
[   41s] 8 |  (action (run propcc %{dep:gtkAction.props})))
[   41s] Command [194] exited with code 2:
[   41s] $ (cd _build/default/src && ../.bin/propcc ./gtkAction.props)
[   41s] Uncaught exception: Sys_error("o./gtkActionProps.ml: No such file or directory")
[   41s] Fatal error: exception Sys_error("o./gtkActionProps.ml: No such file or directory")

[shonfeder]

Thanks for the input!

There is a breaking change in this release, but it is only expected to impact packages that make invalid assumptions, or incorrect use, of the %{bin:...} variable. I'll dig in to these results to be sure, but at first glance it looks like all these packages fall into that bucket. I think we'll need upper bounds or patch releases for the affected packages, unless we decided we need to backtrack on this change (which I would not currently advocate for). I'll right up a summary and proposed action plan once the results are in here.

[olafhering]

The missing_endscope failure suggests that not only cppo becomes ./cppo, but also the argument $1 may becomes ./$1 (via (run %{bin:cppo} %{<})). Maybe I misinterpret the cppo sources. As you said, people familiar with the sources need to work this out.

[shonfeder]

not only cppo becomes ./cppo, but also the argument $1 may becomes ./$1 (via (run %{bin:cppo} %{<})).

That's correct. The underlying issues is that prior to dune 3.24, paths to files in the current directory were not represented with a preceding ./, which meant the representation was ambiguous (in many shells) between executables to be looked up in the environment and paths. We have fixed this ambiguity in dune 3.24, but tests or build rules that where making invalid assumptions about the concrete representation of these paths are now breaking as a result.

See ocaml/dune#15156 for details.

[olafhering]

Hopefully this change did consider the arguments for executables. It is up to each one how something that is (or looks like) a path is processed by that executable. As it can be seen in the case of cppo, it may process a relative path just fine.

[shonfeder]

Hopefully this change did consider the arguments for executables. It is up to each one how something that is (or looks like) a path is processed by that executable. As it can be seen in the case of cppo, it may process a relative path just fine.

As it can be seen in the case of cppo, it may process a relative path just fine.

I am not sure what you mean here. Indeed it was in consideration for arguments to executables that this change was made. See ocaml/dune#15147 and ocaml/dune#14724 for prior discussion.

The general view we have converged on is as follows: these variables have always expanded to paths, but the path representation was not unambiguous. It is now unambiguous. If build configurations were making invalid assumptions about the path representation that breaks once we make the representation unambiguous, this is a misconfiguration.

[shonfeder]

Here's the initial error analysis. Failures are grouped by error message. Tons of these are downstream of just a few libraries (e.g., the ones we have already identified). Patches and bounds will be needed (unless we reverse course on the release, but I am still convinced about our analysis):

https://gist.github.com/shonfeder/09f28716f60bcf4d1a2eb3b066ddc825

[shonfeder]

Triage and remediation is now complete, with results and activities documented in https://gist.github.com/shonfeder/09f28716f60bcf4d1a2eb3b066ddc825 (the work reported there exceeds the character limit of github comments).

Since no regressions or unexpected breaking changes have been identified, I think that this release can be published as soon as the following remediation PR has been merged, adding with-test filters to affected packages:

#30157

The other PRs into the opam repo mentioning this one are unrelated cleanups to package data, which just happened to be caught by the CI here.

[raphael-proust]

thanks for the release and all the work patching all the small issues that popped up <3

[shonfeder]

Thanks for the review and merge!