Remove the Global Build Lock

[joelreymont]

Issue #12731: Remove the Global Build Lock

Current Behavior

Dune uses a global build_system_mutex that serializes all build requests. When watch mode holds this lock, RPC requests wait for the lock to be released before executing.

Observed Problem

When a developer runs dune build --watch and an IDE sends an RPC request (e.g., for type information), the RPC request waits for the watch mode build to complete before starting execution.

Example timeline:

• 00:00 - Watch mode acquires lock, starts build
• 00:02 - RPC request arrives, waits for lock
• 00:05 - Watch mode releases lock
• 00:05 - RPC request acquires lock, executes

The RPC request waits 3 seconds in this scenario.

Proposed Change

Remove the global build_system_mutex and replace it with a build coordinator that allows multiple build sessions to execute concurrently.

Implementation Details

See PR #12730 for implementation.

Correctness Mechanisms

Existing mechanisms that prevent race conditions:

• Path locks prevent concurrent writes to the same file
• Memo system provides first-write-wins semantics for rule execution
• Atomic file operations remain unchanged

Progress Tracking

Progress tracking moved to Build_coordinator to prevent duplicate counting when multiple sessions build shared rules via Memo.

Promotion and Pending Targets

Centralized promotion queue with deduplication (first registration wins).
Reference-counted pending target tracking (file deleted when count reaches zero).

Testing Limitation

Cram tests execute sequentially and cannot verify concurrent execution. Manual testing required for:

• Watch mode + concurrent RPC requests
• Multiple RPC clients
• IDE integration with watch mode

Use Case

Primary use case: IDE responsiveness during watch mode builds. When dune build --watch is running and an LSP request arrives, the request can execute without waiting for the watch build to complete.

[joelreymont]

Reference implementation is in PR #12730.

[rgrinberg]

First can you remove all the AI fluff from your ticket? It would be a lot easier to take this work seriously. To take one example, there's no benefit whatsoever for CI. CI doesn't use RPC, and even if it did, the entire supposed speed up can be achieved by doing dune build @install @runtest @doc. Same for IDE responsiveness. I could elaborate more, but I'm not sure if would be a good use of my time to point out nonsense in AI generated texts.

I'll respond to the motivation behind this ticket though, as it is a valid concern regarding RPC usability. Internally, we've taken the following route to make RPC more usable:

1. Removed the distinction between active/passive watch modes
2. Introduced the notion of a "sticky goal". The sticky goal is the one that attempts to keep up to date wrt to file system changes. Non sticky goals are one off (hence not "sticky").
3. RPC requests append more goals to the currently running goal. If there's no running goal, then one is created. RPC results are no longer serialized like they are currently. This was a decision that was made just to make it easy to write tests.
4. There's a Build_loop module that handles watch mode builds, instead of having it spread across 3 modules like we have now.

I've only skimmed through implementation so my comments might not be entirely accurate. First, there's actually some good parts to it (build id's, co-ordination) although there's obvious nonsense as well (the changes to promotion and pending targets). The fundamental issues are that progress reporting isn't going to work because rule execution is is shared between your "build sessions", but is only counted towards the session that started the rule. Another serious issue is that you're going to re-run actions that fail "non reproducibly" (see memo) more than once.

All of this stuff can be fixed, but I fear that it will require difficult work where gpt will need far more guidance and hand holding. Doing this will require a much bigger investment into understanding the codebase on your part, and a sizeable effort on our part as reviewers. I would prefer if you tried to work on a simpler problem in dune first, so that we can see early on how productive this process is going to be, and whether we should invest our time into it.

Sorry if any of this sounds harsh. I do generally agree with you that AI can be a significant productivity boost. But it is not a substitute for having an understanding of the project.

[joelreymont]

Robert,

I agree with you.

I got a large amount of AI credits that expire in 2 days and I've been trying to use them for the greater good.

I'll remove the fluff and address your comments. Will make new commits to simplify review.

Let me know if you have anything to add on the technical side!

[joelreymont]

I have a suggestion...

There's some good in this PR and more can be squeezed out of it. It doesn't require prompting as much as a description of what's wrong and guidance on the right direction.

I would appreciate this guidance so that I can squeeze as much good as possible of of this already completed work!

[joelreymont]

Fluff removed.

[rgrinberg]

I'm afraid I don't know what part of this work has tangible value on its own. My guess is probably not a whole lot in its current state. It might be a good reference for somebody working on RPC down the line though.