Dynamic Dependency Resolution with map_directory

I was working on providing a ruleset for Lean4 and ran into a fairly common issue. Lean's source files declare a topological ordering (include statements); a module must be compiled after all the modules it depends on are compiled. Problem? Bazel requires all action dependencies to be declared during the analysis phase, and the source files are not available to be read at analysis time.

Enter map_directory

While Bazel's action graph is fixed at analysis time, map_directory lets you defer part of action creation to execution time. You declare placeholder directories during analysis, then a callback inspects those directories after they're populated and creates the actual compilation actions. Now, if someone happened to encode dependency information into those directories in some prior action, then the callback in map directory can use that dependency information to order the compile actions.

We do this with a 2 phase approach. The first quickly parses the include statements, resolves the dependency DAG, and encodes it in marker files. The second utilizes map_directory along with the marker files to create the actions with

Encoding the DAG

Before map_directory runs, a resolver action scans all source files for import statements and constructs the dependency graph. It assigns each file a slot (index 0 through N-1) and writes marker files like examples.app.main.deps_2_4.marker—file at slot 0 imports files at slots 2 and 4. Numeric slots keep filenames compact; dots encode paths since map_directory flattens directory structure.

For example, given these source files:

Slot 0: examples/app/main.src     → imports types.src, utils.src
Slot 1: examples/core/base.src    → no imports
Slot 2: examples/core/types.src   → no imports
Slot 3: examples/lib/helper.src   → imports base.src
Slot 4: examples/lib/utils.src    → imports base.src

The resolver writes:

examples.app.main.deps_2_4.marker    # main depends on slots 2 (types) and 4 (utils)
examples.core.base.deps.marker       # base has no dependencies
examples.core.types.deps.marker      # types has no dependencies
examples.lib.helper.deps_1.marker    # helper depends on slot 1 (base)
examples.lib.utils.deps_1.marker     # utils depends on slot 1 (base)

Creating the compile actions

The map_directory callback runs at execution time with full visibility into the marker directory. It iterates over the markers, reconstructs the dependency graph, and creates compile actions using template_ctx.run(). Actions within a single map_directory callback can declare dependencies on each other's outputs. The callback declares output files with template_ctx.declare_file(), then references those files as inputs to dependent actions.

For example, the callback reads examples.app.main.deps_2_4.marker and creates:

Action: compile examples/app/main.src → examples/app/main.out
  inputs:  examples/app/main.src, examples/core/types.out (slot 2), examples/lib/utils.out (slot 4)
  outputs: examples/app/main.out

And it works?

This solution:

• correctly orders the compilation commands,
• maintains incremental and parallel builds, and
• doesn't require the use of manual multiphase builds like gazelle.

Is it hacky? Oh yes.