Splitting a very large build file

[jodersky]

I'm working on a monorepo which has a few projects which need to interact, but where developers mostly work on one project day-to-day. So far, everything has been configured in one build.sc at the root of all projects, but that file is getting quite large. I would like to split it, and have seen that mill supports this in the form of Foreign Modules.

My initial idea was to transform the repository from a structure like:

a/
b/
c/
build.sc

to something like

a/build.sc
b/build.sc
c/build.sc

and then have the builds reference each other when needed.

This effectively creates separate mill builds, which unfortunately has a couple of drawbacks:

• You can no longer run things from the root; you'll need to cd into a corrsponding subproject to invoke mill.
• Subprojects no longer share any caches; every submodule will have its own copy of cache of any other modules it refers to in out/foreign-modules. This makes it impossible to share compilation results between projects, instead if two projects depend on a third, they will both need to recompile the third one.

Maybe foreign modules are the wrong approach to splitting up a large build config. Does someone have a suggestion on how to effectively split up with a very large build configuration regrouping many quasi-independent projects?

[lefou]

If your only concern is the size of the build.sc but running mill from the top level directory is fine, then you could consider just moving some module definitions into separate files. These need to be included into the top level build.sc though. It's probably a good idea to move common logic into yet another script file, which is included by all of the other scripts.

One caveat though, you can't instantiate modules in these other script files. Instead you may need to just declare them as trait and in the top level build.sc need to define objects which extends these traits. That means, that references to other "external" modules need to stay abstract. Here is the outline of that concept:

// file: a/Modules.sc
trait a extends Module {
  object aa extends ScalaModule {
  }
}

// file: b/Modules.sc
import $file.^.a.{Modules => A}

trait b extends Module {
  // abstract reference to external module
  def a: A.a

  object bb extends ScalaModule {
    // use the still abstract reference to external modules
    def moduleDeps = Seq(a.aa)
  }
}


// file: build.sc
import $file.a.{Modules => A}
import $file.b.{Modules => B}

object a extends A.a
object b extends B.b {
  override def a = a
}

This looks a bit complicated and it is. But it's also in reach with what's possible with current mill. Because of the limitations you described, I never even tried to use mill's foreign module concept.

[jodersky]

Thanks for your suggestion. I think that

One caveat though, you can't instantiate modules in these other script files.

very aptly describes a limitation I'm running into. In an ideal situation, I'm thinking of defining these modules in different files, importing them into the main build.sc and then not having to redefine them (and also maybe use a mill wrapper script that always cd's into the root dir, which I think you already showed me once :) ).

I'll go with something along the lines of what you suggest right now.

[lefou]

What's probably needed is a rethink and redesign of the foreign module concept. In its current form it's IMHO rarely useful.

[lihaoyi]

Yeah I don't use foreign modules either. Someone contributed it, and they kinda work, but they definitely haven't had the same amount of effort gone into making them a polished and broadly useful feature as the rest of Mill has had.

Could use some re-imagining if anyone has the need, to come up with a consistent story for what the problem is and how we expect people to tackle it using Mill. For me personally, even my larger builds like https://github.com/handsonscala/build or https://github.com/com-lihaoyi/Ammonite are still bearable to work with in a single build.sc file, so I haven't been forced to do this myself.

[lefou]

Maybe, just swapping the out directory for foreign modules (from my/out/foreign/their to their/out) is enough? This should solve the caching issue. And some nice way to make them callable from mills cmdline (Somehow binding/"mounting" them into the module tree).