Simple YAML-backed modules and single-file script modules, second iteration (#5951)

This PR implements a way to define "Simple Modules" based on `mill.yaml`
files or single `.java`/`.scala`/`.kt` files with `//|` build headers.
This should make Mill more appealing for small projects, where the use
of a `build.mill` file adds significant boilerplate and complexity. Most
small projects need minimal customization of the build, and so the full
Scala `.mill` syntax provides no value over a more lightweight
config-only approach. This PR also allows interop between simple YAML
modules and custom module classes written in `mill-build/src/`, to allow
a gradual transition to the more flexible programmatic configuration


## Simple Module with `build.mill.yaml`

`mill.yaml`
```yaml
extends: ["mill.javalib.JavaModule"]
mvnDeps:
- "net.sourceforge.argparse4j:argparse4j:0.9.0"
- "org.thymeleaf:thymeleaf:3.1.1.RELEASE"
```

`test/mill.yaml`
```yamlextends: "mill.javalib.JavaModule.Junit4"
moduleDeps: ["build"]
mvnDeps:
- "com.google.guava:guava:33.3.0-jre"
```
```console
> ./mill run --text hello
<h1>hello</h1>

> ./mill .:run --text hello # `.` for explicit root module, `:` as new external module separator


> ./mill test
...
+ foo.FooTests...simple ...  "<h1>hello</h1>"
+ foo.FooTests...escaping ...  "<h1>&lt;hello&gt;</h1>"

> ./mill test:testForked # explicit task
```

## Single-File Module with `.java`, `.scala`, or `.kt`

`Foo.scala`
```scala
//| mvnDeps:
//| - "com.lihaoyi::scalatags:0.13.1"
//| - "com.lihaoyi::mainargs:0.7.6"
import scalatags.Text.all.*
import mainargs.{main, ParserForMethods}

object Foo {
  def generateHtml(text: String) = {
    h1(text).toString
  }

  @main
  def main(text: String) = {
    println(generateHtml(text))
  }

  def main(args: Array[String]): Unit = ParserForMethods(this).runOrExit(args)
}
```
```console
> ./mill Foo.scala --text hello
compiling 1 Scala source to...
<h1>hello</h1>

> ./mill Foo.scala:run --text hello
<h1>hello</h1>

> ./mill show Foo.scala:assembly # show the output of the assembly task
".../out/Foo.scala/assembly.dest/out.jar"

> java -jar ./out/Foo.scala/assembly.dest/out.jar --text hello
<h1>hello</h1>
```

## Implementation

* Mill considers `build.mill.yaml` and `package.mill.yaml` files
similarly to `build.mill` and `package.mill`
* Mill code-generates Scala code for a `package` object for each
`build.mill.yaml`/`package.mill.yaml` file: `extends` becomes the
inheritance list, `moduleDeps` becomes `moduleDeps`, and everything else
becomes `def key = Task.Literal("""value""")`
* A bunch of the `upickle.default.ReadWriter`s have been made more
lenient to allow convenient YAML/JSON input of their values: `PathRef`s,
the various `mill.javalib.publish.*` types,
* `.mill.yaml` and `.mill` can depend on each other and inter-operate,
since in the end they all expand into Scala code
* Single-file scripts continue to be modelled as `ExternalModule`s
instantiated during the `Resolve` phase. These cannot be depended upon
by `.mill.yaml`/`.mill` modules

Supersedes #5836

### Advantages
1. This allows use of normal `trait` modules without needing to package
them up into special `class`es, so it can be applied much more generally
to dozens of `trait`s available in the Mill libraries
2. This allows multiple inheritance, e.g. `extends:
[mill.javalib.JavaModule, mill.javalib.PublishModule]`
3. Modules defined in `build/package.mill.yaml` can interop much more
cleanly with `build/package.mill` files, including type-safe references
in either direction
4. We can handle things like requiring abstract methods to be defined in
the YAML, since the YAML keys get code-gened into method implementations

### Disadvantages
1. Generated code can slow down compiles, whereas previous reflective
instantiation was basically instant
2. Compile errors in generated code may be confusing to the user, since
they didn't write the generated code themselves
3. Script files will still need to be instantiated reflectively from
`class`es, as the ad-hoc nature of scripts means they may live in
arbitrary files that are only discovered pretty late (e.g in the middle
of execution of a `show` command that resolves things) that cannot be
easily discovered during Mill's `walkBuildFiles` step
4. Script files cannot use test frameworks for now, though they can be
tested via normal downstream scripts with `main` methods. This could be
added in future

## Docs and Testing

`.mill.yaml` files are largely covered by example tests: these also form
the documentation, which takes over the first two examples in
`{javalib,scalalib,kotlinlib}/intro.html`, and on an additional page
`{javalib,scalalib,kotlinlib}/simple.html`.

Since the main logic around these `.mill.yaml` files runs as part of
`MillBuildBootstrap`/`MillBuildRootModule/`CodeGen`, they cannot be
easily exercised with unit tests, and so these example tests will have
to do for now

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

Author: lihaoyi

ci/test-dist-run.sh

@@ -2,7 +2,7 @@
 
 set -eux
 
-EXAMPLE=example/scalalib/basic/6-realistic
+EXAMPLE=example/scalalib/basic/7-realistic
 
 rm -rf $EXAMPLE/out
 

core/api/src/mill/api/ExternalModule.scala

@@ -22,7 +22,8 @@ abstract class ExternalModule(using
       millFile0
     ) {
 
-  assert(
+  private[mill] def allowNestedExternalModule = false
+  if (!allowNestedExternalModule) assert(
     !" #".exists(millModuleEnclosing0.value.contains(_)),
     "External modules must be at a top-level static path, not " + millModuleEnclosing0.value
   )

core/api/src/mill/api/ModuleCtx.scala

@@ -41,6 +41,7 @@ object ModuleCtx extends LowPriCtx {
   trait Wrapper {
     def moduleCtx: ModuleCtx
     private[mill] def moduleLinearized: Seq[Class[?]]
+    private[mill] def buildOverrides: Map[String, ujson.Value] = Map()
   }
   private case class Impl(
       enclosing: String,

core/api/src/mill/api/PathRef.scala

@@ -197,26 +197,30 @@ object PathRef {
       storeSerializedPaths(p)
       p.toString()
     },
-    s => {
-      val Array(prefix, valid0, hex, pathString) = s.split(":", 4)
+    {
+      case s"$prefix:$valid0:$hex:$pathString" if prefix == "ref" || prefix == "qref" =>
 
-      val path = os.Path(pathString)
-      val quick = prefix match {
-        case "qref" => true
-        case "ref" => false
-      }
-      val validOrig = valid0 match {
-        case "v0" => Revalidate.Never
-        case "v1" => Revalidate.Once
-        case "vn" => Revalidate.Always
-      }
-      // Parsing to a long and casting to an int is the only way to make
-      // round-trip handling of negative numbers work =(
-      val sig = java.lang.Long.parseLong(hex, 16).toInt
-      val pr = PathRef(path, quick, sig, revalidate = validOrig)
-      validatedPaths.value.revalidateIfNeededOrThrow(pr)
-      storeSerializedPaths(pr)
-      pr
+        val path = os.Path(pathString)
+        val quick = prefix match {
+          case "qref" => true
+          case "ref" => false
+        }
+        val validOrig = valid0 match {
+          case "v0" => Revalidate.Never
+          case "v1" => Revalidate.Once
+          case "vn" => Revalidate.Always
+        }
+        // Parsing to a long and casting to an int is the only way to make
+        // round-trip handling of negative numbers work =(
+        val sig = java.lang.Long.parseLong(hex, 16).toInt
+        val pr = PathRef(path, quick, sig, revalidate = validOrig)
+        validatedPaths.value.revalidateIfNeededOrThrow(pr)
+        storeSerializedPaths(pr)
+        pr
+      case s =>
+        mill.api.BuildCtx.withFilesystemCheckerDisabled(
+          PathRef(os.Path(s, mill.api.BuildCtx.workspaceRoot))
+        )
     }
   )
 

core/api/src/mill/api/Task.scala

@@ -289,6 +289,52 @@ object Task {
     ): Simple[T] = ${ Macros.taskResultImpl[T]('t)('rw, 'ctx, '{ persistent }) }
   }
 
+  // The extra `(x: T) = null` parameter list is necessary to make type inference work
+  // right, ensuring that `T` is fully inferred before implicit resolution starts
+  def Literal[T](s: String)(using
+      x: T = null.asInstanceOf[T]
+  )(using li: LiteralImplicit[T]): Task.Simple[T] = {
+    assert(li.ctx != null, "Unable to resolve context")
+    assert(li.writer != null, "Unable to resolve JSON writer")
+    assert(li.reader != null, "Unable to resolve JSON reader")
+    new Task.Input[T](
+      (_, _) => Result.Success(upickle.default.read[T](s)(using li.reader)),
+      li.ctx,
+      li.writer,
+      None
+    )
+  }
+
+  class LiteralImplicit[T](
+      val reader: upickle.default.Reader[T],
+      val writer: upickle.default.Writer[T],
+      val ctx: ModuleCtx
+  )
+  object LiteralImplicit {
+    // Use a custom macro to perform the implicit lookup so we have more control over implicit
+    // resolution failures. In this case, we want to fall back to `null` if an implicit search
+    // fails so we can provide a good error message
+    implicit inline def create[T]: LiteralImplicit[T] = ${ createImpl[T] }
+
+    private def createImpl[T: Type](using Quotes): Expr[LiteralImplicit[T]] = {
+      import quotes.reflect.*
+
+      def summonOrNull[U: Type]: Expr[U] = {
+        Implicits.search(TypeRepr.of[U]) match {
+          case s: ImplicitSearchSuccess => s.tree.asExprOf[U] // Use the found given
+          case _: ImplicitSearchFailure =>
+            '{ null.asInstanceOf[U] } // Includes both NoMatchingImplicits and AmbiguousImplicits
+        }
+      }
+
+      val readerExpr = summonOrNull[upickle.default.Reader[T]]
+      val writerExpr = summonOrNull[upickle.default.Writer[T]]
+      val ctxExpr = summonOrNull[ModuleCtx]
+
+      '{ new LiteralImplicit[T]($readerExpr, $writerExpr, $ctxExpr) }
+    }
+  }
+
   abstract class Ops[+T] { this: Task[T] =>
     def map[V](f: T => V): Task[V] = new Task.Mapped(this, f)
     def filter(f: T => Boolean): Task[T] = this

core/api/src/mill/api/internal/RootModule.scala

@@ -26,24 +26,37 @@ abstract class RootModule()(using
   // Provided for IDEs to think that one is available and not show errors in
   // build.mill/package.mill even though they can't see the codegen
   def millDiscover: Discover = sys.error("RootModule#millDiscover must be overridden")
+
+  override def buildOverrides: Map[String, ujson.Value] = baseModuleInfo.buildOverrides
 }
 
 @internal
 object RootModule {
   class Info(
       val projectRoot: os.Path,
       val output: os.Path,
-      val topLevelProjectRoot: os.Path
+      val topLevelProjectRoot: os.Path,
+      @com.lihaoyi.unroll val buildOverrides: Map[String, ujson.Value] = Map()
   ) {
+
     def this(
         projectRoot0: String,
         output0: String,
-        topLevelProjectRoot0: String
+        topLevelProjectRoot0: String,
+        headerData: String
     ) = this(
       os.Path(projectRoot0),
       os.Path(output0),
-      os.Path(topLevelProjectRoot0)
+      os.Path(topLevelProjectRoot0),
+      upickle.read[Map[String, ujson.Value]](headerData)
     )
+
+    def this(
+        projectRoot0: String,
+        output0: String,
+        topLevelProjectRoot0: String
+    ) = this(projectRoot0, output0, topLevelProjectRoot0, "{}")
+
     implicit val millMiscInfo: Info = this
   }
 

core/api/src/mill/api/internal/RootModule0.scala

@@ -30,6 +30,7 @@ abstract class RootModule0(
   // `Discover` needs to be defined by every concrete `BaseModule` object, to gather
   // compile-time metadata about the tasks and commands at for use at runtime
   protected def millDiscover: Discover
+  def discover: Discover = millDiscover
 
   // We need to propagate the `Discover` object implicitly throughout the module tree
   // so it can be used for override detection

core/constants/src/mill/constants/CodeGenConstants.java

@@ -23,18 +23,19 @@ public class CodeGenConstants {
    * The name of the root build file
    */
   public static final List<String> rootBuildFileNames =
-      List.of("build.mill", "build.mill.scala", "build.sc");
+      List.of("build.mill.yaml", "build.mill", "build.mill.scala", "build.sc");
 
   /**
    * The name of any sub-folder build files
    */
   public static final List<String> nestedBuildFileNames =
-      List.of("package.mill", "package.mill.scala", "package.sc");
+      List.of("package.mill.yaml", "package.mill", "package.mill.scala", "package.sc");
 
   /**
    * The extensions used by build files
    */
-  public static final List<String> buildFileExtensions = List.of("mill", "mill.scala", "sc");
+  public static final List<String> buildFileExtensions =
+      List.of("mill.yaml", "mill", "mill.scala", "sc");
 
   /**
    * The user-facing name for the root of the module tree.

core/constants/src/mill/constants/Util.java

@@ -5,6 +5,8 @@
 import java.lang.reflect.InvocationTargetException;
 import java.lang.reflect.Method;
 import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
 import java.util.ArrayList;
@@ -84,15 +86,20 @@ private static String throwBuildHeaderError(
         + ": " + line + "\n" + msg);
   }
 
-  public static String readBuildHeader(java.nio.file.Path buildFile, String errorFileName) {
+  public static String readBuildHeader(Path buildFile, String errorFileName) {
+    return readBuildHeader(buildFile, errorFileName, false);
+  }
+
+  public static String readBuildHeader(
+      Path buildFile, String errorFileName, boolean allowNonBuild) {
     try {
-      java.util.List<String> lines = java.nio.file.Files.readAllLines(buildFile);
+      java.util.List<String> lines = Files.readAllLines(buildFile);
       boolean readingBuildHeader = true;
       java.util.List<String> output = new ArrayList<>();
       for (int i = 0; i < lines.size(); i++) {
         String line = lines.get(i);
         if (!line.startsWith("//|")) readingBuildHeader = false;
-        else if (!buildFile.getFileName().toString().startsWith("build.")) {
+        else if (!allowNonBuild && !buildFile.getFileName().toString().startsWith("build.")) {
           throwBuildHeaderError(
               errorFileName,
               i,

core/eval/src/mill/eval/EvaluatorImpl.scala

@@ -24,7 +24,13 @@ import mill.resolve.Resolve
 final class EvaluatorImpl private[mill] (
     private[mill] val allowPositionalCommandArgs: Boolean,
     private[mill] val selectiveExecution: Boolean = false,
-    private val execution: Execution
+    private val execution: Execution,
+    scriptModuleResolver: (
+        String,
+        String => Option[mill.Module],
+        Boolean,
+        Option[String]
+    ) => Seq[Result[mill.api.ExternalModule]]
 ) extends Evaluator {
 
   private[mill] def workspace = execution.workspace
@@ -40,9 +46,18 @@ final class EvaluatorImpl private[mill] (
   def withBaseLogger(newBaseLogger: Logger): Evaluator = new EvaluatorImpl(
     allowPositionalCommandArgs,
     selectiveExecution,
-    execution.withBaseLogger(newBaseLogger)
+    execution.withBaseLogger(newBaseLogger),
+    scriptModuleResolver
   )
 
+  private[mill] def resolveSingleModule(s: String): Option[mill.Module] = {
+    resolveModulesOrTasks(Seq(s), SelectMode.Multi)
+      .toOption
+      .toSeq
+      .flatten
+      .collectFirst { case Left(m) => m }
+  }
+
   /**
    * Takes query selector tokens and resolves them to a list of [[Segments]]
    * representing concrete tasks or modules that match that selector
@@ -59,7 +74,8 @@ final class EvaluatorImpl private[mill] (
         scriptArgs,
         selectMode,
         allowPositionalCommandArgs,
-        resolveToModuleTasks
+        resolveToModuleTasks,
+        scriptModuleResolver = scriptModuleResolver(_, resolveSingleModule, _, _)
       )
     }
   }
@@ -75,7 +91,8 @@ final class EvaluatorImpl private[mill] (
         scriptArgs,
         selectMode,
         allowPositionalCommandArgs,
-        resolveToModuleTasks
+        resolveToModuleTasks,
+        scriptModuleResolver = scriptModuleResolver(_, resolveSingleModule, _, _)
       )
     }
   }
@@ -97,7 +114,8 @@ final class EvaluatorImpl private[mill] (
           scriptArgs,
           selectMode,
           allowPositionalCommandArgs,
-          resolveToModuleTasks
+          resolveToModuleTasks,
+          scriptModuleResolver = scriptModuleResolver(_, resolveSingleModule, _, _)
         )
       }
     }
@@ -115,7 +133,8 @@ final class EvaluatorImpl private[mill] (
           scriptArgs,
           selectMode,
           allowPositionalCommandArgs,
-          resolveToModuleTasks
+          resolveToModuleTasks,
+          scriptModuleResolver = scriptModuleResolver(_, resolveSingleModule, _, _)
         )
       }
     }
@@ -276,7 +295,8 @@ final class EvaluatorImpl private[mill] (
             rootModule,
             scriptArgs,
             selectMode,
-            allowPositionalCommandArgs
+            allowPositionalCommandArgs,
+            scriptModuleResolver = scriptModuleResolver(_, resolveSingleModule, _, _)
           )
         }
       }