Mill Plugin Dependancy Resolution

[Quafadas]

I was experimenting with mdoc-in-mill. What I noticed was that if I run mdoc by spawning a new JVM and manage its class path myself, it works well. Pseudo code

    mill.util.Jvm.callProcess(
      mainClass = "mdoc.Main",
      classPath = mdocLibs_,// ++ Seq(jsSiteModule.mdocJsProperties().path),
      jvmArgs = forkArgs(),
      env = forkEnv(),
      mainArgs = args
    )

With mill 1.0 and scala 3, it seems that it should also now be possible to consume mdoc as a library in a mill plugin. This seems to offer many upsides (e.g. not managing class paths, no new JVM to spawn) and means it can be more "idiomatically mill". By adding mdoc as a dependancy to the build I got this working. Pseudocode

   mdoc.SbtMain.main(
     args = mdocArgs().toArray ++ Seq("--out", Task.dest.toString())
   )

It passes unit testing locally, but after publishing the plugin and and consuming in an "application" build,

I hit the following error;
java.lang.NoClassDefFoundError: scala/meta/internal/io/PathIO$

I'm quite sure, that this is from scala meta commons, which is not published for scala3.

So, it appears that the way mill resolves the transitive dependancies when consuming a plugin, is different, to the way that they are resolved at plugin compile / build / publish time.

Firstly; is that fair? I can imagine a lot of reasons for that to make sense best but I guess it's best to start from the start. Is there a way I might go about understanding that mechanism and / or working around this? It's not important per se, as the original strategy works, but I thought it should have worked as a library and that triggered my curiosity, hence the question.

[lefou]

Plugins are essentially just dependencies defined in the meta-builds. So you can inspect as usual like this (outputs from Mill itself):

> mill --meta-level 1 show mvnDeps
[
  "com.github.lolgab::mill-mima_mill1:0.2.0",
  "net.sourceforge.htmlcleaner:htmlcleaner:2.29",
  "com.lihaoyi::mill-contrib-buildinfo:1.0.3",
  "com.goyeau::mill-scalafix_mill1:0.6.0",
  "org.jsoup:jsoup:1.21.1"
]

> mill --meta-level 1 show compileClasspath
[
  ...
  "qref:v1:fa84a4b6:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/github/lolgab/mill-mima_mill1_3/0.2.0/mill-mima_mill1_3-0.2.0.jar",
  "qref:v1:282fd8a5:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/net/sourceforge/htmlcleaner/htmlcleaner/2.29/htmlcleaner-2.29.jar",
  "qref:v1:f8c5f07f:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/mill-contrib-buildinfo_3/1.0.3/mill-contrib-buildinfo_3-1.0.3.jar",
  "qref:v1:2c9eb85b:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/goyeau/mill-scalafix_mill1_3/0.6.0/mill-scalafix_mill1_3-0.6.0.jar",
  "qref:v1:f671a3de:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jsoup/jsoup/1.21.1/jsoup-1.21.1.jar",
  ...
]

>  mill --meta-level 1 show runMvnDeps
[
  "com.lihaoyi::mill-core-eval:1.0.3"
]

> mill --meta-level 1 show runClasspath
[
  ...
  "qref:v1:282fd8a5:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/net/sourceforge/htmlcleaner/htmlcleaner/2.29/htmlcleaner-2.29.jar",
  "qref:v1:f8c5f07f:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/mill-contrib-buildinfo_3/1.0.3/mill-contrib-buildinfo_3-1.0.3.jar",
  "qref:v1:2c9eb85b:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/goyeau/mill-scalafix_mill1_3/0.6.0/mill-scalafix_mill1_3-0.6.0.jar",
  "qref:v1:f671a3de:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jsoup/jsoup/1.21.1/jsoup-1.21.1.jar",
  "qref:v1:5a3ed00a:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/mill-core-eval_3/1.0.3/mill-core-eval_3-1.0.3.jar",
  ...
]

Please notice the difference between compile and runtime. The former is what is used for compilation, the latter what's seen by the JVM at Mill execution time.

You should inspect if the output you get reflect what you expect. If transitive dependencies are missing, have a look how they are defined in the published artifacts (inspect the poms or their build setup).

If something is missing: Do you have some dependencies accidentally defined as compileMvnDeps (which ends up as <scope>provided</scope> in the pom.xml) or do they have the <optional> flag in the pom.xml?

[Quafadas]

Awesome answer... need to set aside some headspace to investigate properly with it. Thanks...

[Quafadas]

I'm really struggling to follow this. I must be doing something dumb somewhere, but I can't see it.

//| mill-version: 1.0.3-jvm
//| mill-jvm-version: 17
//| mvnDeps:
//| - org.scalameta::mdoc:2.7.2

package build

import mill._, scalalib._

object foo extends ScalaModule {

  def scalaVersion = "3.7.2"
}

object M extends MdocModule with ScalaModule {

  def scalaVersion = "3.7.2"

  def moduleDeps = Seq(foo)
}

The mvnDeps task fails.

simon@Simons-Mac-mini mill_mdoc_2 % mill --meta-level 1 show mvnDeps
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::load has been called by mill.launcher.JLineNativeLoader in an unnamed module (file:/Users/simon/.cache/mill/download/1.0.3)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

Another Mill process is running 'IDE:buildTargetCompile', waiting for it to be done...
[1/1, 1 failed] ============================== show mvnDeps ==============================
1 tasks failed
show java.nio.file.NoSuchFileException: /Users/simon/Code/mill_mdoc_2/out/mill-build/mvnDeps.json
    java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)

What's stranger, is that the compileClasspath and runClasspath invocations, are successful! This breaks my "intuition" such that it is, about how mill works, as I expected the class paths to be dependant on the "mvnDeps" task, which, according to "plan" it is.

In fact, the classpaths even contain everything I would have expected to be needed, for mdoc to run. The the smallest repro I have is here; but I'm out of my depth.

https://github.com/Quafadas/mill_mdoc_2

[lefou]

That is strange. I opened issue #5745

[Quafadas]

@lefou : Thankyou, for taking the time to look