Different versions of a depedency in meta build

[arminsumo]

What is the recommended way to have different versions of a particular dependency in the meta build?
eg I have

//| mvnDeps:
//| - org.openapitools:openapi-generator:4.2.3

now I also need org.openapitools:openapi-generator:6.6.0 for a different module...

I was trying to define this in a separate package in the mill-build like this:

build.mill
versions.mill
  /mill-build
     mill.build
     openapiv4
         package.mill
         src/
     openapiv6
         package.mill
         src/

so that each of the respective package.mills could have their own separate version of the generator, but this seems to not work, as I can only access direct sources of mill-build from the main project, not any of the nested ones.

A bit tangential, but it also seems to not be possible to shared code down to the mill-build. For example, mill-build can't access the versions.mill file. This can lead to a bunch of code duplication.

One thing I could think of is forgoing the mill-build thing entirely, and doing some hacky thing like this

  def compileMvnDeps = Seq(mvn"org.openapitools:openapi-generator:6.6.0")
  def getCodegen = Task {defaultResolver().classpath(compileMvnDeps()).head}

and then use URLClassLoader to use it, but obviously it's not clean/ideal.

Basically, I want to be able to write locally defined plugins, which can have their own set of independent dependencies, that I can use in my main project without having to publish those separately.
Is there a good way to achieve this?

[lihaoyi]

@arminsumo typically for such scenarios, we use classloaders or JVM subprocesses to isolate the conflicting classpaths. See the docs below for examples:

• https://mill-build.org/mill/extending/running-jvm-code.html

You can also look for examples of this in the Mill codebase, e.g. https://github.com/com-lihaoyi/mill/blob/main/libs/javalib/classgraph-worker/src/mill/javalib/classgraph/impl/ClassgraphWorkerImpl.scala

[arminsumo]

I see, that was kind of what I was assuming as well.
It would be great to have this kind of feature in the future.. being able to define more than one independent "mill-build" like a normal package.mill (or being able to have packages within mill-build) would be amazing. For my case (very large gradle monorepo project with many legacy/custom and new plugins intertwined), it would definitely make migration much simpler. The other parts of mill are a breeze for the most part, but migrating the plugins can be quite difficult.

[lihaoyi]

Yeah there's definitely more that can be done to make this easier, although there are some hard boundaries that are enforced by the JVM only allowing a single version of any class on a classpath. It's not immediately obvious what the best way forward is, but maybe as you progress in migrating you'll end up with some helpers built on top of mill.util.Jvm that you can upstream to make things simpler for others

[arminsumo]

I'm not as well versed in build tool internals as you are, so it may not be the correct way to think about it, but some things I have observed:

1. In a single project, each module (ie package.mill) can have same compile time dependencies at different version, even if they depend on each other.
2. plugins pulled from externally can have different versions have same compile dependencies. Eg I could have something like

//| mill-version: 1.0.2
//| mvnDeps: ["io.github::openapi-4-mill-plugin::0.1.0",  "io.github::openapi-6-mill-plugin::0.1.0"]

where each plugin has a compile time dependency on the respective openapi generator versions.
From this we know different versions of the same classpath are fine, as long as those are isolated from each other.

So the only difference is that we want to define/build those locally as part of the main project build, instead of having them defined and pulled from externally. Seems like a very similar concept to the build.mill/package.mill system that's already in place to create independent modules, but we just want to use that in the build system itself, rather than to the application code. It seems like we are already achieving something like this with the suggest class loader strategy, so seems like it's possible to do that?

But I am probably missing something crucial about the internals, that makes this unfeasible in reality.