Task.fork.async, how does Context get passed into it and where can it write to?

[sebhajek]

So I have been experimenting with using mill to build C/C++, it works well enough but I attempted to add parallelism into the compilation itself, and ran into some issues with how Task.ctx().fork.async and Task.ctx().fork.awaitAll work... This is my compile tasks code:

  def parallelCompile: Boolean = false

  def compile =
    Task(persistent = true) {
      val compileParams = sourceFiles().map { input =>
        val outputPath = Task.dest / input.name.concat(objectExtension.prepended('.'))

        val includeFlags = includeDirs()
          .concat(sources())
          .map(incDir => "-I".concat(incDir.path.toString))
          .toSeq

        val command = Seq("gcc", "-c")
          .appended(input.path.path.toString)
          .appended("-o".concat(outputPath.toString))
          .concat(includeFlags)

        val callCompiler: () => CompilationCommand = () => {
          os.call(cmd = command)
          CompilationCommand(PathRef(outputPath), command, input)
        }

        val asyncParams
          : (Logger => CompilationCommand) => Future[CompilationCommand] = {
          Task.ctx().fork.async(dest = Task.dest, input.name, command.mkString(" "))
        }

        (callCompiler, asyncParams)
      }
      if parallelCompile then {
        val compileJobs = compileParams.map { case (callCompiler, asyncParams) =>
          asyncParams { logger => callCompiler.apply() }
        }
        Task.ctx().fork.awaitAll(compileJobs.toSeq).toSet
      } else compileParams.map { case (callCompiler, _) => callCompiler.apply() }
    }

not particularly pretty, it's still more of a PoC than anything, but it works... atleast as long as I don't run it with parallelCompile = true. When I try to do the task spawning, about half the time I finish compilation just fine, the other half, I get:

$ ./mill __.clean && ./mill __.compile
[1/1] clean
[1/1] ============================== __.clean ==============================
[50/56, 3 failed] ============================== __.compile ==============================
3 tasks failed
[52] lua.compile java.lang.RuntimeException: Writing to out/lua/compile.dest not allowed during execution of `lua.luac.compile`.
Normal `Task`s can only write to files within their `Task.dest` folder, only `Task.Command`s can write to other arbitrary files.
[51] lua.libraries.compile java.lang.RuntimeException: Writing to out/lua/libraries/compile.dest not allowed during execution of `lua.luac.compile`.
Normal `Task`s can only write to files within their `Task.dest` folder, only `Task.Command`s can write to other arbitrary files.
[54] lua.luac.compile java.lang.RuntimeException: Writing to out/lua/luac/compile.dest not allowed during execution of `lua.compile`.
Normal `Task`s can only write to files within their `Task.dest` folder, only `Task.Command`s can write to other arbitrary files.

Can someone enlighten me on why this is happening? The docs don't really explain how exactly the context is being passed into the spawned futures (or atleast I didn't understand it from there) and I assume the original task from which they were spawned still has to be running because of the await, so that kinda confuses me...

Also is there any other way to do something like this while still remaining idiomatic to mill?

Thanks for your help, any insights are really appreciated.