Add First Class BuildConfig file support

[DamianReeves]

I've been thinking of a common pattern in build systems and IDE integration and if there is something mill can do to help. Often times there is a config file that represents a project config (this may be a JSON, YAML, TOML, XML, or HOCON file for example or even a custom format like go.mod). Language support in IDEs usually expect this file to exist beside the program sources (think package.json for Node, cargo.toml for Rust, go.mod for Go Lang, requirements.txt for Python, or moon.mod.json/moon.pkg.json for Moonbit). You are often in the case where you want to have build tasks that can change aspects of the build, but you also need the developer to be able to modify the file (it is a source file in a way and a build config in another way).
One can try and copy the file into the task destination folder and manually sync things so mill maintains build isolation, but what if the user or IDE changes a setting that modifies the config. This feels like a common and fundamental concept in adding mill integration with existing language ecosystems, and I wonder if similar to Task.sources if mill should have native support for Task.buildConfig to make synchronization, maintaining the incremental build support of mill, and proper dependency invalidation in these cases simple.
I feel like this foundational tool would make it easier to add support to mill for a large set of tools that function as build config without mill users having to setup their own process of appeasing the IDE by keeping files where it expects it (i.e. mot in mill's out folder), but also appeasing mill by maintaining some form of isolation to support incremental builds.
Thoughts?

[lefou]

Do you have a more concrete example, that makes it more clear what kind of setup and workflow you do expect?

[DamianReeves]

I'll have to think more on the solution. Part of the problem, is that the solution is not straightforward to me.

Like I look at the current PublishModule in the javascriptlib and it builds its own JSON structure internally and only produces the package.json in the task's output folder, but this is not what the IDE and other ecosystem tools expects.

[lefou]

I think what's missing is a nice write up, how the JS ecosystem works and how Mill could hook into it. (The former is more interesting currently.) If that means, we should not manage an isolated package.json but need to sync some internal state with an shared package.json as part of calculating what needs to be done, so be it. The current system is one solution made by people who know JS stuff, but it's probably not the only way we can do it. To speak for myself, who thinks I do understand Mill's concepts and capabilities very well, once I get a clear understanding of the system to adapt to (e.g. JS), I think I can come up with a nice mapping. But I do not have the full picture of JS tooling right now. E.g. Which tools are there and use what files? What is standardized and what is specific to some tools? Which are the most common tools we should focus on? What files are inputs, outputs or shared configuration? Are shared configuration files meant to be understood and edited by human or tools? And various other things.