Sharing outputs between subprojects

[aSemy]

It's very easy to share outputs between projects in a way that breaks Gradle optimisations, or ordering.

1. Reference tasks, or task outputs from other projects
2. Hardcode configuration names between provider/consumer projects
3. Directly using files or directories from another project

These workarounds are really easy to do, but they break Gradle optimisations and caching, and can make refactoring more difficult.

I've always found the Gradle docs are quite confusing on this topic. They contain a lot of information, assume a lot of knowledge, start with a "don't do this!" example, and are generally not easy for me to follow. (This example was more clear, but it's now been removed).

But sharing outputs is not as hard as the docs make it seem - aside from some weird names for things.

I've written a more to-the-point guide, and I'd like to contribute it to this project. I think it's too large to include in the main README though. What do you think?

(Click to expand) Gradle - Sharing outputs between projects SAFELY

('Configurations' doesn't mean 'how to configure your project', it's more like how a naval fleet might have a 'battle configuration', with ships in specific positions to engage the enemy, or a 'restock at the harbour' configuration.)

1. Create some 'variant attributes'. Again, the docs make them sound way more complicated than they are. They're just key-value tags, used to differentiate between different files. Files might be tagged as 'Java source files' or 'JaCoCo coverage data'.

   The values can be any string value, and Gradle provides some default keys which we can re-use. (Custom keys can be manually registered, which is more effort than it's worth.)

   Any Configuration can be tagged with these attributes. Gradle will use the tags to play matchmaker between the Configurations.

   It's nice to define the attribute-tags in buildSrc so they can be more easily re-used.

   // buildSrc/src/main/kotlin/distributions/attributes.kt
   package distributions
   
   import org.gradle.api.artifacts.Configuration
   import org.gradle.api.attributes.Usage.USAGE_ATTRIBUTE
   import org.gradle.api.model.ObjectFactory
   import org.gradle.kotlin.dsl.*
   
   fun Configuration.factorioModAttributes(objects: ObjectFactory): Configuration =
     attributes {
       attribute(USAGE_ATTRIBUTE, objects.named("my.project.factorio_mod"))
     }

   USAGE_ATTRIBUTE is a built-in Gradle attribute-key, which we can re-use - so long as we make the value distinct enough, it won't clash!

2. Configurations are used to both provide and consume artifacts. It's a bit awkward trying to remember which combinations of booleans are needed (there is a table (Table 1. Configuration roles), but it's always hard to remember). So I like creating some helper utils - again, in buildSrc.

   // buildSrc/src/main/kotlin/distributions/configurationUtils.kt
   package distributions
   
   import org.gradle.api.artifacts.Configuration
   import org.gradle.kotlin.dsl.*
   
   /** Mark this [Configuration] as one that will be consumed by other subprojects. */
   fun Configuration.asProvider() {
     isVisible = false
     isCanBeResolved = false
     isCanBeConsumed = true
   }
   
   /** Mark this [Configuration] as one that will consume (also known as 'resolving') artifacts from other subprojects */
   fun Configuration.asConsumer() {
     isVisible = false
     isCanBeResolved = true
     isCanBeConsumed = false
   }

   This makes it easier to declare Configurations as outgoing providers, or incoming consumers.

3. In the 'provider' project, create an 'outgoing' Configuration, that will be consumed by other projects. Use register(), because this configuration will be resolved on-demand. Add files using the Configuration.outgoing property.

   // subproject-factorio-mod/build.gradle.kts
   import distributions.asProvider
   import distributions.factorioModAttributes
   
   val factorioModProvider by configurations.registering<Configuration> {
     asProvider()
     factorioModAttributes(objects)
     outgoing.artifact(tasks.distZip.flatMap { it.archiveFile }) // using 'map'/'flatMap'
   }

   When adding outgoing artifacts, try to use map() on task providers to get the output, so Gradle can be clever and automatically run the task when the 'outgoing' Configuration is consumed.

4. Create an 'incoming' Configuration that will consume Configurations from other projects.

   Use create() so the configuration plays nicely with the Kotlin DSL (see next step). The handy mnemonic I use is "Create Consumers"

   // subproject-factorio-server/build.gradle.kts
   import distributions.asConsumer
   import distributions.factorioModAttributes
   
   val factorioMod by configurations.creating<Configuration> {
     asConsumer()
     description = "Consumes Factorio Mod zip files from other subprojects"
     factorioModAttributes(objects)
   }

5. In the consumer project, add dependencies on another project using the 'incoming' Configuration.

   // subproject-factorio-server/build.gradle.kts
   
   dependencies {
     factorioMod(project(":subproject-factorio-mod"))
   }

6. The 'incoming' Configuration can now be resolved in a task, which will trigger the provider project to run its task and share them.

   // subproject-factorio-server/build.gradle.kts
   
   val deployModToFactorioServer by tasks.registering(Copy::class) {
     description = "Copy the mod to the Factorio Docker server"
   
     dependsOn(factorioMod)
   
     from(
       { factorioMod.incoming.artifacts.artifactFiles.files }
     )
     into(layout.projectDirectory.dir("factorio-server/mods"))
   
     doLast {
       logger.lifecycle("Copying mods ${source.files} into $destinationDir")
     }
   }