Add experimental Kotlin incremental compilation#1316
Add experimental Kotlin incremental compilation#1316erikkerber wants to merge 10 commits intobazel-contrib:masterfrom
Conversation
erikkerber
force-pushed
the
ek/incremental
branch
from
0e33330 to
9fe2ada
Compare
| errStream: java.io.PrintStream, | ||
| vararg args: String, | ||
| ): ExitCode { | ||
| System.setProperty("zip.handler.uses.crc.instead.of.timestamp", "true") |
There was a problem hiding this comment.
Do you need to set this, or is this pulled from the K2JVM impl?
There was a problem hiding this comment.
Pulled from BazelK2JVMCompiler. It hits the same internals as the existing implementation on the Kotlin side, and figured if it was still needed there, it may be needed here.
| // The incremental directory is roughly analogous to a Gradle transform cache directory. Must be unique per compilation target. | ||
| val incrementalDirectory = Paths.get("").toAbsolutePath().parent.resolve("_kotlin_incremental/$label/$incrementalId").toFile() | ||
|
|
||
| val kotlinService = CompilationService.loadImplementation(this.javaClass.classLoader!!) |
There was a problem hiding this comment.
Is it possible/safe to keep the CompilationService implementation around?
There was a problem hiding this comment.
Do you mean simply not loading it every compile task? I'm sure that's safe, yeah.
erikkerber
force-pushed
the
ek/incremental
branch
6 times, most recently
from
b05d750 to
bbd1f67
Compare
Replace commented out snapshot generation code Add new jar dependencies needed for incremental compilation Temporarily force persistent working directory for incremental compilation
erikkerber
force-pushed
the
ek/incremental
branch
from
bbd1f67 to
eec887d
Compare
Invalidate snapshot when underlying (abi) jar changes
Properly hook up flags to control build tool API / incremental compilation enabling
Do not create snapshots for non-incremental scenario
Restore temporary working directory logic
|
Thanks for your contributions to rules_kotlin! |
3 participants
Note
This is presented as a POC, but nowhere near merge-ready. In its current state, it may compile your apps and you may see significant build time improvements.
This adds an experimental implementation of incremental compilation via the new "Build Tools API" (BTAPI). The BTAPI itself can replace the direct usage of
K2JVMCompiler, but is also required to leverage incremental compilation.Initial Design
In short:
BazelK2JVMCompilernamedBuildToolsApiCompiler. This invokes the BTAPI.<output base>/execroot/_kotlin_incremental/)build_tools_apito enable building with the new BTAPI (only)incremental_compilationto then add incremental compilationThe rest is fairly trivial usage of the BTAPI, a little of which requires some tinkering and tribal knowledge to know how exactly it is supposed to be used. I did my best to document my knowledge, with brevity.
Performance
The main driver of performance gains are incremental builds that only build a subset of Kotlin files in a target. The incremental data passed in to subsequent builds gives an additional level of work avoidance beyond the action-based caching in Bazel.
I hope to get more comprehensive numbers, but our main test case was a sample incremental build that was ~4x faster in Gradle due to only needing to recompile very small slices of a chain of Kotlin modules. With incremental, Bazel matches what we see with Gradle.
4x isn't a fixed number, it could be greater or smaller depending on how many Kotlin files are in how many dependent targets, and how many files can skip compilation based on the affected classpaths in a build.
TODO
CompilationTask. It works, but not idea.