File-backed incremental compilation for faster rebuilds

[am11]

MSBuild and the Roslyn compiler server keep running for a while after compilation and retain intermediate compilation data in cache. This makes rebuilds fast for a short time. Eventually, the MSBuild server shuts down, and a new process must rebuild the caches, making full recompilation inevitable even for a single line of code change.

This is usually unnoticeable for small projects with a few dozen source files but becomes very noticeable for large projects like System.CoreLib in the runtime repo. A file-driven option that stores intermediate compilation results on disk could let rebuilds only recompile changed units, keeping rebuild times low even after caches expire.

[333fred]

I have good news for you: #82881

[jjonescz]

Yes, but I'm saying that MSBuild does that today too. It skips C# compiler on projects whose inputs (like .cs files) didn't change. You can see that in a binlog for example. If you are seeing MSBuild rebuilding some projects unnecessarily, the problem might be elsewhere (like that some inputs actually changed unexpectedly) and the compiler cache likely won't help with that. So I'd suggest you start by investigating what's causing unexpected rebuilds for you today.

[am11]

I was actually working on dotnet/runtime#126222 and making small tweaks in CoreLib's .cs files. Even minor edits require deliberate changes and careful thinking that can take 10 to 15 minutes or more. By the time I trigger a rebuild, the MSBuild/Roslyn compiler server has usually shut down, making the rebuild noticeably slower.

[jjonescz]

I guess we could expose a way to configure the server's keepalive timeout. But still, that scenario doesn't sound like something that would be fixed by the caching compiler to me, but to be certain I would need to see the binlogs to see what is taking time in your rebuilds.

[jjonescz]

so even for a single large project like System.CoreLib

Sorry, I misread this - no, neither MSBuild nor the new compiler cache can handle this - if inputs to a single compilation change, both just re-run the compiler.

[jjonescz]

But I guess prolonging the server's keepalive timeout would help you. Feel free to open a feature request for that (if such issue doesn't already exist). Or you can write a script which periodically builds some small projects in the background to keep the server alive :D