Code coverage dropped to ~20% after bpart changes

[giordano]

Based on https://coveralls.io/builds/71969345 and https://app.codecov.io/gh/JuliaLang/julia/commit/f209eba244d55afbf7aeff298434deba4fcbe30a, f209eba (merge commit of #57150) code coverage of Base dropped from >90% to ~20%.

[inkydragon]

Seems related #57053

[giordano]

Seems related #57053

Is it? f209eba was pushed 14 days after you opened that issue.

[IanButterworth]

Just checking, @Keno is this on your radar, given the push to get the scheduled CI back in good shape. At least understanding why it happened.

i.e. latest is at 25% which fails the coverage job (the last remaining solid failure currently) https://buildkite.com/julialang/julia-master-scheduled/builds/1227#0198741a-00f6-4b76-b238-42c2e0d944a0/2121-2486

[IanButterworth]

Coveralls has the narrowest good/bad state:

85% https://coveralls.io/builds/71918402 - 6cd750d
17% https://coveralls.io/builds/71969345 - f209eba

So somewhere in 6cd750d...f209eba

And the distrubuted changes JuliaLang/Distributed.jl@c613685...51e5297

[Keno]

We've determined that the root cause here is a failure of the sysimage to start up in time. It was never particularly fast, but did get slower in 1.12 because we changed at what point the JIT kicks in. The bootstrap code has a special escape hatch to get around this, but it doesn't get triggered by --sysimage-native-code=no. In JuliaCI/julia-buildkite#471 we're trying to just build the full sysimage with coverage which would obsolete that concern. If that doesn't work, we can try to play some games with the cache and either invalidate it or properly implement the flags based cache partitioning.

[vtjnash]

It would be slightly dangerous to build with the code-coverage flag, though I think we fixed most of the bugs there, we also take precautions to ensure it cannot affect the generated code (it is a runtime flag, not a compiler one), so sysimage-native-code is required regardless.

In v1.12, our plan was to stop using sysimage-native-code=no entirely for this (it is already deprecated) and fallback instead to adding an internal "drop all caches" function, which increments the world age then drops all caches, as if by adding a method that had intersected with everything. (c.f. implementation in staticdata.jl which already does this for pkgimages, just not yet for Base)

[Keno]

fallback instead to adding an internal "drop all caches" function, which increments the world age then drops all caches

I had proposed this, before @IanButterworth proposed just building the sysimage with the flag which I said sounded simpler if it worked (and it kind of seems to have, since the build at least ran). I do think capping the code instances is fine as well though.

[IanButterworth]

It produced 70% coverage and ran in basically the same time as a normal build and test run https://coveralls.io/github/JuliaLang/julia?branch=IanButterworth%3Aib/coverage_fix

[IanButterworth]

Fixed. Back up to ~78%