Use a separate cache entry for OS independent files

[quyykk]

I was surprised to learn that run-vcpkg stores the vcpkg port and versioning files alongside the vcpkg exectuable in its cache. This makes the cache at least 70MB with lots of duplication across OSs. Since these files without the executable are OS independent, they should be stored in one (separate) cache usable by every OS (this has recently become possible).

Thoughts? Thanks!

[lukka]

@quyykk this is an interesting idea indeed, and I wonder if we really need caching the vcpkg repository content (along with the vcpkg executable), especially since most of the time would have be spent building the vcpkg executable which is not the case anymore since a prebuilt version is downloaded, so it already works similarly like a cache.

To disable and experiment without run-vcpkg's caching, it suffices to set doNotCache:true.

I'll experiment with it.

[quyykk]

I was just testing without caching and indeed, restoring from cache takes ~10s and downloading the vcpkg repo also takes ~10s + ~1s (the executable).

[quyykk]

There's not always a prebuilt version available (with Linux ARM being the biggest missing). Maybe the doNotCache input could be inverted?

[lukka]

@quyykk good idea, i can make by default doNotCache:true by default.

[lukka]

@quyykk i am going to disable the cache of vcpkg+ports file by default and while testing this out, i noticed that the GH Action cache service is being throttled and refusing to answer too many requests altogether (not sure if this is done per user or per repo) with http status 429, for example: https://github.com/lukka/CppBuildTasks-Validation/actions/runs/4592317371/jobs/8109258929#step:5:94

How is vcpkg reacting to 429 http status? It should be retrying or at least warning that it failed the upload and why.

I think there before there was one request per run-vpkg run, and now there are many requests per run-vcpkg run, each for each package. When using boost in vcpkg.json, as I am doing now on my final validation workflows for run-vcpkg and run-cmake, I see that sometimes packages are not being uploaded (looks like they silently fail, because they should be uploaded after been built for the first time), and then subsequent runs are building it again because they were not uploaded successfully before.

[lukka]

@quyykk with release https://github.com/lukka/run-vcpkg/releases/tag/v11.0 the cache of vcpkg's exe and port files is disabled by default, and can be turned out manually by the user (doNotCache:false).

it may be ideal that the flag is turned on automatically by the action though: #193

[quyykk]

How is vcpkg reacting to 429 http status? It should be retrying or at least warning that it failed the upload and why.

I don't think it retries, it just fails. It should say that it uploaded 0 packages to the cache though I believe. There is a retry mechanism when restoring from what I can tell.

[lukka]

@quyykk yes, when the upload to the cache fails for a single port, it does not report an error but it says "Uploaded 0 package(s) to GHA in 123 ms.".

The takeaway is that when using lot of ports (e.g. the boost metapackage) with the cache granularity of one entry per port, hitting the GitHub Action cache rate limits is happening quite often leading to some sporadic failure, which once relaunched manually later then they succeed.

[quyykk]

Yeah, Boost has a lot of header only libraries that take no time to build. I'll see if I can add a retry mechanism to vcpkg-tool 😄

[lukka]

@quyykk if retrying is a viable option, sounds good. Otherwise if retrying could lead to worsening the problem, it may be sufficient to write in the output the reason/failure that the built package has not been saved on the GH cache. A summary at the end would be a nice to have, e.g.,: 40/42 packages saved on GHA cache, 2 failed: name1, name2.

I think all aforementioned suggestions could be a request on vcpkg-tool issue section.