--experimental_remote_repo_contents_cache

[fmeum]

Use this discussion to provide feedback on the new remote repository cache introduced in Bazel 9 as an experimental feature.

FAQs

How do I enable the remote repo contents cache?

• Enable the --experimental_remote_repo_contents_cache startup flag.
• Use any remote cache (both HTTP and gRPC are supported) via --remote_cache or --remote_executor.

Which repo rules are supported?

• A repository rule has to return repository_ctx.repo_metadata(reproducible = True) to be eligible for caching.
• As a further limitation of the current experimental implementation, only repo rules without any dependencies added at runtime (e.g., via repository_ctx.watch or .getenv) are supported. We hope to lift this restriction in the future. You can check a repo with canonical name @@<repo_name> for runtime deps by opening the file $(bazel info output_base)/external/@<repo_name>.marker - all lines after the first one are runtime deps.

[keith]

As a starting point I have 2 initial questions:

• do we need to worry about anything on the remote exec service side that needs to be added for this?
• is there an easy way to spot out repository_rules that have the watch / getenv requirements that maybe don't need them so we could change them to be compatible with this?

[fmeum]

I started an FAQ list with answers to these questions.

[malt3]

The commit message contains this:

Repositories are cached in a regular remote cache as AC entries

Do I interpret this correctly to mean that I need to allow Bazel to write to the remote AC directly?
I think many companies only allow the remote executor to set AC entries for the remote cache to avoid attackers poisoning the remote action cache.

Since repository rules still run locally (unless they are cached), this design makes a lot of sense.
Just trying to get confirmation.

[fmeum]

That is correct. My general assumption has been that you would allow CI machines to write AC entries and those would then also be able to populate the remote repo contents. Devs could then consume those just fine. Are you really referring to a setup in which Bazel itself never writes AC entries but remote executors do so directly?

[malt3]

Are you really referring to a setup in which Bazel itself never writes AC entries but remote executors do so directly?

Yes that's precisely correct. I believe BuildBuddy and BuildBarn support this configuration.

[malt3]

I think the feature is still great and I'll enable it in most environments, but I think it's important to point out what it entails.

[fmeum]

This would be difficult to support since it's completely unclear how to remotely execute (even a reproducible) repo rule - this would essentially require turning Bazel into a standalone Starlark interpreter of sorts. I guess at that point it would be easier to set up a trusted "repo rule evaluation" machine running regular Bazel?

[malt3]

Yes, sorry. I think we are talking about different things.
I was merely saying that it's a common setup to prevent writes to the AC from Bazel.
I agree that it's extremely hard to work around that limitation.

[sluongng]

I discussed this with @Wyverald at BazelCon:

This feature is really good at helping CI operators reduce the typical size of their CI environment. By not having to materialize the external/ dir on disk and delegating most of the builds to the RBE server, the CI workspace can be shrunk down significantly.

The current implementation relies on an Action Cache(AC) write to the remote cache. AC in most enterprise setups is heavily gated and can only be written from a trusted CI worker/pipeline. This is recommended since 2018 through this talk https://youtu.be/5a0ENnZivo0?t=1041

When a user were to introduce a new external dependency, or upgrade a current one, the trusted CI worker machine will have to (a) download the new external deps and (b) calculate the new AC entry to update the remote cache accordingly. This process may require significant disk space to store the downloaded data.

Because of this, CI server operators, like myself, will see a typical cached CI workspace using very little disk space, but occasionally, there can be workspaces with 10-50x the disk size. This makes it really hard for us to schedule resources for these CI environments. In addition to this, our CI offering (BuildBuddy Workflows) utilizes a VM snapshot system, which may retain the disk content after each build. The bigger the VM, the harder it is to do the snapshot and distribute it among multiple workers.

For this reason, I think the current setup can be improved:

a. Providing separate tools to manually create and populate the AC entries without Bazel. Dependency upgrades / newly added can then go through a separate CI step that would prepopulate the AC entry. This means that all Bazel Workspaces in CI will get a cache hit and expect no disk inflation.

b. Allowing the reproducible repo rules to run remotely using RBE. This would let us offload the downloads and preparation steps to the remote worker. All downloads are expected to be verified by a checksum to ensure reproducibility.

[fmeum]

The challenge is that almost no repo rule does just download_and_extract. Look at all the logic coming before and after that in http_archive:

bazel/tools/build_defs/repo/http.bzl

Line 200 in b8c8d55

download_info = ctx.download_and_extract(

We would probably have to introduce a new simplified version of http_archive without these extra features and special case it in Bazel. At that point it seems easier to use the approach of e.g. rules_js and extract a downloaded file in a build action. This reduces the problem to http_file with integrity, which we could conceivably make a special case in Bazel.

[fmeum]

I just realized that an alternative that could work would be to optimistically inject the remote result of extraction and then carry on with the repo rule execution, carefully tracking operations that may be influenced by these files not being present on disk. For example, this would rule out patch_cmds, but that's acceptable. This would be pretty complex to build - I hope that the remote repo contents cache is good enough, but if not, this could be considered.

[sluongng]

For extraction, I suspect we should implement FetchDirectory in the remote asset api as part of the work?

That means the remote server will download and extract, then give Bazel a tree digest to selectively download what it needs.

[fmeum]

Yes, that way we would avoid having to ship a standalone extractor binary.

[sluongng]

If we want to be fancy, we can do it couple ways:

1. FetchDirectory rpc
2. RemoteExec (network enabled) with extractor binary and Directory output
3. Local extraction -> remote AC cache the mapping between downloaded blob and output dir digest?

and fall through 1 -> 2 -> 3?

Once we have the tree digest, we can call GetTree to get the structure of the tree and selectively download only stuff we need (BUILD and bzl files)

[sluongng]

I could be wrong, but the request metadata seems to be a little bit off from the usual convention:

https://cs.opensource.google/bazel/bazel/+/60bc017bca7dcdc6662061d591b544abac8d6107:src/main/java/com/google/devtools/build/lib/remote/RemoteRepoContentsCacheImpl.java;l=269-270

I think the repo name should be placed in the label and the action id should be a fixed constant (i.e. repository_content)

[fmeum]

You are right. Passing in anything but an action ID would probably require a bit of refactoring work to avoid having to synthesize an ActionAnalysisMetadata object. But it would be the correct approach. The mnemonic could also be set to the repo rule name.