Move Zig /tmp cache to a Bazel repository

[vardaro]

At this time, the hermetic_cc_toolchain toolchain directs the Zig cache to the host machines /tmp directory. This directory is referenced absolutely and remains external to Bazel, leading to reproducibility issues. This change proposes directing the Zig build cache to a known Bazel repository, eliminating the dependency on /tmp.

At this time, Zig input environment variables are composed in the zig_repository implementation, however I think we could move this logic into zig_repository_cache, and encode those variables in the zig.env file it creates. This is similar to how Gazelle resolves Go environment variables. I didn't do it here to keep the PR scoped to eliminating /tmp.

I tested this on Datadog's internal monorepo, but is there a more established process to test these changes?

[CLAassistant]

All committers have signed the CLA.

[motiejus]

Consider updating the setup instructions in the README and adding a "fixes #83" (if I understand this PR right)

[motiejus]

This has been bugging my mind for longer than I want to admit. Thanks. :)

Will leave a real review to @linzhp.

[aherrmann]

Doesn't this inject an absolute path to Bazel's output base into the action key of every action that uses this toolchain?
This would likely induce cache misses with a shared remote cache across machines.
Also, how does this interact with remote execution setups?

[vardaro]

Thanks for pointing this out! Indeed, it does leak ugly sandboxing details into the action invocation.

Perhaps we can make this a relative path, starting from /external?. I tried this locally, where cache_prefix becomes something like external/zig_sdk_cache. This did not work OOTB, and I think the reason is because ZIG_LOCAL_CACHE_DIR and ZIG_GLOBAL_CACHE_DIR are not intended to handle relative paths. In which case, perhaps we can modify zig-wrapper.zig to reconnect the relative path to the absolute path of the output base within the action sandbox? FWIW, it appears that the wrapper does something similar to this to compute ZIG_LIB_DIR.

[aherrmann]

I think the reason is because ZIG_LOCAL_CACHE_DIR and ZIG_GLOBAL_CACHE_DIR are not intended to handle relative paths.

FWIW I used to use relative local and global cache paths in rules_zig until aherrmann/rules_zig#253 . The reason for the switch was unrelated to the relative paths and instead had to do with Zig 0.12 producing dangling symlinks in the cache which Bazel doesn't allow for build outputs...

That said, it's not hard to believe that relative cache dirs could be problematic, e.g. I encountered ziglang/zig#19284 in a 0.12 pre-release build. Converting to an absolute path in the wrapper sounds like a good middle ground.

[linzhp]

we can modify zig-wrapper.zig to reconnect the relative path to the absolute path of the output base within the action sandbox

I like this approach. Maybe we can set ZIG_LOCAL_CACHE_DIR and ZIG_GLOBAL_CACHE_DIR to $(pwd)/$HERMETIC_CC_TOOLCHAIN_CACHE_PREFIX in zig-wrapper.zig?

[jadenPete]

Hi, @vardaro. We're seeing failing builds with hermetic_cc_toolchain because our actions are executed under different users and /tmp/zig-cache is owned by the first user to create it. I'm hoping that having the cache owned by Bazel will fix it. Are there any plans to revive this PR?

[vardaro]

Hi, @vardaro. We're seeing failing builds with hermetic_cc_toolchain because our actions are executed under different users and /tmp/zig-cache is owned by the first user to create it. I'm hoping that having the cache owned by Bazel will fix it. Are there any plans to revive this PR?

Hey, I'm no longer working at the employer funding this work. I'm surprised this is still relevant, I recall it being fixed by another PR?

[jadenPete]

Thanks for your response! I think it is, as /tmp/zig-cache is still created on the latest uber/hermetic_cc_toolchain.

If you're no longer working on this, would you mind closing this PR? I'm going to take a stab rebasing this branch on the latest main and fixing the absolute path issue .