[bazel] Add support for bzlmod module system

[aaronmondal]

We are currently working on making the Clang/LLVM Bazel overlay available via the bzlmod system. We noticed that parts of our code may be desirable to have in the Clang/LLVM overlay in this repo.

What works

• We have a working MODULE.bazel file here (ignore the rules_ll specific stuff).
• An extensions.bzl file to apply the overlay etc. is here, mostly copied from configure.bzl.

What doesn't work

I would expect that we would need to fix the following issues before something like this would be acceptable for upstreaming:

• Users will need to be able to specify whether they want zlib etc. This should be simple to fix by exposing related functionality as extensions.
• Registries do not work with labels in http_archive because of this bug: Label resolution fails by repo mapping when used by a repo rule inside a module extension bazelbuild/bazel#14659. Workarounds are possible, but not pretty.

Conclusion

Please let me know whether bzlmod support is something that would have a place in the bazel overlay in this repository. If it is, please also let me know which issues, apart from those already mentioned we would have to address.

[fmeum]

@aaronmondal If you test with recent rolling releases of Bazel, local overrides of modules also get a repo name with a suffix, which means that you can more easily reproduce issues caused by that locally.

bazelbuild/bazel#15553 contains some ideas on how to handle bzlmod repo names in copts.

[aaronmondal]

@fmeum Thanks for the infos! The first one should make testing a lot easier. The second one may be able to fix the issue with //clang:ast, and may help with //clang:builtin_headers_gen. I'll post an update as soon as I implemented it on our side 😊

[aaronmondal]

Ok things seem to be working now. I updated the initial post and will send a patch for review.

[aaronmondal]

Some groundwork proposed in https://reviews.llvm.org/D136496. Also opened an issue in the BCR: bazelbuild/bazel-central-registry#206

For those interested, I am upstreaming from https://github.com/eomii/bazel-eomii-registry/tree/llvm-project-overlay/modules/llvm-project-overlay.

[fmeum]

@aaronmondal Maybe I just haven't figured out Phabricator, but I think I'm unable to comment on the LLVM revision, so I'm going to comment here.

You should indeed be able to drop the Label constructors except for the one line where you pass the label to repository_ctx.path. Other than that, the PR looks good to me and should be a no-op in WORKSPACE builds.

The way you patch a repo through a symlink looks slightly "adventurous" and I'm not 100% convinced it won't break some assumptions Bazel makes about external repos, but if it's only going to be used by LLVM devs, I think that would be acceptable.

[aaronmondal]

@fmeum

You should indeed be able to drop the Label constructors except for the one line where you pass the label to repository_ctx.path. Other than that, the PR looks good to me and should be a no-op in WORKSPACE builds.

Yeah that was indeed an oversight. I keep losing track of all the changes locally in all the different repos 😅

The way you patch a repo through a symlink looks slightly "adventurous" and I'm not 100% convinced it won't break some assumptions Bazel makes about external repos, but if it's only going to be used by LLVM devs, I think that would be acceptable.

The easiest variant is for sure to use a http_archive. Patching the symlinked copy works with the initial build, but breaks rebuilds since the original sources will be modified. This is especially annoying when one wants to test changes via a local_override for the llvm-project. The symlinked copy should only be available when one uses no patches. This is an error in the current implementation in the demo registry where I accidentally swapped some if-else statements that check for this.

[keith]

@aaronmondal where did you leave this one, are you interested in working on this still? I might be interested in picking it up