Skip to content

Commit 2666d9c

Browse files
committed
docs: populate install section
1 parent 5eadd00 commit 2666d9c

File tree

1 file changed

+87
-30
lines changed

1 file changed

+87
-30
lines changed

README.md

Lines changed: 87 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -8,26 +8,12 @@ Using Protocol Buffers with Bazel has always been difficult.
88
Also, Bazel does not ship with a hermetic toolchain, so you may have a handful of developers whose Bazel build is broken.
99
- Nearly every Bazel user has waited for `protoc` to compile from sources many, MANY times.
1010
This universally slows down builds, especially due to issues like https://github.com/bazelbuild/bazel/issues/7095 where it is observed to be easily cache-busted.
11-
- The versioning of the protobuf module on Bazel Central Registry has fallen behind and contains many patches.
12-
As of mid-March 2024 there are [7 patches](https://github.com/bazelbuild/bazel-central-registry/tree/main/modules/protobuf/23.1/patches)
13-
which essentially constitute a fork of the protobuf repo specifically for publishing to BCR.
11+
- The protobuf Bazel module is quite complex and maintenance and support from the protobuf team has been inconsistent.
12+
By using pre-built artifacts, Bazel users can follow the same well-tested as users of other build systems.
1413
- Relying on the protobuf runtime for each language from the `@com_google_protobuf` repo forces you to use
1514
the same version of the runtime for all languages in a monorepo, and matching protoc.
16-
This makes it difficult to migrate to a monorepo, allowing some applications to move from their separate repo without changing their
17-
dependency versions.
18-
19-
The key observations:
20-
21-
- `protoc` has always been distributed as pre-built binaries on https://github.com/protocolbuffers/protobuf/releases
22-
- That distribution includes the "well known types" such as `timestamp.proto`
23-
- The protobuf runtimes for each language are distributed to the appropriate package manager such as npm or PyPI.
24-
25-
Bazel 7 introduced `--incompatible_enable_proto_toolchain_resolution` to allow us fetch `protoc` rather than re-build it!
26-
That flag ALSO decouples how each language rule locates the runtime.
27-
Thanks to that flag, this repo simply contains a toolchain that resolves those pre-built binaries.
28-
29-
See `examples` for several language rules like `py_proto_library` and `java_proto_library`.
30-
There is NO dependency on `@com_google_protobuf` anywhere.
15+
This makes it difficult to migrate to a monorepo, allowing some applications to move from their separate repo without
16+
changing their dependency versions.
3117

3218
## Support matrix
3319

@@ -38,8 +24,91 @@ There is NO dependency on `@com_google_protobuf` anywhere.
3824
| Rust | | https://github.com/bazelbuild/rules_rust/issues/2627 |
3925
| Go | | https://github.com/bazelbuild/rules_go/issues/3895 |
4026

27+
## Installation
28+
29+
Fetch this module using instructions from the release you wish to use:
30+
<https://github.com/aspect-build/toolchains_protoc/releases>
31+
32+
Enable the toolchain support by adding this to `.bazelrc`:
33+
34+
```
35+
# Introduced in Bazel 7.
36+
common --incompatible_enable_proto_toolchain_resolution
37+
```
38+
39+
### For each language, follow these steps
40+
41+
Since the user installs the proto runtimes through their existing package manager setup,
42+
the toolchain registration happens in your repository as well.
43+
44+
First, fetch the official protobuf runtime that Google publishes to package registries,
45+
using whatever Bazel rule you chose for interacting with package managers
46+
(e.g. `maven_install` or `pip.parse`):
47+
48+
- Python: https://pypi.org/project/protobuf
49+
- Java: https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java
50+
- JavaScript: https://www.npmjs.com/package/protobufjs
51+
- Go: https://pkg.go.dev/google.golang.org/protobuf/runtime
52+
53+
Second, declare the toolchain in a `BUILD` file. It looks like the following
54+
(where `LANG`, `--flag_to_protoc`, and `runtime` are replaced
55+
with appropriate values for the language and the label of the runtime you installed).
56+
57+
You can choose a Bazel package where this goes; we recommend `/tools/protoc/BUILD.bazel`.
58+
59+
```starlark
60+
load("@rules_proto//proto:defs.bzl", "proto_lang_toolchain")
61+
proto_lang_toolchain(
62+
name = "protoc_LANG_toolchain",
63+
command_line = "--flag_to_protoc=%s",
64+
progress_message = "Generating LANG proto_library %{label}",
65+
runtime = "@some-external//lib",
66+
)
67+
```
68+
69+
Finally, in the same `BUILD` file, declare the registration target...
70+
71+
```starlark
72+
toolchain(
73+
name = "protoc_LANG_toolchain.registration",
74+
toolchain = ":protoc_LANG_toolchain",
75+
# This type should be declared by the language rules:
76+
toolchain_type = "@rules_LANG//path/to/proto:toolchain_type",
77+
)
78+
```
79+
80+
...and then register it, either in `MODULE.bazel` or `WORKSPACE`:
81+
82+
```starlark
83+
register_toolchains("//tools/protoc:all")
84+
```
85+
86+
See `examples` for several language rules like `py_proto_library` and `java_proto_library`.
87+
88+
Note that there is NO dependency on `@com_google_protobuf` anywhere.
89+
90+
### Troubleshooting
91+
92+
What if you still see that protoc is compiling? This means that there is still a transitive dependency on the
93+
`com_google_protobuf` module, likely from some macro call in your `WORKSPACE` file.
94+
95+
> TODO: explain how to track down where the `com_google_protobuf` dependency is coming from.
96+
97+
> TODO: populate a list here of known issues in other Bazel modules.
98+
4199
## Design
42100

101+
### How it works
102+
103+
1. `protoc` has always been distributed as pre-built binaries on https://github.com/protocolbuffers/protobuf/releases
104+
1. That distribution includes the "well known types" such as `timestamp.proto`
105+
1. The protobuf runtimes for each language are distributed to the appropriate package manager such as npm or PyPI.
106+
1. Bazel 7 introduced `--incompatible_enable_proto_toolchain_resolution` to allow us fetch `protoc` rather than re-build it!
107+
That flag ALSO decouples how each built-in language rule (Java, Python, C++, etc.) locates the runtime.
108+
109+
Thanks to that flag, this repo simply contains a toolchain that resolves those pre-built binaries.
110+
In the user's repository, there's a small BUILD file where the toolchain is configured.
111+
43112
### Questioning why Bazel is different
44113

45114
Protobuf works fine under many build tools, using the releases and runtime libraries shipped by the protobuf team.
@@ -50,15 +119,3 @@ As with many other tools such as Swagger and GraphQL, the Bazel community is sel
50119

51120
This toolchain shows that there's no need to treat Bazel as a “special” build system vs. all the others that protobuf users rely on.
52121
https://protobuf.dev/reference/ is sufficient documentation for everyone.
53-
54-
## Installation
55-
56-
Make sure your `.bazelrc` contains
57-
58-
```
59-
# Introduced in Bazel 7.
60-
common --incompatible_enable_proto_toolchain_resolution
61-
```
62-
63-
Follow instructions from the release you wish to use:
64-
<https://github.com/aspect-build/toolchains_protoc/releases>

0 commit comments

Comments
 (0)