feat(gazelle): Gazelle plugin generates py_proto_library #3057

shaldengeki · 2025-07-05T00:45:27Z

Please go over this with a fine-toothed comb! This is my first contribution to rules_python / the gazelle plugin, and while I've worked in Gazelle before, I'm pretty unfamiliar with the Python plugin's architecture.

This adds support in the Gazelle plugin for generating py_proto_library rules automatically, if there are any proto_library rules detected in a given package. We do this via a new Gazelle directive, python_generate_proto, which defaults to true, and controls whether these rules are generated.

See the tests in testdata/directive_python_generate_proto for examples.

By default, we source the py_proto_library rule from the @protobuf repository. I think this the intended long-term home of the rule? Users are expected to use gazelle:map_kind to change this if need be.

I haven't done anything here to support resolution of imports of py_proto_library. I think this is worth landing first, to save folks from having to maintain these by hand. But this should lay the foundation for resolving that in #1703.

…of scaffolding & tests

… the singular input proto

…opy is deprecated

…lign with the other directives

…ic to implement this

CHANGELOG.md

gazelle/python/testdata/directive_python_generate_proto/test.yaml

gazelle/python/generate.go

...lle/python/testdata/directive_python_generate_proto/test7_removes_when_unnecessary/BUILD.out

gazelle/python/testdata/directive_python_generate_proto/README.md

gazelle/python/testdata/directive_python_generate_proto/test1_default_with_proto/foo.proto

gazelle/README.md

gazelle/python/kinds.go

shaldengeki · 2025-07-05T22:24:39Z

While writing a test covering the bzlmod integration I ran into a small blocker, so I PRed a fix for that separately: #3060

shaldengeki · 2025-07-12T05:39:45Z

Can you elaborate on why you you can't add the bzlmod tests without bumping gazelle / rules_go / go version? From what I see in cdeafde, I don't see any reason why the bzlmod tests won't work.

Yep! Should've made this clear earlier. You can reproduce the errors I ran into by checking out 086c6aa (the last commit before I undid the upgrade & removed the tests), and undoing the upgrade:

$ bazel test //...
# ...
FAIL: //python:python_test_directive_python_generate_proto_bzlmod_protobuf_renamed (Exit 1) (see /home/charles/.cache/bazel/_bazel_charles/25da5a3f5a375d8b77b073daf0c17156/execroot/_main/bazel-out/k8-fastbuild/testlogs/python/python_test_directive_python_generate_proto_bzlmod_protobuf_renamed/test.log)
FAIL: //python:python_test_directive_python_generate_proto_bzlmod_protobuf (Exit 1) (see /home/charles/.cache/bazel/_bazel_charles/25da5a3f5a375d8b77b073daf0c17156/execroot/_main/bazel-out/k8-fastbuild/testlogs/python/python_test_directive_python_generate_proto_bzlmod_protobuf/test.log)
INFO: From Testing //python:python_test_directive_python_generate_proto_bzlmod_protobuf:
==================== Test output for //python:python_test_directive_python_generate_proto_bzlmod_protobuf:
--- FAIL: TestGazelleBinary (0.00s)
    --- FAIL: TestGazelleBinary/directive_python_generate_proto_bzlmod_protobuf (0.00s)
        python_test.go:169: expected gazelle exit code: 0
            got: 1
        python_test.go:179: expected gazelle stderr: 
            got: gazelle: -repo_root not specified, and WORKSPACE cannot be found: file does not exist
        python_test.go:144: "" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/BUILD" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/MODULE.bazel" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/README.md" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/foo.proto" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf/test.yaml" exists
FAIL
================================================================================
INFO: From Testing //python:python_test_directive_python_generate_proto_bzlmod_protobuf_renamed:
==================== Test output for //python:python_test_directive_python_generate_proto_bzlmod_protobuf_renamed:
--- FAIL: TestGazelleBinary (0.00s)
    --- FAIL: TestGazelleBinary/directive_python_generate_proto_bzlmod_protobuf_renamed (0.01s)
        python_test.go:169: expected gazelle exit code: 0
            got: 1
        python_test.go:179: expected gazelle stderr: 
            got: gazelle: -repo_root not specified, and WORKSPACE cannot be found: file does not exist
        python_test.go:144: "" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/BUILD" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/MODULE.bazel" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/README.md" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/foo.proto" exists
        python_test.go:144: "/directive_python_generate_proto_bzlmod_protobuf_renamed/test.yaml" exists
FAIL
================================================================================

which reads to me like Gazelle doesn't know about bzlmod, and is just looking for the WORKSPACE.

…ng generate.go

shaldengeki · 2025-07-12T06:11:29Z

@dougthor42 one change I just made -- I double-checked how the golang & java extensions work, and they both generate e.g. foo_proto -> foo_(go|java)_proto, so I made our default here foo_py_proto, instead of foo_py_pb2 (which came from my own idiosyncratic naming in my personal repo).

I think that makes sense? And it'll make it just a tiny bit easier to resolve imports down the line.

dougthor42 · 2025-07-12T06:18:44Z

https://bazel.build/reference/be/protocol-buffer#py_proto_library:

The convention is to name the `py_proto_library` rule `foo_py_pb2`, when it is wrapping `proto_library` rule `foo_proto`

IIRC this comes from the fact that `protoc` makes `foo_pb2.py` files and they're imported in Python with `import foo_pb2`.

shaldengeki · 2025-07-12T06:21:40Z

Ah, dang it. I knew that came from somewhere.

It is... an unfortunate naming convention, I'll venture? I can put it back, but it's weird because a) it doesn't match other languages, and b) it doesn't even fit the actual naming of the generated file, which is like foo_pb2.py (while the convention is foo_py_pb2).

Let me know if you'd prefer I revert it.

dougthor42

Let me know if you'd prefer I revert it.

Yeah let's revert 3e0fdcc and we'll use #3081 to support naming things like foo_py_proto

Otherwise this LGTM. We just have to figure out that CI failure...

dougthor42 · 2025-07-12T21:11:47Z

which reads to me like Gazelle doesn't know about bzlmod, and is just looking for the WORKSPACE.

Ah yes I see the error and repro'd it locally. Turns out you still need a dummy (empty) WORKSPACE file in the test dir. You can re-add those test cases if you'd like, as long as you're not bumping versions.

With the empty WORKSPACE file, things pass for me locally so hopefully they pass CI too.

This reverts commit 3e0fdcc.

…support" This reverts commit cdeafde.

dougthor42

Love it, thank you so much! 🎉

shaldengeki · 2025-07-13T18:23:12Z

@dougthor42 Thanks for the review -- really excited to get this in!

I just noticed, as part of doing #3081, that I'm technically not following the convention you linked. I'm generating foo_proto_py_pb2 instead of foo_py_pb2. I just pushed a change that fixes this, both in the plugin and also in a bunch of examples where I noticed this as well. Hope that's OK?

…y_proto_library` naming (#3093) Closes #3081. This adds support in the Gazelle plugin for controlling how the generated `py_proto_library` rules are named; support for these was originally added in #3057. We do this via a new Gazelle directive, `python_proto_naming_convention`, which is similar to `python_library_naming_convention` and the like, except it interpolates `$proto_name$`, which is the `proto_library` rule minus any trailing `_proto`. We default to `$proto_name$_py_pb2`. For instance, for a `proto_library` named `foo_proto`, the default value would generate `foo_py_pb2`, aligning with [the convention stated in the Bazel docs.](https://bazel.build/reference/be/protocol-buffer#py_proto_library)

…ation and usage (#3132) Part of #3082 2nd of probably 5 PRs. + Migrate installation and usage info from `gazelle/README.md` to `gazelle/docs/installation_and_usage.md` + Slight rewording and reformatting of the `Example` section. + Reorganized and modernized the `MODULE.bazel` and `BUILD.bazel` examples: + less details on rules_python as that's available in other docs + default to gazelle multilang support now that #3057 is merged. + Mechanical updates: + Wrap at ~80 chars + Use MyST directives and roles.

shaldengeki added 13 commits July 2, 2025 23:18

[wip] Start work on generating py_proto_library, by creating a bunch …

c85c56f

…of scaffolding & tests

Remove test cases that don't make any sense; py_proto_library runs on…

8b5a91b

… the singular input proto

Add additional test fixtures required to run tests

d8f9b1a

[wip] Start working on generate.go

caf5d6e

Fix visibility in tests

b712696

Load py_proto_library from protobuf repository, since rules_proto's c…

3186761

…opy is deprecated

Remove unneessary code

e073055

Delete some more unused stuff

aa9f4aa

Update CHANGELOG.md

b6df17b

Have a sudden change-of-heart and call it python_generate_proto, to a…

d794ac1

…lign with the other directives

Update README.md

cad6849

Don't repeatedly initialize a variable

a0e4f5f

Add test enforcing that unnecessary rules are removed, and change log…

161cc51

…ic to implement this

shaldengeki requested review from dougthor42, aignas and rickeylev as code owners July 5, 2025 00:45

shaldengeki added 2 commits July 4, 2025 20:48

Add a tiny comment

fc7fa08

Merge branch 'main' into gazelle-plugin-generates-py-proto-library

ad4f3f7

dougthor42 reviewed Jul 5, 2025

View reviewed changes

shaldengeki added 5 commits July 5, 2025 17:48

Default python_generate_proto to false, and update tests

4853207

In README.md add details section

9a44ef3

Plugin detects configured name for protobuf directory

362dea9

Update Gazelle, to bring in bzlmod support

4e41e0e

Update Gazelle, to bring in bzlmod support

dc27cda

shaldengeki mentioned this pull request Jul 5, 2025

build(gazelle): Update Gazelle, to bring in bzlmod support #3060

Closed

shaldengeki added 4 commits July 5, 2025 18:28

Add tests covering bzlmod & renaming functionality

5ee804a

Remove unnecessary licenses

e5fbed8

Add trailing newlines to some protobuf

a2cb993

Add test covering "deletes only the py_proto_library rule"

7c2706c

shaldengeki added 3 commits July 12, 2025 01:45

Wrap README.md to 80 chars per line, and be less-disruptive to existi…

763ef9d

…ng generate.go

Fix some language again

b3805a9

Use _py_proto suffix, to align with java, go naming schemes

3e0fdcc

shaldengeki mentioned this pull request Jul 12, 2025

Gazelle plugin should have a configurable naming convention for py_proto_library targets #3081

Closed

dougthor42 reviewed Jul 12, 2025

View reviewed changes

shaldengeki added 5 commits July 12, 2025 22:16

Revert "Use _py_proto suffix, to align with java, go naming schemes"

0c9d436

This reverts commit 3e0fdcc.

Revert "Undo gazelle upgrade, and remove tests that depend on bzlmod …

eb87801

…support" This reverts commit cdeafde.

Undo version upgrade

6791d4b

Fix a test's load ordering

66966fe

Check in empty WORKSPACE files to make tests pass

d61aad1

shaldengeki requested a review from dougthor42 July 13, 2025 02:22

dougthor42 approved these changes Jul 13, 2025

View reviewed changes

dougthor42 mentioned this pull request Jul 13, 2025

RBE Failures related to protobuf, abseil-cpp, and #include <filesystem> #3084

Closed

_proto_py_pb2 -> _py_pb2

b0039ee

dougthor42 approved these changes Jul 13, 2025

View reviewed changes

Merge branch 'main' into gazelle-plugin-generates-py-proto-library

fa0d564

dougthor42 enabled auto-merge July 14, 2025 23:52

dougthor42 added this pull request to the merge queue Jul 14, 2025

Merged via the queue into bazel-contrib:main with commit dd6550f Jul 15, 2025
3 checks passed

shaldengeki mentioned this pull request Jul 15, 2025

feat(gazelle): python_proto_naming_convention directive controls py_proto_library naming #3093

Merged

dougthor42 mentioned this pull request Aug 4, 2025

docs(gazelle): Migrate Gazelle docs to ReadTheDocs, part 2/5: installation and usage #3132

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(gazelle): Gazelle plugin generates py_proto_library #3057

feat(gazelle): Gazelle plugin generates py_proto_library #3057

Uh oh!

shaldengeki commented Jul 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shaldengeki commented Jul 5, 2025

Uh oh!

shaldengeki commented Jul 12, 2025 •

edited

Loading

Uh oh!

shaldengeki commented Jul 12, 2025

Uh oh!

dougthor42 commented Jul 12, 2025 via email •

edited

Loading

Uh oh!

shaldengeki commented Jul 12, 2025 •

edited

Loading

Uh oh!

dougthor42 left a comment

Uh oh!

dougthor42 commented Jul 12, 2025

Uh oh!

dougthor42 left a comment

Uh oh!

shaldengeki commented Jul 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat(gazelle): Gazelle plugin generates py_proto_library #3057

feat(gazelle): Gazelle plugin generates py_proto_library #3057

Uh oh!

Conversation

shaldengeki commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shaldengeki commented Jul 5, 2025

Uh oh!

shaldengeki commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shaldengeki commented Jul 12, 2025

Uh oh!

dougthor42 commented Jul 12, 2025 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shaldengeki commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dougthor42 left a comment

Choose a reason for hiding this comment

Uh oh!

dougthor42 commented Jul 12, 2025

Uh oh!

dougthor42 left a comment

Choose a reason for hiding this comment

Uh oh!

shaldengeki commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shaldengeki commented Jul 5, 2025 •

edited

Loading

shaldengeki commented Jul 12, 2025 •

edited

Loading

dougthor42 commented Jul 12, 2025 via email •

edited

Loading

shaldengeki commented Jul 12, 2025 •

edited

Loading

shaldengeki commented Jul 13, 2025 •

edited

Loading