[Question] How to use protoc instead of buf in a python project

[jmspereira]

Hey everyone, sorry if this issue is duplicated, but I didn't find any that addressed the entire process or I am missing something when using protoc instead of buf. I have a python project with a set of protobuf files, I am compiling them with:

protoc <protobuf-files> 
-I/usr/local/include/protovalidate  
--fatal_warnings 
--python_out=/<outpath> 
--pyi_out=/<outpath>  
--include_imports 
--descriptor_set_out=<outpath>/gen

Where the include_imports and the descriptor_set_out were set as documented. However, even when I define the PYTHONPATH to the gen file, still does not work, with the following error:

>>> import protovalidate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/lib/python3.11/site-packages/protovalidate/__init__.py", line 15, in <module>
    from protovalidate import validator
  File "/venv/lib/python3.11/site-packages/protovalidate/validator.py", line 19, in <module>
    from buf.validate import expression_pb2  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'buf'

This is likely because this is not generating the validate_pb2.py but a binary file. So, what is the correct process in order to use protovalidate with protoc in a python project?

Thanks!

[stefanvanburen]

hi @jmspereira — if I understand correctly, you're generating the descriptor set to <outpath>/gen, and the python generated code to /<outpath>. You'd want to point your PYTHONPATH to wherever the python generated code is generated to (in this case, /<outpath>) — can you see if that works, or clarify your setup?

[jmspereira]

Hi @stefanvanburen, in the python_out (/<outpath>) only the files corresponding to the messages that I pass are available (not the ones in the imports, which I am assuming is what you want to be generated).

[stefanvanburen]

hey @jmspereira — I think what you need to do is also generate the protovalidate proto module, as it is not included in the wheel. You'll probably want a separate protoc invocation (some prior art in #147 (comment)) to generate that output, and then point your PYTHONPATH to that generated code. Let me know if that helps!

[jmspereira]

I can try to do that, but that will add more complexity to the build process of the final package. What is the reason for the compiled validated module not being included in the python package?

[stefanvanburen]

We've considered including it in the past, but that would tie protovalidate-python's releases to a particular protobuf gencode version. By not including it, users can generate the code that matches their version of protobuf.

[jmspereira]

Ok. I changed the project setup for, during the build, downloading the protovalidate package and adding the validated module to the list of messages to be compiled. Everything is working, thanks!