Bring back the compiler (still in a different Python package), switch to uv (#106)

* Move betterproto

* Add back the compiler

* Switch to uv

* Add lock file

* Add uv lock file

* Update pyproject

* Fix CI

* Temporary fix

* Fix workflow name

* Fix test skipping on windows

* Fix code quality

* Fix Python version

* Fix action

* Remove comment

* Compile tests in CI

* Move documentation

* Fix documentation workflow

* Fix Python version

* Update lockfile

* Bump version

* Update pyprojects

* Update README

* Fix release script for the lib

Author: AdrienVannson

.github/workflows/ci.yml

@@ -13,63 +13,45 @@ permissions:
 
 jobs:
   tests:
-    name: ${{ matrix.os }} / ${{ matrix.python-version }} / Rust codec ${{ matrix.rust-codec }}
+    name: ${{ matrix.os }} / ${{ matrix.python-version }}
     runs-on: ${{ matrix.os }}-latest
     strategy:
       fail-fast: false
       matrix:
         os: [Ubuntu, MacOS, Windows]
         python-version: ['3.10', '3.11', '3.12', '3.13']
-        rust-codec: ['disabled'] # TODO add 'enabled' when the Rust codec is ready
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v5
+      - name: Install uv with Python ${{ matrix.python-version }}
+        uses: astral-sh/setup-uv@v5
         with:
+          version: "0.7.5"
           python-version: ${{ matrix.python-version }}
 
-      - name: Get full Python version
-        id: full-python-version
-        shell: bash
-        run: echo "version=$(python -c "import sys; print('-'.join(str(v) for v in sys.version_info))")" >> "$GITHUB_OUTPUT"
-
-      - name: Install poetry
-        shell: bash
-        run: |
-          python -m pip install poetry
-          echo "$HOME/.poetry/bin" >> $GITHUB_PATH
-
-      - name: Configure poetry
-        shell: bash
-        run: poetry config virtualenvs.in-project true
-
-      - name: Set up cache
-        uses: actions/cache@v4
-        id: cache
-        with:
-          path: .venv
-          key: venv-${{ runner.os }}-${{ steps.full-python-version.outputs.version }}-rust-codec-${{ matrix.rust-codec }}-${{ hashFiles('**/poetry.lock') }}
+      - name: Install betterproto2
+        working-directory: ./betterproto2
+        run: uv sync --locked --all-extras --all-groups
 
-      - name: Ensure cache is healthy
-        if: steps.cache.outputs.cache-hit == 'true'
-        shell: bash
-        run: poetry run pip --version >/dev/null 2>&1 || rm -rf .venv
+      - name: Install betterproto2_compiler
+        working-directory: ./betterproto2_compiler
+        run: uv sync --locked --all-extras --all-groups
 
-      - name: Install dependencies
-        if: matrix.rust-codec == 'disabled'
+      - name: Test betterproto2_compiler
+        working-directory: ./betterproto2_compiler
         shell: bash
-        run: poetry install -E all
+        run: uv run poe test
 
-      - name: Install dependencies with the Rust codec
-        if: matrix.rust-codec == 'enabled'
+      - name: Compile test files
+        working-directory: ./betterproto2_compiler
         shell: bash
-        run: poetry install -E all -E rust-codec
+        run: uv run poe generate
 
-      - name: Download compiled test files
+      - name: Move compiled files to betterproto2
         shell: bash
-        run: poetry run poe pull-compiled-tests
+        run: mv betterproto2_compiler/tests/output_betterproto betterproto2_compiler/tests/output_betterproto_pydantic betterproto2_compiler/tests/output_reference betterproto2/tests 
 
       - name: Execute test suite
+        working-directory: ./betterproto2
         shell: bash
-        run: poetry run poe test
+        run: uv run poe test

.github/workflows/code-quality.yml

@@ -12,42 +12,24 @@ permissions:
   contents: read
 
 jobs:
-  check-formatting:
+  code-quality:
     name: Check code/doc formatting
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v4
-    - uses: actions/setup-python@v5
     - uses: pre-commit/[email protected]
 
-    - name: Install poetry
-      shell: bash
-      run: |
-        python -m pip install poetry
-        echo "$HOME/.poetry/bin" >> $GITHUB_PATH
-
-    - name: Configure poetry
-      shell: bash
-      run: poetry config virtualenvs.in-project true
-
-    - name: Set up cache
-      uses: actions/cache@v3
-      id: cache
+    - name: Install uv
+      uses: astral-sh/setup-uv@v5
       with:
-        path: .venv
-        key: venv-quality-${{ hashFiles('**/poetry.lock') }}
+        version: "0.7.5"
+        python-version: "3.10"
 
-    - name: Ensure cache is healthy
-      if: steps.cache.outputs.cache-hit == 'true'
-      shell: bash
-      run: poetry run pip --version >/dev/null 2>&1 || rm -rf .venv
-
-    - name: Install dependencies
-      shell: bash
-      run: poetry install -E all
-
-    - uses: pre-commit/[email protected]
+    - name: Install the project
+      working-directory: ./betterproto2
+      run: uv sync --locked --all-extras --all-groups
 
     - name: Pyright
+      working-directory: ./betterproto2
       shell: bash
-      run: poetry run poe typecheck
+      run: uv run poe typecheck

.github/workflows/documentation.yml

@@ -16,35 +16,16 @@ jobs:
           git config user.name github-actions[bot]
           git config user.email 41898282+github-actions[bot]@users.noreply.github.com
 
-      - name: Set up Python 3.12
-        uses: actions/setup-python@v5
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
         with:
-          python-version: 3.12
+          version: "0.7.5"
+          python-version: "3.10"
 
-      - name: Install poetry
-        shell: bash
-        run: |
-          python -m pip install poetry
-          echo "$HOME/.poetry/bin" >> $GITHUB_PATH
-
-      - name: Configure poetry
-        shell: bash
-        run: poetry config virtualenvs.in-project true
-
-      - name: Set up cache
-        uses: actions/cache@v3
-        id: cache
-        with:
-          path: .venv
-          key: venv-documentation-${{ hashFiles('**/poetry.lock') }}
-      
-      - name: Ensure cache is healthy
-        if: steps.cache.outputs.cache-hit == 'true'
-        shell: bash
-        run: poetry run pip --version >/dev/null 2>&1 || rm -rf .venv
-
-      - name: Install dependencies
-        shell: bash
-        run: poetry install
+      - name: Install betterproto2
+        working-directory: ./betterproto2
+        run: uv sync --locked --all-extras --all-groups
 
-      - run: poetry run mkdocs gh-deploy --force
+      - name: Build and deploy documentation
+        working-directory: ./betterproto2
+        run: uv run mkdocs gh-deploy --force

.github/workflows/release.yml

@@ -8,9 +8,9 @@ on:
       - '**'
 
 jobs:
-  packaging:
-    if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
-    name: Distribution
+  publish_betterproto2:
+    if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/lib-v')
+    name: Publish betterproto2 to PyPI
     runs-on: ubuntu-latest
 
     permissions:
@@ -23,16 +23,17 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python 3.10
-        uses: actions/setup-python@v5
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
         with:
+          version: "0.7.5"
           python-version: "3.10"
 
-      - name: Install poetry
-        run: python -m pip install poetry
-
       - name: Build package
-        run: poetry build
+        working-directory: ./betterproto2
+        run: uv build
 
       - name: Publish package distributions to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1.12
+        with:
+          packages-dir: betterproto2/dist

.pre-commit-config.yaml

@@ -6,6 +6,6 @@ repos:
     rev: v0.9.3
     hooks:
       - id: ruff-format
-        args: ["--diff", "src", "tests"]
+        args: ["--diff", "betterproto2/src", "betterproto2/tests", "betterproto2_compiler/src", "betterproto2_compiler/tests"]
       - id: ruff
-        args: ["src", "tests"]
+        args: ["betterproto2/src", "betterproto2/tests", "betterproto2_compiler/src", "betterproto2_compiler/tests"]

README.md

@@ -4,7 +4,7 @@
 
 > :warning: `betterproto2` is a fork of the original [`betterproto`](https://github.com/danielgtaylor/python-betterproto) repository. It is a major redesign of the library, allowing to fix several bugs and to support new features.
 > 
-> However, it is still in active developement. The documentation is not complete and a few breaking changes still are likely to be done. There is still work to do and the project is still subject to breaking changes.
+> However, it is still in active developement. The documentation is not complete, there is still work to do and the project is still subject to breaking changes.
 
 This project aims to provide an improved experience when using Protobuf / gRPC in a modern Python environment by making use of modern language features and generating readable, understandable, idiomatic Python code. It will not support legacy features or environments (e.g. Protobuf 2). The following are supported:
 

docs/api.md renamed to betterproto2/docs/api.md

@@ -0,0 +1,24 @@
+# Development
+
+This page targets the betterproto maintainers.
+
+## Recompiling the lib proto files
+
+After some updates in the compiler, it might be useful to recompile the standard Google proto files used by the
+compiler. As the proto files are distributed with `protoc`, their path might depend on your installation.
+
+```bash
+mkdir lib
+protoc \
+    --python_betterproto2_out=lib \
+    -I /usr/include/ \
+    /usr/include/google/protobuf/*.proto
+```
+
+!!! warning
+    These proto files are written with the `proto2` syntax, which is not supported by betterproto. For the compiler to
+    work, you need to manually patch the generated file to mark the field `oneof_index` in `Field` and
+    `FieldDescriptorProto` optional.
+
+In the compiler, you also need to compile the [plugin.proto](https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/compiler/plugin.proto)
+file in `src/betterproto2_compiler/lib/google.protobug/compiler/__init__.py`.

betterproto2/docs/development.md

@@ -1,29 +1,111 @@
 Getting Started
 ===============
 
-## Installation
+## Compilation
+
+
+### Install protoc
 
-The package `betterproto2` can be simply installed from PyPI using `pip`:
+The betterproto2 compiler is a plugin of `protoc`, you first need to [install](https://grpc.io/docs/protoc-installation/) it.
+
+You can also use it from `grpcio-tools`:
 
 ```sh
-pip install betterproto2[all]
+pip install grpcio-tools
+```
+
+### Install `betterproto2_compiler`
+
+It is possible to install `betterproto2_compiler` using pip:
+
+```
+pip install betterproto2_compiler
+```
+
+### Compile a proto file
+
+Create the following `example.proto` file.
+
+```proto
+syntax = "proto3";
+
+package helloworld;
+
+message HelloWorld {
+    string message = 1;
+}
+
+service HelloService {
+    rpc SayHello (HelloWorld) returns (HelloWorld);
+}
+```
+
+You should be able to compile it using:
+
+```
+mkdir lib
+protoc -I . --python_betterproto2_out=lib example.proto
+```
+
+If you installed `protoc` with `grpc-tools`, the command will be:
+
 ```
+mkdir lib
+python -m grpc.tools.protoc -I . --python_betterproto2_out=lib example.proto
+```
+
+#### Service compilation
+
+##### Clients
+
+By default, for each service, betterproto will generate a synchronous client. Both synchronous and asynchronous clients
+are supported.
+
+  - Synchronous clients rely on the `grpcio` package. Make sure to enable the `grpcio` extra package when installing
+    betterproto2 to use them.
+  - Asynchronous clients rely on the `grpclib` package. Make sure to enable the `grpclib` extra package when installing
+    betterproto2 to use them.
 
-!!! info
-    The package is compatible with all Python versions from 3.10 to 3.13.
+To choose which clients to generate, use the `client_generation` option of betterproto. It supports the following
+values:
 
+  - `none`: Clients are not generated.
+  - `sync`: Only synchronous clients are generated.
+  - `async`: Only asynchronous clients are generated.
+  - `sync_async`: Both synchronous and asynchronous clients are generated.
+        Asynchronous clients are generated with the Async suffix.
+  - `async_sync`: Both synchronous and asynchronous clients are generated.
+        Synchronous clients are generated with the Sync suffix.
+  - `sync_async_no_default`: Both synchronous and asynchronous clients are generated.
+        Synchronous clients are generated with the Sync suffix, and asynchronous clients are generated with the Async
+        suffix.
 
-## Compiling proto files
+For example, `protoc -I . --python_betterproto2_out=lib example.proto --python_betterproto2_opt=client_generation=async`
+will only generate asynchronous clients.
 
-Follow the documentation of [betterproto2 compiler](https://betterproto.github.io/python-betterproto2-compiler/getting-started/) to compile your proto files.
+##### Servers
+
+By default, betterproto will not generate server base classes. To enable them, set the `server_generation` option to
+`async` with `--python_betterproto2_opt=server_generation=async`.
+
+These base classes will be asynchronous and rely on `grpclib`. To use them, make sure to install `betterproto2` with the
+`grpclib` extra package.
+
+
+## Installation
+
+The package `betterproto2` can be installed from PyPI using `pip`:
+
+```sh
+pip install betterproto2[all]
+```
 
 !!! warning
     Make sure that the proto files were generated with a version of `betterproto2_compiler` that is compatible with your
     version of `betterproto2`.
 
     The version `0.x.y` of `betterproto` is compatible with the version `0.a.b` of the compiler if and only if `a=b`.
 
-
 ## Basic usage
 
 If you successfuly compiled the `example.proto` file from the compiler documentation, you should now be able to use it!