Skip to content

Commit 0dc494e

Browse files
abravalheridavidhewitt
abravalheri
and
davidhewitt
authored
Update README/docs to use pyproject.toml + separate setup.py docs (#356)
* Modify setup.py example to be easier to write a tutorial * Add separated docs about setup.py usage * Add 'docs' session to noxfile to facilitate development * Update README to use pyproject.toml instead of setup.py * Update building_wheels docs to avoid using setup.py * Remove reference to setup.py in API docs * Fix formating in example * Fix toml syntax error in README * Apply suggestions from code review Co-authored-by: David Hewitt <[email protected]> * Tweak README based on review --------- Co-authored-by: David Hewitt <[email protected]>
1 parent 8da3512 commit 0dc494e

File tree

13 files changed

+599
-86
lines changed

13 files changed

+599
-86
lines changed

‎README.md‎

Lines changed: 138 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -10,83 +10,159 @@
1010
Compile and distribute Python extensions written in Rust as easily as if
1111
they were written in C.
1212

13-
## Setup
13+
## Quickstart
1414

15-
For a complete example, see
16-
[html-py-ever](https://github.com/PyO3/setuptools-rust/tree/main/examples/html-py-ever).
15+
The following is a very basic tutorial that shows how to use `setuptools-rust` in `pyproject.toml`.
16+
It assumes that you already have a bunch of Python and Rust files that you want
17+
to distribute. You can see examples for these files in the
18+
[`examples/hello-world`](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world)
19+
directory in the [github repository](https://github.com/PyO3/setuptools-rust).
20+
The [PyO3 docs](https://pyo3.rs) have detailed information on how to write Python
21+
modules in Rust.
1722

18-
First, you need to create a bunch of files:
19-
20-
### setup.py
21-
22-
```python
23-
from setuptools import setup
24-
from setuptools_rust import Binding, RustExtension
25-
26-
setup(
27-
name="hello-rust",
28-
version="1.0",
29-
rust_extensions=[RustExtension("hello_rust.hello_rust", binding=Binding.PyO3)],
30-
packages=["hello_rust"],
31-
# rust extensions are not zip safe, just like C-extensions.
32-
zip_safe=False,
33-
)
23+
```
24+
hello-world
25+
├── python
26+
│ └── hello_world
27+
│ └── __init__.py
28+
└── rust
29+
└── lib.rs
3430
```
3531

36-
For a complete reference of the options supported by the `RustExtension` class, see the
37-
[API reference](https://setuptools-rust.readthedocs.io/en/latest/reference.html).
32+
Once the implementation files are in place, we need to add a `pyproject.toml`
33+
file that tells anyone that wants to use your project how to build it.
34+
In this file, we use an [array of tables](https://toml.io/en/v1.0.0#array-of-tables)
35+
(TOML jargon equivalent to Python's list of dicts) for ``[[tool.setuptools-rust.ext-modules]]``,
36+
to specify different extension modules written in Rust:
3837

39-
### pyproject.toml
4038

4139
```toml
40+
# pyproject.toml
4241
[build-system]
43-
requires = ["setuptools", "wheel", "setuptools-rust"]
42+
requires = ["setuptools", "setuptools-rust"]
43+
build-backend = "setuptools.build_meta"
44+
45+
[project]
46+
name = "hello-world"
47+
version = "1.0"
48+
49+
[tool.setuptools.packages]
50+
# Pure Python packages/modules
51+
find = { where = ["python"] }
52+
53+
[[tool.setuptools-rust.ext-modules]]
54+
# Private Rust extension module to be nested into the Python package
55+
target = "hello_world._lib" # The last part of the name (e.g. "_lib") has to match lib.name in Cargo.toml,
56+
# but you can add a prefix to nest it inside of a Python package.
57+
path = "Cargo.toml" # Default value, can be omitted
58+
binding = "PyO3" # Default value, can be omitted
59+
py-limited-api = "auto" # Default value, can be omitted
4460
```
4561

46-
### MANIFEST.in
62+
Each extension module should map directly into the corresponding `[lib]` table on the
63+
[Cargo manifest file](https://doc.rust-lang.org/cargo/reference/manifest.html):
4764

48-
This file is required for building source distributions
49-
50-
```text
51-
include Cargo.toml
52-
recursive-include src *
65+
```toml
66+
# Cargo.toml
67+
[package]
68+
name = "hello-world"
69+
version = "0.1.0"
70+
edition = "2018"
71+
72+
[dependencies]
73+
pyo3 = { version = "0.19.2", features = ["extension-module"] }
74+
75+
[lib]
76+
name = "_lib" # private module to be nested into Python package,
77+
# needs to match the name of the function with the `[#pymodule]` attribute
78+
path = "rust/lib.rs"
79+
crate-type = ["cdylib"] # required for shared library for Python to import from.
80+
81+
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
82+
# See also PyO3 docs on writing Cargo.toml files at https://pyo3.rs
5383
```
5484

55-
## Usage
56-
57-
You can use same commands as for c-extensions. For example:
85+
You will also need to tell Setuptools that the Rust files are required to build your
86+
project from the [source distribution](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html).
87+
That can be done either via `MANIFEST.in` (see example below) or via a plugin like
88+
[`setuptools-scm`](https://pypi.org/project/setuptools-scm/).
5889

5990
```
60-
>>> python ./setup.py develop
61-
running develop
62-
running egg_info
63-
writing hello-rust.egg-info/PKG-INFO
64-
writing top-level names to hello_rust.egg-info/top_level.txt
65-
writing dependency_links to hello_rust.egg-info/dependency_links.txt
66-
reading manifest file 'hello_rust.egg-info/SOURCES.txt'
67-
writing manifest file 'hello_rust.egg-info/SOURCES.txt'
68-
running build_ext
69-
running build_rust
70-
cargo build --manifest-path extensions/Cargo.toml --features python3
71-
Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
72-
73-
Creating /.../lib/python3.6/site-packages/hello_rust.egg-link (link to .)
74-
75-
Installed hello_rust
76-
Processing dependencies for hello_rust==1.0
77-
Finished processing dependencies for hello_rust==1.0
91+
# MANIFEST.in
92+
include Cargo.toml
93+
recursive-include rust *.rs
7894
```
7995

80-
Or you can use commands like `bdist_wheel` (after installing `wheel`). See also [the notes in the documentation about building wheels](https://setuptools-rust.readthedocs.io/en/latest/building_wheels.html).
81-
82-
Cross-compiling is also supported, using one of [`crossenv`](https://github.com/benfogle/crossenv), [`cross`](https://github.com/rust-embedded/cross) or [`cargo-zigbuild`](https://github.com/messense/cargo-zigbuild).
83-
For examples see the `test-crossenv` and `test-cross` and `test-zigbuild` Github actions jobs in [`ci.yml`](https://github.com/PyO3/setuptools-rust/blob/main/.github/workflows/ci.yml).
84-
85-
By default, `develop` will create a debug build, while `install` will create a release build.
86-
87-
## Commands
96+
With these files in place, you can install the project in a virtual environment
97+
for testing and making sure everything is working correctly:
98+
99+
```powershell
100+
# cd hello-world
101+
python3 -m venv .venv
102+
source .venv/bin/activate # on Linux or macOS
103+
.venv\Scripts\activate # on Windows
104+
python -m pip install -e .
105+
python
106+
>>> import hello_world
107+
# ... try running something from your new extension module ...
108+
# ... better write some tests with pytest ...
109+
```
88110

89-
- `build` - Standard build command will also build all rust extensions.
90-
- `build_rust` - Command builds all rust extensions.
91-
- `clean` - Standard clean command executes cargo clean for all rust
92-
extensions.
111+
## Next steps and final remarks
112+
113+
- When you are ready to distribute your project, have a look on
114+
[the notes in the documentation about building wheels](https://setuptools-rust.readthedocs.io/en/latest/building_wheels.html).
115+
116+
- Cross-compiling is also supported, using one of
117+
[`crossenv`](https://github.com/benfogle/crossenv),
118+
[`cross`](https://github.com/rust-embedded/cross) or
119+
[`cargo-zigbuild`](https://github.com/messense/cargo-zigbuild).
120+
For examples see the `test-crossenv` and `test-cross` and `test-zigbuild` Github actions jobs in
121+
[`ci.yml`](https://github.com/PyO3/setuptools-rust/blob/main/.github/workflows/ci.yml).
122+
123+
- You can also use `[[tool.setuptools-rust.bins]]` (instead of `[[tool.setuptools-rust.ext-modules]]`),
124+
if you want to distribute a binary executable written in Rust (instead of a library that can be imported by the Python runtime).
125+
Note however that distributing both library and executable (or multiple executables),
126+
may significantly increase the size of the
127+
[wheel](https://packaging.python.org/en/latest/glossary/#term-Wheel)
128+
file distributed by the
129+
[package index](https://packaging.python.org/en/latest/glossary/#term-Package-Index)
130+
and therefore increase build, download and installation times.
131+
Another approach is to use a Python entry-point that calls the Rust
132+
implementation (exposed via PyO3 bindings).
133+
See the [hello-world](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world)
134+
example for more insights.
135+
136+
- For a complete reference of the configuration options, see the
137+
[API reference](https://setuptools-rust.readthedocs.io/en/latest/reference.html).
138+
You can use any parameter defined by the `RustExtension` class with
139+
`[[tool.setuptools-rust.ext-modules]]` and any parameter defined by the
140+
`RustBin` class with `[[tool.setuptools-rust.bins]]`; just remember to replace
141+
underscore characters `_` with dashes `-` in your `pyproject.toml` file.
142+
143+
- `Cargo.toml` allow only one `[lib]` table per file.
144+
If you require multiple extension modules you will need to write multiple `Cargo.toml` files.
145+
Alternatively you can create a single private Rust top-level module that exposes
146+
multiple submodules (using [PyO3's submodules](https://pyo3.rs/v0.19.2/module#python-submodules)),
147+
which may also reduce the size of the build artifacts.
148+
You can always keep your extension modules private and wrap them in pure Python
149+
to have fine control over the public API.
150+
151+
- If want to include both `[[tool.setuptools-rust.bins]]` and `[[tool.setuptools-rust.ext-modules]]`
152+
in the same macOS wheel, you might have to manually add an extra `build.rs` file,
153+
see [PyO3/setuptools-rust#351](https://github.com/PyO3/setuptools-rust/pull/351)
154+
for more information about the workaround.
155+
156+
- For more examples, see:
157+
- [`hello-world`](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world):
158+
a more complete version of the code used in this tutorial that mixes both
159+
`[[tool.setuptools-rust.ext-modules]]` and `[[tool.setuptools-rust.bins]]`
160+
in a single distribution.
161+
- [`html-py-ever`](https://github.com/PyO3/setuptools-rust/tree/main/examples/html-py-ever):
162+
a more advanced example that uses Rust crates as dependencies.
163+
- [`rust_with_cffi`](https://github.com/PyO3/setuptools-rust/tree/main/examples/rust_with_cffi):
164+
uses both Rust and [CFFI](https://cffi.readthedocs.io/en/latest/).
165+
- [`namespace_package`](https://github.com/PyO3/setuptools-rust/tree/main/examples/namespace_package):
166+
integrates Rust-written modules into PEP 420 namespace packages.
167+
- [`hello-world-script`](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world-script):
168+
uses Rust only for creating binary executables, not library modules.

‎docs/building_wheels.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,12 @@
11
# Building wheels
22

3-
Because `setuptools-rust` is an extension to `setuptools`, the standard `setup.py bdist_wheel` command is used to build distributable wheels. These wheels can be uploaded to PyPI using standard tools such as [twine](https://github.com/pypa/twine).
3+
Because `setuptools-rust` is an extension to `setuptools`, the standard [`python -m build`](https://pypa-build.readthedocs.io/en/stable/) command
4+
(or [`pip wheel --no-deps . --wheel-dir dist`](https://pip.pypa.io/en/stable/cli/pip_wheel/)) can be used to build distributable wheels.
5+
These wheels can be uploaded to PyPI using standard tools such as [twine](https://github.com/pypa/twine).
46

5-
`setuptools-rust` supports building for the [PEP 384](https://www.python.org/dev/peps/pep-0384/) "stable" (aka "limited") API when the `--py-limited-api` option is passed to `setup.py bdist_wheel`. If using PyO3 bindings for `RustExtension`, then the correct [`pyo3/abi3`](https://pyo3.rs/v0.14.5/features.html#abi3) sub-feature is automatically enabled. In this way, abi3 wheels can be uploaded to make package distributors' roles easier, and package users installing from source with `python setup.py install` can use optimizations specific to their Python version.
7+
`setuptools-rust` supports building for the [PEP 384](https://www.python.org/dev/peps/pep-0384/) "stable" (aka "limited") API when the `py_limited_api` option is set on the `[bdist_wheel]` section of `setup.cfg`.
8+
If using PyO3 bindings for `RustExtension`, then the correct [`pyo3/abi3`](https://pyo3.rs/v0.14.5/features.html#abi3) sub-feature is automatically enabled.
9+
In this way, abi3 wheels can be uploaded to make package distributors' roles easier, and package users installing from source with `pip install .` can use optimizations specific to their Python version.
610

711
This chapter of the documentation explains two possible ways to build wheels for multiple Python versions below.
812

@@ -52,7 +56,7 @@ It is possible to use any of the `manylinux` docker images: `manylinux1`, `manyl
5256

5357
### Binary wheels on macOS
5458

55-
For building wheels on macOS it is sufficient to run the `bdist_wheel` command, i.e. `setup.py bdist_wheel`.
59+
For building wheels on macOS it is sufficient to use one of the default `python -m build` or `pip wheel --no-deps . --wheel-dir dist` commands.
5660

5761
To build `universal2` wheels set the `ARCHFLAGS` environment variable to contain both `x86_64` and `arm64`, for example `ARCHFLAGS="-arch x86_64 -arch arm64"`. Wheel-building solutions such as [`cibuildwheel`][cibuildwheel] set this environment variable automatically.
5862

‎docs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
:hidden:
55
66
README.md
7+
setuppy_tutorial
78
building_wheels
89
reference
910
```

‎docs/setuppy_tutorial.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Usage with `setup.py`
2+
3+
While `pyproject.toml`-based configuration will be enough for most projects,
4+
sometimes you may need to use custom logic and imperative programming during the build.
5+
For those scenarios, `setuptools` also allows you to specify project configuration
6+
via `setup.py` in addition to `pyproject.toml`.
7+
8+
The following is a very basic tutorial that shows how to use `setuptools-rust` in
9+
your `setup.py`.
10+
11+
12+
## Basic implementation files
13+
14+
Let's start by assuming that you already have a bunch of Python and Rust files[^1]
15+
that you would like to package for distribution in PyPI inside of a project directory
16+
named `hello-world-setuppy`[^2][^3]:
17+
18+
[^1]: To know more about how to write Rust to be integrated into Python packages,
19+
please have a look on the [PyO3 docs](https://pyo3.rs)
20+
[^2]: You can have a look on the
21+
[examples/hello-world-setuppy](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world-setuppy)
22+
directory in the `setuptools-rust` repository.
23+
[^3]: If you are an experienced Python or Rust programmer, you may notice that we
24+
avoid using the `src` directory and explicitly instruct Setuptools and Cargo to
25+
look into the `python` and `rust` directories respectively.
26+
Since both Python and Rust ecosystem will try to claim the `src` directory as
27+
their default, we prefer to be explicit and avoid confusion.
28+
29+
30+
```
31+
hello-world-setuppy
32+
├── Cargo.lock
33+
├── Cargo.toml
34+
├── python
35+
│ └── hello_world
36+
│ └── __init__.py
37+
└── rust
38+
└── lib.rs
39+
```
40+
41+
```{literalinclude} ../examples/hello-world-setuppy/python/hello_world/__init__.py
42+
:language: python
43+
```
44+
45+
```{literalinclude} ../examples/hello-world-setuppy/rust/lib.rs
46+
:language: rust
47+
```
48+
49+
```{literalinclude} ../examples/hello-world-setuppy/Cargo.toml
50+
:language: toml
51+
```
52+
53+
54+
## Adding files to support packaging
55+
56+
Now we start by adding a `pyproject.toml` which tells anyone that wants to use
57+
our project to use `setuptools` and `setuptools-rust` to build it:
58+
59+
```{literalinclude} ../examples/hello-world-setuppy/pyproject.toml
60+
:language: toml
61+
```
62+
63+
… and a [`setup.py` configuration file](https://setuptools.pypa.io/en/latest/references/keywords.html)
64+
that tells Setuptools how to build the Rust extensions using our `Cargo.toml` and `setuptools-rust`:
65+
66+
```{literalinclude} ../examples/hello-world-setuppy/setup.py
67+
:language: python
68+
```
69+
70+
For a complete reference of the options supported by the `RustExtension` class, see the
71+
[API reference](https://setuptools-rust.readthedocs.io/en/latest/reference.html).
72+
73+
74+
We also add a [`MANIFEST.in` file](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html)
75+
to control which files we want in the source distribution[^4]:
76+
77+
```{literalinclude} ../examples/hello-world-setuppy/MANIFEST.in
78+
```
79+
80+
[^4]: Alternatively you can also use `setuptools-scm` to add all the files under revision control
81+
to the `sdist`, see the [docs](https://pypi.org/project/setuptools-scm/) for more information.
82+
83+
84+
## Testing the extension
85+
86+
With these files in place, you can install the project in a virtual environment
87+
for testing and making sure everything is working correctly:
88+
89+
90+
```powershell
91+
# cd hello-world-setuppy
92+
python3 -m venv .venv
93+
source .venv/bin/activate # on Linux or macOS
94+
.venv\Scripts\activate # on Windows
95+
python -m pip install -e .
96+
python -c 'import hello_world; print(hello_world.sum_as_string(5, 7))' # => 12
97+
# ... better write some tests with pytest ...
98+
```
99+
100+
101+
## Next steps and final remarks
102+
103+
- When you are ready to distribute your project, have a look on
104+
[the notes in the documentation about building wheels](https://setuptools-rust.readthedocs.io/en/latest/building_wheels.html).
105+
106+
- You can also use a [`RustBin`](https://setuptools-rust.readthedocs.io/en/latest/reference.html) object
107+
(instead of a `RustExtension`), if you want to distribute a binary executable
108+
written in Rust (instead of a library that can be imported by the Python runtime).
109+
Note however that distributing both library and executable (or multiple executables),
110+
may significantly increase the size of the
111+
[wheel](https://packaging.python.org/en/latest/glossary/#term-Wheel)
112+
file distributed by the
113+
[package index](https://packaging.python.org/en/latest/glossary/#term-Package-Index)
114+
and therefore increase build, download and installation times.
115+
Another approach is to use a Python entry-point that calls the Rust
116+
implementation (exposed via PyO3 bindings).
117+
See the [hello-world](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world)
118+
example for more insights.
119+
120+
- If want to include both `RustBin` and `RustExtension` same macOS wheel, you might have
121+
to manually add an extra `build.rs` file, see [PyO3/setuptools-rust#351](https://github.com/PyO3/setuptools-rust/pull/351)
122+
for more information about the workaround.
123+
124+
- Since the adoption of {pep}`517`, running `python setup.py ...` directly as a CLI tool is
125+
[considered deprecated](https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html).
126+
Nevertheless, `setup.py` can be safely used as a configuration file
127+
(the same way `conftest.py` is used by `pytest` or `noxfile.py` is used by `nox`).
128+
There is a different mindset that comes with this change, though:
129+
for example, it does not make sense to use `sys.exit(0)` in a `setup.py` file
130+
or use a overarching `try...except...` block to re-run a failed build with different parameters.

0 commit comments

Comments
 (0)