Skip to content

Commit 4db3074

Browse files
lwasserlwasser
committed
Add: more edits from massive pr
Update package-structure-code/python-package-structure.md Co-authored-by: David Nicholson <[email protected]> Phew! so many comments - more edits and cleanup pyproj page
1 parent e114a1b commit 4db3074

6 files changed

+410
-278
lines changed

‎package-structure-code/complex-python-package-builds.md

Lines changed: 69 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,57 @@
11
# Complex Python package builds
22

3+
This guide is focused on packages that are either pure-python or that
4+
have a few simple extensions in another language such as C or C++.
5+
6+
If your package is more complex, [you may want to refer to this guide
7+
created by Ralf Gommers on Python packaging.](https://pypackaging-native.github.io/)
8+
9+
## Pure Python Packages vs. packages with extensions in other languages
10+
11+
You can classify Python package complexity into three general categories. These
12+
categories can in turn help you select the correct package front-end and
13+
back end tools.
14+
15+
1. **Pure-python packages:** these are packages that only rely on Python to function. Building a pure Python package is simpler. As such, you can chose a tool below that
16+
has the features that you want and be done with your decision!
17+
2. **Python packages with non-Python extensions:** These packages have additional components called extensions written in other languages (such as `C` or `C++`). If you have a package with non-python extensions, then you need to select a build back-end tool that allows you to add additional build steps needed to compile your extension code. Further, if you wish to use a front-end tool to support your workflow, you will need to select a tool that
18+
supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back end of choice. We will discuss this at a high level on the complex builds page.
19+
3.**Python packages that have extensions written in different languages (e.g. fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
20+
or [meson-python](https://mesonbuild.com/Python-module.html) to build. NOTE: you can use meson-python with PDM.
21+
22+
23+
<!--
24+
On this page, we will focus on using front-end tools to package pure python
25+
packages. We will note if a package does have the flexibility to support other
26+
back-ends and in turn more complex builds (*mentioned in #2 and #3 above*). -->
27+
<!--
28+
## COmbine the two sets of statement below...
29+
ELI:
30+
PDM supports C/Cython extensions too: https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels
31+
32+
It does this by allowing you to write a python script that gets injected into a setuptools build process :) so that's not necessarily the greatest choice. It's a bit like using setuptools directly. ;)
33+
34+
Ralf:
35+
Hatch only supports pure Python packages as of now. setuptools is still a very reasonable choice, and okay if all you have is a few C/Cython extensions. But I'd say you should probably recommend meson-python and scikit-build-core as the two best tools for building packages containing compiled extensions.
36+
337
438
* link to ralf's blog and book on complex builds
539
* keep this page high level so we don't get weight downsides
640
* can use the examplePy repo stefan and I are working on that will test various build combinations
741
42+
*****
43+
44+
ELI: It would be more accurate to say that PDM supports using PDM and setuptools at the same time, so you run setuptools to produce the C extensions and then PDM receives the compiled extension files (.so, .pyd) and packages it up alongside the pure python files.
45+
46+
Hatch - https://hatch.pypa.io/latest/config/build/#build-hooks uild hooks
847
48+
Ralf -
49+
Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build backend hooks in pyproject.toml, it's pretty much a dead end.
50+
****
951
1052
53+
HEnry: Poetry will move to PEP 621 configuration in version 2.
54+
1155
* pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back end.
1256
* poetry's support for C extensions is not fully developed and documented (yet). * Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back end like meson to build via a build hook. so while some have used it with other back end builds it's not ideal for this application
1357
* pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
@@ -25,6 +69,14 @@ part of your packaging steps. These tools also support some C and C++
2569
extensions.
2670
2771
72+
OFEK - Why use hatchlin vs pdm back end -
73+
File inclusion is more configurable and easier by default
74+
There is already a rich ecosystem of plugins and a well-thought-out interface
75+
Consistency since the official Python packaging tutorial uses Hatchling by default
76+
77+
78+
Henry -
79+
The scikit-hep cookie provides 11 backends including flit-core and hatchling, and I've moved packaging to flit-core, and lots of other things to hatchling, and I can say that hatching's defaults are much nicer than flit-core's. Hatching uses .gitignore to decide what to put in the SDist. Flit-core basically tries to keep its hands off of adding defaults, so you have to configure everything manually. To make it even more confusing, if you use flit instead of a standard tool like build, it will switch to using VCS and those ignored files won't be added - meaning it is really easy to have a project that doesn't support build, including various GitHub Actions. Hatchling wins this by a ton.
2880
2981
<!-- TODO: add - compatible with other build back ends eg pdm can work with hatchling
3082
@@ -59,9 +111,7 @@ CORRECTIONS:
59111
pdm doesn't use plugins. Hatch does.
60112
pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
61113
62-
-->
63114
64-
```{note}
65115
??
66116
Poetry supports extensions written in other languages but this functionality is
67117
currently undocumented.
@@ -74,4 +124,20 @@ package builds.
74124
Some front-end packaging tools, such as PDM, allow you to use other
75125
build back-ends such as **meson** and **scikit-build**.
76126
77-
```
127+
128+
me:
129+
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back end.
130+
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back end like meson to build via a build hook.
131+
PDM and hatch both offer a plugin type approach to support custom build steps
132+
PDM (right now) is the only tool that supports other back ends (hatch is working on this - 2 minor releases away)
133+
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
134+
@eli-schwartz eli-schwartz 3 weeks ago
135+
PDM and hatch both offer a plugin type approach to support custom build steps
136+
137+
ELI:
138+
pdm doesn't use plugins. Hatch does.
139+
pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
140+
141+
142+
https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels
143+
-->

‎package-structure-code/intro.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,7 @@ the `src` directory.
9595
Intro <self>
9696
9797
Python package structure <python-package-structure>
98+
pyproject.toml Package Metadata <pyproject-toml-python-package-metadata>
9899
What are SDist & Wheel Files? <python-package-distribution-files-sdist-wheel>
99100
Package Build Tools <python-package-build-tools>
100101
Complex Builds <complex-python-package-builds>

‎package-structure-code/pyproject-toml-python-package-metadata.md

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Use a pyproject.toml file for your package configuration & metadata
2+
3+
The standard file that Python packages use to specify build requirements and
4+
metadata is called a pyproject.toml. The pyproject.toml file has become the
5+
standard for declaring Python package metadata (including dependencies) rather
6+
than using a setup.py file (or a setup.py + setup.cfg file).
7+
8+
As such you should try to [include all project based metadata and build system specifications in a `pyproject.toml` file.](https://packaging.python.org/en/latest/specifications/declaring-project-metadata/) Using setup.py to manage both package set up and
9+
hold metadata [can cause problems with package development.](https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html)
10+
11+
12+
```{admonition} Benefits of using a pyproject.toml file
13+
:class: tip
14+
15+
1. Because setup.py has a mixture of code and metadata, it will be run twice when
16+
building your package. First it will be run to extract metadata (dependencies). Then it will be run to build your package.
17+
1. Including your package's metadata in a separate human-readable `pyproject.toml` format also allows someone to view the project's metadata without
18+
running any Python code.
19+
```
20+
21+
A pyproject.toml is written in [TOML (Tom's Obvious, Minimal Language) format](https://toml.io/en/). TOML is an easy-to-read structure that is founded on key: value pairs.
22+
23+
Each section in the pyproject.toml file contains a `[table identifier]`.
24+
Below that table identifier are key value pairs that
25+
support configuration for that particular table.
26+
27+
<!-- setup.cfg for project metadata is being deprecated - set setuptools guide and
28+
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html
29+
pypa -
30+
https://packaging.python.org/en/latest/specifications/declaring-project-metadata/ -->
31+
32+
```{note}
33+
<!-- [PEP518 describes the move away from setup.py to the pyproject.toml file.](https://peps.python.org/pep-0518/) -->
34+
Python package standards are moving away from including both package metadata
35+
and Python code needed to set up a package in the same **setup.py** file.
36+
Instead we are moving towards using a **proproject.toml** file.
37+
38+
In some cases where a build is complex, a **setup.py** file may still be
39+
required. While this guide will not cover complex builds, we will provide
40+
resources working with complex builds in the future.
41+
42+
<!-- https://github.com/pyOpenSci/python-package-guide/pull/23#discussion_r1071541329
43+
ELI: A complex build could mean running a python script that processes some data file and produces a pure python module file.
44+
45+
Probably not common in the scientific community specifically, but I've seen quite a few setup.py files that contain custom build stages which e.g. build gettext locale catalogs.
46+
47+
The main point is that it is more "complex" than simply copying files or directories as-is into the built wheel.
48+
-->
49+
```
50+
51+
## Example pyproject.toml
52+
53+
Below is an example build configuration for a Python project. This setup
54+
requires:
55+
56+
* **setuptools** to create the package structure,
57+
* **wheel** which is used by `setuptools` to create the [**.whl** (wheel) file](https://realpython.com/python-wheels/).
58+
* **setuptools build** to "build" the package
59+
* **setuptools_scm** to manage package version updates
60+
61+
In the example below `[build-system]` is the first table
62+
of values. It has two keys that specify the build front end and backend for a package:
63+
64+
1. `requires =`
65+
1. `build-backend =`
66+
67+
```
68+
[build-system]
69+
requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2"]
70+
build-backend = "setuptools.build_meta"
71+
72+
[project]
73+
name = "examplePy"
74+
authors = [
75+
{name = "Some Maintainer", email = "[email protected]"}
76+
]
77+
maintainers = [{name = "All the contributors"}]
78+
license = {text = "BSD 3-Clause"}
79+
description = "An example Python package used to support Python packaging tutorials"
80+
keywords = ["pyOpenSci", "python packaging"]
81+
readme = "README.md"
82+
83+
dependencies = [
84+
"dependency-package-name-1",
85+
"dependency-package-name-2",
86+
]
87+
```
88+
89+
90+
Notice that you also can specify dependencies in this file.
91+
92+
93+
A major benefit of the pyproject.toml file is that it makes is transparent
94+
95+
1. what build system you are using to create your package
96+
2. what dependencies you need
97+
98+
99+
The package metadata including authors, keywords, etc is also easy to read.
100+
Below you can see the same toml file that uses a different build system (PDM).
101+
Notice how simple it is to swap out the tools needed to build this package!
102+
103+
```
104+
[build-system]
105+
requires = ["pdm-pep517>=1.0.0"]
106+
build-backend = "pdm.pep517.api"
107+
108+
[project]
109+
name = "examplePy"
110+
authors = [
111+
{name = "Some Maintainer", email = "[email protected]"}
112+
]
113+
maintainers = [{name = "All the contributors"}]
114+
license = {text = "BSD 3-Clause"}
115+
description = "An example Python package used to support Python packaging tutorials"
116+
keywords = ["pyOpenSci", "python packaging"]
117+
readme = "README.md"
118+
119+
dependencies = [
120+
"dependency-package-name-1",
121+
"dependency-package-name-2",
122+
]
123+
```
124+
125+
126+
127+
```{note}
128+
[Click here to read about our packaging documentation requirements.](/package-structure-code/python-package-build-tools)
129+
```
130+
131+
<!-- TODO: add link to section on build tools when it exists and
132+
turn this into button:
133+
134+
We discuss build tools for python package more here.
135+
-->
136+
137+
138+
139+
<!-- TODO:
140+
1. add some links to packages that are using a purely toml config
141+
1. link to our example package once it's further along
142+
-->

0 commit comments

Comments
 (0)