You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: package-structure-code/complex-python-package-builds.md
+1-130Lines changed: 1 addition & 130 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,134 +14,5 @@ back-end tools.
14
14
1.**Pure-python packages:** these are packages that only rely on Python to function. Building a pure Python package is simpler. As such, you can chose a tool below that
15
15
has the features that you want and be done with your decision!
16
16
2.**Python packages with non-Python extensions:** These packages have additional components called extensions written in other languages (such as C or C++). If you have a package with non-python extensions, then you need to select a build back-end tool that allows you to add additional build steps needed to compile your extension code. Further, if you wish to use a front-end tool to support your workflow, you will need to select a tool that
17
-
supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back-end of choice. We will discuss this at a high level on the complex builds page.
18
-
3.**Python packages that have extensions written in different languages (e.g. Fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
17
+
supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back-end of choice. We will discuss this at a high level on the complex builds page. 3.**Python packages that have extensions written in different languages (e.g. Fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
19
18
or [meson-python](https://mesonbuild.com/Python-module.html) to build. NOTE: you can use meson-python with PDM.
20
-
21
-
<!--
22
-
On this page, we will focus on using front-end tools to package pure python
23
-
packages. We will note if a package does have the flexibility to support other
24
-
back-ends and in turn more complex builds (*mentioned in #2 and #3 above*). -->
It does this by allowing you to write a python script that gets injected into a setuptools build process :) so that's not necessarily the greatest choice. It's a bit like using setuptools directly. ;)
31
-
32
-
Ralf:
33
-
Hatch only supports pure Python packages as of now. setuptools is still a very reasonable choice, and okay if all you have is a few C/Cython extensions. But I'd say you should probably recommend meson-python and scikit-build-core as the two best tools for building packages containing compiled extensions.
34
-
35
-
36
-
* link to ralf's blog and book on complex builds
37
-
* keep this page high level so we don't get weight downsides
38
-
* can use the examplePy repo stefan and I are working on that will test various build combinations
39
-
40
-
*****
41
-
42
-
ELI: It would be more accurate to say that PDM supports using PDM and setuptools at the same time, so you run setuptools to produce the C extensions and then PDM receives the compiled extension files (.so, .pyd) and packages it up alongside the pure python files.
Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build back-end hooks in pyproject.toml, it's pretty much a dead end.
48
-
****
49
-
50
-
51
-
HEnry: Poetry will move to PEP 621 configuration in version 2.
52
-
53
-
* pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-backend, hatchling and poetry's build back-end.
54
-
* poetry's support for C extensions is not fully developed and documented (yet). * Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook. so while some have used it with other back-end builds it's not ideal for this application
55
-
* pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
56
-
* hatch both offers a plugin type approach to support custom build steps
57
-
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
58
-
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
59
-
60
-
61
-
62
-
**Setuptools** is the oldest tool in the above list. While it doesn't have a
63
-
friendly user front end, because "OG" tool that has been used for Python packaging for over a decade, we discuss it here.
64
-
65
-
**Hatch** and PDM are newer, more modern tool that support customization of any
66
-
part of your packaging steps. These tools also support some C and C++
67
-
extensions.
68
-
69
-
70
-
OFEK - Why use hatchlin vs pdm back-end -
71
-
File inclusion is more configurable and easier by default
72
-
There is already a rich ecosystem of plugins and a well-thought-out interface
73
-
Consistency since the official Python packaging tutorial uses Hatchling by default
74
-
75
-
76
-
Henry -
77
-
The scikit-hep cookie provides 11 back-ends including flit-core and hatchling, and I've moved packaging to flit-core, and lots of other things to hatchling, and I can say that hatching's defaults are much nicer than flit-core's. Hatching uses .gitignore to decide what to put in the sdist. Flit-core basically tries to keep its hands off of adding defaults, so you have to configure everything manually. To make it even more confusing, if you use flit instead of a standard tool like build, it will switch to using VCS and those ignored files won't be added - meaning it is really easy to have a project that doesn't support build, including various GitHub Actions. Hatchling wins this by a ton.
78
-
79
-
<!-- TODO: add - compatible with other build back-ends eg pdm can work with hatchling
80
-
81
-
Eli:
82
-
poetry: supports it, but is undocumented and uses setuptools under the hood, they plan to change how this works and then document it
83
-
pdm-back-end: supports it, and documents it -- and also uses setuptools under the hood
84
-
hatchling: permits you to define hooks for you to write your own custom build steps, including to build C++ extensions
85
-
86
-
-->
87
-
88
-
<!-- from eli about pdm
89
-
It would be more accurate to say that PDM supports using PDM and setuptools at the same time, so you run setuptools to produce the C extensions and then PDM receives the compiled extension files (.so, .pyd) and packages it up alongside the pure Python files.
From ralf: There are no silver bullets here yet, no workflow tool is complete. Both Hatch and PDM are single-author tools, which is another concern. @eli-schwartz's assessment is unfortunately correct here I believe (at a high level at least, not sure about details). Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build back-end hooks in pyproject.toml, it's pretty much a dead end.
95
-
96
-
-->
97
-
98
-
<!--TODO Add examples of builds using each of the tools below?
99
-
100
-
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
101
-
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook.
102
-
PDM and hatch both offer a plugin type approach to support custom build steps
103
-
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
104
-
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
105
-
106
-
CORRECTIONS:
107
-
pdm doesn't use plugins. Hatch does.
108
-
pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
109
-
110
-
111
-
??
112
-
Poetry supports extensions written in other languages but this functionality is
113
-
currently undocumented.
114
-
115
-
Tools such as Setuptools, PDM, Hatch and Poetry all have some level of support
116
-
for C and C++ extensions.
117
-
Some Python packaging tools,
118
-
such as **Flit** and the **flit-core** build back-end only support pure-Python
119
-
package builds.
120
-
Some front-end packaging tools, such as PDM, allow you to use other
121
-
build back-ends such as **meson** and **scikit-build**.
122
-
123
-
124
-
me:
125
-
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
126
-
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook.
127
-
PDM and hatch both offer a plugin type approach to support custom build steps
128
-
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
129
-
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
130
-
@eli-schwartz eli-schwartz 3 weeks ago
131
-
PDM and hatch both offer a plugin type approach to support custom build steps
132
-
133
-
ELI:
134
-
pdm doesn't use plugins. Hatch does.
135
-
pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
ELI: A complex build could mean running a python script that processes some data file and produces a pure python module file.
143
-
144
-
Probably not common in the scientific community specifically, but I've seen quite a few setup.py files that contain custom build stages which e.g. build gettext locale catalogs.
145
-
146
-
The main point is that it is more "complex" than simply copying files or directories as-is into the built wheel.
<imgsrc="../images/python-package-tools-decision-tree.png"alt="Figure showing... will finish this once we are all happy with the figure and it's not going to change more..."width="700px">
@@ -53,44 +51,6 @@ and **scikit-build** support complex builds with custom steps. If your
53
51
build is particularly complex (i.e. you have more than a few `C`/`C++`
54
52
extensions), then we suggest you use **meson.build** or **scikit-build**.
55
53
56
-
<!--
57
-
### Build front-ends
58
-
59
-
Build front-ends have a user-friendly interface that allow you to perform
60
-
common Python packaging tasks such as building your package, creating an
61
-
environment to run package tests and build documentation, and pushing to PyPI.
62
-
63
-
For instance, you can use **Flit**, **Hatch**, **Poetry** and **PDM** to both build your
64
-
package and to publish your package to PyPI (or test PyPI). However, if you
65
-
want a tool that also support environment management and versioning your package,
66
-
then you might prefer to use **Hatch**, **Poetry** or **PDM**.
67
-
68
-
Using a tool like **Flit**, **Hatch**, **Poetry** or **PDM** will simplify your workflow.
69
-
70
-
Example to build your package with **Flit**:
71
-
72
-
`flit build`
73
-
74
-
Example to publish to PyPI:
75
-
`flit publish --repository testpypi`
76
-
77
-
In the Python package build space **setuptools** is
78
-
the "OG" -the original tool that everyone used to use.
79
-
With a tool like `setuptools` you have the flexibility
80
-
to publish python pure python packages and packages with custom build steps. However, you will also need to use other tools. For example, you will use `twine` to publish to PyPI.
81
-
82
-
## An ecosystem of Python build tools
83
-
84
-
Below we introduce several of the most commonly used
85
-
Python packaging build tools. Each tool has various
86
-
features that might make you chose to use it
87
-
or not use it. There is no right or wrong tool to use
88
-
as far as pyOpenSci is concerned. We are just trying to
89
-
help you find the tool that works best for
90
-
your workflow.
91
-
Example build steps using setuptools:
92
-
======= -->
93
-
94
54
### Python package build front-ends
95
55
96
56
A packaging front-end tool refers to a tool that makes it easier for you to
@@ -214,17 +174,6 @@ questions:
214
174
NOTE: You can also use Hatch for non pure python builds but you will need to
215
175
write your own plugin for this support.
216
176
217
-
<!-- ### Build tools for Python packages with complex build steps
218
-
If your package is not pure Python, or it has complex build steps (or build
219
-
steps that you need to customize), then you should consider using:
220
-
221
-
* Setuptools
222
-
* Hatch
223
-
* PDM
224
-
225
-
These tools allow you to customize your workflow with build steps needed
226
-
to compile code. -->
227
-
228
177
## Python packaging tools summary
229
178
230
179
<!-- NOTE - add language around the front end means that you have less individual tools in your build - such as nox / make with hatch -->
@@ -379,9 +328,6 @@ Build your sdist and wheel distributions|✅| Hatch will build the sdist and whe
379
328
380
329
_\*\* There is some argument about this approach placing a burden on maintainers to create a custom build system. But others appreciate the flexibility_
381
330
382
-
<!-- QUESTION: Does hatch allow you to use other envt managers like conda?? i don't see that it does
383
-
so it might be similar to Poetry in that regard -->
384
-
385
331
### Why you might not want to use Hatch
386
332
387
333
There are a few features that hatch is missing that may be important for some.
@@ -427,7 +373,7 @@ Install your package in editable mode|✅|Poetry supports installing your packag
427
373
Build your sdist and wheel distributions|✅|Poetry will build your sdist and wheel distributions using `poetry build`
428
374
```
429
375
430
-
<!-- TODO: update this given responses here: https://github.com/python-poetry/poetry/discussions/7525 -->
376
+
<!-- TODO: responses here on poetry's future dev work: https://github.com/python-poetry/poetry/discussions/7525 -->
431
377
432
378
### Challenges with Poetry
433
379
@@ -516,79 +462,3 @@ when using setuptools. For instance:
516
462
\*Setuptools also will include all of the files in your package
517
463
repository if you do not explicitly tell it to exclude files using a
518
464
**MANIFEST.in** file
519
-
520
-
<!-- From stefan: build, run tests on built version, load the built version into
521
-
Python (?how is this different from install??), make editable install, build
522
-
wheel, build sdist -->
523
-
524
-
<!--
525
-
The example below is taken from [this thread in GitHub](https://github.com/py-pkgs/py-pkgs/issues/95#issuecomment-1035584750).
526
-
527
-
```toml
528
-
[tool.poetry.dependencies]
529
-
python = ">=3.6" # This is muddled in as a dependency, while it's not like the others
530
-
numpy = ">=1.13.3"
531
-
typing_extensions = { version = ">=3.7", python = "<3.8" }
532
-
533
-
sphinx = {version = "^4.0", optional = true}
534
-
sphinx_book_theme = { version = ">=0.0.40", optional = true }
535
-
sphinx_copybutton = { version = ">=0.3.1", optional = true }
536
-
pytest = { version = ">=6", optional = true }
537
-
importlib_metadata = { version = ">=1.0", optional = true, python = "<3.8" } # TOML error to add an ending comma or new line, even if this gets long
538
-
boost-histogram = { version = ">=1.0", optional = true }
539
-
540
-
[tool.poetry.dev-dependencies]
541
-
pytest = ">=5.2" # All the optional stuff above doesn't help here!
0 commit comments