You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -61,7 +60,6 @@ Learn about our open peer review process
61
60
:::
62
61
::::
63
62
64
-
65
63
<!-- I think this is the end of the header - below begins the next grid-->
66
64
67
65
::::{grid} 1 1 2 2
@@ -92,7 +90,6 @@ commonly used in the scientific Python community.
92
90
All of the modern tools discussed in this guide will help you build an efficient packaging workflow. This section helps you select the tool that will work best for you.
93
91
:::
94
92
95
-
96
93
:::{grid-item-card}
97
94
:link: CONTRIBUTING
98
95
:link-type: doc
@@ -106,6 +103,7 @@ contribute.
106
103
::::
107
104
108
105
## Who this guidebook is for
106
+
109
107
We assume that you are here because you are:
110
108
111
109
1. Looking for guidance on creating a Python package.
@@ -118,9 +116,9 @@ Well, friend, you've come to the right place!
118
116
119
117
This guidebook contains:
120
118
121
-
* Explanation for "Good enough" minimum requirements associated with being reviewed by pyOpenSci
122
-
* Explanation of better and best practices in case you want to set the bar higher for your package (which we hope you will)!
123
-
* A curated list of resources to help you get your package into documented, usable and tested shape.
119
+
- Explanation for "Good enough" minimum requirements associated with being reviewed by pyOpenSci
120
+
- Explanation of better and best practices in case you want to set the bar higher for your package (which we hope you will)!
121
+
- A curated list of resources to help you get your package into documented, usable and tested shape.
124
122
125
123
## Where this guide is headed
126
124
@@ -131,9 +129,6 @@ Good meets the requirements. Going beyond the minimum can make package maintenan
131
129
This guide is now a work in progress. If you have ideas of things you'd like
132
130
to see here, [we invite you to open an issue on GitHub that details any changes or additions that you'd like to see.](https://github.com/pyOpenSci/python-package-guide/issues).
Copy file name to clipboardExpand all lines: package-structure-code/complex-python-package-builds.md
+19-24Lines changed: 19 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,15 +9,13 @@ In the future, we want to provide resources for packaging workflows that require
9
9
10
10
You can classify Python package complexity into three general categories. These
11
11
categories can in turn help you select the correct package front-end and
12
-
backend tools.
12
+
back-end tools.
13
13
14
14
1.**Pure-python packages:** these are packages that only rely on Python to function. Building a pure Python package is simpler. As such, you can chose a tool below that
15
-
has the features that you want and be done with your decision!
15
+
has the features that you want and be done with your decision!
16
16
2.**Python packages with non-Python extensions:** These packages have additional components called extensions written in other languages (such as C or C++). If you have a package with non-python extensions, then you need to select a build back-end tool that allows you to add additional build steps needed to compile your extension code. Further, if you wish to use a front-end tool to support your workflow, you will need to select a tool that
17
-
supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back end of choice. We will discuss this at a high level on the complex builds page.
18
-
3.**Python packages that have extensions written in different languages (e.g. fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
19
-
or [meson-python](https://mesonbuild.com/Python-module.html) to build. NOTE: you can use meson-python with PDM.
20
-
17
+
supports additional build setps. In this case, you could use setuptools. However, we suggest that you chose build tool that supports custom build steps such as Hatch with Hatchling or PDM. PDM is an excellent choice as it allows you to also select your build back-end of choice. We will discuss this at a high level on the complex builds page. 3.**Python packages that have extensions written in different languages (e.g. fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL)** These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as [scikit-build](https://scikit-build.readthedocs.io/en/latest/)
18
+
or [meson-python](https://mesonbuild.com/Python-module.html) to build. NOTE: you can use meson-python with PDM.
21
19
22
20
<!--
23
21
On this page, we will focus on using front-end tools to package pure python
@@ -45,17 +43,17 @@ ELI: It would be more accurate to say that PDM supports using PDM and setuptools
Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build backend hooks in pyproject.toml, it's pretty much a dead end.
46
+
Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build back-end hooks in pyproject.toml, it's pretty much a dead end.
49
47
****
50
48
51
49
52
50
HEnry: Poetry will move to PEP 621 configuration in version 2.
53
51
54
-
* pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-backend, hatchling and poetry's build backend.
55
-
* poetry's support for C extensions is not fully developed and documented (yet). * Poetry doesn't offer a way to facilitate "communication" between poetry front end and another backend like meson to build via a build hook. so while some have used it with other backend builds it's not ideal for this application
52
+
* pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
53
+
* poetry's support for C extensions is not fully developed and documented (yet). * Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook. so while some have used it with other back-end builds it's not ideal for this application
56
54
* pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
57
55
* hatch both offers a plugin type approach to support custom build steps
58
-
PDM (right now) is the only tool that supports other backends (hatch is working on this - 2 minor releases away)
56
+
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
59
57
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
60
58
61
59
@@ -68,42 +66,40 @@ part of your packaging steps. These tools also support some C and C++
68
66
extensions.
69
67
70
68
71
-
OFEK - Why use hatchlin vs pdm backend -
69
+
OFEK - Why use hatchlin vs pdm back-end -
72
70
File inclusion is more configurable and easier by default
73
71
There is already a rich ecosystem of plugins and a well-thought-out interface
74
72
Consistency since the official Python packaging tutorial uses Hatchling by default
75
73
76
74
77
75
Henry -
78
-
The scikit-hep cookie provides 11 backends including flit-core and hatchling, and I've moved packaging to flit-core, and lots of other things to hatchling, and I can say that hatching's defaults are much nicer than flit-core's. Hatching uses .gitignore to decide what to put in the SDist. Flit-core basically tries to keep its hands off of adding defaults, so you have to configure everything manually. To make it even more confusing, if you use flit instead of a standard tool like build, it will switch to using VCS and those ignored files won't be added - meaning it is really easy to have a project that doesn't support build, including various GitHub Actions. Hatchling wins this by a ton.
76
+
The scikit-hep cookie provides 11 back-ends including flit-core and hatchling, and I've moved packaging to flit-core, and lots of other things to hatchling, and I can say that hatching's defaults are much nicer than flit-core's. Hatching uses .gitignore to decide what to put in the sdist. Flit-core basically tries to keep its hands off of adding defaults, so you have to configure everything manually. To make it even more confusing, if you use flit instead of a standard tool like build, it will switch to using VCS and those ignored files won't be added - meaning it is really easy to have a project that doesn't support build, including various GitHub Actions. Hatchling wins this by a ton.
79
77
80
-
<!-- TODO: add - compatible with other build backends eg pdm can work with hatchling
78
+
<!-- TODO: add - compatible with other build back-ends eg pdm can work with hatchling
81
79
82
80
Eli:
83
81
poetry: supports it, but is undocumented and uses setuptools under the hood, they plan to change how this works and then document it
84
-
pdm-backend: supports it, and documents it -- and also uses setuptools under the hood
82
+
pdm-back-end: supports it, and documents it -- and also uses setuptools under the hood
85
83
hatchling: permits you to define hooks for you to write your own custom build steps, including to build C++ extensions
86
84
87
85
-->
88
86
89
-
90
-
91
87
<!-- from eli about pdm
92
88
It would be more accurate to say that PDM supports using PDM and setuptools at the same time, so you run setuptools to produce the C extensions and then PDM receives the compiled extension files (.so, .pyd) and packages it up alongside the pure Python files.
From ralf: There are no silver bullets here yet, no workflow tool is complete. Both Hatch and PDM are single-author tools, which is another concern. @eli-schwartz's assessment is unfortunately correct here I believe (at a high level at least, not sure about details). Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build backend hooks in pyproject.toml, it's pretty much a dead end.
93
+
From ralf: There are no silver bullets here yet, no workflow tool is complete. Both Hatch and PDM are single-author tools, which is another concern. @eli-schwartz's assessment is unfortunately correct here I believe (at a high level at least, not sure about details). Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build back-end hooks in pyproject.toml, it's pretty much a dead end.
98
94
99
95
-->
100
96
101
97
<!--TODO Add examples of builds using each of the tools below?
102
98
103
-
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build backend.
104
-
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another backend like meson to build via a build hook.
99
+
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
100
+
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook.
105
101
PDM and hatch both offer a plugin type approach to support custom build steps
106
-
PDM (right now) is the only tool that supports other backends (hatch is working on this - 2 minor releases away)
102
+
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
107
103
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
108
104
109
105
CORRECTIONS:
@@ -125,10 +121,10 @@ build back-ends such as **meson** and **scikit-build**.
125
121
126
122
127
123
me:
128
-
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build backend.
129
-
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another backend like meson to build via a build hook.
124
+
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
125
+
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook.
130
126
PDM and hatch both offer a plugin type approach to support custom build steps
131
-
PDM (right now) is the only tool that supports other backends (hatch is working on this - 2 minor releases away)
127
+
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
132
128
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
133
129
@eli-schwartz eli-schwartz 3 weeks ago
134
130
PDM and hatch both offer a plugin type approach to support custom build steps
@@ -141,7 +137,6 @@ pdm and poetry both rely on setuptools for C extensions. pdm's support claims to
0 commit comments