Skip to content

Commit aa3b076

Browse files
tbirdsotbirdso
authored
Merge pull request #243 from tbirdso/module-deps-script
ENH: Support iterative ITK module dependencies
2 parents 22c8e60 + 1ada74e commit aa3b076

12 files changed

+577
-80
lines changed

‎README.md‎

Lines changed: 104 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,36 @@
1-
ITK Python Package
2-
==================
1+
# ITK Python Package
32

43
This project provides a `setup.py` script to build ITK Python binary
54
packages and infrastructure to build ITK external module Python
65
packages.
76

8-
ITK is an open-source, cross-platform system that provides developers
7+
The Insight Toolkit (ITK) is an open-source, cross-platform system that provides developers
98
with an extensive suite of software tools for image analysis.
9+
More information is available on the [ITK website](https://itk.org/)
10+
or at the [ITK GitHub homepage](https://github.com/insightSoftwareConsortium/ITK).
1011

11-
Installation
12-
------------
12+
## Using ITK Python Packages
1313

14-
To install the ITK Python package:
14+
ITKPythonPackage scripts can be used to produce [Python](https://www.python.org/) packages
15+
for ITK and ITK external modules. The resulting packages can be
16+
hosted on the [Python Package Index (PyPI)](https://pypi.org/)
17+
for easy distribution.
18+
19+
### Installation
20+
21+
To install baseline ITK Python packages:
1522

1623
```sh
17-
pip install itk
24+
> pip install itk
1825
```
1926

20-
Usage
21-
-----
27+
To install ITK external module packages:
2228

23-
### Simple example script
29+
```sh
30+
> pip install itk-<module_name>
31+
```
32+
33+
### Using ITK in Python scripts
2434

2535
```python
2636
import itk
@@ -36,15 +46,97 @@ Usage
3646
itk.imwrite(median, output_filename)
3747
```
3848

49+
### Other Resources for Using ITK in Python
50+
3951
See also the [ITK Python Quick Start
4052
Guide](https://itkpythonpackage.readthedocs.io/en/master/Quick_start_guide.html).
41-
There are also many [downloadable examples documented in
42-
Sphinx](https://itk.org/ITKExamples/search.html?q=Python).
53+
There are also many [downloadable examples on the ITK examples website](https://examples.itk.org/search.html?q=Python).
4354

4455
For more information on ITK's Python wrapping, [an introduction is
4556
provided in the ITK Software
4657
Guide](https://itk.org/ITKSoftwareGuide/html/Book1/ITKSoftwareGuide-Book1ch3.html#x32-420003.7).
4758

59+
## Building with ITKPythonPackage
60+
61+
ITK reusable workflows are available to build and package Python wheels as
62+
part of Continuous Integration (CI) via Github Actions runners.
63+
Those workflows can handle the overhead of fetching, configuring, and
64+
running ITKPythonPackage build scripts for most ITK external modules.
65+
See [ITKRemoteModuleBuildTestPackageAction](https://github.com/InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction)
66+
for more information.
67+
68+
For special cases where ITK reusable workflows are not a good fit,
69+
ITKPythonPackage scripts can be directly used to build Python wheels
70+
to target Windows, Linux, and MacOS platforms. See
71+
[ITKPythonPackage ReadTheDocs](https://itkpythonpackage.readthedocs.io/en/master/Build_ITK_Module_Python_packages.html)
72+
documentation for more information on building wheels by hand.
73+
74+
## Frequently Asked Questions
75+
76+
### What target platforms and architectures are supported?
77+
78+
ITKPythonPackage currently supports building wheels for the following platforms and architectures:
79+
- Windows 10 x86_64 platforms
80+
- Windows 11 x86_64 platforms
81+
- MacOS 10.9+ x86_64 platforms
82+
- MacOS 11.0+ arm64 platforms
83+
- Linux glibc 2.17+ (E.g. Ubuntu 18.04+) x86_64 platforms
84+
- _Coming Soon: Linux ARM platforms_
85+
86+
### What should I do if my target platform/architecture does not appear on the list above?
87+
88+
Please open an issue in the [ITKPythonPackage issue tracker](https://github.com/InsightSoftwareConsortium/ITKPythonPackage/issues)
89+
for discussion, and consider contributing either time or funding to support
90+
development. The ITK open source ecosystem is driven through contributions from its community members.
91+
92+
### What is an ITK external module?
93+
94+
The Insight Toolkit consists of several baseline module groups for image analysis
95+
including filtering, I/O, registration, segmentation, and more. Community members
96+
can extend ITK by developing an ITK "external" module which stands alone in a separate
97+
repository and its independently built and tested. An ITK external module which
98+
meets community standards for documentation and maintenance may be included in the
99+
ITK build process as an ITK "remote" module to make it easier to retrieve and build.
100+
101+
Visit [ITKModuleTemplate](https://github.com/insightSoftwareConsortium/ITKmoduletemplate)
102+
to get started creating a new ITK external module.
103+
104+
### How can I make my ITK C++ filters available in Python?
105+
106+
ITK uses SWIG to wrap C++ filters for use in Python.
107+
See [Chapter 9 in the ITK Software Guide](https://itk.org/ITKSoftwareGuide/html/Book1/ITKSoftwareGuide-Book1ch9.html)
108+
or visit [ITKModuleTemplate](https://github.com/insightSoftwareConsortium/ITKmoduletemplate)
109+
to get started on writing `.wrap` files.
110+
111+
After you've added wrappings for your external module C++ filters
112+
you may build and distribute Python packages automatically with
113+
[ITKRemoteModuleBuildTestPackageAction](https://github.com/InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction)
114+
or manually with ITKPythonPackage scripts.
115+
116+
### What makes building ITK external module wheels different from building ITK wheels?
117+
118+
In order to build an ITK external module you must have first built ITK for the same target platform.
119+
However, building ITK modules and wrapping them for Python can take a very long time!
120+
To avoid having to rebuilt ITK before building every individual external module,
121+
artifacts from the ITK build process (headers, source files, wrapper outputs, and more) are
122+
packaged and cached as [ITKPythonBuilds](https://github.com/insightSoftwareConsortium/ITKpythonbuilds)
123+
releases.
124+
125+
In order to build Python wheels for an ITK external module, ITKPythonPackage scripts
126+
first fetch the appropriate ITK Python build artifacts along with other necessary
127+
tools. Then, the module can be built, packaged, and distributed on [PyPI](https://pypi.org/).
128+
129+
### My external module has a complicated build process. Is it supported by ITKPythonPackage?
130+
131+
Start by consulting the [ITKPythonPackage ReadTheDocs](https://itkpythonpackage.readthedocs.io/en/master/Build_ITK_Module_Python_packages.html)
132+
documentation and the [ITKPythonPackage issue tracker](https://github.com/InsightSoftwareConsortium/ITKPythonPackage/issues)
133+
for discussion related to your specific issue.
134+
135+
If you aren't able to find an answer for your specific case, please start a discussion the
136+
[ITK Discourse forum](https://discourse.itk.org/) for help.
137+
138+
## Additional Information
139+
48140
- Free software: Apache Software license
49141
- Documentation: <http://itkpythonpackage.readthedocs.org>
50142
- Source code: <https://github.com/InsightSoftwareConsortium/ITKPythonPackage>

‎docs/Build_ITK_Module_Python_packages.rst‎

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ can be found in the `ITK Software Guide
3636
GitHub automated CI package builds
3737
==================================
3838

39-
Freely available GitHub Action continous integration (CI) build and test
39+
Freely available GitHub Actions continous integration (CI) build and test
4040
services for open source repositories are provided by
4141
`GitHub <https://github.com/>`_. These services will build and test the C++
4242
code for your module and also generate Linux, macOS, and Windows Python
@@ -52,6 +52,11 @@ Section.
5252
.. figure:: images/GitHubActionArtifacts.png
5353
:alt: GitHub Action Artifacts
5454

55+
Reusable workflows available in
56+
[ITKRemoteModuleBuildTestPackageAction](https://github.com/InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction)
57+
can be used to handle the build-test-package process
58+
for a majority of ITK external modules with minimal extra development.
59+
5560
Upload the packages to PyPI
5661
----------------------------
5762

@@ -188,7 +193,7 @@ Then, build the wheels::
188193
Windows
189194
-------
190195

191-
First, install Microsoft Visual Studio 2015, Git, and CMake, which should be added to the system PATH environmental variable.
196+
First, install Microsoft Visual Studio 2022, Git, and CMake, which should be added to the system PATH environmental variable.
192197

193198
Open a PowerShell terminal as Administrator, and install Python::
194199

@@ -201,3 +206,33 @@ In a PowerShell prompt, run the `windows-build-wheels.ps1` script::
201206
PS C:\Windows> cd C:\ITKMyModule
202207
PS C:\ITKMyModule> git clone https://github.com/InsightSoftwareConsortium/ITKPythonPackage.git IPP
203208
PS C:\ITKMyModule> .\ITKPythonPackage\scripts\windows-download-cache-and-build-module-wheels.ps1
209+
210+
Other Notes
211+
-----------
212+
213+
ITK modules sometimes depend on third-party libraries. To include third-party libraries
214+
in development wheels for distribution, first add the library path to `LD_LIBRARY_PATH`
215+
on Linux, `DYLD_LIBRARY_PATH` on MacOS, or `PATH` on Windows. Then, run the platform
216+
build script.
217+
218+
ITK modules sometimes depend on other ITK modules. For instance, to build
219+
[ITKBSplineGradient](https://github.com/InsightSoftwareConsortium/ITKBSplineGradient)
220+
the user must first build ITK and then [ITKMeshToPolyData](https://github.com/InsightSoftwareConsortium/ITKmeshtopolydata).
221+
ITKPythonPackage scripts support iterative prerequisite ITK module dependencies with the `ITK_MODULE_PREQ`
222+
environment variable.
223+
224+
For Python build scripts, the ordered list of ITK module dependencies must be formatted as follows:
225+
226+
```
227+
ITK_MODULE_PREQ=<module_org>/<module_name>@<module_tag>:<module_org>/<module_name>@<module_tag>:...
228+
```
229+
230+
Where
231+
- `module_org` is the name of a Github organization to use to fetch the module, i.e. "InsightSoftwareConsortium";
232+
- `module_name` is the name of the module, i.e. "ITKMeshToPolyData";
233+
- `module_tag` is the git tag or commit hash to use to fetch the module, i.e. "v1.0.0"
234+
235+
Module names must be provided in order of dependencies for the build to succeed.
236+
237+
For more information see the
238+
[build scripts directory](https://github.com/InsightSoftwareConsortium/ITKPythonPackage/tree/master/scripts).

‎scripts/dockcross-manylinux-build-module-deps.sh‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
#!/bin/bash
2+
3+
########################################################################
4+
# Run this script in an ITK external module directory to generate
5+
# build artifacts for prerequisite ITK external modules.
6+
#
7+
# Module dependencies are built in a flat directory structure regardless
8+
# of recursive dependencies. Prerequisite sources are required to be passed
9+
# in the order in which they should be built.
10+
# For example, if ITKTargetModule depends on ITKTargetModuleDep2 which
11+
# depends on ITKTargetModuleDep1, the output directory structure
12+
# will look like this:
13+
#
14+
# / ITKTargetModule
15+
# -- / ITKTargetModuleDep1
16+
# -- / ITKTargetModuleDep2
17+
# ..
18+
#
19+
# ===========================================
20+
# ENVIRONMENT VARIABLES
21+
#
22+
# - `ITK_MODULE_PREQ`: Prerequisite ITK modules that must be built before the requested module.
23+
# Format is `<org_name>/<module_name>@<module_tag>:<org_name>/<module_name>@<module_tag>:...`.
24+
# For instance, `export ITK_MODULE_PREQ=InsightSoftwareConsortium/[email protected]`
25+
#
26+
########################################################################
27+
28+
# Initialize variables
29+
30+
script_dir=$(cd $(dirname $0) || exit 1; pwd)
31+
if [[ ! -f "${script_dir}/dockcross-manylinux-download-cache-and-build-module-wheels.sh" ]]; then
32+
echo "Could not find download script to use for building module dependencies!"
33+
exit 1
34+
fi
35+
36+
source "${script_dir}/dockcross-manylinux-set-vars.sh"
37+
38+
# Temporarily update prerequisite environment variable to prevent infinite recursion.
39+
ITK_MODULE_PREQ_TOPLEVEL=${ITK_MODULE_PREQ}
40+
ITK_MODULE_NO_CLEANUP_TOPLEVEL=${ITK_MODULE_NO_CLEANUP}
41+
export ITK_MODULE_PREQ=""
42+
export ITK_MODULE_NO_CLEANUP="ON"
43+
44+
########################################################################
45+
# Build ITK module dependencies
46+
47+
for MODULE_INFO in ${ITK_MODULE_PREQ_TOPLEVEL//:/ }; do
48+
MODULE_ORG=`(echo ${MODULE_INFO} | cut -d'/' -f 1)`
49+
MODULE_NAME=`(echo ${MODULE_INFO} | cut -d'@' -f 1 | cut -d'/' -f 2)`
50+
MODULE_TAG=`(echo ${MODULE_INFO} | cut -d'@' -f 2)`
51+
52+
MODULE_UPSTREAM=https://github.com/${MODULE_ORG}/${MODULE_NAME}.git
53+
echo "Cloning from ${MODULE_UPSTREAM}"
54+
git clone ${MODULE_UPSTREAM}
55+
56+
# Reuse cached build archive instead of redownloading.
57+
# Build archives are usually ~2GB so it is reasonable to move
58+
# instead of redownloading.
59+
if [[ `(compgen -G ./ITKPythonBuilds-linux*.tar.zst)` ]]; then
60+
mv ITKPythonBuilds-linux*.tar.zst ${MODULE_NAME}/
61+
fi
62+
63+
pushd ${MODULE_NAME}
64+
git checkout ${MODULE_TAG}
65+
cp ../dockcross-manylinux-download-cache-and-build-module-wheels.sh .
66+
if [[ -d ../ITKPythonPackage ]]; then
67+
ln -s ../ITKPythonPackage
68+
ln -s ./ITKPythonPackage/oneTBB-prefix
69+
fi
70+
71+
echo "Building module dependency ${MODULE_NAME}"
72+
./dockcross-manylinux-download-cache-and-build-module-wheels.sh $@
73+
popd
74+
75+
echo "Cleaning up module dependency"
76+
cp ./${MODULE_NAME}/include/* include/
77+
78+
# Cache build archive
79+
if [[ `(compgen -G ./ITKPythonBuilds-linux*.tar.zst)` ]]; then
80+
rm -f ./${MODULE_NAME}/ITKPythonBuilds-linux*.tar.zst
81+
else
82+
mv ./${MODULE_NAME}/ITKPythonBuilds-linux*.tar.zst .
83+
fi
84+
85+
# Cache ITKPythonPackage build scripts
86+
if [[ ! -d ./ITKPythonPackage ]]; then
87+
mv ./${MODULE_NAME}/ITKPythonPackage .
88+
ln -s ./ITKPythonPackage/oneTBB-prefix .
89+
fi
90+
91+
done
92+
93+
# Restore environment variable
94+
export ITK_MODULE_PREQ=${ITK_MODULE_PREQ_TOPLEVEL}
95+
ITK_MODULE_PREQ_TOPLEVEL=""
96+
export ITK_MODULE_NO_CLEANUP=${ITK_MODULE_NO_CLEANUP_TOPLEVEL}
97+
ITK_MODULE_NO_CLEANUP_TOPLEVEL=""
98+
99+
# Summarize disk usage for debugging
100+
du -sh ./* | sort -hr | head -n 20
101+
102+
echo "Done building ITK external module dependencies"

0 commit comments

Comments
 (0)