Skip to content

Commit 7d6cd62

Browse files
vepadulanovepadulano
committed
[build] Provide way to build Python wheels of ROOT
This commit introduces the infrastructure required to build Python wheels from the ROOT project. In turn, these can be used to effectively install the package via `pip install ROOT`. For more details, see https://indico.cern.ch/event/1338689/contributions/6010410/. A Python wheel is a binary archive with the compiled artifacts of a package build. When installing the ROOT package via pip, the package manager will simply decompress the archive and all the ROOT libraries will be ready for use in the user's Python environment. In order to enable building a Python wheel of ROOT, the following changes have been made. First, it is required that all the components of the software do not explicitly link against the Python shared library (e.g. 'libpython.so'). The requirement comes from PEP503 which establishes a portable way to build Python wheels across different platforms, see https://peps.python.org/pep-0513/#libpythonx-y-so-1. Second, the directory structure of the destination user Python environment follows a semi-defined logic, where all artifacts of a package are unpacked at a root directory e.g. `site-packages` inside a sub-directory with the package name, e.g. `site-packages/ROOT`. In order to respect this structure, proper RPATH management of the ROOT libraries and, most notably, the CPython extensions (libROOTPythonizations and cppyy) is required. The `-Drpath=ON` build option is used in the build of the wheel. The CPython extensions are also modified with the right RPATH location. This is handled by building ROOT with the CMake variables CMAKE_INSTALL_BINDIR, CMAKE_INSTALL_INCLUDEDIR, CMAKE_INSTALL_LIBDIR and CMAKE_INSTALL_PYTHONDIR following the `site-packages/ROOT` structure during the build process and then moving the entire directory created this way directly to the target user Python environment directory. On top of being able to use ROOT as a Python package, the project provides a few CLI tools such as `root`. This first commit demonstrates how to also expose these tools in the Python packaging way, i.e. declaring a Python script to the build backend that invokes the correct command. This is defined in the [project.scripts] section of the newly created pyproject.toml file. At the moment, only the `root` command is going to be available from the Python wheel, other commands can be added in the future in a similar way. The wheel of ROOT is currently built via setuptools and using the `manylinux` platform tag to create portable wheels that can work across any Linux x86_64 distribution. One modification particular to the setuptools build is the declaration of a dummy CPython extension module in the setuptools `setup` function. This extensions has no code and takes no part in the build process, but it is enough to signal to the build backend that this package needs to be built for a specific Python version (required at the moment to ensure compatibility of the CPython extensions with the target system). Another issue is pypa/distutils#284, at this moment it looks like more recent versions of setuptools break this CMake extension compatibility. This should be addressed in the future. A new github workflow is written to automatically create Python wheels for every Python version supported by ROOT: 3.8, 3.9, 3.10, 3.11, 3.12 and 3.13 at the time of writing. The workflow is set to run nightly and can be later adjusted to run on specific triggers, e.g. during a PR build with a special label.
1 parent 85e74b8 commit 7d6cd62

File tree

6 files changed

+293
-2
lines changed

6 files changed

+293
-2
lines changed

‎.github/workflows/cibuildwheel-impl/action.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
name: 'cibuildwheel-impl'
2+
description: 'Run cibuildwheel to produce a wheel depending on input tag'
3+
inputs:
4+
build-tag:
5+
description: 'The tag for this build'
6+
required: true
7+
8+
runs:
9+
using: "composite"
10+
steps:
11+
- name: Build wheel
12+
uses: pypa/[email protected]
13+
env:
14+
CIBW_BUILD: ${{ inputs.build-tag }}
15+
16+
- name: Upload wheel
17+
uses: actions/upload-artifact@v4
18+
with:
19+
name: ${{ inputs.build-tag }}
20+
path: wheelhouse/*

‎.github/workflows/python_wheel_build.yml‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
name: 'ROOT Python wheels'
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
branch:
7+
description: 'The branch for which the Python wheel has to be generated'
8+
type: string
9+
required: true
10+
default: "master"
11+
push:
12+
branches: experimental-pip-install-root
13+
schedule:
14+
- cron: '01 1 * * *'
15+
16+
jobs:
17+
build-wheels:
18+
runs-on: ubuntu-latest
19+
strategy:
20+
matrix:
21+
target: [cp38-manylinux_x86_64, cp39-manylinux_x86_64, cp310-manylinux_x86_64, cp311-manylinux_x86_64, cp312-manylinux_x86_64, cp313-manylinux_x86_64]
22+
name: ${{ matrix.target }}
23+
steps:
24+
- uses: actions/checkout@v4
25+
- uses: ./.github/workflows/cibuildwheel-impl
26+
with:
27+
build-tag: ${{ matrix.target }}
28+
29+
create-and-upload-wheel-registry:
30+
needs: build-wheels
31+
runs-on: ubuntu-latest
32+
steps:
33+
34+
- name: Download produced wheels
35+
uses: actions/download-artifact@v4
36+
with:
37+
merge-multiple: true
38+
39+
- name: Install required system packages
40+
run: sudo apt-get install -y krb5-user xrootd-client
41+
42+
- name: Setup Python
43+
uses: actions/setup-python@v5
44+
45+
- name: Create PEP503-compliant wheel registry
46+
run: |
47+
mkdir -p wheelhouse
48+
mv *.whl wheelhouse
49+
python -m pip install --upgrade pip
50+
python -m pip install --user simple503
51+
simple503 wheelhouse
52+
53+
- name: Compute number of files to upload
54+
id: nfiles
55+
run: echo "NFILES=$(find wheelhouse -maxdepth 1 | wc -l)" >> "$GITHUB_OUTPUT"
56+
57+
- name: Upload wheels to EOS
58+
env:
59+
RWEBEOS_KT: ${{ secrets.RWEBEOS_KT }}
60+
KT_FILE_NAME: /tmp/decoded.keytab
61+
EOS_PATH: /eos/project/r/root-eos/www/experimental-python-wheels
62+
EOS_ENDPOINT: root://eosproject-r.cern.ch
63+
KRB5CCNAME: /tmp/krb5cc
64+
NFILES: ${{ steps.nfiles.outputs.NFILES }}
65+
working-directory: ${{ env.WORKDIR }}
66+
run: |
67+
echo +++ Content
68+
ls
69+
echo +++ Retrieving the secret
70+
echo ${RWEBEOS_KT} | base64 -d > ${KT_FILE_NAME}
71+
echo +++ Creating the token
72+
kinit -p ${{ secrets.KRB5USER }}@${{ secrets.KRB5REALM }} -kt ${KT_FILE_NAME}
73+
echo +++ Running the copy
74+
xrdcp --parallel ${NFILES} -rf wheelhouse/* ${EOS_ENDPOINT}/${EOS_PATH}/

‎bindings/pyroot/pythonizations/python/ROOT/_rootcli.py‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
import os
2+
import subprocess
3+
import sys
4+
5+
ROOT_HOME = os.path.dirname(__file__)
6+
7+
8+
def main() -> None:
9+
# Find the ROOT executable from the current installation directory
10+
bindir = os.path.join(ROOT_HOME, "bin")
11+
rootexe = os.path.join(bindir, "root.exe")
12+
if not os.path.exists(rootexe):
13+
raise FileNotFoundError(
14+
f"Could not find 'root.exe' executable in directory '{bindir}'. "
15+
"Something is wrong in the ROOT installation."
16+
)
17+
# Make sure command line arguments are preserved
18+
args = [rootexe] + sys.argv[1:]
19+
# Run the actual ROOT executable and return the exit code to the main Python process
20+
out = subprocess.run(args)
21+
return out.returncode
22+
23+
24+
if __name__ == "__main__":
25+
sys.exit(main())

‎cmake/modules/SearchInstalledSoftware.cmake‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -676,7 +676,11 @@ set(Python3_FIND_FRAMEWORK LAST)
676676

677677
# Even if we don't build PyROOT, one still need python executable to run some scripts
678678
list(APPEND python_components Interpreter)
679-
if(pyroot OR tmva-pymva)
679+
if(pyroot AND NOT (tpython OR tmva-pymva))
680+
# We have to only look for the Python development module in order to be able to build ROOT with a pip backend
681+
# In particular, it is forbidden to link against libPython.so, see https://peps.python.org/pep-0513/#libpythonx-y-so-1
682+
list(APPEND python_components Development.Module)
683+
elseif(tpython OR tmva-pymva)
680684
list(APPEND python_components Development)
681685
endif()
682686
if(tmva-pymva)
@@ -1759,7 +1763,7 @@ endif(tmva)
17591763
#---Check for PyROOT---------------------------------------------------------------------
17601764
if(pyroot)
17611765

1762-
if(Python3_Development_FOUND)
1766+
if(Python3_Development.Module_FOUND)
17631767
message(STATUS "PyROOT: development package found. Building for version ${Python3_VERSION}")
17641768
else()
17651769
if(fail-on-missing)

‎pyproject.toml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
[build-system]
2+
requires = ["cmake", "setuptools<72", "wheel", "numpy"]
3+
4+
[project]
5+
name = "ROOT"
6+
version = "0.1a8"
7+
requires-python = ">=3.8"
8+
maintainers = [
9+
{name = "Vincenzo Eduardo Padulano", email = "[email protected]"}
10+
]
11+
description = "An open-source data analysis framework used by high energy physics and others."
12+
readme = "README.md"
13+
license = {file = "LICENSE"}
14+
dependencies = [
15+
"numpy",
16+
]
17+
18+
# Demonstrate how to expose ROOT CLI tools the Python way. For now only include
19+
# `root`, other tools can be added later using the same approach.
20+
[project.scripts]
21+
root = "ROOT._rootcli:main"
22+
23+
[tool.cibuildwheel]
24+
# Increase pip debugging output
25+
build-verbosity = 1
26+
manylinux-x86_64-image = "manylinux_2_28"
27+
28+
# Install system libraries
29+
[tool.cibuildwheel.linux]
30+
before-all = "dnf install -y epel-release && /usr/bin/crb enable && dnf install -y openssl-devel libX11-devel libXpm-devel libXft-devel libXext-devel libuuid-devel libjpeg-devel giflib-devel"

‎setup.py‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
"""
2+
setuptools-based build of ROOT.
3+
4+
This script uses setuptools API to steer a custom CMake build of ROOT. All build
5+
configuration options are specified in the class responsible for building. A
6+
custom extension module is injected in the setuptools setup to properly generate
7+
the wheel with CPython extension metadata. Note that ROOT is first installed via
8+
CMake into a temporary directory, then the ROOT installation artifacts are moved
9+
to the final Python environment installation path, which often starts at
10+
${ENV_PREFIX}/lib/pythonXX.YY/site-packages, before being packaged as a wheel.
11+
"""
12+
13+
import os
14+
import pathlib
15+
import shlex
16+
import subprocess
17+
import tempfile
18+
19+
from setuptools import Extension, find_packages, setup
20+
from setuptools.command.build import build as _build
21+
from setuptools.command.install import install as _install
22+
23+
# Get the long description from the README file
24+
SOURCE_DIR = pathlib.Path(__file__).parent.resolve()
25+
LONG_DESCRIPTION = (SOURCE_DIR / "README.md").read_text(encoding="utf-8")
26+
27+
BUILD_DIR = tempfile.mkdtemp()
28+
INSTALL_DIR = tempfile.mkdtemp()
29+
30+
# Name given to an internal directory within the build directory
31+
# used to mimick the structure of the target installation directory
32+
# in the user Python environment, usually named "site-packages"
33+
ROOT_BUILD_INTERNAL_DIRNAME = "mock_site_packages"
34+
35+
36+
class ROOTBuild(_build):
37+
def run(self):
38+
_build.run(self)
39+
40+
# Configure ROOT build
41+
configure_command = shlex.split(
42+
"cmake "
43+
# gminimal=ON enables only a minimal set of components (cling+core+I/O+graphics)
44+
"-Dgminimal=ON -Dasimage=ON -Dopengl=OFF "
45+
"-Druntime_cxxmodules=ON -Drpath=ON -Dfail-on-missing=ON " # Generic build configuration
46+
# Explicitly turned off components, even though they are already off because of gminimal, we want to keep
47+
# them listed here for documentation purposes:
48+
# - tmva-pymva, tpython: these components link against libPython, forbidden for manylinux compatibility,
49+
# see https://peps.python.org/pep-0513/#libpythonx-y-so-1
50+
# - thisroot_scripts: the thisroot.* scripts are broken if CMAKE_INSTALL_PYTHONDIR!=CMAKE_INSTALL_LIBDIR
51+
"-Dtmva-pymva=OFF -Dtpython=OFF -Dthisroot_scripts=OFF "
52+
"-Dbuiltin_nlohmannjson=ON -Dbuiltin_tbb=ON -Dbuiltin_xrootd=ON " # builtins
53+
"-Dbuiltin_lz4=ON -Dbuiltin_lzma=ON -Dbuiltin_zstd=ON -Dbuiltin_xxhash=ON " # builtins
54+
"-Dpyroot=ON -Ddataframe=ON -Dxrootd=ON -Dssl=ON -Dimt=ON "
55+
"-Droofit=ON "
56+
# Next 4 paths represent the structure of the target binaries/headers/libs
57+
# as the target installation directory of the Python environment would expect
58+
f"-DCMAKE_INSTALL_BINDIR={ROOT_BUILD_INTERNAL_DIRNAME}/ROOT/bin "
59+
f"-DCMAKE_INSTALL_INCLUDEDIR={ROOT_BUILD_INTERNAL_DIRNAME}/ROOT/include "
60+
f"-DCMAKE_INSTALL_LIBDIR={ROOT_BUILD_INTERNAL_DIRNAME}/ROOT/lib "
61+
f"-DCMAKE_INSTALL_PYTHONDIR={ROOT_BUILD_INTERNAL_DIRNAME} "
62+
f"-DCMAKE_INSTALL_PREFIX={INSTALL_DIR} -B {BUILD_DIR} -S {SOURCE_DIR}"
63+
)
64+
subprocess.run(configure_command, check=True)
65+
66+
# Run build with CMake
67+
build_command = f"cmake --build {BUILD_DIR} -j{os.cpu_count()}"
68+
subprocess.run(shlex.split(build_command), check=True)
69+
70+
71+
class ROOTInstall(_install):
72+
def _get_install_path(self):
73+
if hasattr(self, "bdist_dir") and self.bdist_dir:
74+
install_path = self.bdist_dir
75+
else:
76+
install_path = self.install_lib
77+
78+
return install_path
79+
80+
def run(self):
81+
_install.run(self)
82+
83+
install_cmd = f"cmake --build {BUILD_DIR} --target install"
84+
subprocess.run(shlex.split(install_cmd), check=True)
85+
86+
install_path = self._get_install_path()
87+
88+
# Copy ROOT installation tree to the ROOT package directory in the pip installation path
89+
self.copy_tree(os.path.join(INSTALL_DIR, ROOT_BUILD_INTERNAL_DIRNAME), install_path)
90+
91+
root_package_dir = os.path.join(install_path, "ROOT")
92+
93+
# After the copy of the "mock" package structure from the ROOT installations, these are the
94+
# leftover directories that still need to be copied
95+
self.copy_tree(os.path.join(INSTALL_DIR, "cmake"), os.path.join(root_package_dir, "cmake"))
96+
self.copy_tree(os.path.join(INSTALL_DIR, "etc"), os.path.join(root_package_dir, "etc"))
97+
self.copy_tree(os.path.join(INSTALL_DIR, "fonts"), os.path.join(root_package_dir, "fonts"))
98+
self.copy_tree(os.path.join(INSTALL_DIR, "icons"), os.path.join(root_package_dir, "icons"))
99+
self.copy_tree(os.path.join(INSTALL_DIR, "macros"), os.path.join(root_package_dir, "macros"))
100+
self.copy_tree(os.path.join(INSTALL_DIR, "man"), os.path.join(root_package_dir, "man"))
101+
self.copy_tree(os.path.join(INSTALL_DIR, "README"), os.path.join(root_package_dir, "README"))
102+
self.copy_tree(os.path.join(INSTALL_DIR, "tutorials"), os.path.join(root_package_dir, "tutorials"))
103+
self.copy_file(os.path.join(INSTALL_DIR, "LICENSE"), os.path.join(root_package_dir, "LICENSE"))
104+
105+
def get_outputs(self):
106+
outputs = _install.get_outputs(self)
107+
return outputs
108+
109+
110+
class DummyExtension(Extension):
111+
"""
112+
Dummy CPython extension for setuptools setup.
113+
114+
In order to generate the wheel with CPython extension metadata (i.e.
115+
producing one wheel per supported Python version), setuptools requires that
116+
at least one CPython extension is declared in the `ext_modules` kwarg passed
117+
to the `setup` function. Usually, declaring a CPython extension triggers
118+
compilation of the corresponding sources, but in this case we already do
119+
that in the CMake build step. This class defines a dummy extension that
120+
can be declared to setuptools while avoiding any further compilation step.
121+
"""
122+
123+
def __init__(_):
124+
super().__init__(name="Dummy", sources=[])
125+
126+
127+
pkgs = find_packages("bindings/pyroot/pythonizations/python") + find_packages(
128+
"bindings/pyroot/cppyy/cppyy/python", include=["cppyy"]
129+
)
130+
131+
s = setup(
132+
long_description=LONG_DESCRIPTION,
133+
package_dir={"": "bindings/pyroot/pythonizations/python", "cppyy": "bindings/pyroot/cppyy/cppyy/python"},
134+
packages=pkgs,
135+
# Crucial to signal this is not a pure Python package
136+
ext_modules=[DummyExtension()],
137+
cmdclass={"build": ROOTBuild, "install": ROOTInstall},
138+
)

0 commit comments

Comments
 (0)