Skip to content

Commit e38b8c0

Browse files
vepadulanovepadulano
committed
[build] Provide way to build Python wheels of ROOT
pip install ROOT: the first seed new setup for cibuildwheel Automate version of C++ Python extension libs First steps towards building Python wheels Removing the need to link against libPython is a requirement for manylinux. A run of `cibuildwheel` with this configuration produces a wheel, which can be reinstalled on another Python virtual environment in the same machine. The problem is that the wheel is recognized as a pure Python wheel and the C++ libraries are just copied verbatim over. We need to understand how to build according to the Python version. Convince cibuildwheel ROOT is not pure Python Apparently just declaring a C extension module in the setuptools setup function is enough. Even if that extension has nothing to do with the project and doesn't exist anywhere. Enabling RDataFrame and RooFit Rename the project to ROOT TPython requires linking against libPython setuptools >= 72 breaks cmake extension compatibility See pypa/distutils#284 , the issue is marked as solved but it is not really. use non-specific directories Cleanup setup.py and check for C++ compiler Preserve cmd line args to ROOT cli Establish full list of build options with fail-on-missing Document some of the choices Update list of Python versions for cibuildwheel Rebase to master and add github workflow Run workflow on pull_request Tweak directory location in workflow Another test for the workflow Another change in workflow Try with push Remove quotes Try extending types of pull_request Just run workflow on push Correct artifact directory Add PEP503-compliant registry creation Add cron schedule and enable all wheel builds Use self-hosted nodes with ccache Specify container Use python virtual environment Use Python from the container image Help cibuildwheel run Reuse official setup python action Try another container Try to run as user in ci image Try with user ubuntu Fix for cmake and new workflow strategy Mock workflow Attempt workflow parallelization Streamline workflow steps Fix usage of f-string for older Python versions Fix download artifact Use special build option for wheel build Remove workaround for importing submodule Avoid using wrong compiler command on Windows Create target directory structure in installation folder This allows removing the extra RPATH manipulations in our CPython extensions Address review comments
1 parent 6db21cf commit e38b8c0

File tree

6 files changed

+286
-2
lines changed

6 files changed

+286
-2
lines changed

‎.github/workflows/cibuildwheel-impl/action.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
name: 'cibuildwheel-impl'
2+
description: 'Run cibuildwheel to produce a wheel depending on input tag'
3+
inputs:
4+
build-tag:
5+
description: 'The tag for this build'
6+
required: true
7+
8+
runs:
9+
using: "composite"
10+
steps:
11+
- name: Build wheel
12+
uses: pypa/[email protected]
13+
env:
14+
CIBW_BUILD: ${{ inputs.build-tag }}
15+
16+
- name: Upload wheel
17+
uses: actions/upload-artifact@v4
18+
with:
19+
name: ${{ inputs.build-tag }}
20+
path: wheelhouse/*

‎.github/workflows/python_wheel_build.yml‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
name: 'ROOT Python wheels'
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
branch:
7+
description: 'The branch for which the Python wheel has to be generated'
8+
type: string
9+
required: true
10+
default: "experimental-pip-install-root"
11+
push:
12+
branches: experimental-pip-install-root
13+
schedule:
14+
- cron: '01 1 * * *'
15+
16+
jobs:
17+
build-wheels:
18+
runs-on: ubuntu-latest
19+
strategy:
20+
matrix:
21+
target: [cp38-manylinux_x86_64, cp39-manylinux_x86_64, cp310-manylinux_x86_64, cp311-manylinux_x86_64, cp312-manylinux_x86_64, cp313-manylinux_x86_64]
22+
name: ${{ matrix.target }}
23+
steps:
24+
- uses: actions/checkout@v4
25+
- uses: ./.github/workflows/cibuildwheel-impl
26+
with:
27+
build-tag: ${{ matrix.target }}
28+
29+
create-and-upload-wheel-registry:
30+
needs: build-wheels
31+
runs-on: ubuntu-latest
32+
steps:
33+
34+
- name: Download produced wheels
35+
uses: actions/download-artifact@v4
36+
with:
37+
merge-multiple: true
38+
39+
- name: Install required system packages
40+
run: sudo apt-get install -y krb5-user xrootd-client
41+
42+
- name: Setup Python
43+
uses: actions/setup-python@v5
44+
45+
- name: Create PEP503-compliant wheel registry
46+
run: |
47+
mkdir -p wheelhouse
48+
mv *.whl wheelhouse
49+
python -m pip install --upgrade pip
50+
python -m pip install --user simple503
51+
simple503 wheelhouse
52+
53+
- name: Compute number of files to upload
54+
id: nfiles
55+
run: echo "NFILES=$(find wheelhouse -maxdepth 1 | wc -l)" >> "$GITHUB_OUTPUT"
56+
57+
- name: Upload wheels to EOS
58+
env:
59+
RWEBEOS_KT: ${{ secrets.RWEBEOS_KT }}
60+
KT_FILE_NAME: /tmp/decoded.keytab
61+
EOS_PATH: /eos/project/r/root-eos/www/experimental-python-wheels
62+
EOS_ENDPOINT: root://eosproject-r.cern.ch
63+
KRB5CCNAME: /tmp/krb5cc
64+
NFILES: ${{ steps.nfiles.outputs.NFILES }}
65+
working-directory: ${{ env.WORKDIR }}
66+
run: |
67+
echo +++ Content
68+
ls
69+
echo +++ Retrieving the secret
70+
echo ${RWEBEOS_KT} | base64 -d > ${KT_FILE_NAME}
71+
echo +++ Creating the token
72+
kinit -p ${{ secrets.KRB5USER }}@${{ secrets.KRB5REALM }} -kt ${KT_FILE_NAME}
73+
echo +++ Running the copy
74+
xrdcp --parallel ${NFILES} -rf wheelhouse/* ${EOS_ENDPOINT}/${EOS_PATH}/

‎bindings/pyroot/pythonizations/python/ROOT/_rootcli.py‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
import os
2+
import subprocess
3+
import sys
4+
5+
ROOT_HOME = os.path.dirname(__file__)
6+
7+
8+
def main() -> None:
9+
# Find the ROOT executable from the current installation directory
10+
bindir = os.path.join(ROOT_HOME, "bin")
11+
rootexe = os.path.join(bindir, "root.exe")
12+
if not os.path.exists(rootexe):
13+
raise FileNotFoundError(
14+
f"Could not find 'root.exe' executable in directory '{bindir}'. "
15+
"Something is wrong in the ROOT installation."
16+
)
17+
# Make sure command line arguments are preserved
18+
args = [rootexe] + sys.argv[1:]
19+
# Run the actual ROOT executable and return the exit code to the main Python process
20+
out = subprocess.run(args)
21+
return out.returncode
22+
23+
24+
if __name__ == "__main__":
25+
sys.exit(main())

‎cmake/modules/SearchInstalledSoftware.cmake‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -676,7 +676,11 @@ set(Python3_FIND_FRAMEWORK LAST)
676676

677677
# Even if we don't build PyROOT, one still need python executable to run some scripts
678678
list(APPEND python_components Interpreter)
679-
if(pyroot OR tmva-pymva)
679+
if(pyroot AND NOT (tpython OR tmva-pymva))
680+
# We have to only look for the Python development module in order to be able to build ROOT with a pip backend
681+
# In particular, it is forbidden to link against libPython.so, see https://peps.python.org/pep-0513/#libpythonx-y-so-1
682+
list(APPEND python_components Development.Module)
683+
elseif(tpython OR tmva-pymva)
680684
list(APPEND python_components Development)
681685
endif()
682686
if(tmva-pymva)
@@ -1759,7 +1763,7 @@ endif(tmva)
17591763
#---Check for PyROOT---------------------------------------------------------------------
17601764
if(pyroot)
17611765

1762-
if(Python3_Development_FOUND)
1766+
if(Python3_Development.Module_FOUND)
17631767
message(STATUS "PyROOT: development package found. Building for version ${Python3_VERSION}")
17641768
else()
17651769
if(fail-on-missing)

‎pyproject.toml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
[build-system]
2+
requires = ["cmake", "setuptools<72", "wheel", "ninja", "numpy"]
3+
4+
[project]
5+
name = "ROOT"
6+
version = "0.1a7"
7+
8+
requires-python = ">=3.8"
9+
authors = [
10+
{name = "Vincenzo Eduardo Padulano", email="[email protected]"}
11+
]
12+
maintainers = [
13+
{name = "Vincenzo Eduardo Padulano", email = "[email protected]"}
14+
]
15+
description = "An open-source data analysis framework used by high energy physics and others."
16+
readme = "README.md"
17+
license = {file = "LICENSE"}
18+
19+
[project.scripts]
20+
root = "ROOT._rootcli:main"
21+
22+
[tool.cibuildwheel]
23+
# Increase pip debugging output
24+
build-verbosity = 1
25+
manylinux-x86_64-image = "manylinux_2_28"
26+
27+
# Install system libraries
28+
[tool.cibuildwheel.linux]
29+
before-all = "dnf install -y epel-release && /usr/bin/crb enable && dnf install -y openssl-devel libX11-devel libXpm-devel libXft-devel libXext-devel libuuid-devel libjpeg-devel giflib-devel"

‎setup.py‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
"""
2+
setuptools-based build of ROOT.
3+
4+
This script uses setuptools API to steer a custom CMake build of ROOT. All build
5+
configuration options are specified in the class responsible for building. A
6+
custom extension module is injected in the setuptools setup to properly generate
7+
the wheel with CPython extension metadata. Note that ROOT is first installed via
8+
CMake into a temporary directory, then the ROOT installation artifacts are moved
9+
to the final Python environment installation path, which often starts at
10+
${ENV_PREFIX}/lib/pythonXX.YY/site-packages, before being packaged as a wheel.
11+
"""
12+
13+
import os
14+
import pathlib
15+
import shlex
16+
import subprocess
17+
import tempfile
18+
19+
from setuptools import Extension, find_packages, setup
20+
from setuptools.command.build import build as _build
21+
from setuptools.command.install import install as _install
22+
23+
# Get the long description from the README file
24+
SOURCE_DIR = pathlib.Path(__file__).parent.resolve()
25+
LONG_DESCRIPTION = (SOURCE_DIR / "README.md").read_text(encoding="utf-8")
26+
27+
BUILD_DIR = tempfile.mkdtemp()
28+
INSTALL_DIR = tempfile.mkdtemp()
29+
30+
# Name given to an internal directory within the build directory
31+
# used to mimick the structure of the target installation directory
32+
# in the user Python environment, usually named "site-packages"
33+
ROOT_BUILD_INTERNAL_DIRNAME = "mock_site_packages"
34+
35+
36+
class ROOTBuild(_build):
37+
def run(self):
38+
_build.run(self)
39+
40+
# Configure ROOT build
41+
configure_command = shlex.split(
42+
"cmake -GNinja -Dccache=ON "
43+
"-Dgminimal=ON -Dasimage=ON -Dopengl=OFF " # Graphics
44+
"-Druntime_cxxmodules=ON -Drpath=ON -Dfail-on-missing=ON " # Generic build configuration
45+
"-Dtmva-pymva=OFF -Dtpython=OFF " # Turn off explicitly components that link against libPython
46+
"-Dbuiltin_nlohmannjson=ON -Dbuiltin_tbb=ON -Dbuiltin_xrootd=ON " # builtins
47+
"-Dbuiltin_lz4=ON -Dbuiltin_lzma=ON -Dbuiltin_zstd=ON -Dbuiltin_xxhash=ON " # builtins
48+
"-Dpyroot=ON -Ddataframe=ON -Dxrootd=ON -Dssl=ON -Dimt=ON "
49+
"-Droofit=ON "
50+
# Next 4 paths represent the structure of the target binaries/headers/libs
51+
# as the target installation directory of the Python environment would expect
52+
f"-DCMAKE_INSTALL_BINDIR={ROOT_BUILD_INTERNAL_DIRNAME}/ROOT/bin "
53+
f"-DCMAKE_INSTALL_INCLUDEDIR={ROOT_BUILD_INTERNAL_DIRNAME}/ROOT/include "
54+
f"-DCMAKE_INSTALL_LIBDIR={ROOT_BUILD_INTERNAL_DIRNAME}/ROOT/lib "
55+
f"-DCMAKE_INSTALL_PYTHONDIR={ROOT_BUILD_INTERNAL_DIRNAME} "
56+
f"-DCMAKE_INSTALL_PREFIX={INSTALL_DIR} -B {BUILD_DIR} -S {SOURCE_DIR}"
57+
)
58+
subprocess.run(configure_command, check=True)
59+
60+
# Run build with CMake
61+
build_command = f"cmake --build {BUILD_DIR} -j{os.cpu_count()}"
62+
subprocess.run(shlex.split(build_command), check=True)
63+
64+
65+
class ROOTInstall(_install):
66+
def _get_install_path(self):
67+
if hasattr(self, "bdist_dir") and self.bdist_dir:
68+
install_path = self.bdist_dir
69+
else:
70+
install_path = self.install_lib
71+
72+
return install_path
73+
74+
def run(self):
75+
_install.run(self)
76+
77+
install_cmd = f"cmake --build {BUILD_DIR} --target install"
78+
subprocess.run(shlex.split(install_cmd), check=True)
79+
80+
install_path = self._get_install_path()
81+
82+
# Copy ROOT installation tree to the ROOT package directory in the pip installation path
83+
self.copy_tree(os.path.join(INSTALL_DIR, ROOT_BUILD_INTERNAL_DIRNAME), install_path)
84+
85+
root_package_dir = os.path.join(install_path, "ROOT")
86+
87+
# After the copy of the "mock" package structure from the ROOT installations, these are the
88+
# leftover directories that still need to be copied
89+
self.copy_tree(os.path.join(INSTALL_DIR, "cmake"), os.path.join(root_package_dir, "cmake"))
90+
self.copy_tree(os.path.join(INSTALL_DIR, "etc"), os.path.join(root_package_dir, "etc"))
91+
self.copy_tree(os.path.join(INSTALL_DIR, "fonts"), os.path.join(root_package_dir, "fonts"))
92+
self.copy_tree(os.path.join(INSTALL_DIR, "icons"), os.path.join(root_package_dir, "icons"))
93+
self.copy_tree(os.path.join(INSTALL_DIR, "macros"), os.path.join(root_package_dir, "macros"))
94+
self.copy_tree(os.path.join(INSTALL_DIR, "man"), os.path.join(root_package_dir, "man"))
95+
self.copy_tree(os.path.join(INSTALL_DIR, "README"), os.path.join(root_package_dir, "README"))
96+
self.copy_tree(os.path.join(INSTALL_DIR, "tutorials"), os.path.join(root_package_dir, "tutorials"))
97+
self.copy_file(os.path.join(INSTALL_DIR, "LICENSE"), os.path.join(root_package_dir, "LICENSE"))
98+
99+
def get_outputs(self):
100+
outputs = _install.get_outputs(self)
101+
return outputs
102+
103+
104+
class DummyExtension(Extension):
105+
"""
106+
Dummy CPython extension for setuptools setup.
107+
108+
In order to generate the wheel with CPython extension metadata (i.e.
109+
producing one wheel per supported Python version), setuptools requires that
110+
at least one CPython extension is declared in the `ext_modules` kwarg passed
111+
to the `setup` function. Usually, declaring a CPython extension triggers
112+
compilation of the corresponding sources, but in this case we already do
113+
that in the CMake build step. This class defines a dummy extension that
114+
can be declared to setuptools while avoiding any further compilation step.
115+
"""
116+
117+
def __init__(_):
118+
super().__init__(name="Dummy", sources=[])
119+
120+
121+
pkgs = find_packages("bindings/pyroot/pythonizations/python") + find_packages(
122+
"bindings/pyroot/cppyy/cppyy/python", include=["cppyy"]
123+
)
124+
125+
s = setup(
126+
long_description=LONG_DESCRIPTION,
127+
package_dir={"": "bindings/pyroot/pythonizations/python", "cppyy": "bindings/pyroot/cppyy/cppyy/python"},
128+
packages=pkgs,
129+
# Crucial to signal this is not a pure Python package
130+
ext_modules=[DummyExtension()],
131+
cmdclass={"build": ROOTBuild, "install": ROOTInstall},
132+
)

0 commit comments

Comments
 (0)