Build a custom wheel for different variants

[rdavidams]

Hello,
Is there some documentation somewhere about how to regenerate a wheel package with different variants?
All I can find is how to compile with cmake from sources, which works great, but when it comes to building the actual wheel, I encountered some difficulties.

I have a Dockerfile of my experiments and it seems to fail to link drjit at the end of the compilation process.
Am I doing something wrong?

Thanks a lot.

FROM ubuntu:20.04

# Source code
RUN apt update && apt install -y git
RUN git clone -b v3.1.0 --recursive https://github.com/mitsuba-renderer/mitsuba3

# Compiling tools
RUN DEBIAN_FRONTEND=noninteractive apt install -y git build-essential clang-10 libc++-10-dev libc++abi-10-dev cmake ninja-build libpng-dev libjpeg-dev libpython3-dev python3-distutils python3-pip
RUN export CC=clang-10 export CXX=clang++-10

# Build wheel
RUN pip wheel /mitsuba3 --wheel-dir=/mitsuba_wheel -v

Traceback (most recent call last):
File "/tmp/pip-req-build-d4ixbw8i/_skbuild/linux-x86_64-3.8/cmake-build/python/mitsuba/__init__.py", line 107, in __getattribute__
  _import('mitsuba.mitsuba_' + variant + '_ext'),
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
File "<frozen importlib._bootstrap>", line 556, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1166, in create_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: pybind11::detail::get_type_info: unable to find type info for "drjit::Array<float, 0ul>"

[njroussel]

Hi @rdavidams

I wouldn't expect that specific error message. Can you even build the variant "normally" (local cmake build) ?

We build our wheels using cibuildwheel. You can see the exact commands we run here: https://github.com/mitsuba-renderer/mitsuba3/blob/master/.github/workflows/wheels.yml. It doesn't use docker, but cibuildwheel uses docker on Linux for each individual build so you'll want to be careful if you try using it (docker-in-docker issues).

Just to be clear, the mitsuba wheel is built without drjit which is it's own package that is simply set as a requirement/dependency to mitsuba.

[rdavidams]

Thanks a lot for the explanations.

With your help, I successfully managed to build a wheel with a custom variant by adding an extra cmake option into the setup.py.

I have attached the patch bellow. Would you consider adding this option in future releases? I think I might not be the only one who would be interested in building custom pip packages for specific variants in the future.

diff --git a/setup.py b/setup.py
index f0c3bcc8..9f563ab5 100644
--- a/setup.py
+++ b/setup.py
@@ -36,6 +36,7 @@ mi_cmake_toolchain_file = os.environ.get("MI_CMAKE_TOOLCHAIN_FILE", "")
 mi_drjit_cmake_dir = os.environ.get("MI_DRJIT_CMAKE_DIR", "")
 mi_srgb_coeff_file = os.environ.get("MI_SRGB_COEFF_FILE", "")
 mi_python_stubs_dir = os.environ.get("MI_PYTHON_STUBS_DIR", "")
+mi_default_variants = os.environ.get("MI_DEFAULT_VARIANTS", "scalar_rgb;scalar_spectral;cuda_ad_rgb;llvm_ad_rgb")
 pathlib.Path("./mitsuba").mkdir(exist_ok=True)
 
 setup(
@@ -56,7 +57,8 @@ setup(
         f'-DCMAKE_TOOLCHAIN_FILE={mi_cmake_toolchain_file}',
         f'-DMI_DRJIT_CMAKE_DIR:STRING={mi_drjit_cmake_dir}',
         f'-DMI_SRGB_COEFF_FILE:STRING={mi_srgb_coeff_file}',
-        f'-DMI_PYTHON_STUBS_DIR:STRING={mi_python_stubs_dir}'
+        f'-DMI_PYTHON_STUBS_DIR:STRING={mi_python_stubs_dir}',
+        f'-DMI_DEFAULT_VARIANTS:STRING={mi_default_variants}'
     ],
     install_requires=[f"drjit=={drjit_version}"],
     packages=['mitsuba'],

And here is the Dockerfile (the patch above must be put into the same directory):

FROM quay.io/pypa/manylinux2014_x86_64:2022-10-25-fbea779

# Setup Python
ENV PATH=$PATH:/opt/python/cp38-cp38/bin/

# Source code
RUN git clone -b v3.1.0 --recursive https://github.com/mitsuba-renderer/mitsuba3
ADD ./mi_default_variants.patch /mi_default_variants.patch
RUN cd /mitsuba3 && git apply /mi_default_variants.patch
ENV MI_DEFAULT_VARIANTS scalar_rgb

# Build whith pip
RUN pip wheel /mitsuba3 --no-deps -v

[njroussel]

Great 🎉 !

Eventually, we would like to have packages on PyPI for "every" variant such that users can install whatever they need. This requires a bit more work on the build system such that variants are standalone shared libraries that can be linked into a main mitsuba library. This should also make what you were trying to do easier.

[rdavidams]

Hello @njroussel,
I am bringing back this thread to let you know about our latest developments.
I successfully managed to build my pip package for the variants: [scalar_rgb;scalar_spectral;cuda_ad_rgb;llvm_ad_rgb;cuda_ad_spectral].
However, when I try to remove some of those variants and keep only cuda_ad_spectral, then it fails. Same goes for cuda_ad_rgb as a single variant.
The error is the following:

[701/947] Building CXX object src/bsdfs/CMakeFiles/retarder.dir/retarder.cpp.o
  [702/947] Building CXX object src/bsdfs/CMakeFiles/twosided.dir/twosided.cpp.o
  [703/947] Linking CXX shared library libmitsuba.so
  FAILED: libmitsuba.so
  : && /opt/rh/devtoolset-10/root/usr/bin/c++ -fPIC -fdiagnostics-color=always -O3 -DNDEBUG -flto=auto -fno-fat-lto-objects   -shared -Wl,-soname,libmitsuba.so -o libmitsuba.so src/core/CMakeFiles/mitsuba-core.dir/string.cpp.o src/core/CMakeFiles/mitsuba-core.dir/appender.cpp.o src/core/CMakeFiles/mitsuba-core.dir/argparser.cpp.o src/core/CMakeFiles/mitsuba-core.dir/bitmap.cpp.o src/core/CMakeFiles/mitsuba-core.dir/class.cpp.o src/core/CMakeFiles/mitsuba-core.dir/dstream.cpp.o src/core/CMakeFiles/mitsuba-core.dir/filesystem.cpp.o src/core/CMakeFiles/mitsuba-core.dir/formatter.cpp.o src/core/CMakeFiles/mitsuba-core.dir/fresolver.cpp.o src/core/CMakeFiles/mitsuba-core.dir/fstream.cpp.o src/core/CMakeFiles/mitsuba-core.dir/jit.cpp.o src/core/CMakeFiles/mitsuba-core.dir/logger.cpp.o src/core/CMakeFiles/mitsuba-core.dir/mmap.cpp.o src/core/CMakeFiles/mitsuba-core.dir/tensor.cpp.o src/core/CMakeFiles/mitsuba-core.dir/mstream.cpp.o src/core/CMakeFiles/mitsuba-core.dir/object.cpp.o src/core/CMakeFiles/mitsuba-core.dir/plugin.cpp.o src/core/CMakeFiles/mitsuba-core.dir/profiler.cpp.o src/core/CMakeFiles/mitsuba-core.dir/progress.cpp.o src/core/CMakeFiles/mitsuba-core.dir/properties.cpp.o src/core/CMakeFiles/mitsuba-core.dir/qmc.cpp.o src/core/CMakeFiles/mitsuba-core.dir/rfilter.cpp.o src/core/CMakeFiles/mitsuba-core.dir/spectrum.cpp.o src/core/CMakeFiles/mitsuba-core.dir/stream.cpp.o src/core/CMakeFiles/mitsuba-core.dir/struct.cpp.o src/core/CMakeFiles/mitsuba-core.dir/thread.cpp.o src/core/CMakeFiles/mitsuba-core.dir/transform.cpp.o src/core/CMakeFiles/mitsuba-core.dir/util.cpp.o src/core/CMakeFiles/mitsuba-core.dir/xml.cpp.o src/core/CMakeFiles/mitsuba-core.dir/zstream.cpp.o src/core/CMakeFiles/mitsuba-core.dir/dither-matrix256.cpp.o src/render/CMakeFiles/mitsuba-render.dir/bsdf.cpp.o src/render/CMakeFiles/mitsuba-render.dir/emitter.cpp.o src/render/CMakeFiles/mitsuba-render.dir/endpoint.cpp.o src/render/CMakeFiles/mitsuba-render.dir/film.cpp.o src/render/CMakeFiles/mitsuba-render.dir/imageblock.cpp.o src/render/CMakeFiles/mitsuba-render.dir/integrator.cpp.o src/render/CMakeFiles/mitsuba-render.dir/medium.cpp.o src/render/CMakeFiles/mitsuba-render.dir/mesh.cpp.o src/render/CMakeFiles/mitsuba-render.dir/microfacet.cpp.o src/render/CMakeFiles/mitsuba-render.dir/phase.cpp.o src/render/CMakeFiles/mitsuba-render.dir/sampler.cpp.o src/render/CMakeFiles/mitsuba-render.dir/scene.cpp.o src/render/CMakeFiles/mitsuba-render.dir/sensor.cpp.o src/render/CMakeFiles/mitsuba-render.dir/shape.cpp.o src/render/CMakeFiles/mitsuba-render.dir/texture.cpp.o src/render/CMakeFiles/mitsuba-render.dir/spiral.cpp.o src/render/CMakeFiles/mitsuba-render.dir/srgb.cpp.o src/render/CMakeFiles/mitsuba-render.dir/optix_api.cpp.o src/render/CMakeFiles/mitsuba-render.dir/shapegroup.cpp.o src/render/CMakeFiles/mitsuba-render.dir/volume.cpp.o src/render/CMakeFiles/mitsuba-render.dir/volumegrid.cpp.o src/render/CMakeFiles/mitsuba-render.dir/optixdenoiser.cpp.o src/render/CMakeFiles/mitsuba-render.dir/librender_ptx.cpp.o  -Wl,-rpath,/builds/martigny/tech_fwk/asl/third_party/mitsuba3/mitsuba3/_skbuild/linux-x86_64-3.10/cmake-build:/tmp/pip-build-env-eq524l9s/overlay/lib/python3.10/site-packages/drjit:  -Wl,--no-undefined  libpng-mitsuba.so  /usr/lib64/libz.so  /usr/lib64/libm.so  libpugixml.so  libjpeg-mitsuba.so  libIlmImf.so  libImath.so  libHalf.so  libIexMath.so  libIlmThread.so  libIex.so  -pthread  /usr/lib64/libz.so  libasmjit-mitsuba.so  -lpthread  -lrt  -ldl  /tmp/pip-build-env-eq524l9s/overlay/lib/python3.10/site-packages/drjit/libdrjit-core.so  /tmp/pip-build-env-eq524l9s/overlay/lib/python3.10/site-packages/drjit/libnanothread.so  ext/rgb2spec/librgb2spec.a  libembree3.so && :
  /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: /tmp/libmitsuba.so.RdLs6q.ltrans0.ltrans.o: in function `mitsuba::Bitmap::resample(mitsuba::Bitmap*, mitsuba::ReconstructionFilter<float, mitsuba::Color<float, 3ul> > const*, std::pair<mitsuba::FilterBoundaryCondition, mitsuba::FilterBoundaryCondition> const&, std::pair<float, float> const&, mitsuba::Bitmap*) const':
  <artificial>:(.text+0x851d): undefined reference to `mitsuba::ReconstructionFilter<float, mitsuba::Color<float, 3ul> >::m_class'
  /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: <artificial>:(.text+0x8d8d): undefined reference to `mitsuba::ReconstructionFilter<float, mitsuba::Color<float, 3ul> >::m_class'
  /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: <artificial>:(.text+0x8ebd): undefined reference to `mitsuba::ReconstructionFilter<float, mitsuba::Color<float, 3ul> >::m_class'
  collect2: error: ld returned 1 exit status

It's not too problematic as there is a way to build. However the package might end up bigger than what it might really require for our use case. Is it something that is familiar to you? Shall I open an issue?

[njroussel]

I think scalar_rgb is a hard requirement right now, for a few different reasons that aren't about to change soon. So, I think at best you can have [scalar_rgb;cuda_ad_spectral].

Keep me updated :)

[rdavidams]

Great, it works perfectly fine with [scalar_rgb;cuda_ad_spectral].
Thanks again! :)