Wheel repair question

[ErichZimmer]

Hi all,

I am trying to perform wheel repairing on my first c++/python project. The dynamic libraries are built on build time and do not exist beforehand. Here is a quick folder layout.

project_folder
project_folder/extern (third party packages, some managed by vcpkg)
project_folder/openpiv_cxx(namespace python package)
project_folder/src (pure c++/c code)
project_folder/setup.py
project_folder/vcpkg.json (list of external dependencies)

Using scikit-build and cmake, this compiles into the following folder structure.

my_proj/_libs (contains the built runtime dependencies)
my_project/submodule1 (contains pybind11 extensions linked to dynamic libraries)
my_proj/set_path.py (adds _libs and dynamic libraries to path so it can be found)

However, when performing wheel repair, delocate can't find the libraries in the _libs folder. Even using

CIBW_REPAIR_WHEEL_COMMAND_MACOS: >
  DYLD_LIBRARY_PATH=$REPAIR_LIBRARY_PATH delocate-listdeps {wheel} &&
  DYLD_LIBRARY_PATH=$REPAIR_LIBRARY_PATH delocate-wheel --require-archs {delocate_archs} -w {dest_dir} {wheel}

and setting REPAIR_LIBRSRY_PATH, I still have issues because of locating the temporary dir in which the wheel is built. I saw this script in a previous answer, but I would like to stay away from bash scripts. Is there a way to locate the dynamic libraries built under _libs in CIBW, or should I simply use bash scripts to circumnavigate my current issue? Or should I simply restructure my project in a way CIBW can somehow work?

[ErichZimmer]

Here is a short copy of the log.

  + DYLD_LIBRARY_PATH=$LIBRARY_PATH delocate-listdeps /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/built_wheel/OpenPIV_cxx-0.3.2-cp38-cp38-macosx_10_14_x86_64.whl && DYLD_LIBRARY_PATH=$LIBRARY_PATH delocate-wheel --require-archs x86_64 -w /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/repaired_wheel /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/built_wheel/OpenPIV_cxx-0.3.2-cp38-cp38-macosx_10_14_x86_64.whl
  
  @rpath/libopenpivcore.dylib not found:
    Needed by: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpl72oe3y4/openpiv_cxx/process/_process_cpp.cpython-38-darwin.so
    Search path:
      
  
  @rpath/libopenpivcore.dylib not found:
    Needed by: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpl72oe3y4/openpiv_cxx/process/_process_cpp.cpython-38-darwin.so
    Search path:
      
  @rpath/libopenpivcore.dylib not found, requested by /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpl72oe3y4/openpiv_cxx/process/_process_cpp.cpython-38-darwin.so
  ERROR:delocate.libsana:
  @rpath/libopenpivcore.dylib not found:
    Needed by: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpu5f1xels/wheel/openpiv_cxx/process/_process_cpp.cpython-38-darwin.so
    Search path:
      
  ERROR:delocate.libsana:
  @rpath/libopenpivcore.dylib not found:
    Needed by: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpu5f1xels/wheel/openpiv_cxx/process/_process_cpp.cpython-38-darwin.so
    Search path:
      
  ERROR:delocate.libsana:@rpath/libopenpivcore.dylib not found, requested by /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpu5f1xels/wheel/openpiv_cxx/process/_process_cpp.cpython-38-darwin.so
  Traceback (most recent call last):
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/build/venv/bin/delocate-wheel", line 8, in <module>
      sys.exit(main())
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/build/venv/lib/python3.8/site-packages/delocate/cmd/delocate_wheel.py", line 128, in main
      copied = delocate_wheel(
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 643, in delocate_wheel
      copied_libs = delocate_path(
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 465, in delocate_path
      lib_dict = tree_libs_from_directory(
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/build/venv/lib/python3.8/site-packages/delocate/libsana.py", line 384, in tree_libs_from_directory
      return _tree_libs_from_libraries(
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-r_tepl4z/cp38-macosx_x86_64/build/venv/lib/python3.8/site-packages/delocate/libsana.py", line 325, in _tree_libs_from_libraries
      raise delocate.delocating.DelocationError(
  delocate.delocating.DelocationError: Could not find all dependencies.

[ErichZimmer]

To answer my question, restructuring my package worked. I followed what the maintainers of Freud did and installed the shared libraries directly into the sub-modules so CIBW can repair it.

[joerick]

Thanks for the follow up!

[ChrisBarker-NOAA]

Asking again for a similar problem, three years later.

I'm trying to build a python package that is Cython wrapping a C lib that also requires a bunch of dependencies.

the lib is libgd, which is pretty common on Linux systems, so Linux isn't too hard.

For years, I've been doing it all with conda, as conda-forge has a libgd package, and all is good.

However, I've got users that want a pure PyPi solution. I haven't done it for all these years because the dependencies are a pain, particularly on Windows.

However I recently discovered vcpkg, which should make it all easy (or at least, doable). vcpkg has libgd, so that's all ready to go.

And we set up the CI to build wheels, with vcpkg building the deps, and scikit-build-core building the package. And that all seems to work.

However, when we get to the audit-wheel step, it can't find the libs that vcpkg built:

  FileNotFoundError: Unable to find library: libgd.dll
  repairing C:\Users\runneradmin\AppData\Local\Temp\cibw-run-nz427842\cp310-win32\built_wheel\py_gd-2.3.3-cp310-cp310-win32.whl
  finding DLL dependencies

I've had no luck figuring this out.

Making it SO much harder is that my local Windows machine is locked down, and I can't even get vcpkg to work at al locally, so can't tell what it's doing.

Are there instruction somewhere on how to use vcpkg to provided dependencies for biulding wheels?

There have to be others doing this ....

Here's the issue in the project:

NOAA-ORR-ERD/py_gd#34

@henryiii : have you made any progress on this?