Building wheels on MacOS arm64 for Fortran/C project fails at link-time

[gnikit]

We are in the process of generating wheels for the Fortran Package Manager (fpm: https://github.com/fortran-lang/fpm).
The project is written in Fortran with a bit of C, so we were hoping to use cmake, scikit-build and cibuildwheel since integration between the 3 is quite seamless.

However, the MacOS arm64 build has been causing us grief (see workflow log: https://github.com/awvwgk/fpm-on-wheels/actions/runs/4535252232/jobs/7990474203 and the full cmake project on https://github.com/awvwgk/fpm-on-wheels/tree/build/cmake). Building the bdist_wheel directly (on an arm mac) with python3 setup.py bdist_wheel works fine, are we missing something?

Also, since we don't actually have any Python in our code it would be ideal if we built wheels without any mention to the Python ABI i.e. py3-none-... do you know if that is easily doable with cibuildwheel (and/or scikit-build)?

Thanks for all the great work you've been doing on cibuildwheel and friends.

[Czaki]

I'm not an expert of compiling on macOS, but log explicitly says that part of code is x86-64. Also, I see that linking contains /usr/local/Cellar/gcc@11/11.3.0/lib/gcc/11/libgomp.dylib that looks like brew-installed files that most probably will be x86-64.

py3-none- - the cmake wheels are built using cibuildwheel and produces such wheels see: https://github.com/scikit-build/cmake-python-distributions

[gnikit]

Yes we indeed use OpenMP (for parallel compilation in fpm). Linking the x86-64 version of OpenMP on the arm64 wheel is definitely problematic. Even if we disable OpenMP see we still fail. I suppose it is for the same reason, more or less. gfortran (at /usr/local/bin/gfortran-11) is probably not present on arm64. Any ideas how to address this?

py3-none- - the cmake wheels are built using cibuildwheel and produces such wheels see: https://github.com/scikit-build/cmake-python-distributions

Thanks for the link. I suppose you are referring to the scripts/repair_wheel.py or is there something else I am missing?

[Czaki]

log suggest that this files are problematic:

    ld: warning: ignoring file CMakeFiles/fpm.dir/fpm/src/filesystem_utilities.c.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
    ld: warning: ignoring file CMakeFiles/fpm.dir/fpm/src/ptycheck/isatty.c.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
    ld: warning: ignoring file CMakeFiles/fpm.dir/fpm/src/ptycheck/iscygpty.c.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64

The only way that I know is to download build artifacts and inspect files to validate architecture. Or get an intel machine and try to setup a cross-compile.

py3-none- - the cmake wheels are built using cibuildwheel and produces such wheels see: scikit-build/cmake-python-distributions

Thanks for the link. I suppose you are referring to the scripts/repair_wheel.py or is there something else I am missing?

I do not dive in this code. but it looks like @henryiii could be a good person to ask. If I correctly remember last wheel release brings better support for this.

[henryiii]

scikit-build-core supports this directly via a configuration parameter. Classic scikit-build is limited to setuptools, so it can't directly produce py3-none, but you can retag the wheel (starting with wheel 0.40):

$ wheel tags --python-tag py3 --abi-tag none --remove <filename>.whl

You can't get dependencies from brew for building cross-compiled wheels (and in general, they aren't designed for redistributable binaries either). So you'll need a Universal version of g-fortran ideally compiled with a good setting for MACOSX_DEPLOYMENT_TARGET. Usually this means you need to build the dep yourself - though always check to see if someone supplies binaries. OpenMP's not too hard, and I think gfortran probably isn't too bad either, though I think I've only done it on linux. Probably should cache the outputs.

[Czaki]

here is the script to build openmp https://github.com/Czaki/imagecodecs_build/blob/master/build_utils/build_libomp.sh, but it may need to be modified for cross-compilation.

[gnikit]

Thank you both for your help, it's much appreciated.

@henryiii I wasn't aware that wheel could do this, that's neat. I will see if we can transfer to scikit-build-core over scikit-build (if not now in the not so distant future).

As for the cross-compilation on MacOS for gfortran, I think the problem might be running deeper than that. From what I could tell from a bit of reading, there are ongoing (and currently incomplete) efforts from the maintainers of GCC to create a cross-platform compiler from x86-64 MacOS to arm64 see: https://github.com/iains/gcc-darwin-arm64 .

I will have to untangle all the information I've come upon to see there is a workaround. I seem to recall that universal2 binary compiled successfully but failed on the wheel repairing stage. I will investigate further.