Building Gaffer on MacOS 10.15

[jedypod]

Hello!
I'm trying to build Gaffer on MacOS 10.15.7 for x86_64. I'm still working on getting all the dependencies compiled. I have gotten pretty far, all the way up to OpenShadingLanguage:

python3 build.py
Project OpenSSL is up to date : skipping
Project LibFFI is up to date : skipping
Project Python is up to date : skipping
Project Boost is up to date : skipping
Project Imath is up to date : skipping
Project OpenEXR is up to date : skipping
Project HDF5 is up to date : skipping
Project Alembic is up to date : skipping
Project BitstreamVera is up to date : skipping
Project Blosc is up to date : skipping
Project CMark is up to date : skipping
Project PyBind11 is up to date : skipping
Project LibJPEG-Turbo is up to date : skipping
Project LibTIFF is up to date : skipping
Project LibPNG is up to date : skipping
Project OpenJPEG is up to date : skipping
Project Expat is up to date : skipping
Project YAML-CPP is up to date : skipping
Project PyString is up to date : skipping
Project ZLib is up to date : skipping
Project Minizip is up to date : skipping
Project OpenColorIO is up to date : skipping
Project LibRaw is up to date : skipping
Project PugiXML is up to date : skipping
Project Fmt is up to date : skipping
Project OpenImageIO is up to date : skipping
Project LLVM is up to date : skipping
Project Partio is up to date : skipping
Building project OpenShadingLanguage

Unfortunately I'm running up against a brick wall there with this error:

[ 22%] Generating llvm_ops_host.bc
cd /Users/jed/dev/compile/dependencies-7.0.0/OpenShadingLanguage/working/OpenShadingLanguage-1.12.14.0/src/liboslexec && /Users/jed/dev/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/bin/clang++ -DOSL_FAST_MATH=1 -DOSL_INTERNAL=1 -DOIIO_DISABLE_DEPRECATED=1 -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DOIIO_HIDE_FORMAT=1 -DOSL_LLVM_VERSION=111 -DOSL_LLVM_FULL_VERSION="11.1.0" -Dosoparse=OSL_v1_12_osoparse -Doslparse=OSL_v1_12_oslparse -std=c++17 -fno-use-cxa-atexit -I/Users/jed/dev/compile/dependencies-7.0.0/OpenShadingLanguage/working/OpenShadingLanguage-1.12.14.0/src/liboslexec -I/Users/jed/dev/compile/dependencies-7.0.0/OpenShadingLanguage/working/OpenShadingLanguage-1.12.14.0/src/include -I/Users/jed/dev/compile/dependencies-7.0.0/OpenShadingLanguage/working/OpenShadingLanguage-1.12.14.0/gafferBuild/include -I/Users/jed/dev/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include -I/Users/jed/dev/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include -I/Users/jed/dev/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include -DOSL_COMPILING_TO_BITCODE=1 -Wno-deprecated-register -Wno-ignored-attributes -Wno-unknown-attributes -O3 -fno-math-errno -S -emit-llvm -o /Users/jed/dev/compile/dependencies-7.0.0/OpenShadingLanguage/working/OpenShadingLanguage-1.12.14.0/gafferBuild/src/liboslexec/llvm_ops_host.s llvm_ops.cpp
llvm_ops.cpp:77:10: fatal error: 'cstddef' file not found
#include <cstddef>
         ^~~~~~~~~
1 error generated.
make[2]: *** [src/liboslexec/llvm_ops_host.bc] Error 1
make[1]: *** [src/liboslexec/CMakeFiles/oslexec.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

I'm guessing it's something to do with the include paths, since it can't find cstddef. I've tried manually adding them to the include paths with " -D CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES=/Library/Developer/CommandLineTools/usr/include/c++/v1" (Pretty sure this is the right include directory). I've tried recompiling LLVM with a newer clang. I've tried many things and nothing seems to change the error :(

Any advice would be greatly appreciated. Thanks!

[johnhaddon]

This rings a bell - there's some commented code in the OSL source that I think was dealing with an identical problem in earlier MacOS versions :

https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/b2b5e5eabaf473c508b3133c5147847788838513/src/cmake/llvm_macros.cmake#L64-L76

We also used to have our own patch on top of that, adapting it to our build environment :

https://github.com/GafferHQ/dependencies/blob/e04f136c4f66dda012fe3bbc91d6d9d9d56fd960/OpenShadingLanguage/patches/llvmCompileFlagsMac.patch.

Note that this patch is out of date (it applies to a different file than the commented code above), but it might be useful to see that we needed a slightly different variant at one point.

[johnhaddon]

For what it's worth, I'm building Gaffer on Monterey (12.5.1) with Xcode 13.4, so you might want to try a VM with that setup. My other tip would be not to make a patch file until you've got it working, because it's quite a painful way to work. What I do is when the build fails, I go into the OpenShadingLanguage/working/... directory and start issuing modified cmake and make commands myself (based on the ones that config.py would emit. Then I can just directly edit any relevant files and rerun. Once it works, then I make a patch file from what I've done.

This thread also contains various things I've needed in the past, and which people are claiming fixes the wchar.h error. Might be worth a look - the xcode-select one definitely rings a bell.

What we'd like to do is build a "blessed" VM for this that everyone could use, like the Docker container we have for Linux. But MacOS is very much a personal-time-only thing for us, so finding the time is problematic.

[murraystevenson]

/Library/Developer/CommandLineTools/usr/include/c++/v1/wchar.h:118:15: fatal error: 'wchar.h' file not found
#include_next <wchar.h>

This rings a few very faint bells from my time building on an Intel mac in the 10.14/10.15 era. I think I worked around it by setting the SDKROOT environment variable to hint to CMake where the macOS platform SDK could be found.

export SDKROOT="`xcrun --sdk macosx --show-sdk-path`"

[jedypod]

Thank you both for your help!
With the following two commands I've gotten OpenShadingLanguage to compile successfully.

sudo xcode-select --switch /Library/Developer/CommandLineTools/
export SDKROOT="`xcrun --sdk macosx --show-sdk-path`"

For what it's worth, I'm building Gaffer on Monterey (12.5.1) with Xcode 13.4, so you might want to try a VM with that setup.

Yes I actually already built gaffer and dependencies successfully on macos 12 (it was pretty easy). I wanted to do a version that was a bit more widely compatible though, which is why I'm on this masochistic circus ride.

What we'd like to do is build a "blessed" VM for this that everyone could use, like the Docker container we have for Linux. But MacOS is very much a personal-time-only thing for us, so finding the time is problematic.

Oh that's interesting! Is this something you would be open to assistance with? I would be keen to help if I can. Though I admit I'd probably need more info about how the github runner action vms / docker containers are set up... maybe I can summarize in excruciating detail all of the steps needed
to get the compile working as a starting point.

Resuming the circus ride: On my MacOS 11 VM I've managed to get all the way up to compiling Cortex. And I'm running into another error. First an issue saying it can't find scons:

mv /Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/lib/python/pxr /Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/python       
Building project Cortex                                                                                                                                                  
curl -L https://github.com/ImageEngine/cortex/archive/refs/tags/10.5.0.0.tar.gz > Cortex/archives/10.5.0.0.tar.gz                                                        
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                          
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                            
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0                                                                                           
100 19.7M    0 19.7M    0     0  7922k      0 --:--:--  0:00:02 --:--:-- 10.4M                                                                                           
scons  -j 16 CXX=`which g++` CXXSTD=c++17 INSTALL_PREFIX=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos INSTALL_DOC_DIR=/Users/jed/compile/dependen
cies-7.0.0/gafferDependencies-7.0.0-macos/doc/cortex INSTALL_PYTHON_DIR=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/python INSTALL_IECORE_OPS=''
 PYTHON_CONFIG=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/bin/python3-config PYTHON=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.
0.0-macos/bin/python BOOST_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/boost LIBPATH=/Users/jed/compile/dependencies-7.0.0/
gafferDependencies-7.0.0-macos/lib BOOST_LIB_SUFFIX='' OPENEXR_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include ILMBASE_INCLUDE_
PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/Imath VDB_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-m
acos/include TBB_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include OIIO_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gaffer
Dependencies-7.0.0-macos/include OSL_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include BLOSC_INCLUDE_PATH=/Users/jed/compile/depe
ndencies-7.0.0/gafferDependencies-7.0.0-macos/include FREETYPE_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/freetype2 WITH_G
L=1 GLEW_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/GL NUKE_ROOT= USD_LIB_PREFIX=usd_ ENV_VARS_TO_IMPORT='LD_LIBRARY_PATH 
TERM' OPTIONS=''  PYTHON_LIB_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/lib/Python.framework/Versions/3.8/lib PYTHON_LINK_FLAGS=-lpython3.
8 SAVE_OPTIONS=gaffer.options                                                                                                                                            
/bin/sh: scons: command not found                                                                                                                                        
Traceback (most recent call last):                                                                                                                                       
  File "./build.py", line 487, in <module>                                                                                                                               
    __buildProjects( args.projects, configs, buildDir )                                                                                                                  
  File "./build.py", line 368, in __buildProjects                                                                                                                        
    walk( project, configs, buildDir )                                                                                                                                   
  File "./build.py", line 360, in walk                                                                                                                                   
    __buildProject( project, configs[project], buildDir )                                                                                                                
  File "./build.py", line 257, in decorated                                                                                                                              
    return f( *args, **kw )                                                                                                                                              
  File "./build.py", line 315, in __buildProject                                                                                                                         
    subprocess.check_call( command, shell = True, env = environment )                                                                                                    
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call                                                
    raise CalledProcessError(retcode, cmd)                                                                                                                               
subprocess.CalledProcessError: Command 'scons  -j 16 CXX=`which g++` CXXSTD=c++17 INSTALL_PREFIX=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos INS
TALL_DOC_DIR=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/doc/cortex INSTALL_PYTHON_DIR=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-
7.0.0-macos/python INSTALL_IECORE_OPS='' PYTHON_CONFIG=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/bin/python3-config PYTHON=/Users/jed/compile/
dependencies-7.0.0/gafferDependencies-7.0.0-macos/bin/python BOOST_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/boost LIBPAT
H=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/lib BOOST_LIB_SUFFIX='' OPENEXR_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependenc
ies-7.0.0-macos/include ILMBASE_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/Imath VDB_INCLUDE_PATH=/Users/jed/compile/depen
dencies-7.0.0/gafferDependencies-7.0.0-macos/include TBB_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include OIIO_INCLUDE_PATH=/Use
rs/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include OSL_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include BL
OSC_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include FREETYPE_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependenc
ies-7.0.0-macos/include/freetype2 WITH_GL=1 GLEW_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/GL NUKE_ROOT= USD_LIB_PREFIX=u
sd_ ENV_VARS_TO_IMPORT='LD_LIBRARY_PATH TERM' OPTIONS=''  PYTHON_LIB_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/lib/Python.framework/Versi
ons/3.8/lib PYTHON_LINK_FLAGS=-lpython3.8 SAVE_OPTIONS=gaffer.options' returned non-zero exit status 127.

Then, this error after I installed scons using system python pip python -m pip install scons, and adding the folder where the scons binary is installed onto $PATH: export PATH=/Users/jed/Library/Python/3.8/bin/:$PATH

Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Checking for C++ header file boost/version.hpp... no
ERROR : unable to find the boost headers, check BOOST_INCLUDE_PATH.
Traceback (most recent call last):
  File "./build.py", line 487, in <module>
    __buildProjects( args.projects, configs, buildDir )
  File "./build.py", line 368, in __buildProjects
    walk( project, configs, buildDir )
  File "./build.py", line 360, in walk
    __buildProject( project, configs[project], buildDir )
  File "./build.py", line 257, in decorated
    return f( *args, **kw )
  File "./build.py", line 315, in __buildProject
    subprocess.check_call( command, shell = True, env = environment )
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'scons  -j 16 CXX=`which g++` CXXSTD=c++17 INSTALL_PREFIX=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos INSTALL_DOC_DIR=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/doc/cortex INSTALL_PYTHON_DIR=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/python INSTALL_IECORE_OPS='' PYTHON_CONFIG=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/bin/python3-config PYTHON=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/bin/python BOOST_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/boost LIBPATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/lib BOOST_LIB_SUFFIX='' OPENEXR_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include ILMBASE_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/Imath VDB_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include TBB_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include OIIO_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include OSL_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include BLOSC_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include FREETYPE_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/freetype2 WITH_GL=1 GLEW_INCLUDE_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/include/GL NUKE_ROOT= USD_LIB_PREFIX=usd_ ENV_VARS_TO_IMPORT='LD_LIBRARY_PATH TERM' OPTIONS=''  PYTHON_LIB_PATH=/Users/jed/compile/dependencies-7.0.0/gafferDependencies-7.0.0-macos/lib/Python.framework/Versions/3.8/lib PYTHON_LINK_FLAGS=-lpython3.8 SAVE_OPTIONS=gaffer.options' returned non-zero exit status 1.

I've verifeid that BOOST_INCLUDE_PATH is valid and has the right stuff in it (I think). I'm guessing it's probably something to do with using system scons instead of one that seems to not be installed in the main installation/compilation path... but it's a long shot. I've tried installing scons with the compile directory's python. Again.. at a bit of a loss here. Any ideas would be greatly appreciated!

[johnhaddon]

Oh that's interesting! Is this something you would be open to assistance with? I would be keen to help if I can.

Yes, we'd definitely be interested in help with making the MacOS builds more officially supported, and a standard VM would be a great place to start. I should note though that our appetite for masochism extends only as far as supporting Apple silicon - we don't intend to support Intel builds as well.

ERROR : unable to find the boost headers, check BOOST_INCLUDE_PATH.

In the Cortex source directory, next to the SConstruct file, there should be a file called config.log. The contents of that file will have more details about why Boost isn't being found. It's possible that it's as simple as us using a compiler flag that isn't supported by the old compiler (the checks work by trying to compile a small file with #include <boost/blahdyblah.hpp in it).

[johnhaddon]

I should note though that our appetite for masochism extends only as far as supporting Apple silicon - we don't intend to support Intel builds as well.

I think the true masochism starts when we get into code signing and notarisation and all that jazz.