Symbol not found: _aligned_alloc on MacOS 10.9.5

[jonoomph]

A user reported this to me on MacOS 10.9.5 running our latest Mac builds. When launching OpenShot, an error during loading PyQt5.QtCore.so, looking for a missing symbol: Symbol not found: _aligned_alloc. It is expecting that symbol in /usr/lib/System.B.dylib.

We are currently building with GCC 8.4 on MacOS 10.15 (Catalina), and using the MacOS 10.9 SDK while building. Hopefully this can be solved with some compiler flags, or something simple. Our current CMake command used on our Mac builder:

cmake -DCMAKE_CXX_FLAGS=-I\ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -D"CMAKE_INSTALL_PREFIX:PATH=$CI_PROJECT_DIR/build/install-x64" -DCMAKE_CXX_COMPILER=/usr/local/opt/gcc@8/bin/g++-8 -DCMAKE_C_COMPILER=/usr/local/opt/gcc@8/bin/gcc-8 -DCMAKE_PREFIX_PATH=/usr/local/qt5/5.5/clang_64 -DPYTHON_INCLUDE_DIR=/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m -DPYTHON_LIBRARY=/Library/Frameworks/Python.framework/Versions/3.6/lib/libpython3.6.dylib -DPYTHON_MODULE_PATH=python -DPython_FRAMEWORKS=/Library/Frameworks/Python.framework/ -D"CMAKE_BUILD_TYPE:STRING=Release" -D"CMAKE_OSX_SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk" -D"CMAKE_OSX_DEPLOYMENT_TARGET=10.9" -D"CMAKE_INSTALL_RPATH_USE_LINK_PATH=1" -D"ENABLE_RUBY=0" ../

[jonoomph]

@ferdnyc I would love your thoughts on this, if you have any. This just feels like a "CMake" or "GCC flag" type fix... your specialty! haha.

[jonoomph]

I can also verify this error happens on MacOS 10.13 (High Sierra). But interestingly, the error message includes a hint: libstdc++.6.dylib (which was built for Mac OS X 10.15):

ImportError: dlopen(/Applications/OpenShot Video Editor.app/Contents/MacOS/lib/python3.6/PyQt5.QtCore.so, 2): Symbol not found: _aligned_alloc
  Referenced from: /Applications/OpenShot Video Editor.app/Contents/MacOS/libstdc++.6.dylib (which was built for Mac OS X 10.15)
  Expected in: /usr/lib/libSystem.B.dylib
 in /Applications/OpenShot Video Editor.app/Contents/MacOS/libstdc++.6.dylib

[ferdnyc]

Hmm. First thought: Support for Python 3.6 was officially added in cx_Freeze 5.0.1. cx_Freeze 5.0 might just be too old.

[ferdnyc]

What happens if you take cx_Freeze out of the mix and just try to run OpenShot from the source tree? That is, from a Mac Terminal, do a

cd /path/to/the/cloned/openshot-qt/src
PYTHONPATH=/path/to/libopenshot/build_dir/src/bindings/python/ python3 launch.py

?

[ferdnyc]

Hmm. It looks like neither PyQt5.QtCore.so nor anything else should be linked against libstdc++, which was deprecated in XCode 10 and has been on Apple's "avoid" pile for longer than that. They want people to migrate to their libc++, and have apparently been forcing the issue more of late.

How/when was PyQt5 built for this system? If it wasn't compiled on the MacOS release currently installed, it may need to be rebuilt. Finding a way to work -stdlib=libc++ (or -DCMAKE_CXX_FLAGS="-stdlib=libc++") into its build configs might also help.

[ferdnyc]

Also, regarding aligned_alloc specifically, that's apparently a not-uncommon source of pain for people building code on MacOS. But just doing a sed -e 's/aligned_alloc/malloc/g' is apparently effective at working around its availability issues, at least sometimes.

[jonoomph]

I think the PyQt and Qt releases are older and almost certainly were compiled on an ancient toolchain. Perhaps that is a great place to start, updating Python, cx_freeze, Qt, Sip, and PyQt. Of course, this will break our Mac builds again for a few days while I sort through the chaos, but it makes since that I need to update the dependencies.

[jonoomph]

@ferdnyc Okay, this is actually a nightmare situation, similar to our older Linux builder. Qt 5.5 is the last official release to contain qtwebkit. Of course, I was using the prebuild MacOS version from qt.io (brew versions had issues with packaging and deployment), and manually compiling Sip and PyQt. The oldest official MacOS build I can get is Qt 5.9.9, which of course, has long since removed QtWebKit. I'm having trouble finding source code for any of these old versions, from qt.io or riverbank. So...

Long story short, I think it's finally time for QtWebEngine!!!! This is too painful...

[ferdnyc]

@jonoomph Nahh, in some ways QtWebKit is better supported now than when it was part of Qt. It's just an external project, which lives here: https://github.com/qtwebkit/qtwebkit

[ferdnyc]

You may even be able to use their binary package: qtwebkit-MacOS-MacOS_10_13-Clang-MacOS-MacOS_10_13-X86_64.7z

Binary packages should be unpacked inside installation of Qt 5.14.1 which you can obtain at https://www.qt.io/download-qt-installer.

...If that doesn't work, AIUI building it is fairly easy if you have a working Qt install, it's just a matter of unpack source, qmake, make, make install and it becomes part of Qt. Then you need to rebuild PyQt5 on top of that Qt, to pick up the QtWebKit and QtWebKitWidgets modules..

[jonoomph]

FYI: I have a QtWebEngine version of OpenShot running locally now! However, I'm converting all the synchronous calls to async, with callback methods. Might take a few hours to fully flush through it all. But so far, seems to be working great.

[jeffski]

FYI: I have a QtWebEngine version of OpenShot running locally now! However, I'm converting all the synchronous calls to async, with callback methods. Might take a few hours to fully flush through it all. But so far, seems to be working great.

@jonoomph, does this mean you are rebuilding the entire interface using HTML?

[ferdnyc]

@jeffski No, just the timeline, which is already in HTML. (He's switching the backend from QtWebKit to QtWebEngine.)

[ferdnyc]

@jonoomph

You know this means we're going to HAVE to upgrade the Linux builder too, now, right? QtWebEngine was pretty much broken in Qt 5.2. (It was broken until Qt 5.6, from what I hear.)

...Not that we shouldn't upgrade the Linux builder regardless, to move it off an EOL'd Ubuntu release, of course.

[jonoomph]

@ferdnyc Yup, this is probably also the time to drop support for some older Linux distros with old versions of glib. I'm open to recommendations on what version of Ubuntu (but I would prefer we keep it on Ubuntu, since I most experienced with it). Not sure what distro AppImage recommends these days... I guess I need to do some research.

I'm already so deep in this from completely rebuilding the Mac builder, I figure, bring it on. Let's rebuild all the builders!