TSan-instrumented build instructions

[ngoldbaum]

On a Mac, it's surprisingly tricky to build a nontrivial stack using TSan instrumentation. I'm putting notes here but probably a lot of this information should live in the guide?

In case anyone ever needs to, here's how you can manually build CPython, NumPy, and SciPy with thread sanitizer instrumentation on a Mac. I think the same thing should work for address sanitizer as well. I have llvm installed via homebrew. I'm using pyenv to build Python from source with the following build definition in a file named 3.14t-dev-tsan. You can build 3.15 with the same definition pointed at the main branch instead of the 3.14 branch.

export PYTHON_BUILD_FREE_THREADING=1
export PYTHON_BUILD_CONFIGURE_WITH_OPENSSL=1
export PYTHON_BUILD_TCLTK_USE_PKGCONFIG=1
export PYTHON_BUILD_CONFIGURE_WITH_DSYMUTIL=1
export CONFIGURE_OPTS="--with-thread-sanitizer"
export CC=/opt/homebrew/opt/llvm/bin/clang
export CXX=/opt/homebrew/opt/llvm/bin/clang++
install_package "openssl-3.3.0" "https://www.openssl.org/source/openssl-3.3.0.tar.gz#53e66b043322a606abf0087e7699a0e033a37fa13feb9742df35c3a33b18fb02" mac_openssl --if has_broken_mac_openssl
install_package "readline-8.0" "https://ftpmirror.gnu.org/readline/readline-8.0.tar.gz#e339f51971478d369f8a053a330a190781acb9864cf4c541060f12078948e461" mac_readline --if has_broken_mac_readline
install_git "Python-3.14-dev" "https://github.com/python/cpython" 3.14 standard copy_python_gdb ensurepip

You can run the definition with pyenv install 3.14t-dev-tsan in the same folder as the recipe. Next, in a clone of the NumPy repository with all submodules checked out:

pip install -r requirements/build_requirements.txt
LDFLAGS="-L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib/unwind -lunwind" CC=/opt/homebrew/opt/llvm/bin/clang CXX=/opt/homebrew/opt/llvm/bin/clang++ PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig" python -m pip install -vvv . -C'setup-args=-Db_sanitize=thread' -C'setup-args=-Dbuildtype=debugoptimized' --no-build-isolation

And once that build finishes, verify it works by doing import numpy outside of the numpy source tree.

And then you can build SciPy with the following command, in a clone of the SciPy repo with all submodules checked out:

pip install -r requirements/build.txt
LDFLAGS="-L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib/unwind -lunwind" CC=/opt/homebrew/opt/llvm/bin/clang CXX=/opt/homebrew/opt/llvm/bin/clang++ PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig" python -m pip install -vvv . -C'setup-args=-Dc_args=-fsanitize=thread' -C'setup-args=-Dcpp_args=-fsanitize=thread' -C'setup-args=-Dc_link_args=-fsanitize=thread' -C'setup-args=-Dcpp_link_args=-fsanitize=thread' -C'setup-args=-Dbuildtype=debugoptimized' --no-build-isolation

b_sanitize=thread doesn't work for SciPy because of mesonbuild/meson#15381.

[ngoldbaum]

For Pandas, I did:

pip install "versioneer[toml]"
LDFLAGS="-L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib/unwind -lunwind" CC=/opt/homebrew/opt/llvm/bin/clang CXX=/opt/homebrew/opt/llvm/bin/clang++ PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig" python -m pip install -vvv . -C'setup-args=-Db_sanitize=thread' -C'setup-args=-Dbuildtype=debugoptimized' --no-build-isolation

[rgommers]

Cleaner instructions are already in the guide? See https://py-free-threading.github.io/thread_sanitizer/

This seems too detailed and not suitable for inclusion - things like has_broken_mac_openssl are too specific and will change over time, so better in an issue like this.

[ngoldbaum]

Scikit-image depends on Pillow and markupsafe. I compiled both with:

CFLAGS=-fsanitize=thread LDFLAGS="-L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib/unwind -lunwind -fsanitize=thread" CC=/opt/homebrew/opt/llvm/bin/clang CXX=/opt/homebrew/opt/llvm/bin/clang++ PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig" python -m pip install -vvv . --no-build-isolation

And then build scikit-image using:

LDFLAGS="-L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib/unwind -lunwind" CC=/opt/homebrew/opt/llvm/bin/clang CXX=/opt/homebrew/opt/llvm/bin/clang++ PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig" spin build -v --clean -- -Db_sanitize=thread -Dbuildtype=debugoptimized

See scikit-image/scikit-image#8065 for context.