Skip to content

General improvements to Emscripten build and ci #533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 26, 2025
Merged

General improvements to Emscripten build and ci #533

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
296af5c
General improvements to Emscripten llvm build
mcbarton Mar 18, 2025
04d8241
Add clang_build_tools=off to docs
mcbarton Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 17 additions & 113 deletions .github/workflows/emscripten.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,22 @@ jobs:
fail-fast: false
matrix:
include:
- name: ubu24-arm-gcc12-clang-repl-19-emscripten
- name: ubu24-arm-clang-repl-19-emscripten
os: ubuntu-24.04-arm
compiler: gcc-12
clang-runtime: '19'
cling: Off
llvm_enable_projects: "clang;lld"
llvm_targets_to_build: "WebAssembly"
emsdk_ver: "3.1.73"
- name: osx15-arm-clang-clang-repl-19-emscripten
- name: osx15-arm-clang-repl-19-emscripten
os: macos-15
compiler: clang
clang-runtime: '19'
cling: Off
llvm_enable_projects: "clang;lld"
llvm_targets_to_build: "WebAssembly"
emsdk_ver: "3.1.73"
- name: ubu24-x86-gcc12-clang-repl-19-emscripten
- name: ubu24-x86-clang-repl-19-emscripten
os: ubuntu-24.04
compiler: gcc-12
clang-runtime: '19'
cling: Off
llvm_enable_projects: "clang;lld"
Expand Down Expand Up @@ -112,7 +109,7 @@ jobs:
path: |
llvm-project
${{ matrix.cling=='On' && 'cling' || '' }}
key: ${{ env.CLING_HASH }}-${{ runner.os }}-${{ matrix.os }}-${{ matrix.compiler }}-clang-${{ matrix.clang-runtime }}.x-emscripten
key: ${{ env.CLING_HASH }}-${{ runner.os }}-${{ matrix.os }}-clang-${{ matrix.clang-runtime }}.x-emscripten
lookup-only: true

- name: Setup emsdk
Expand All @@ -127,12 +124,6 @@ jobs:
run: |
echo "BUILD_TYPE=Release" >> $GITHUB_ENV
echo "CODE_COVERAGE=0" >> $GITHUB_ENV
os="${{ matrix.os }}"
if [[ "${os}" == "macos"* ]]; then
echo "ncpus=$(sysctl -n hw.ncpu)" >> $GITHUB_ENV
else
echo "ncpus=$(nproc --all)" >> $GITHUB_ENV
fi

- name: Setup default Build Type on Windows
if: ${{ runner.os == 'windows' && steps.cache.outputs.cache-hit != 'true' }}
Expand All @@ -142,92 +133,6 @@ jobs:
$env:ncpus=$([Environment]::ProcessorCount)
echo "ncpus=$env:ncpus" >> $env:GITHUB_ENV

- name: Setup compiler on Linux
if: ${{ runner.os == 'Linux' && steps.cache.outputs.cache-hit != 'true' }}
run: |
# https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
vers="${compiler#*-}"
os_codename="`cat /etc/os-release | grep UBUNTU_CODENAME | cut -d = -f 2`"
##sudo apt update
if [[ "${{ matrix.compiler }}" == *"gcc"* ]]; then
sudo apt install -y gcc-${vers} g++-${vers} lld
echo "CC=gcc-${vers}" >> $GITHUB_ENV
echo "CXX=g++-${vers}" >> $GITHUB_ENV
else
if ! sudo apt install -y clang-${vers}; then
curl https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
echo "deb https://apt.llvm.org/${os_codename}/ llvm-toolchain-${os_codename}-${vers} main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install -y clang-${vers}
fi
echo "CC=clang-${vers}" >> $GITHUB_ENV
echo "CXX=clang++-${vers}" >> $GITHUB_ENV
fi
env:
compiler: ${{ matrix.compiler }}

- name: Setup compiler on macOS
if: ${{ runner.os == 'macOS' && steps.cache.outputs.cache-hit != 'true' }}
run: |
vers="${compiler#*-}"
if [[ "${{ matrix.compiler }}" == *"gcc"* ]]; then
brew install "gcc@$vers"
echo "CC=gcc-${vers}" >> $GITHUB_ENV
echo "CXX=g++-${vers}" >> $GITHUB_ENV
else
export ARCHITECHURE=$(uname -m)
if [[ "$ARCHITECHURE" == "arm64" ]]; then
cling_on=$(echo "${{ matrix.cling }}" | tr '[:lower:]' '[:upper:]')
if [[ "${cling_on}" == "ON" ]]; then
brew install llvm@15
brew remove llvm@18
brew cleanup
#FIXME: Do not believe setting all these environment variables are necessary
# They were set to avoid using Xcodes libc++ and to stop CppInterOp using llvm@18 in tests
echo 'LDFLAGS="-L/opt/homebrew/opt/llvm@15/lib/ -L/opt/homebrew/opt/llvm@15/c++/"' >> $GITHUB_ENV
echo 'CPPFLAGS="-I/opt/homebrew/opt/llvm@15/include"' >> $GITHUB_ENV
echo 'CPATH="/opt/homebrew/include/"' >> $GITHUB_ENV
echo 'LIBRARY_PATH="/opt/homebrew/lib/"' >> $GITHUB_ENV
echo "CC=$(brew --prefix llvm@15)/bin/clang" >> $GITHUB_ENV
echo "CXX=$(brew --prefix llvm@15)/bin/clang++" >> $GITHUB_ENV
else
echo "CC=$(brew --prefix llvm@18)/bin/clang" >> $GITHUB_ENV
echo "CXX=$(brew --prefix llvm@18)/bin/clang++" >> $GITHUB_ENV
fi
else
echo "CC=$(brew --prefix llvm@15)/bin/clang" >> $GITHUB_ENV
echo "CXX=$(brew --prefix llvm@15)/bin/clang++" >> $GITHUB_ENV
fi
fi
echo "SDKROOT=$(xcrun --sdk macosx --show-sdk-path)" >> $GITHUB_ENV
env:
compiler: ${{ matrix.compiler }}

#Section slightly modified version of
#https://github.com/vgvassilev/clad/blob/40d8bec11bde47b14a281078183a4f6147abeac5/.github/workflows/ci.yml#L510C1-L534C10
- name: Setup compiler on Windows
if: ${{ runner.os == 'windows' && steps.cache.outputs.cache-hit != 'true' }}
run: |
if ( "${{ matrix.compiler }}" -imatch "clang" )
{
$ver="${{ matrix.compiler }}".split("-")[1]
choco install llvm --version=$ver --no-progress -my
clang --version
#
$env:CC="clang"
$env:CXX="clang++"
echo "CC=clang" >> $env:GITHUB_ENV
echo "CXX=clang++" >> $env:GITHUB_ENV
}
elseif ( "${{ matrix.compiler }}" -imatch "msvc" )
{
# MSVC is builtin in container image
}
else
{
echo "Unsupported compiler - fix YAML file"
}

- name: Install deps on Windows
if: ${{ runner.os == 'windows' && steps.cache.outputs.cache-hit != 'true' }}
run: |
Expand Down Expand Up @@ -299,8 +204,11 @@ jobs:
-DLLVM_INCLUDE_TESTS=OFF \
-DLLVM_ENABLE_THREADS=OFF \
-G Ninja \
-DLLVM_BUILD_TOOLS=OFF \
-DLLVM_ENABLE_LIBPFM=OFF \
-DCLANG_BUILD_TOOLS=OFF \
../llvm
emmake ninja clang cling lld gtest_main -j ${{ env.ncpus }}
emmake ninja clang cling lld gtest_main
else
# Apply patches
llvm_vers=$(echo "${{ matrix.clang-runtime }}" | tr '[:lower:]' '[:upper:]')
Expand All @@ -325,10 +233,14 @@ jobs:
-DLLVM_INCLUDE_EXAMPLES=OFF \
-DLLVM_INCLUDE_TESTS=OFF \
-DLLVM_ENABLE_THREADS=OFF \
-DLLVM_BUILD_TOOLS=OFF \
-DLLVM_ENABLE_LIBPFM=OFF \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you educate me on what this last flag does and why do we need it ?

Copy link
Collaborator Author

@mcbarton mcbarton Mar 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anutosh491 going by the LLVM documentation

LLVM_ENABLE_LIBPFM:BOOL

    Enable building with libpfm to support hardware counter measurements in LLVM tools. Defaults to ON.

I got the idea for the flag from emsdks build of llvm here https://github.com/emscripten-core/emsdk/blob/b665079cb9ad9eb1704652f962281a7fa1633e2d/emsdk.py#L1124C42-L1124C66 . I thought if its good enough for the developers of Emscripten to think it can be off, then its good enough for CppInterOp.

It also makes sense that if we turn off building the tools, then we should turn off anything which builds stuff to support the tools.

-DCLANG_BUILD_TOOLS=OFF \
-G Ninja \
../llvm
emmake ninja libclang clangInterpreter clangStaticAnalyzerCore lldWasm -j ${{ env.ncpus }}
emmake ninja libclang clangInterpreter clangStaticAnalyzerCore lldWasm
fi
rm -rf ./NATIVE/
cd ../
rm -rf $(find . -maxdepth 1 ! -name "build" ! -name "llvm" ! -name "clang" ! -name ".")
if [[ "${cling_on}" == "ON" ]]; then
Expand Down Expand Up @@ -457,23 +369,20 @@ jobs:
fail-fast: false
matrix:
include:
- name: ubu24-x86-gcc12-clang-repl-19-emscripten_wasm
- name: ubu24-x86-clang-repl-19-emscripten_wasm
os: ubuntu-24.04
compiler: gcc-12
clang-runtime: '19'
cling: Off
micromamba_shell_init: bash
emsdk_ver: "3.1.73"
- name: osx15-arm-clang-clang-repl-19-emscripten_wasm
- name: osx15-arm-clang-repl-19-emscripten_wasm
os: macos-15
compiler: clang
clang-runtime: '19'
cling: Off
micromamba_shell_init: bash
emsdk_ver: "3.1.73"
- name: ubu24-arm-gcc12-clang-repl-19-emscripten_wasm
- name: ubu24-arm-clang-repl-19-emscripten_wasm
os: ubuntu-24.04-arm
compiler: gcc-12
clang-runtime: '19'
cling: Off
micromamba_shell_init: bash
Expand All @@ -484,11 +393,6 @@ jobs:
with:
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Save PR Info on Unix systems
if: ${{ runner.os != 'windows' }}
run: |
Expand Down Expand Up @@ -542,7 +446,7 @@ jobs:
path: |
llvm-project
${{ matrix.cling=='On' && 'cling' || '' }}
key: ${{ env.CLING_HASH }}-${{ runner.os }}-${{ matrix.os }}-${{ matrix.compiler }}-clang-${{ matrix.clang-runtime }}.x-emscripten
key: ${{ env.CLING_HASH }}-${{ runner.os }}-${{ matrix.os }}-clang-${{ matrix.clang-runtime }}.x-emscripten

- name: Emscripten build of CppInterOp on Unix systems
if: ${{ runner.os != 'windows' }}
Expand Down
2 changes: 2 additions & 0 deletions Emscripten-build-instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ emcmake cmake -DCMAKE_BUILD_TYPE=Release \
-DLLVM_INCLUDE_EXAMPLES=OFF \
-DLLVM_INCLUDE_TESTS=OFF \
-DLLVM_ENABLE_THREADS=OFF \
-DLLVM_BUILD_TOOLS=OFF \
-DLLVM_ENABLE_LIBPFM=OFF \
../llvm
emmake make libclang -j $(nproc --all)
emmake make clangInterpreter clangStaticAnalyzerCore -j $(nproc --all)
Expand Down
2 changes: 2 additions & 0 deletions docs/Emscripten-build-instructions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ executing the following
-DLLVM_INCLUDE_EXAMPLES=OFF \
-DLLVM_INCLUDE_TESTS=OFF \
-DLLVM_ENABLE_THREADS=OFF \
-DLLVM_BUILD_TOOLS=OFF \
-DLLVM_ENABLE_LIBPFM=OFF \
../llvm
emmake make libclang -j $(nproc --all)
emmake make clangInterpreter clangStaticAnalyzerCore -j $(nproc --all)
Expand Down
Loading