Frameworks SDK Module Build Scripts

We have build scripts for compiling the Frameworks SDK using system available modules and libraries.

Build Environment

The build environment assumes a user session on {aurora,sunspot}.alcf.anl.gov. Builds are performed in /tmp.

Prior to building, most build scripts will load common modules and environment variables using the setup_build_env routine in ci-lib.sh.

Manual Usage

If not running the CI pipeline, the Framework SDK build scripts can be configured via the following environment variables.

Name	Description	Example
FRAMEWORKS_ROOT_DIR	Directory to store built artifacts	/lus/flare/projects/datascience/frameworks-ci (Lustre allocation on Aurora)
FRAMEWORKS_PYTHON_VERSION	Python version to build against	3.12
FRAMEWORKS_TORCH_VERSION	PyTorch version to build (git ref)	v2.10.0
FRAMEWORKS_TRITON_VERSION	Triton-XPU version to build (git ref)	main
FRAMEWORKS_TORCHCCL_VERSION	torchCCL version to build (git ref)	master
FRAMEWORKS_IPEX_VERSION	IPEX version to build (git ref)	xpu-main
FRAMEWORKS_VLLM_VERSION	vLLM version to build (git ref)	main
FRAMEWORKS_VLLM_XPU_KERNELS_VERSION	vLLM XPU kernels version to build (git ref)	main
FRAMEWORKS_SDK_TESTS_VERSION	Frameworks SDK validation tests version (git ref)	main

Then, run any of the scripts in nightly_wheels. The resultant builds/logs will then be in $FRAMEWORKS_ROOT_DIR/$(whoami).

Adding a Build Script

Build scripts are generally structured similarly.

#!/usr/bin/env bash

source "$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../ci-lib.sh"

# 1) Pull source and gen build environment
gen_build_dir_with_git 'https://github.com/<foo>/<bar>'
setup_build_env

setup_uv_venv # pass needed dependencies (i.e. from prior builds) here

# 2) Set <library> configuration
CC="$(which gcc)"
export CC
CXX="$(which g++)"
export CXX

# 3) Build & Archive
build_bdist_wheel
artifact_out "<bar>-*.whl"

Testing

The test stage of the CI pipeline runs a bats harness (see tests/). Components are tested against an ephemeral uv venv of the wheels built by the pipeline.

Additionally, the frameworks-sdk-tests validation suite is cloned at test time and its default smoke suite runs against the built wheels. Its summary.json results are converted to JUnit XML (tools/frameworks-sdk-tests-junit.py) and ingested by GitLab CI, so individual validation test results show up in the pipeline test report. On aurora.alcf.anl.gov, it can be run manually via:

./tests/bats/bin/bats tests/frameworks-sdk-tests.bats

Wheels

We have scripts to build the following wheels:

• pytorch/
  • pytorch
  • ao¹
  • torchtune¹
• intel/
  • intel-extension-for-pytorch
  • torch-ccl
  • intel-xpu-backend-for-triton
• vllm-project/
  • vllm
  • vllm-xpu-kernels
• mpi4py/mpi4py
• h5py/h5py¹

Build Time(s)

	took (hours)
torch	~ 2:00
ipex	~ 1:00
others2	< 0:30
total	~ 4:00

IPEX Build Bugs

• [❌ TAKE 1]

  # [2025-07-05 @ 23:20] hung (?) (@ 92% > ~ 2 hr)
  #    [ 92%] Linking CXX shared library libxetla_gemm.so]

• [❌ TAKE 2]

  # [2025-07-06 @ 10:30:24] hung (?) (@ 97% )
  #     [ 97%] Built target intel-ext-pt-gpu-op-TripleOps
  # [2025-07-06 @ 11:01] ...[waiting]...
  # [2025-07-06 @ 13:00] job ended :(

• [✅ TAKE 3]

  # [✅ TAKE 3]
  # [2025-07-06 @ 18:00] Successfully built IPEX
  # took: 1h:05m:36s

Footnotes

1. not ran by CI pipeline ↩ ↩² ↩³

2. Others:

   • h5py/h5py (broken ??)
   • intel/torch-ccl
   • mpi4py/mpi4py
   • pytorch/ao
   • pytorch/torchaudio
   • pytorch/torchdata
   • pytorch/torchtune
   • pytorch/torchvision
   ↩