Impossible to compile the package #2383
Description
Hello, I am trying to install transformer-engine using pip install --no-build-isolation transformer_engine[pytorch] on two different systems but I always get compilation errors:
Personal system (Nvidia 3090, torch 2.6):
Building wheels for collected packages: transformer_engine_torch
Building wheel for transformer_engine_torch (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for transformer_engine_torch (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [226 lines of output]
/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/dist.py:289: UserWarning: Unknown distribution option: 'tests_require'
warnings.warn(msg)
running bdist_wheel
Guessing wheel URL: https://github.com/NVIDIA/TransformerEngine/releases/download/v2.9.0/transformer_engine_torch-2.9.0+cu12torch2.6cxx11abiFALSE-cp313-cp313-linux_x86_64.whl
Precompiled wheel not found. Building from source...
running build
running build_ext
building 'transformer_engine_torch' extension
creating /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc
creating /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions
creating /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/multi_tensor
Emitting ninja build file /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/28] c++ -MMD -MF /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/common.o.d -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvcc/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cublas/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cudnn/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_cccl/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/nvtx/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvrtc/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/torch/csrc/api/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/TH -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/THC -I/home/matteo/envs/torch-cuda/include -I/usr/include/python3.13 -c -c /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/common.cpp -o /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/common.o -O3 -fvisibility=hidden -g0 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=transformer_engine_torch -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
FAILED: /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/common.o
c++ -MMD -MF /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/common.o.d -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvcc/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cublas/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cudnn/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_cccl/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/nvtx/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvrtc/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/torch/csrc/api/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/TH -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/THC -I/home/matteo/envs/torch-cuda/include -I/usr/include/python3.13 -c -c /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/common.cpp -o /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/common.o -O3 -fvisibility=hidden -g0 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=transformer_engine_torch -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
In file included from /home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/ATen/cuda/CUDAContextLight.h:6,
from /home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/ATen/cuda/CUDAContext.h:3,
from /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/common.h:12,
from /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/common.cpp:7:
/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include/cuda_runtime_api.h:148:10: fatal error: crt/host_defines.h: File o directory non esistente
148 | #include "crt/host_defines.h"
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
[2/28] c++ -MMD -MF /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/apply_rope.o.d -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvcc/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cublas/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cudnn/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_cccl/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/nvtx/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvrtc/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/torch/csrc/api/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/TH -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/THC -I/home/matteo/envs/torch-cuda/include -I/usr/include/python3.13 -c -c /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/apply_rope.cpp -o /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/apply_rope.o -O3 -fvisibility=hidden -g0 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=transformer_engine_torch -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
FAILED: /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/apply_rope.o
c++ -MMD -MF /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/apply_rope.o.d -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvcc/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cublas/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cudnn/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_cccl/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/nvtx/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvrtc/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/torch/csrc/api/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/TH -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/THC -I/home/matteo/envs/torch-cuda/include -I/usr/include/python3.13 -c -c /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/apply_rope.cpp -o /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/apply_rope.o -O3 -fvisibility=hidden -g0 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=transformer_engine_torch -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
In file included from /home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/ATen/cuda/CUDAContextLight.h:6,
from /home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/ATen/cuda/CUDAContext.h:3,
from /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/../common.h:12,
from /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/../extensions.h:12,
from /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/apply_rope.cpp:7:
/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include/cuda_runtime_api.h:148:10: fatal error: crt/host_defines.h: File o directory non esistente
148 | #include "crt/host_defines.h"
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
[3/28] c++ -MMD -MF /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/activation.o.d -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvcc/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cublas/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cudnn/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_cccl/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/nvtx/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvrtc/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/torch/csrc/api/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/TH -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/THC -I/home/matteo/envs/torch-cuda/include -I/usr/include/python3.13 -c -c /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/activation.cpp -o /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/activation.o -O3 -fvisibility=hidden -g0 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=transformer_engine_torch -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
FAILED: /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/activation.o
c++ -MMD -MF /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/activation.o.d -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvcc/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cublas/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cudnn/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_cccl/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/nvtx/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_nvrtc/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/common_headers/common/include -I/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/torch/csrc/api/include -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/TH -I/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/THC -I/home/matteo/envs/torch-cuda/include -I/usr/include/python3.13 -c -c /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/activation.cpp -o /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build/temp.linux-x86_64-cpython-313/csrc/extensions/activation.o -O3 -fvisibility=hidden -g0 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=transformer_engine_torch -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
In file included from /home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/ATen/cuda/CUDAContextLight.h:6,
from /home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/include/ATen/cuda/CUDAContext.h:3,
from /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/../common.h:12,
from /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/../extensions.h:12,
from /tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/csrc/extensions/activation.cpp:6:
/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/nvidia/cuda_runtime/include/cuda_runtime_api.h:148:10: fatal error: crt/host_defines.h: File o directory non esistente
148 | #include "crt/host_defines.h"
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "<string>", line 118, in run
File "/usr/lib/python3.13/urllib/request.py", line 214, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
~~~~~~~^^^^^^^^^^^
File "/usr/lib/python3.13/urllib/request.py", line 189, in urlopen
return opener.open(url, data, timeout)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/urllib/request.py", line 495, in open
response = meth(req, response)
File "/usr/lib/python3.13/urllib/request.py", line 604, in http_response
response = self.parent.error(
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.13/urllib/request.py", line 533, in error
return self._call_chain(*args)
~~~~~~~~~~~~~~~~^^^^^^^
File "/usr/lib/python3.13/urllib/request.py", line 466, in _call_chain
result = func(*args)
File "/usr/lib/python3.13/urllib/request.py", line 613, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/utils/cpp_extension.py", line 2209, in _run_ninja_build
subprocess.run(
~~~~~~~~~~~~~~^
command,
^^^^^^^^
...<3 lines>...
check=True,
^^^^^^^^^^^
env=env)
^^^^^^^^
File "/usr/lib/python3.13/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
main()
~~~~^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
return _build_backend().build_wheel(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
wheel_directory, config_settings, metadata_directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/build_meta.py", line 435, in build_wheel
return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/build_meta.py", line 423, in _build
return self._build_with_temp_dir(
~~~~~~~~~~~~~~~~~~~~~~~~~^
cmd,
^^^^
...<3 lines>...
self._arbitrary_args(config_settings),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/build_meta.py", line 404, in _build_with_temp_dir
self.run_setup()
~~~~~~~~~~~~~~^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/build_meta.py", line 512, in run_setup
super().run_setup(setup_script=setup_script)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/build_meta.py", line 317, in run_setup
exec(code, locals())
~~~~^^^^^^^^^^^^^^^^
File "<string>", line 159, in <module>
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 186, in setup
return run_commands(dist)
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
dist.run_commands()
~~~~~~~~~~~~~~~~~^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
self.run_command(cmd)
~~~~~~~~~~~~~~~~^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
~~~~~~~~~~~^^
File "<string>", line 135, in run
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
self.run_command("build")
~~~~~~~~~~~~~~~~^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
self.distribution.run_command(command)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
~~~~~~~~~~~^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
self.distribution.run_command(command)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
~~~~~~~~~~~^^
File "/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build_tools/build_ext.py", line 131, in run
super().run()
~~~~~~~~~~~^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/command/build_ext.py", line 99, in run
_build_ext.run(self)
~~~~~~~~~~~~~~^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
self.build_extensions()
~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/pip-install-wt_vvn5q/transformer-engine-torch_c18f5f484dfb4d589bf55a29582498d4/build_tools/build_ext.py", line 208, in build_extensions
super().build_extensions()
~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/utils/cpp_extension.py", line 900, in build_extensions
build_ext.build_extensions(self)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
self._build_extensions_serial()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
self.build_extension(ext)
~~~~~~~~~~~~~~~~~~~~^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
_build_ext.build_extension(self, ext)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/Cython/Distutils/build_ext.py", line 136, in build_extension
super().build_extension(ext)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 565, in build_extension
objects = self.compiler.compile(
sources,
...<5 lines>...
depends=ext.depends,
)
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/utils/cpp_extension.py", line 713, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
sources=sources,
^^^^^^^^^^^^^^^^
...<7 lines>...
verbose=True,
^^^^^^^^^^^^^
with_cuda=with_cuda)
^^^^^^^^^^^^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/utils/cpp_extension.py", line 1869, in _write_ninja_file_and_compile_objects
_run_ninja_build(
~~~~~~~~~~~~~~~~^
build_directory,
^^^^^^^^^^^^^^^^
...<2 lines>...
# that failed to build but there isn't a good way to get it here.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error_prefix='Error compiling objects for extension')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/matteo/envs/torch-cuda/lib/python3.13/site-packages/torch/utils/cpp_extension.py", line 2225, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for transformer_engine_torch
Failed to build transformer_engine_torch
[notice] A new release of pip is available: 25.2 -> 25.3
[notice] To update, run: pip install --upgrade pip
error: failed-wheel-build-for-install
× Failed to build installable wheels for some pyproject.toml based projects
╰─> transformer_engine_torch
University's server (4 Nvidia A40): trying with
CUDNN_PATH=/home/matteo/.local/lib/python3.10/site-packages/nvidia/cudnn/include
CPLUS_INCLUDE_PATH=/home/matteo/.local/lib/python3.10/site-packages/nvidia/cudnn/include
but in both cases:
File "/home/matteo/miniconda3/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 96, in run
_build_ext.run(self)
File "/home/matteo/miniconda3/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
self.build_extensions()
File "/tmp/pip-install-v0d6dw0z/transformer-engine-torch_c11d10be70a74e29a480b47d50cf42b5/build_tools/build_ext.py", line 208, in build_extensions
super().build_extensions()
File "/home/matteo/miniconda3/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 900, in build_extensions
build_ext.build_extensions(self)
File "/home/matteo/miniconda3/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
self._build_extensions_serial()
File "/home/matteo/miniconda3/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
self.build_extension(ext)
File "/home/matteo/miniconda3/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 261, in build_extension
_build_ext.build_extension(self, ext)
File "/home/matteo/miniconda3/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 565, in build_extension
objects = self.compiler.compile(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/matteo/miniconda3/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 713, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/home/matteo/miniconda3/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1869, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/home/matteo/miniconda3/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2225, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for transformer_engine_torch
Failed to build transformer_engine_torch
ERROR: Could not build wheels for transformer_engine_torch, which is required to install pyproject.toml-based projects
It looks it doesn't find cudnn.h even though I am pointing it to the right location
/home/matteo/miniconda3/lib/python3.12/site-packages/torch/include/ATen/cudnn/cudnn-wrapper.h:3:10: fatal error: cudnn.h: No such file o
r directory
3 | #include <cudnn.h>
| ^~~~~~~~~
compilation terminated.
Is it a bug or am I doing something wrong? Thanks.