Compile Error!!!Urgent need for assistance

[woshiame]

Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/gn_op/build.ninja...
Building extension module gn_op...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF custom_gn.o.d -DTORCH_EXTENSION_NAME=gn_op -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -Ofast -funroll-all-loops -march=native -c /workspace/channels-last-groupnorm/src/gnnhwc/csrc/custom_gn.cpp -o custom_gn.o
[2/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output gn_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=gn_op -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -use_fast_math -extra-device-vectorization -extended-lambda -lineinfo -src-in-ptx -std=c++17 -c /workspace/channels-last-groupnorm/src/gnnhwc/csrc/gn_kernel.cu -o gn_kernel.cuda.o
FAILED: gn_kernel.cuda.o
/usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output gn_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=gn_op -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -use_fast_math -extra-device-vectorization -extended-lambda -lineinfo -src-in-ptx -std=c++17 -c /workspace/channels-last-groupnorm/src/gnnhwc/csrc/gn_kernel.cu -o gn_kernel.cuda.o
nvcc error : 'ptxas' died due to signal 11 (Invalid memory reference)
nvcc error : 'ptxas' core dumped
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 2105, in _run_ninja_build
subprocess.run(
File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/channels-last-groupnorm/src/gnnhwc/custom_gn.py", line 7, in
gn_op = load(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1312, in load
return _jit_compile(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1722, in _jit_compile
_write_ninja_file_and_build_library(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1834, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 2121, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'gn_op'

[latentCall145]

Hi, some questions:

• What's the output of nvidia-smi?
• What's the output of nvcc -V?

If you're running on Google Colab, I'd recommend restarting your runtime after installing ninja and my repo. One more thing, if your GPU is older than Turing/Volta (e.g. you're using K40, K80, P100, etc.), there is no point in using channels last groupnorm because they don't support float16.

[woshiame]

1.nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
2.nvidia-smi
Thu Nov 14 14:56:14 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A10 On | 00000000:61:00.0 Off | 0 |
| 0% 39C P0 56W / 150W | 4MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A10 On | 00000000:62:00.0 Off | 0 |
| 0% 35C P0 41W / 150W | 4MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA A10 On | 00000000:A1:00.0 Off | 0 |
| 0% 41C P0 59W / 150W | 4MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA A10 On | 00000000:A2:00.0 Off | 0 |
| 0% 41C P0 58W / 150W | 4MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

[woshiame]

In addition, I found that Nvidia's Apex library seems to have similar functionality, the link is as follows: https://github.com/NVIDIA/apex/blob/master/apex/contrib/group_norm/group_norm.py May I ask if there is any difference in efficiency between the two libraries

[latentCall145]

Unfortunately, I could not reproduce your error with python 3.10 and cuda 12.1. Maybe try installing pytorch with conda and then install cudatoolkit (conda install cudatoolkit -c nvidia; this is for nvcc) to see if there's any improvement.

As for apex, I haven't tried it out. I'd assume apex is decently faster since it has more kernels for specialized GN input shapes, but if the math is correct on how many loads and stores GN forward and backward needs, the margin should be closer to 30% faster and not 200% faster.