Skip to content

Comments

feat: compute-sanitizer in build#1735

Merged
vgvassilev merged 2 commits intovgvassilev:masterfrom
fogsong233:add-cuda-check
Feb 23, 2026
Merged

feat: compute-sanitizer in build#1735
vgvassilev merged 2 commits intovgvassilev:masterfrom
fogsong233:add-cuda-check

Conversation

@fogsong233
Copy link
Contributor

Also replace a deprecated in lit.cfg with python std.

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@fogsong233
Copy link
Contributor Author

fogsong233 commented Feb 20, 2026

https://github.com/vgvassilev/clad/actions/runs/22232602574/job/64316060988?pr=1735
a new mem problem is found by sanitizer, should it be fixed in this PR, or another new?
Re-run in local, the output is here

> compute-sanitizer ./build/test/CUDA/GradientKernels.out
========= COMPUTE-SANITIZER
10
10
Use execute() for non-global CUDA kernels
Use execute_kernel() for global CUDA kernels
5, 5, 5, 5, 5
5, 5, 5, 5, 5
5, 5, 5, 5, 5
5, 5, 5, 5, 5
5, 5, 5, 5, 5
25, 0, 0, 0, 0
50.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00
50.00
50.00
no error
50.00
5.00, 5.00, 5.00
5.00, 5.00, 5.00
50.00
50.00
50.00
5.00, 5.00, 5.00
Writing result of out[0]
Writing result of out[1]
Writing result of out[2]
Writing result of out[3]
Writing result of out[4]
Writing result of out[5]
Writing result of out[6]
Writing result of out[7]
Writing result of out[8]
Writing result of out[9]
60.00, 0.00, 0.00
5, 5, 5
20, 20, 20, 20, 20
20, 25, 45, 15, 15
4
========= Invalid __global__ read of size 4 bytes
=========     at [clone injective_reassignment_grad(int *, void *)] injective_reassignment_grad(int *, int *)+0xe40
=========     by thread (0,0,0) in block (0,0,0)
=========     Address 0x702000a04 is out of bounds
=========     and is 1 bytes after the nearest allocation at 0x702000a00 of size 4 bytes
=========     Device Frame:injective_reassignment_grad(int *, void *)+0x670
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:cuLaunchKernel [0x2e7380]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame: [0x156e4]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:cudaLaunchKernel [0x74631]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:_ZN4clad25execute_with_default_argsILb1EJEPFvPiPvEJRS1_S5_EJS1_S2_ETnNSt9enable_ifIXT_EbE4typeELb1EEENS_15function_traitsIT1_E11return_typeENS_4listIJDpT0_EEESA_NSD_IJDpT3_EEEb4dim3SK_mP11CUstream_stDpOT2_ [0x9c6a]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:void clad::CladFunction<void (*)(int*, void*), clad::NoObject, true, false>::execute_helper<void (*)(int*, void*), int*&, int*&>(void (*)(int*, void*), bool, dim3, dim3, int*&, int*&) const [0x9ab3]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:_ZN4clad12CladFunctionIPFvPiPvENS_8NoObjectELb1ELb0EE14execute_kernelIJRS1_S8_ES4_EENSt9enable_ifIXntsr3std7is_sameIT0_PNS_10NoFunctionEEE5valueEvE4typeE4dim3SF_DpOT_ [0x8465]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:main [0x5600]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaMemcpy.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaMemcpy [0x6f330]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x562e]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
-1078510768
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaMemcpy.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaMemcpy [0x6f330]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x575d]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaMemcpy.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaMemcpy [0x6f330]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x577a]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaLaunchKernel.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaLaunchKernel [0x7466e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:_ZN4clad25execute_with_default_argsILb1EJEPFvPiPvEJRS1_S5_EJS1_S2_ETnNSt9enable_ifIXT_EbE4typeELb1EEENS_15function_traitsIT1_E11return_typeENS_4listIJDpT0_EEESA_NSD_IJDpT3_EEEb4dim3SK_mP11CUstream_stDpOT2_ [0x9c6a]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:void clad::CladFunction<void (*)(int*, void*), clad::NoObject, true, false>::execute_helper<void (*)(int*, void*), int*&, int*&>(void (*)(int*, void*), bool, dim3, dim3, int*&, int*&) const [0x9ab3]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:_ZN4clad12CladFunctionIPFvPiPvENS_8NoObjectELb1ELb0EE14execute_kernelIJRS1_S8_ES4_EENSt9enable_ifIXntsr3std7is_sameIT0_PNS_10NoFunctionEEE5valueEvE4typeE4dim3SF_DpOT_ [0x8465]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:main [0x583f]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaMemcpy.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaMemcpy [0x6f330]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x586d]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
415744953
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x58f7]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x5903]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x590f]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x591b]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x5927]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x5933]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x5993]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x599f]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x59ab]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x59b7]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= Program hit cudaErrorUnknown (error 999) due to "unknown error" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x555843]
=========                in /usr/lib/wsl/drivers/nvltsi.inf_amd64_acf18fc358278bf9/libcuda.so.1.1
=========     Host Frame:cudaFree [0x56a4e]
=========                in /usr/local/cuda-12.3/lib64/libcudart.so.12
=========     Host Frame:main [0x59c3]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
=========     Host Frame:__libc_start_call_main in ../sysdeps/nptl/libc_start_call_main.h:74 [0x2a1ca]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:__libc_start_main in ../csu/libc-start.c:347 [0x2a28b]
=========                in /lib/x86_64-linux-gnu/libc.so.6
=========     Host Frame:_start [0x1235]
=========                in /home/kacent/project/clad/build/test/CUDA/GradientKernels.out
========= 
========= ERROR SUMMARY: 17 errors

@codecov
Copy link

codecov bot commented Feb 21, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Copy link
Owner

@vgvassilev vgvassilev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Lgtm!

@vgvassilev vgvassilev merged commit 1fe0125 into vgvassilev:master Feb 23, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants