Cross compile CUDA for Jetson Orin

[xuyanwen2012]

Hi,

最近在Jetson Orin Nano (aarch64)上写CUDA (12.2)，xmake非常好用。但是Jetson编译是在是太慢了，所以我希望在x86的Ubuntu上用Xmake Cross compile CUDA。

• 纯C++的program我已经成功用xmake cross compile在aarch64上了
• 我已经查阅Nvidia的资料并且成功的按照以下步骤手动cross compile了，但是我希望把我的操作使用xmake里面（项目是xmake写的），尝试了几次都没搞定。

首先我有aarch64-linux-gnu, 然后我也有Nvidia提供的 /usr/local/cuda/12-2.2/targets/aarch64-linux

如果我手动cross compile就是

# step 1: Cross compile ".cu" files
/usr/local/cuda-12.2/bin/nvcc -c -Xcompiler "-fPIC" src/foo.cu -ccbin=aarch64-linux-gnu-g++ -gencode arch=compute_87,code=sm_87 -o foo.o

# Step 2: make static lib 
aarch64-linux-gnu-ar rcs libfoo.a foo.o

# Step 3: Cross compile ".cpp" files
aarch64-linux-gnu-g++ -o my_exe main.o -L. -lfoo

# Step 4: Link all together 
aarch64-linux-gnu-g++ -o my_exe main.o -L. -lfoo -L/usr/local/cuda-12.2/targets/aarch64-linux/lib -lcudart

以上成功在Jetson运行了，完全没问题。

我希望可以通过xmake直接把project cross compile了

我知道普通纯C++的项目cross compile要 xmake f -p cross --sdk=/usr/aarch64-linux-gnu --cxx=aarch64-linux-gnu-g++ -a arm64 --ld=aarch64-linux-gnu-g++这个我试了也没问题。

但是每次尝试cross compile cuda就有问题/以及不知道如何针对这个情况config xmake

xmake.lua就这样，

set_policy("build.cuda.devlink", true)
add_cuflags("-Xcompiler -fPIC", "-ccbin=aarch64-linux-gnu-g++")

target("foo_lib")
    set_kind("static")
    add_includedirs("include")
    add_headerfiles("include/foo.h")
    add_files("src/foo.cu")
    add_cugencodes("compute_87, sm_87")

target("cuda-cross-jetson")
    set_kind("binary")
    add_includedirs("include")
    add_files("src/main.cpp")
    add_deps("foo_lib")

这是我的尝试

doremy@sweet ~/p/cuda-cross-jetson> xmake b -v foo_lib 
checking for gcc ... /usr/bin/gcc
checking for zig ... no
checking for zig ... no
checking for Cuda SDK directory ... /usr/local/cuda-12.2/
checking for /usr/local/cuda-12.2/bin/nvcc ... ok
checking for the cuda compiler (cu) ... nvcc
checking for /usr/local/cuda-12.2/bin/nvcc ... ok
checking for flags (-O3) ... ok
checking for flags (-Xcompiler -fPIC) ... ok
checking for flags (-ccbin=aarch64-linux-gnu-g++) ... ok
checking for flags (-rdc=true) ... ok
checking for flags (-gencode arch=compute_87,code=sm_87) ... ok
[ 25%]: compiling.release src/foo.cu
/usr/local/cuda-12.2/bin/nvcc -c -O3 -Iinclude -I/usr/local/cuda-12.2/include -Xcompiler -fPIC -ccbin=aarch64-linux-gnu-g++ -m64 -rdc=true -gencode arch=compute_87,code=sm_87 -o build/.objs/foo_lib/linux/aarch64/release/src/foo.cu.o src/foo.cu
checking for flags (-MMD -MF) ... ok
checking for /usr/local/cuda-12.2/bin/nvcc ... ok
checking for the cuda linker (culd) ... nvcc
checking for flags (-gencode arch=compute_87,code=sm_87) ... ok
[ 50%]: devlinking.release foo_lib_gpucode.cu.o
/usr/local/cuda-12.2/bin/nvcc -o build/.objs/foo_lib/linux/aarch64/release/rules/cuda/devlink/foo_lib_gpucode.cu.o build/.objs/foo_lib/linux/aarch64/release/src/foo.cu.o -L/usr/local/cuda-12.2/lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl -m64 -gencode arch=compute_87,code=sm_87 -dlink
[ 50%]: archiving.release libfoo_lib.a
aarch64-linux-gnu-ar -cr build/linux/aarch64/release/libfoo_lib.a build/.objs/foo_lib/linux/aarch64/release/src/foo.cu.o build/.objs/foo_lib/linux/aarch64/release/rules/cuda/devlink/foo_lib_gpucode.cu.o
[100%]: build ok, spent 3.898s

doremy@sweet ~/p/cuda-cross-jetson> xmake b -v cuda-cross-jetson 
checking for aarch64-linux-gnu-g++ ... /usr/bin/aarch64-linux-gnu-g++
checking for flags (-fPIC) ... ok
checking for flags (-fvisibility-inlines-hidden) ... ok
checking for flags (-O3) ... ok
checking for flags (-DNDEBUG) ... ok
[ 37%]: cache compiling.release src/main.cpp
aarch64-linux-gnu-g++ -c -fvisibility=hidden -fvisibility-inlines-hidden -O3 -Iinclude -DNDEBUG -o build/.objs/cuda-cross-jetson/linux/aarch64/release/src/main.cpp.o src/main.cpp
checking for flags (-MMD -MF) ... ok
checking for flags (-fdiagnostics-color=always) ... ok
checking for flags (-fPIC) ... ok
checking for flags (-Wl,-rpath=/usr/local/cuda-12.2/lib) ... ok
[ 75%]: linking.release cuda-cross-jetson
aarch64-linux-gnu-g++ -o build/linux/aarch64/release/cuda-cross-jetson build/.objs/cuda-cross-jetson/linux/aarch64/release/src/main.cpp.o -L/usr/local/cuda-12.2/lib -Lbuild/linux/aarch64/release -Wl,-rpath=/usr/local/cuda-12.2/lib -s -lfoo_lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: build/linux/aarch64/release/libfoo_lib.a(foo_lib_gpucode.cu.o): Relocations in generic ELF (EM: 62)
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: build/linux/aarch64/release/libfoo_lib.a(foo_lib_gpucode.cu.o): Relocations in generic ELF (EM: 62)
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: build/linux/aarch64/release/libfoo_lib.a(foo_lib_gpucode.cu.o): Relocations in generic ELF (EM: 62)
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: build/linux/aarch64/release/libfoo_lib.a: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
error: execv(aarch64-linux-gnu-g++ -o build/linux/aarch64/release/cuda-cross-jetson build/.objs/cuda-cross-jetson/linux/aarch64/release/src/main.cpp.o -L/usr/local/cuda-12.2/lib -Lbuild/linux/aarch64/release -Wl,-rpath=/usr/local/cuda-12.2/lib -s -lfoo_lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl) failed(1)

希望有人可以指导以下，万分感谢

[star-hengxing]

最后链接命令还是用的 host lib，说明 xmake 没支持 cuda cross compile，可以来个 pr 支持一下

临时方法可以在 target on_config 把 linkdirs 和 rpathdirs 去掉，改成 arm64 的路径

[xuyanwen2012]

{
    ld = aarch64-linux-gnu-g++
    host = linux
    ccache = true
    mode = release
    plat = linux
    arch = arm64
    kind = static
    ndk_stdcxx = true
    buildir = build
    cuda = /usr/local/cuda-12.2/
    cxx = aarch64-linux-gnu-g++
}

是这样写么

on_config("linux", function (target)
    target:set("linkdirs", "/usr/local/cuda-12.2/targets/aarch64-linux/lib")    
    target:set("rpathdirs", "/usr/local/cuda-12.2/targets/aarch64-linux/lib")    
end)

...
[ 50%]: archiving.release libfoo_lib.a
/usr/bin/ar -cr build/linux/arm64/release/libfoo_lib.a build/.objs/foo_lib/linux/arm64/release/src/foo.cu.o build/.objs/foo_lib/linux/arm64/release/rules/cuda/devlink/foo_lib_gpucode.cu.o
checking for flags (-fPIC) ... ok
checking for flags (-Wl,-rpath=/usr/local/cuda-12.2/targets/aarch64-linux/lib) ... ok
[ 75%]: linking.release cuda-cross-jetson
aarch64-linux-gnu-g++ -o build/linux/arm64/release/cuda-cross-jetson build/.objs/cuda-cross-jetson/linux/arm64/release/src/main.cpp.o -L/usr/local/cuda-12.2/targets/aarch64-linux/lib -Wl,-rpath=/usr/local/cuda-12.2/targets/aarch64-linux/lib -s -lfoo_lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: cannot find -lfoo_lib: No such file or directory
collect2: error: ld returned 1 exit status
error: execv(aarch64-linux-gnu-g++ -o build/linux/arm64/release/cuda-cross-jetson build/.objs/cuda-cross-jetson/linux/arm64/release/src/main.cpp.o -L/usr/local/cuda-12.2/targets/aarch64-linux/lib -Wl,-rpath=/usr/local/cuda-12.2/targets/aarch64-linux/lib -s -lfoo_lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl) failed(1)

现在看上去linkdirs和rpathdirs以及改成了target aarch64的路径了，然后找不到foo_lib了。

[xuyanwen2012]

我试着加了个add_ldflags("-L$(projectdir)/build/linux/arm64/release")然后又回到原来的问题了

aarch64-linux-gnu-g++ -o build/linux/arm64/release/cuda-cross-jetson build/.objs/cuda-cross-jetson/linux/arm64/release/src/main.cpp.o -L/usr/local/cuda-12.2/targets/aarch64-linux/lib -Wl,-rpath=/usr/local/cuda-12.2/targets/aarch64-linux/lib -s -lfoo_lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl -L/home/doremy/projects/cuda-cross-jetson/build/linux/arm64/release
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/doremy/projects/cuda-cross-jetson/build/linux/arm64/release/libfoo_lib.a(foo_lib_gpucode.cu.o): Relocations in generic ELF (EM: 62)
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/doremy/projects/cuda-cross-jetson/build/linux/arm64/release/libfoo_lib.a(foo_lib_gpucode.cu.o): Relocations in generic ELF (EM: 62)
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/doremy/projects/cuda-cross-jetson/build/linux/arm64/release/libfoo_lib.a(foo_lib_gpucode.cu.o): Relocations in generic ELF (EM: 62)
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/doremy/projects/cuda-cross-jetson/build/linux/arm64/release/libfoo_lib.a: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

[star-hengxing]

Relocations in generic ELF (EM: 62)

还是交叉编译的错误，可能是 ar 用错导致的

/usr/bin/ar -cr build/linux/arm64/release/libfoo_lib.a build/.objs/foo_lib/linux/arm64/release/src/foo.cu.o build/.objs/foo_lib/linux/arm64/release/rules/cuda/devlink/foo_lib_gpucode.cu.o

对比你给出正确的编译命令，应该要把 ar 设置为 aarch64-linux-gnu-ar