onnxruntime-linux-aarch64 with gpu support

[quangsonle]

I am trying to run a yolo-based model converted to Onnx format on Nvidia jetson nano. My code works well on a ubuntu 20 PC

the only thing i changed is, instead of onnxruntime-linux-x64-gpu-1.11.0 for the PC, i am using onnxruntime-linux-aarch64 for jetson.

It works but I got "GPU is not supported by your ONNXRuntime build. Fallback to CPU" and of course, very slow.

my question is, is there any version of gpu support for onnxruntime-linux-aarch64?

[andys-playground]

Seconded! Looking for onnxruntime-linux-arm64-gpu to take advantage of GPUs on aarch devices, notably Jetson devices from nvidia.

Their forums (https://forums.developer.nvidia.com/t/how-can-i-run-onnxruntime-c-api-in-jetson-os/201722/4) repeatedly refer to https://elinux.org/Jetson_Zoo#ONNX_Runtime, which is very python-oriented, not c/c++.

Is there a workaround? Do I need to build onnx from source on my target system?

[andys-playground]

I ended up solving this for my situation, at least for now.

References:

• Build on Jetson Xavier NX does not produce c++ API include and libs #9371
• https://onnxruntime.ai/docs/build/eps.html#nvidia-jetson-tx1tx2nanoxavier
• https://forums.developer.nvidia.com/t/how-can-i-run-onnxruntime-c-api-in-jetson-os/201722/5

I did the build on the Jetson Target (AGX Orin in my situation).

Assuming you have a default filesystem layout, steps to recreate:

1. clone onnxruntime repo
2. checkout the tagged version you want
3. init and update submodules recursively
4. Run their build.sh script with the following
   ./build.sh --config Release --update --build --parallel --build_wheel --build_shared_lib --use_tensorrt --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu/ --tensorrt_home /usr/lib/aarch64-linux-gnu/
5. Note, the build process will download more source, so internet access is needed.
6. Once the build process is complete, the Linux/Release/ folder will contain the .so files needed to link a c++ application, and can be copied out to an install destination folder.
7. The header files are in the repo, and can be copied out to an install destination folder

[csukuangfj]

I ended up solving this for my situation, at least for now.

References:

• Build on Jetson Xavier NX does not produce c++ API include and libs #9371
• https://onnxruntime.ai/docs/build/eps.html#nvidia-jetson-tx1tx2nanoxavier
• https://forums.developer.nvidia.com/t/how-can-i-run-onnxruntime-c-api-in-jetson-os/201722/5

I did the build on the Jetson Target (AGX Orin in my situation).

Assuming you have a default filesystem layout, steps to recreate:

1. clone onnxruntime repo
2. checkout the tagged version you want
3. init and update submodules recursively
4. Run their build.sh script with the following
   ./build.sh --config Release --update --build --parallel --build_wheel --build_shared_lib --use_tensorrt --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu/ --tensorrt_home /usr/lib/aarch64-linux-gnu/
5. Note, the build process will download more source, so internet access is needed.
6. Once the build process is complete, the Linux/Release/ folder will contain the .so files needed to link a c++ application, and can be copied out to an install destination folder.
7. The header files are in the repo, and can be copied out to an install destination folder

Thanks! It works!

[snnn]

Our team doesn't maintain or publish packages for Nvidia Jetson devices. However, you get one from https://elinux.org/Jetson_Zoo#ONNX_Runtime .

[csukuangfj]

https://elinux.org/Jetson_Zoo#ONNX_Runtime

The above link contains only packages for Python and it lacks of libonnxruntime.so so it cannot be used for linking in C/C++ projects.
@snnn

[csukuangfj]

I have collected pre-built libs for some versions of onnxruntime with GPU support. You can find them at

CUDA 10.2 + cudnn8

https://github.com/csukuangfj/onnxruntime-libs/releases/download/v1.11.0/onnxruntime-linux-aarch64-gpu-1.11.0.tar.bz2

CUDA 11.4 + cudnn8

https://github.com/csukuangfj/onnxruntime-libs/releases/download/v1.16.0/onnxruntime-linux-aarch64-gpu-1.16.0.tar.bz2

CUDA 12.6 + cudnn9

https://github.com/csukuangfj/onnxruntime-libs/releases/download/v1.18.1/onnxruntime-linux-aarch64-gpu-cuda12-1.18.1.tar.bz2

[klimgeorge]

Does the os version mater when you build them cause i need to run it on a jetson nano, Ubuntu 18.04. The Cuda version i have is 10.2 and when i try that version i get
The given version [18] is not supported, only version 1 to 11 is supported in this build.

when i try the other two versions i get the following errors:

cuda 12.6

/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_rwlock_wrlock@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `stat64@GLIBC_2.33'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_create@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `__libc_single_threaded@GLIBC_2.32'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `dlclose@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `log2@GLIBC_2.29'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pow@GLIBC_2.29'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `dlsym@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_rwlock_unlock@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_setspecific@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `dlopen@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `exp@GLIBC_2.29'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_rwlock_init@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `dlerror@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_setaffinity_np@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_attr_setstacksize@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `log@GLIBC_2.29'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_join@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_rwlock_rdlock@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `stat@GLIBC_2.33'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `fstat@GLIBC_2.33'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_once@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_rwlock_destroy@GLIBC_2.34'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pthread_key_create@GLIBC_2.34'
collect2: error: ld returned 1 exit status

version cuda 11.4
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `log2@GLIBC_2.29'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `pow@GLIBC_2.29'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `exp@GLIBC_2.29'
/usr/local/onnxruntime/lib/libonnxruntime.so: undefined reference to `log@GLIBC_2.29'
collect2: error: ld returned 1 exit status```

any idea what could it be the problem?

[snnn]

You need to use the onnxruntime C/C++ headers from v1.18.1, or older.

[klimgeorge]

even if i use an older version there's not a single build available for linux-aarch64-gpu, my code runs fine with the base linux-aarch64-v1.19.0, but it doesn't use the gpu, i even tried building from source, but i keep getting

  The compiler doesn't support BFLOAT16!!!

and i can't update jetpack version (on 4.6.6) cause i'm on the jetson nano and that's the latest version for it