Pytorch CUDA compatibity foor aarch64 architecture

[npeltier-scub]

Hello,

I have discover and test your library for a robotic project with face recognition. I used the face recognition example (https://docs.djl.ai/master/examples/docs/face_recognition.html) on x86_64 linux Ubuntu 22.04 with CPU (not GPU) : execution times are very correct for me : ~150ms per prediction.

Now, I want to test this project on a Jetson NANO for my robotic project (aarch64 system with CUDA GPU). But I saw this architecture is not supported for DJL / Pytorch (https://djl.ai/docs/development/dependency_management.html).

Do you know if it will be supported in the near future ?

Thanks.

Nicolas

[frankfliu]

There are few way you can try:

1. convert your model to ONNX and use DJL's OnnxRuntime engine to run the inference
2. pip install pytorch on Jetson, and use PYTORCH_LIBRARY_PATH env var to point the libtorch.so directory. I didn't try this, not sure if this work or not. If it doesn't work, then most likely you need build DJL's JNI from the source.

[npeltier-scub]

Thanks for you reply.
I will try to test the second option. I am a noob in AI and I don't know ONNX. I just try to execute DJL samples on Jetson Nano.

To build DJL's JNI, I have to follow this instructions ? http://djl.ai/engines/pytorch/pytorch-native/#maclinux-1

[npeltier-scub]

I managed to run the program on the NANO with the CPU: 2 seconds per prediction... :-(

I tried the first option following these instructions to install PyTorch (https://qengineering.eu/install-pytorch-on-jetson-nano.html) and adding PYTORCH_LIBRARY_PATH env var. I have these errors :

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Loading:     100% |████████████████████████████████████████|
[INFO ] - Downloading jni https://publish.djl.ai/pytorch/1.12.1/jnilib/0.19.0/linux-aarch64/cu102-precxx11/libdjl_torch.so to cache ...
Exception in thread "main" ai.djl.engine.EngineException: Failed to load PyTorch native library
	at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:82)
	at ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:40)
	at ai.djl.engine.Engine.getEngine(Engine.java:181)
	at ai.djl.Model.newInstance(Model.java:99)
	at ai.djl.repository.zoo.BaseModelLoader.createModel(BaseModelLoader.java:189)
	at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:152)
	at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:168)
	at org.example.Main.main(Main.java:48)
Caused by: java.lang.IllegalStateException: Cannot download jni files: https://publish.djl.ai/pytorch/1.12.1/jnilib/0.19.0/linux-aarch64/cu102-precxx11/libdjl_torch.so
	at ai.djl.pytorch.jni.LibUtils.downloadJniLib(LibUtils.java:503)
	at ai.djl.pytorch.jni.LibUtils.findJniLibrary(LibUtils.java:241)
	at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:78)
	at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:53)
	... 7 more
Caused by: java.io.FileNotFoundException: https://publish.djl.ai/pytorch/1.12.1/jnilib/0.19.0/linux-aarch64/cu102-precxx11/libdjl_torch.so
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1920)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
	at java.base/java.net.URL.openStream(URL.java:1165)
	at ai.djl.util.Utils.openUrl(Utils.java:459)
	at ai.djl.util.Utils.openUrl(Utils.java:443)
	at ai.djl.pytorch.jni.LibUtils.downloadJniLib(LibUtils.java:497)
	... 10 more

It looks better because the program tries to download aarch64 version for CUDA ... that does not exist : normal !

So, now, i will try the second option. I will get there one day ! ;-)

[npeltier-scub]

I tried the second option but I had these errors :

Command: bash build.sh 1.12.1 cu116 cxx11 aarch64
2022-11-11T19:01:35.503+0100 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: STARTING
2022-11-11T19:01:35.504+0100 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Waiting until process started: command 'bash'.
2022-11-11T19:01:35.514+0100 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: STARTED
2022-11-11T19:01:35.514+0100 [DEBUG] [org.gradle.process.internal.ExecHandleRunner] waiting until streams are handled...
2022-11-11T19:01:35.514+0100 [INFO] [org.gradle.process.internal.DefaultExecHandle] Successfully started process 'command 'bash''
2022-11-11T19:01:35.518+0100 [ERROR] [system.err] +++ dirname build.sh
2022-11-11T19:01:35.534+0100 [ERROR] [system.err] ++ cd .
2022-11-11T19:01:35.536+0100 [ERROR] [system.err] ++ pwd
2022-11-11T19:01:35.537+0100 [ERROR] [system.err] + WORK_DIR=/home/jetson/deep-java-library/djl/engines/pytorch/pytorch-native
2022-11-11T19:01:35.537+0100 [ERROR] [system.err] + export WORK_DIR
2022-11-11T19:01:35.537+0100 [ERROR] [system.err] + NUM_PROC=1
2022-11-11T19:01:35.541+0100 [ERROR] [system.err] ++ command -v nproc
2022-11-11T19:01:35.543+0100 [ERROR] [system.err] + [[ -n /usr/bin/nproc ]]
2022-11-11T19:01:35.544+0100 [ERROR] [system.err] ++ nproc
2022-11-11T19:01:35.554+0100 [ERROR] [system.err] + NUM_PROC=4
2022-11-11T19:01:35.555+0100 [ERROR] [system.err] ++ uname
2022-11-11T19:01:35.555+0100 [ERROR] [system.err] ++ tr '[:upper:]' '[:lower:]'
2022-11-11T19:01:35.558+0100 [ERROR] [system.err] + PLATFORM=linux
2022-11-11T19:01:35.558+0100 [ERROR] [system.err] + VERSION=1.12.1
2022-11-11T19:01:35.558+0100 [ERROR] [system.err] + FLAVOR=cu116
2022-11-11T19:01:35.559+0100 [ERROR] [system.err] + AARCH64_CXX11ABI=-cxx11
2022-11-11T19:01:35.559+0100 [ERROR] [system.err] + CXX11ABI=-cxx11-abi
2022-11-11T19:01:35.559+0100 [ERROR] [system.err] + [[ cxx11 == \p\r\e\c\x\x\1\1 ]]
2022-11-11T19:01:35.559+0100 [ERROR] [system.err] + ARCH=aarch64
2022-11-11T19:01:35.559+0100 [ERROR] [system.err] + [[ ! -d libtorch ]]
2022-11-11T19:01:35.559+0100 [ERROR] [system.err] + [[ linux == \l\i\n\u\x ]]
2022-11-11T19:01:35.560+0100 [ERROR] [system.err] + [[ ! cu116 =~ ^(cpu|cu102|cu111|cu113|cu116)$ ]]
2022-11-11T19:01:35.560+0100 [ERROR] [system.err] + [[ aarch64 == \a\a\r\c\h\6\4 ]]
2022-11-11T19:01:35.563+0100 [ERROR] [system.err] + jar xv
2022-11-11T19:01:35.565+0100 [ERROR] [system.err] + curl -s https://djl-ai.s3.amazonaws.com/publish/pytorch/1.12.1/libtorch-cxx11-shared-with-deps-1.12.1-aarch64.zip
2022-11-11T19:01:36.069+0100 [ERROR] [system.err] + [[ 1.12.1 =~ ^1\.10\..*|^1\.9\..* ]]
2022-11-11T19:01:36.069+0100 [QUIET] [system.out] ~/deep-java-library/djl/engines/pytorch/pytorch-native ~/deep-java-library/djl/engines/pytorch/pytorch-native
2022-11-11T19:01:36.069+0100 [ERROR] [system.err] + pushd .
2022-11-11T19:01:36.070+0100 [ERROR] [system.err] + rm -rf build
2022-11-11T19:01:36.077+0100 [ERROR] [system.err] + mkdir build
2022-11-11T19:01:36.081+0100 [ERROR] [system.err] + cd build
2022-11-11T19:01:36.081+0100 [ERROR] [system.err] + mkdir classes
2022-11-11T19:01:36.084+0100 [ERROR] [system.err] + javac -sourcepath ../../pytorch-engine/src/main/java/ ../../pytorch-engine/src/main/java/ai/djl/pytorch/jni/PyTorchLibrary.java -h include -d classes
2022-11-11T19:01:38.707+0100 [ERROR] [system.err] + cmake -DCMAKE_PREFIX_PATH=libtorch -DPT_OLD_VERSION= ..
2022-11-11T19:01:38.932+0100 [QUIET] [system.out] -- The C compiler identification is GNU 7.5.0
2022-11-11T19:01:39.137+0100 [QUIET] [system.out] -- The CXX compiler identification is GNU 7.5.0
2022-11-11T19:01:39.153+0100 [QUIET] [system.out] -- Check for working C compiler: /usr/bin/cc
2022-11-11T19:01:39.399+0100 [QUIET] [system.out] -- Check for working C compiler: /usr/bin/cc -- works
2022-11-11T19:01:39.401+0100 [QUIET] [system.out] -- Detecting C compiler ABI info
2022-11-11T19:01:39.644+0100 [QUIET] [system.out] -- Detecting C compiler ABI info - done
2022-11-11T19:01:39.679+0100 [QUIET] [system.out] -- Detecting C compile features
2022-11-11T19:01:40.400+0100 [QUIET] [system.out] -- Detecting C compile features - done
2022-11-11T19:01:40.411+0100 [QUIET] [system.out] -- Check for working CXX compiler: /usr/bin/c++
2022-11-11T19:01:40.686+0100 [QUIET] [system.out] -- Check for working CXX compiler: /usr/bin/c++ -- works
2022-11-11T19:01:40.688+0100 [QUIET] [system.out] -- Detecting CXX compiler ABI info
2022-11-11T19:01:40.959+0100 [QUIET] [system.out] -- Detecting CXX compiler ABI info - done
2022-11-11T19:01:40.992+0100 [QUIET] [system.out] -- Detecting CXX compile features
2022-11-11T19:01:42.221+0100 [QUIET] [system.out] -- Detecting CXX compile features - done
2022-11-11T19:01:42.223+0100 [ERROR] [system.err] Building djl_torch with the host...
2022-11-11T19:01:42.256+0100 [ERROR] [system.err] CMake Error at CMakeLists.txt:13 (find_package):
2022-11-11T19:01:42.256+0100 [ERROR] [system.err]   By not providing "FindTorch.cmake" in CMAKE_MODULE_PATH this project has
2022-11-11T19:01:42.257+0100 [ERROR] [system.err]   asked CMake to find a package configuration file provided by "Torch", but
2022-11-11T19:01:42.257+0100 [ERROR] [system.err]   CMake did not find one.
2022-11-11T19:01:42.257+0100 [ERROR] [system.err] 
2022-11-11T19:01:42.257+0100 [ERROR] [system.err]   Could not find a package configuration file provided by "Torch" with any of
2022-11-11T19:01:42.257+0100 [ERROR] [system.err]   the following names:
2022-11-11T19:01:42.257+0100 [ERROR] [system.err] 
2022-11-11T19:01:42.258+0100 [ERROR] [system.err]     TorchConfig.cmake
2022-11-11T19:01:42.258+0100 [ERROR] [system.err]     torch-config.cmake
2022-11-11T19:01:42.258+0100 [QUIET] [system.out] -- Configuring incomplete, errors occurred!
2022-11-11T19:01:42.258+0100 [QUIET] [system.out] See also "/home/jetson/deep-java-library/djl/engines/pytorch/pytorch-native/build/CMakeFiles/CMakeOutput.log".
2022-11-11T19:01:42.258+0100 [ERROR] [system.err] 
2022-11-11T19:01:42.263+0100 [ERROR] [system.err]   Add the installation prefix of "Torch" to CMAKE_PREFIX_PATH or set
2022-11-11T19:01:42.263+0100 [ERROR] [system.err]   "Torch_DIR" to a directory containing one of the above files.  If "Torch"
2022-11-11T19:01:42.264+0100 [ERROR] [system.err]   provides a separate development package or SDK, be sure it has been
2022-11-11T19:01:42.264+0100 [ERROR] [system.err]   installed.
2022-11-11T19:01:42.264+0100 [ERROR] [system.err] 
2022-11-11T19:01:42.264+0100 [ERROR] [system.err] 
2022-11-11T19:01:42.269+0100 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: FAILED
2022-11-11T19:01:42.270+0100 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Process 'command 'bash'' finished with exit value 1 (state: FAILED)
2022-11-11T19:01:42.271+0100 [LIFECYCLE] [org.gradle.internal.operations.DefaultBuildOperationRunner] 
2022-11-11T19:01:42.271+0100 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Execute doFirst {} action for :engines:pytorch:pytorch-native:compileJNI'
2022-11-11T19:01:42.271+0100 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Executing task ':engines:pytorch:pytorch-native:compileJNI''
2022-11-11T19:01:42.238+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] 
2022-11-11T19:01:42.238+0100 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :engines:pytorch:pytorch-native:compileJNI FAILED

[npeltier-scub]

Hi,
How to convert the model to ONNX ?
Will these methods work ?
https://deci.ai/blog/how-to-convert-a-pytorch-model-to-onnx/
or
https://pytorch.org/docs/stable/onnx.html#example-alexnet-from-pytorch-to-onnx

And is this enough for facial recognition to work with CUDA on Jetson ?

Thanks,
Nicolas

[frankfliu]

You should be able to convert PyTorch model to onnx by following the Blog. I didn't try onnx on Jetson. But it should work based on the document.