Building a Java ONNX CUDA container

[ldave23]

Hi, I want to build a dockerized java app that uses the onnx binding. I was wondering if anyone has experience with this. I built the CUDA docker container first:

docker build -t onnxruntime-cuda -f ./dockerfile/Dockerfile.cuda .

Next in my dockerfile:

FROM onnxruntime-cuda
ENV DEBIAN_FRONTEND=noninteractive
WORKDIR /app
RUN apt-get update && apt-get install -y openjdk-{VERSION}-jre
COPY onnx-app.jar /app/onnx-app.jar
CMD java -jar ./onnx-app.jar

I was able to run this successfully:

docker build -t onnx-java-app
docker run -it --gpus all --rm onnx-java-app

i.e. the ORT session runs. However, the step options.ddCUDA(0) fails:

ai.onnxruntime.OrtException: Error code - ORT_RUNTIME_EXCEPTION - message: /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1193 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

I'm aware that this problem has existed, and that I should probably build the java project from scratch, but I don't know where to start. I'm using the maven repositories for ONNX, so I didn't really have to do any building before, much less with gradle. Do I need to build a jar with gradle from the java sub-folder on my GPU machine, then link it somehow to my maven project? Is this the correct approach?

[Craigacp]

To build ORT for java you should run ./build.sh --update --config Release --build --build_java --parallel --use_cuda --cuda_home <path-to-cuda> --cudnn_home <path-to-cudnn>. That will trigger the gradle build for you after the native library has been built.

[ldave23]

Where should I be building this? I just built ORT on my host machine and then built the maven jarfile, but when I move this into the docker container, the error still persists. Is this supposed to be built in the docker container?

Because on my host I'm running cuda 11.8, whereas docker seems to run 12.x

[Craigacp]

You need to build it with the same CUDA version, preferably in the same place.

[ldave23]

Got it, thanks. I'll build this and see the result.

[ldave23]

The build works. Thanks!