cudnn_home not valid during build

[mfruhner]

Description
I am not able to build the ONNX Backend. I am following the build instructions in the README but the build fails at Step 17.

Triton Information
Main Branch for Trition Version 21.02

To Reproduce

I am running DGX OS 5 (Ubuntu 20.04).

cmake -DCMAKE_INSTALL_PREFIX:PATH=pwd/install -DTRITON_BUILD_ONNXRUNTIME_VERSION=1.6.0 -DTRITO N_BUILD_CONTAINER_VERSION=21.02 ..
make install

Output:

Step 17/24 : RUN ./build.sh ${COMMON_BUILD_ARGS} --update --build --use_cuda --cuda_home "/usr/local/cuda" ---> Running in 3360f12bb769 2021-03-18 11:01:00,463 build [ERROR] - cuda_home and cudnn_home paths must be specified and valid. cuda_home='/usr/local/cuda' valid=True. cudnn_home='None' valid=False The command '/bin/sh -c ./build.sh ${COMMON_BUILD_ARGS} --update --build --use_cuda --cuda_home "/usr/local/cuda"' returned a non-zero code: 1 make[2]: *** [CMakeFiles/ort_target.dir/build.make:81: onnxruntime/lib/libonnxruntime.so.1.6.0] Fehler 1 make[1]: *** [CMakeFiles/Makefile2:158: CMakeFiles/ort_target.dir/all] Fehler 2 make: *** [Makefile:149: all] Fehler 2

Expected behavior
I expect the build to succed.

[CoderHam]

https://github.com/triton-inference-server/onnxruntime_backend/blob/main/tools/gen_ort_dockerfile.py#L93
The build relies on getting CUDNN_VERSION from the base containers here.
I checked that the variable is indeed present in the nvcr.io/nvidia/tritonserver:21.02-py3-min container. Can you share the dockerfile generated by gen_ort_dockerfile.py in your build?

[askhade]

I have encountered this issue too... perhaps it may be a good idea to allow users to enter the path for cudnn similar to cuda.
TRITON_BUILD_CUDNN_HOME (similar to TRITON_BUILD_CUDA_HOME)

[askhade]

@mfruhner : Were you able to find a workaround ? I simply updated the gen_ort_dockerfile.py script tp explicitly include the cudnn path

[GowthamKudupudi]

@mfruhner, did you resolve it? if the build is failing then many should report but only few are here in this thread; are we doing anything wrong? I'm using --no-container-build flag while building triton server; why the build is trying to build ONNXruntime inside a container!

[GowthamKudupudi]

@CoderHam https://paste.ubuntu.com/p/nF3HCcYycR/ is the Dockerfile.ort found in the build folder.

[GowthamKudupudi]

@askhade what should be the value of --cudnn-home?

[askhade]

--cudnn_home should be set to the path to cudnn libs dir

[chandrameenamohan]

@askhade or @GowthamKudupudi Are you able to resolve it? I am trying this in centos7. I am able to build tensorflow1, tensorflow2, python and pytorch backend. But I am getting error when I try to build backend for onnx.
I use this command to build:
./build.py --cmake-dir=/server/build --build-dir=/tmp/citritonbuild --no-container-build --endpoint=http --endpoint=grpc --repo-tag=common:r21.04 --repo-tag=core:r21.04 --repo-tag=backend:r21.04 --repo-tag=thirdparty:r21.04 --backend=onnxruntime:r21.04 --enable-logging --enable-stats --enable-tracing

This is the error:

Step 26/37 : RUN ./build.sh ${COMMON_BUILD_ARGS} --update --build --use_cuda --cuda_home "/usr/local/cuda" --use_tensorrt --tensorrt_home "/usr/src/tensorrt" --use_openvino CPU_FP32
 ---> Running in dc590d08a4e7
2021-08-06 06:22:20,248 tools_python_utils [INFO] - flatbuffers module is not installed. parse_config will not be available
2021-08-06 06:22:20,257 build [ERROR] - cuda_home and cudnn_home paths must be specified and valid.
cuda_home='/usr/local/cuda' valid=True. cudnn_home='None' valid=False
The command '/bin/sh -c ./build.sh ${COMMON_BUILD_ARGS} --update --build --use_cuda --cuda_home "/usr/local/cuda" --use_tensorrt --tensorrt_home "/usr/src/tensorrt" --use_openvino CPU_FP32' returned a non-zero code: 1
make[2]: *** [CMakeFiles/ort_target.dir/build.make:81: onnxruntime/lib/libonnxruntime.so.1.7.1] Error 1
make[2]: Leaving directory '/tmp/citritonbuild/onnxruntime/build'
make[1]: *** [CMakeFiles/Makefile2:158: CMakeFiles/ort_target.dir/all] Error 2
make[1]: Leaving directory '/tmp/citritonbuild/onnxruntime/build'
make: *** [Makefile:149: all] Error 2
error: make install failed

[chandrameenamohan]

@mfruhner : Were you able to find a workaround ? I simply updated the gen_ort_dockerfile.py script tp explicitly include the cudnn path

Can you please explicitly mention where did you edit and how did you run this?
When I run build script. It automatically downloads the onnxruntime from git. What you did to use your local downloaded onnxruntime backend codebase?

[mfruhner]

I didn't try to solve this any further and went on to use something else, sorry.

[askhade]

@chandrameenamohan : I suppose you are hitting this issue because of "--no-container-build" can you remove it and test again?
@CoderHam : Can you pick this up?

[GowthamKudupudi]

The comment by @CoderHam that is liked is the key

[aravindhank11]

Any known fixes on how to pass the cudnn_home with --no-container-build opt?