CMake failed to build inside of container

[akinwilson]

Description

I have a container which has tensorrt installed already, this container can be pulled from:

nvcr.io/nvidia/tensorrt:22.08-py3

when I tried to build the image and extend it to allow onnx to find tensorrt, I am greeted with build errors. Particularly, the cmake command

cmake .. -DTENSORRT_ROOT=/workspace/tensorrt

fails with the following output;

root@25730af9b10e:/workspace/onnx-tensorrt/build# cmake .. -DTENSORRT_ROOT=/workspace/tensorrt
-- The CXX compiler identification is GNU 9.4.0
-- The C compiler identification is GNU 9.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread (found version "3.6.1") 
-- Build type not set - defaulting to Release
-- Found PythonInterp: /usr/bin/python (found version "3.8.10") 
Generated: /workspace/onnx-tensorrt/build/third_party/onnx/onnx/onnx_onnx2trt_onnx-ml.proto
Generated: /workspace/onnx-tensorrt/build/third_party/onnx/onnx/onnx-operators_onnx2trt_onnx-ml.proto
Generated: /workspace/onnx-tensorrt/build/third_party/onnx/onnx/onnx-data_onnx2trt_onnx.proto
-- 
-- ******** Summary ********
--   CMake version                     : 3.16.3
--   CMake command                     : /usr/bin/cmake
--   System                            : Linux
--   C++ compiler                      : /usr/bin/c++
--   C++ compiler version              : 9.4.0
--   CXX flags                         :  -Wall -Wno-deprecated-declarations -Wno-unused-function -Wnon-virtual-dtor
--   Build type                        : Release
--   Compile definitions               : SOURCE_LENGTH=25;ONNX_NAMESPACE=onnx2trt_onnx;__STDC_FORMAT_MACROS
--   CMAKE_PREFIX_PATH                 : 
--   CMAKE_INSTALL_PREFIX              : /usr/local
--   CMAKE_MODULE_PATH                 : 
-- 
--   ONNX version                      : 1.17.0
--   ONNX NAMESPACE                    : onnx2trt_onnx
--   ONNX_USE_LITE_PROTO               : OFF
--   USE_PROTOBUF_SHARED_LIBS          : OFF
--   Protobuf_USE_STATIC_LIBS          : ON
--   ONNX_DISABLE_EXCEPTIONS           : OFF
--   ONNX_DISABLE_STATIC_REGISTRATION  : OFF
--   ONNX_WERROR                       : OFF
--   ONNX_BUILD_TESTS                  : OFF
--   ONNX_BUILD_SHARED_LIBS            : 
--   BUILD_SHARED_LIBS                 : 
-- 
--   Protobuf compiler                 : /usr/bin/protoc
--   Protobuf includes                 : /usr/include
--   Protobuf libraries                : /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread
--   BUILD_ONNX_PYTHON                 : OFF
-- Found CUDA headers at /usr/local/cuda/include
-- Found TensorRT headers at /usr/include/x86_64-linux-gnu
Found TensorRT Python headers at TENSORRT_PYTHON_INCLUDE_DIR-NOTFOUND
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/workspace/onnx-tensorrt/TENSORRT_PYTHON_INCLUDE_DIR
   used as include directory in directory /workspace/onnx-tensorrt
   used as include directory in directory /workspace/onnx-tensorrt

-- Configuring incomplete, errors occurred!
See also "/workspace/onnx-tensorrt/build/CMakeFiles/CMakeOutput.log".
See also "/workspace/onnx-tensorrt/build/CMakeFiles/CMakeError.log".

when I go to investigate the error log file from the build; CMakeError.log, I am shown:

Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /workspace/onnx-tensorrt/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_331d9/fast && /usr/bin/make -f CMakeFiles/cmTC_331d9.dir/build.make CMakeFiles/cmTC_331d9.dir/build
make[1]: Entering directory '/workspace/onnx-tensorrt/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_331d9.dir/src.c.o
/usr/bin/cc   -DCMAKE_HAVE_LIBC_PTHREAD -fPIE   -o CMakeFiles/cmTC_331d9.dir/src.c.o   -c /workspace/onnx-tensorrt/build/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_331d9
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_331d9.dir/link.txt --verbose=1
/usr/bin/cc  -DCMAKE_HAVE_LIBC_PTHREAD    CMakeFiles/cmTC_331d9.dir/src.c.o  -o cmTC_331d9 
/usr/bin/ld: CMakeFiles/cmTC_331d9.dir/src.c.o: in function `main':
src.c:(.text+0x46): undefined reference to `pthread_create'
/usr/bin/ld: src.c:(.text+0x52): undefined reference to `pthread_detach'
/usr/bin/ld: src.c:(.text+0x63): undefined reference to `pthread_join'
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_331d9.dir/build.make:87: cmTC_331d9] Error 1
make[1]: Leaving directory '/workspace/onnx-tensorrt/build/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_331d9/fast] Error 2


Source file was:
#include <pthread.h>

void* test_func(void* data)
{
  return data;
}

int main(void)
{
  pthread_t thread;
  pthread_create(&thread, NULL, test_func, NULL);
  pthread_detach(thread);
  pthread_join(thread, NULL);
  pthread_atfork(NULL, NULL, NULL);
  pthread_exit(NULL);

  return 0;
}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /workspace/onnx-tensorrt/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_73222/fast && /usr/bin/make -f CMakeFiles/cmTC_73222.dir/build.make CMakeFiles/cmTC_73222.dir/build
make[1]: Entering directory '/workspace/onnx-tensorrt/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_73222.dir/CheckFunctionExists.c.o
/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=pthread_create -fPIE   -o CMakeFiles/cmTC_73222.dir/CheckFunctionExists.c.o   -c /usr/share/cmake-3.16/Modules/CheckFunctionExists.c
Linking C executable cmTC_73222
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_73222.dir/link.txt --verbose=1
/usr/bin/cc  -DCHECK_FUNCTION_EXISTS=pthread_create    CMakeFiles/cmTC_73222.dir/CheckFunctionExists.c.o  -o cmTC_73222  -lpthreads 
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_73222.dir/build.make:87: cmTC_73222] Error 1
make[1]: Leaving directory '/workspace/onnx-tensorrt/build/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_73222/fast] Error 2

where I noted that error line /usr/bin/ld: cannot find -lpthreads especially.

Environment

The environment is an image and therefore the error can easily be reproduced. The dockerfile is as follows:

FROM nvcr.io/nvidia/tensorrt:22.08-py3
LABEL maintainer="Akinola Antony Wilson <akinolawilson@gmail.com>"
# Allow passing in decision threshold and model version during build of serving container.
ARG DECISION_THRESHOLD=0.5
ARG MODEL_VERSION="docker-env-model-version"
ARG EXECUTION_PROVIDER="TensorrtExecutionProvider"
# Setting decision_threshold and model_version and env vars 
ENV DECISION_THRESHOLD=${DECISION_THRESHOLD}
ENV MODEL_VERSION=${MODEL_VERSION}
# setting environment variable specifying execution provider, can be: CUDAExecutionProvider, CPUExecutionProvider or TensorrtExecutionProvider

ENV EXECUTION_PROVIDER=${EXECUTION_PROVIDER}
# install utilities
RUN apt-get update && \
    apt-get install --no-install-recommends -y curl
# audio processing dependencies 
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
RUN apt-get install -y libsndfile-dev
# Install python 
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
# install protobuf dependencies
RUN apt-get install protobuf-compiler libprotobuf-dev -y 


# Installing python dependencies
RUN python3 -m pip --no-cache-dir install --upgrade pip && \
    python3 --version && \
    pip3 --version

# install gpu-enabled torch 
RUN pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113 


# check https://github.com/onnx/onnx-tensorrt/issues/354#issuecomment-572279735 ---> 
RUN git clone --recurse-submodules https://github.com/onnx/onnx-tensorrt.git
WORKDIR /workspace/onnx-tensorrt

RUN mkdir build
WORKDIR /workspace/onnx-tensorrt/build

RUN cmake .. -DTENSORRT_ROOT=/workspace/tensorrt

RUN make -j

RUN export LD_LIBRARY_PATH=$PWD:$LD_LIBRARY_PATH

WORKDIR /workspace

COPY ./requirements.txt ./requirements.txt
RUN pip3 --timeout=300 --no-cache-dir install -r requirements.txt


# Copy model files
# COPY ./deploy/model /model


RUN pip3 --timeout=300 --no-cache-dir install -r requirements.txt
# Copy app files
RUN mkdir app model 
COPY ./app /app


COPY ./start.sh /app/start.sh
RUN chmod +x /app/start.sh

WORKDIR /app
EXPOSE 80
ENTRYPOINT ["./start.sh"]

I am not too sure how to proceed, since I had this image built and working around a year ago. Its only now that I have tried to rebuild it, that I am facing these issues.

Is anyone able to help?

[nimdrak]

@akinwilson Hi

Found TensorRT Python headers at TENSORRT_PYTHON_INCLUDE_DIR-NOTFOUND
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/workspace/onnx-tensorrt/TENSORRT_PYTHON_INCLUDE_DIR
   used as include directory in directory /workspace/onnx-tensorrt
   used as include directory in directory /workspace/onnx-tensorrt

As in the error, I think you need to add proper TENSORRT_PYTHON_INCLUDE_DIR.
In my case, It was -DTENSORRT_ROOT=/TensorRT-10.13.0.35/include/.

How about find the directory by using sudo find / -name "NvInfer.h" in the container?