Could not load dynamic library 'libnvinfer.so.6'

[kba]

In a fresh venv on a CPU-only machine, running make test-cli will install ocr4all-pixel-classifier[tf_cpu] as it is supposed to but the actual test fails with

2020-02-27 11:34:41.047310: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-27 11:34:41.047404: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-27 11:34:41.047419: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-02-27 11:34:42.154339: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-27 11:34:42.154378: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-27 11:34:42.154406: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (b-ocr-d): /proc/driver/nvidia/version does not exist
2020-02-27 11:34:42.154585: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-27 11:34:42.160566: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2596990000 Hz
2020-02-27 11:34:42.160761: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4dc6220 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-02-27 11:34:42.160778: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
bash: line 1:  4962 Segmentation fault      ocrd-pc-segmentation -l DEBUG -m mets.xml -I OCR-D-IMG-BIN -O OCR-D-SEG-BLOCK
Makefile:62: recipe for target 'test-cli' failed
make: *** [test-cli] Error 139

Why does it even try to use CUDA?

[crater2150]

I cannot reproduce the segfault by hiding the GPU (using CUDA_VISIBLE_DEVICES=-1). I'll have to check with another machine without libcuda installed. But I also get the errors about the libnvinfer libraries, followed by no CUDA-capable device is detected and tensorflow then proceeds to use the CPU successfully.

Note, that ocr4all-pixel-classifier uses TensorFlow 2.0 for some time now, which doesn't have separate builds for GPU/CPU anymore and autodetects them. The [tf_cpu] or [tf_gpu] extras are ignored, I just forgot to remove it from the Makefile.

Could you try running the following test script in your virtualenv and see if the same error occurs? (example from the tensorflow tutorial)

import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10)
])
predictions = model(x_train[:1]).numpy()
print(predictions)

[kba]

Thanks for the quick response.

Output of your test script before segfault:

2020-02-27 15:34:40.383144: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-02-27 15:34:40.383237: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-02-27 15:34:40.383252: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
2020-02-27 15:34:42.871885: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-27 15:34:42.872656: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-27 15:34:42.872886: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (b-ocr-d): /proc/driver/nvidia/version does not exist
2020-02-27 15:34:42.873316: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-27 15:34:42.880302: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2596990000 Hz
2020-02-27 15:34:42.880615: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3c0f4b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-02-27 15:34:42.880631: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[5]    16409 segmentation fault  python /tmp/test.cuda.py

$ pip show tensorflow

Name: tensorflow
Version: 2.1.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /data/venv3.6-pc-seg/lib/python3.6/site-packages
Requires: tensorboard, wheel, keras-preprocessing, six, grpcio, termcolor, opt-einsum, astor, wrapt, scipy, google-pasta, gast, absl-py, numpy, keras-applications, tensorflow-estimator, protobuf
Required-by:

$ pip show ocr4all-pixel-classifier
Name: ocr4all-pixel-classifier
Version: 0.2.0
Summary: UNKNOWN
Home-page: https://gitlab2.informatik.uni-wuerzburg.de/chw71yx/page-segmentation.git
Author: Christoph Wick, Alexander Hartelt, Alexander Gehrke
Author-email: christoph.wick@informatik.uni-wuerzburg.de, alexander.hartelt@informatik.uni-wuerzburg.de, alexander.gehrke@informatik.uni-wuerzburg.de
License: UNKNOWN
Location: /data/venv3.6-pc-seg/lib/python3.6/site-packages
Requires: scikit-image, numpy, tqdm, dataclasses, Pillow, scipy, efficientnet, opencv-python-headless
Required-by: ocrd-pc-segmentation

[crater2150]

As the test script is straight from the first tensorflow tutorial, this seems to be an upstream bug in Tensorflow, that probably has to be reported there. Can you test if it still happens in the current nightly? (pip uninstall tensorflow && pip install tf-nightly). Otherwise I'll try to reproduce it in a virtual machine.

Our software should work with Tensorflow 2.0.0, maybe a downgrade can help temporarily. If so, notify me, I'll update requirements.txt with a maximum version.

[kba]

ok, I'll try tensorflow==2.0.0 first.

[kba]

I'll try tensorflow==2.0.0 first.

That fixed it. ocr-d-modul-2-segmentierung/page-segmentation#3

[crater2150]

I found out, that tf-2.0 does not yet have GPU detection, it was introduced with 1.15, but 2.0 was released earlier and only got it in 2.1. So I'm reopening the issue and will try to reproduce this myself.

I'll keep the current workaround with limiting the version for now, and re-add the GPU package destinction.

[crater2150]

I tested tensorflow-2.1.0 with a virtual machine running Ubuntu 18.04 / Python 3.6, and with a laptop without dedicated GPU, running pyenv Python versions 3.6.10 and 3.7.6.

In all of those variants the test script produces the same output, which is identical to your result up to the line before segfault, but then continues to run normally, using the CPU. The various "library not found" errors appear, but are non-fatal. Can you provide more information about the environment you're using, e.g OS, CPU architecture?

[kba]

It's a Debian 8 VM, so this might very well be the issue. I'll try again on a less ancient OS as soon as I can and report back.

…

On Thu, Mar 5, 2020, 14:58 Alexander Gehrke ***@***.***> wrote: I tested tensorflow-2.1.0 with a virtual machine running Ubuntu 18.04 / Python 3.6, and with a laptop without dedicated GPU, running pyenv Python versions 3.6.10 and 3.7.6. In all of those variants the test script produces the same output, which is identical to your result up to the line before segfault, but then continues to run normally, using the CPU. The various "library not found" errors appear, but are non-fatal. Can you provide more information about the environment you're using, e.g OS, CPU architecture? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#7?email_source=notifications&email_token=AACCXV3KNWGW6NUO4OEDBLDRF6VXLA5CNFSM4K4YNCK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN5K7VY#issuecomment-595242967>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACCXV6QX34B3ZXUDTKOYU3RF6VXLANCNFSM4K4YNCKQ> .