demo.py script failing in GPU mode

[kaisark]

The demo.py script completes in CPU mode, but I am running into a CUDNN error when the script is running in GPU mode. The first image is processed successfully, but when I close the first image so the second image can be processed, the CUDNN error is generated. Could this be a config issue?

caffe.set_mode_gpu();
caffe.set_device(0)

Thanks.

---

nvidia@tegra-ubuntu:~/caffe/tools$ caffe device_query
I1012 16:42:30.237367 1105 caffe.cpp:470] This is NVCaffe 0.16.4 started at Thu Oct 12 16:42:29 2017
I1012 16:42:30.243551 1105 caffe.cpp:473] CuDNN version: 6021
I1012 16:42:30.243568 1105 caffe.cpp:474] CuBLAS version: 8000
I1012 16:42:30.243578 1105 caffe.cpp:475] CUDA version: 8000
I1012 16:42:30.243589 1105 caffe.cpp:476] CUDA driver version: 8000
I1012 16:42:30.243602 1105 caffe.cpp:110] Querying GPUs
caffe-time-test.txt

nvidia@tegra-ubuntu:~/cviz/mtcnn$ cat run
GLOG_minloglevel=1 ./demo-gpu.py

nvidia@tegra-ubuntu:~/cviz/mtcnn$ ./run
W1012 16:25:42.740932 864 upgrade_proto.cpp:71] Note that future Caffe releases will only support input layers and not input fields.
W1012 16:25:44.495682 864 upgrade_proto.cpp:71] Note that future Caffe releases will only support input layers and not input fields.
W1012 16:25:44.516561 864 upgrade_proto.cpp:71] Note that future Caffe releases will only support input layers and not input fields.

test1.jpg
[1]: 101
[2]: 101
[4]: 101
[4.5]: 101
[5]: 25
[6]: 16
[7]: 16
[8]: 16
2: (16, 5)
[9]: 13
[10]: 13
[11]: 2
3: (2, 5)

test2.jpg
F1012 16:25:46.933244 864 cudnn_conv_layer.cu:51] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM, device 0
*** Check failure stack trace: ***
@ 0x7f81ae4718 google::LogMessage::Fail()
@ 0x7f81ae6614 google::LogMessage::SendToLog()
@ 0x7f81ae4290 google::LogMessage::Flush()
@ 0x7f81ae6eb4 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f826068bc caffe::CuDNNConvolutionLayer<>::Forward_gpu()
@ 0x7f82be8f74 caffe::Layer<>::Forward()
@ 0x7f82518eac caffe::Net::ForwardFromTo()
@ 0x7f82bdd940 boost::python::objects::caller_py_function_impl<>::operator()()
@ 0x7f81b45c04 boost::python::objects::function::call()
@ 0x7f81b45e28 (unknown)
@ 0x7f81b4d5a8 boost::python::handle_exception_impl()
@ 0x7f81b42a34 (unknown)
@ 0x458ee8 PyObject_Call
./run: line 1: 864 Aborted GLOG_minloglevel=1 ./demo-gpu.py