Error when running train.py with lrcn, conv_3d, c3d

[ucsky]

Hello,
I get the following error message when trying to run train.py with lrcn, conv_3d and c3d
2018-05-30 10:36:35.547825: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-05-30 10:36:35.952164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: name: Quadro K5200 major: 3 minor: 5 memoryClockRate(GHz): 0.771 pciBusID: 0000:02:00.0 totalMemory: 7.93GiB freeMemory: 7.78GiB 2018-05-30 10:36:36.268054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 1 with properties: name: Tesla K40c major: 3 minor: 5 memoryClockRate(GHz): 0.745 pciBusID: 0000:03:00.0 totalMemory: 11.17GiB freeMemory: 11.09GiB 2018-05-30 10:36:36.615248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 2 with properties: name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531 pciBusID: 0000:81:00.0 totalMemory: 11.90GiB freeMemory: 11.74GiB 2018-05-30 10:36:36.615562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Device peer to peer matrix 2018-05-30 10:36:36.615610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1051] DMA: 0 1 2 2018-05-30 10:36:36.615620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 0: Y Y N 2018-05-30 10:36:36.615626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 1: Y Y N 2018-05-30 10:36:36.615633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 2: N N Y 2018-05-30 10:36:36.615661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Quadro K5200, pci bus id: 0000:02:00.0, compute capability: 3.5) 2018-05-30 10:36:36.615671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Tesla K40c, pci bus id: 0000:03:00.0, compute capability: 3.5) 2018-05-30 10:36:36.615679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: TITAN X (Pascal), pci bus id: 0000:81:00.0, compute capability: 6.1) Using TensorFlow backend. /usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from floattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 612, in data_generator_task
generator_output = next(self._generator)
File "/home/bob/test/five-video-classification-methods-ucsky/data.py", line 27, in next
return next(self.iterator)
File "/home/bob/test/five-video-classification-methods-ucsky/data.py", line 197, in frame_generator
y.append(self.get_class_one_hot(sample[1]))
File "/home/bob/test/five-video-classification-methods-ucsky/data.py", line 107, in get_class_one_hot
assert len(label_hot) == len(self.classes)
AssertionError

Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 612, in data_generator_task
generator_output = next(self._generator)
File "/home/bob/test/five-video-classification-methods-ucsky/data.py", line 27, in next
return next(self.iterator)
StopIteration

Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 612, in data_generator_task
generator_output = next(self._generator)
File "/home/bob/test/five-video-classification-methods-ucsky/data.py", line 27, in next
return next(self.iterator)
StopIteration

Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 612, in data_generator_task
generator_output = next(self._generator)
File "/home/bob/test/five-video-classification-methods-ucsky/data.py", line 27, in next
return next(self.iterator)
StopIteration

Set model to lrcn
Loading CNN-LSTM model.

---

Layer (type) Output Shape Param #

time_distributed_1 (TimeDist (None, 40, 40, 40, 32) 4736

---

time_distributed_2 (TimeDist (None, 40, 38, 38, 32) 9248

---

time_distributed_3 (TimeDist (None, 40, 19, 19, 32) 0

---

time_distributed_4 (TimeDist (None, 40, 19, 19, 64) 18496

---

time_distributed_5 (TimeDist (None, 40, 19, 19, 64) 36928

---

time_distributed_6 (TimeDist (None, 40, 9, 9, 64) 0

---

time_distributed_7 (TimeDist (None, 40, 9, 9, 128) 73856

---

time_distributed_8 (TimeDist (None, 40, 9, 9, 128) 147584

---

time_distributed_9 (TimeDist (None, 40, 4, 4, 128) 0

---

time_distributed_10 (TimeDis (None, 40, 4, 4, 256) 295168

---

time_distributed_11 (TimeDis (None, 40, 4, 4, 256) 590080

---

time_distributed_12 (TimeDis (None, 40, 2, 2, 256) 0

---

time_distributed_13 (TimeDis (None, 40, 2, 2, 512) 1180160

---

time_distributed_14 (TimeDis (None, 40, 2, 2, 512) 2359808

---

time_distributed_15 (TimeDis (None, 40, 1, 1, 512) 0

---

time_distributed_16 (TimeDis (None, 40, 512) 0

---

dropout_1 (Dropout) (None, 40, 512) 0

---

lstm_1 (LSTM) (None, 256) 787456

---

dense_1 (Dense) (None, 101) 25957

Total params: 5,529,477
Trainable params: 5,529,477
Non-trainable params: 0

---

None
Creating train generator with 7871 samples.
Epoch 1/1000
Traceback (most recent call last):
File "./train.py", line 132, in
main()
File "./train.py", line 128, in main
load_to_memory=load_to_memory, batch_size=batch_size, nb_epoch=nb_epoch)
File "./train.py", line 89, in train
workers=4)
File "/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py", line 88, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/models.py", line 1124, in fit_generator
initial_epoch=initial_epoch)
File "/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py", line 88, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 1877, in fit_generator
str(generator_output))
ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None
`
Please is somebody known what is going on and how I can fix it?

[andruet]

I also suffered this problem and hope someone fix it.

[mertia-himanshu]

Please see [pull request] (#87)

[13293824182]

I modified
assert len(label_hot[0]) == len(self.classes)
return label_hot[0]
but i get the flowing error
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 10485760 values, but the requested
[[Node: gradients/time_distributed_13/Reshape_1_grad/Reshape = _MklReshape[T=DT_FLOAT, Tshape=DT_INT32, _kecalhost/replica:0/task:0/device:CPU:0"](gradients/time_distributed_14/convolution_grad/Conv2DBackpropInput, gradient1_grad/Shape, gradients/time_distributed_14/convolution_grad/Conv2DBackpropInput:1, DMT/_319)]]
Please is somebody known what is going on and how I can fix it?