error on training

[urmikakasi]

This is the output from training- the model is not getting saved due to a callback issue.

4:6:34: Using Inceptionv3 model
{}: Generating image features using inceptionv3 model...
2022-01-14 04:06:34.692740: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/inception_v3/inception_v3_weights_tf_dim_ordering_tf_kernels.h5
96116736/96112376 [==============================] - 1s 0us/step
96124928/96112376 [==============================] - 1s 0us/step
100% 8091/8091 [28:33<00:00, 4.72it/s]
4:35:12: Completed & Saved features for 8091 images successfully
4:35:12: Parsing captions file...
4:35:12: Parsed captions: 40460
4:35:12: Parsed & Saved successfully
4:35:12: Available images for training: 6000
4:35:12: Available captions for training: 30000
4:35:13: Available images for validation: 1000
4:35:13: Available captions for validation: 5000
RNN Model (Decoder) Summary :
Model: "model_1"

---

Layer (type) Output Shape Param # Connected to

input_3 (InputLayer) [(None, 40)] 0 []

input_2 (InputLayer) [(None, 2048)] 0 []

embedding (Embedding) (None, 40, 300) 2213400 ['input_3[0][0]']

dense (Dense) (None, 300) 614700 ['input_2[0][0]']

lstm (LSTM) (None, 40, 256) 570368 ['embedding[0][0]']

repeat_vector (RepeatVector) (None, 40, 300) 0 ['dense[0][0]']

time_distributed (TimeDistribu (None, 40, 300) 77100 ['lstm[0][0]']
ted)

concatenate_2 (Concatenate) (None, 40, 600) 0 ['repeat_vector[0][0]',
'time_distributed[0][0]']

bidirectional (Bidirectional) (None, 512) 1755136 ['concatenate_2[0][0]']

dense_2 (Dense) (None, 7378) 3784914 ['bidirectional[0][0]']

==================================================================================================
Total params: 9,015,618
Trainable params: 9,015,618
Non-trainable params: 0

---

None
steps_train: 94, steps_val: 16
Batch Size: 64
Total Number of Epochs = 20
train_val.py:86: UserWarning: Model.fit_generator is deprecated and will be removed in a future version. Please use Model.fit, which supports generators.
verbose=1)
Epoch 1/20
Traceback (most recent call last):
File "train_val.py", line 86, in
verbose=1)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 2030, in fit_generator
initial_epoch=initial_epoch)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Matrix size-incompatible: In[0]: [3732,1000], In[1]: [2048,300]
[[node model_1/dense/Relu
(defined at /usr/local/lib/python3.7/dist-packages/keras/backend.py:4867)
]] [Op:__inference_train_function_569695]

Errors may have originated from an input operation.
Input Source operations connected to node model_1/dense/Relu:
In[0] model_1/dense/BiasAdd (defined at /usr/local/lib/python3.7/dist-packages/keras/layers/core/dense.py:210)

Operation defined at: (most recent call last)

File "train_val.py", line 86, in
verbose=1)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 2030, in fit_generator
initial_epoch=initial_epoch)

File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1216, in fit
tmp_logs = self.train_function(iterator)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 878, in train_function
return step_function(self, iterator)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 867, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))

File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 860, in run_step
outputs = model.train_step(data)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 808, in train_step
y_pred = self(x, training=True)

File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py", line 1083, in call
outputs = call_fn(inputs, *args, **kwargs)

File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/functional.py", line 452, in call
inputs, training=training, mask=mask)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/functional.py", line 589, in _run_internal_graph
outputs = node.layer(*args, **kwargs)

File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)

File "/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py", line 1083, in call
outputs = call_fn(inputs, *args, **kwargs)

File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)

File "/usr/local/lib/python3.7/dist-packages/keras/layers/core/dense.py", line 213, in call
outputs = self.activation(outputs)

File "/usr/local/lib/python3.7/dist-packages/keras/activations.py", line 311, in relu
return backend.relu(x, alpha=alpha, max_value=max_value, threshold=threshold)

File "/usr/local/lib/python3.7/dist-packages/keras/backend.py", line 4867, in relu
x = tf.nn.relu(x)

2022-01-14 04:35:25.329300: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]

[hibah321]

I've got the same error. Can you help if you've resolved it?

[Anant-mishra1729]

Got the same error. Help needed!

[kunal0230]

Matrix Size Mismatch:
The shape mismatch [3732,1000] vs. [2048,300] likely comes from connecting the InceptionV3 output to the RNN decoder. Double-check that InceptionV3 outputs [None, 2048] for each image, then add a Dense layer or Reshape layer to align it with the decoder’s input.

Callback and Model Saving Issue:
Since fit_generator is deprecated, switch to fit, which should work seamlessly with data generators. Also, make sure ModelCheckpoint or other callbacks are properly configured and compatible with TensorFlow’s current version, especially for saving the model.

CUDA Warning (No GPU Detected):
This message means training is running on CPU instead of GPU. If a GPU is available, use tf.config.list_physical_devices('GPU') to verify its accessibility, or check your environment settings to ensure TensorFlow can detect it.