validation data is getting better accuracy and lower loss, than training data

[CareNetAI]

I am getting rather odd results, where our validation data is getting better accuracy and lower loss, than training data. And this is consistent across different sizes of hidden layers. This is our model:

def build_custom_cnn(input_shape, num_classes):
    model = Sequential([
        Conv2D(32, (3, 3), activation='relu', input_shape=input_shape),
        MaxPooling2D((2, 2)),
        Conv2D(64, (3, 3), activation='relu'),
        MaxPooling2D((2, 2)),
        Conv2D(128, (3, 3), activation='relu'),
        MaxPooling2D((2, 2)),
        Flatten(),
        Dropout(0.5),
        Dense(512, activation='relu'),
        Dropout(0.5),
        Dense(num_classes, activation='softmax')
    ])
    return model

I've tried to remove regularization and dropout, which, as expected, ended in overfitting. I've even tried to decrease the learning rate drastically, with similiar results.

Has anyone seen similar results?

[Mattral]

This indicates the presence of high bias in your dataset. It is underfitting. The solutions to issue are:-

• Probably the network is struggling to fit the training data. Hence, try a little bit bigger network.

• Try a different Deep Neural Network. I mean to say change the architecture a bit.

• Train for longer time.

• Try using advanced optimization algorithms.

• Make sure the is small class imbalance

• This happens when you use Dropout, since the behaviour when training and testing are different.

When training, a percentage of the features are set to zero (50% in your case since you are using Dropout(0.5)). When testing, all features are used (and are scaled appropriately). So the model at test time is more robust - and can lead to higher testing accuracies.

• In addition, whenever possible, check if your results make sense. For example, in case of a n-class classification with categorical cross entropy the loss on the first epoch should be the value of [ -ln(1/n) ] , where n is the number of classes in your case!

Apart your specific case, I believe that apart from the Dropout the dataset split may sometimes result in this situation. Especially if the dataset split is not random (in case where temporal or spatial patterns exist) the validation set may be fundamentally different, i.e less noise or less variance, from the train and thus easier to to predict leading to higher accuracy on the validation set than on training.

Moreover, if the validation set is very small compared to the training then by random the model fits better the validation set than the training.

[CareNetAI]

Thank you