Achieve low loss on validation set but get very low metrics

[phu2007vis]

• At first, I loaded the pretrained weights and evaluated the model. It achieved high accuracy (91%), but the evaluation loss was also high (22.6).
• After fine-tuning the model for several epochs, the evaluation loss decreased rapidly, but the accuracy dropped significantly, which is quite strange.
• In general, if the loss decreases, the metrics may improve or at least drop several %
  

[martianbit]

Hi,

I think the reason is that you're computing ForCausalLMLoss, but the model was trained using ForMaskedLMLoss as criterion (BERT is a masked LM).

Try adding this to get_model.py:

def get_model(encoder_name, decoder_name, max_length, num_decoder_layers=None):
...
    model.loss_type = "ForMaskedLM"
    return model, processor