Reduction/calculation of the total loss

[roskoN]

Hi there,

Thank you for sharing your excellent work!

I am having trouble understanding how the loss is calculated during pretraining. Could you help clarifying some details, please?

So far, I have noticed that for both frame loss (models/EAT_pretraining.py#L705) and utterance/CLS loss (models/EAT_pretraining.py#L659), the code uses MSE, but without reduction (loss = F.mse_loss(x, y, reduction="none")) .

The paper states that the two losses are summed up. But before one can do that, they need to be "reduced" (e.g., sum or mean). Where can I see how the reduction and the calculation of the total loss are performed?

Thank you!

[cwx-worst-one]

Hi,

Apologies, it’s been a while and I don’t quite remember the specifics of how the loss reduction was handled here, but I believe it might be related to the Fairseq framework. I looked into the Fairseq functions related to the Model Criterion and found that in fairseq/fairseq/criterions/model_criterion.py, the forward function of ModelCriterion includes the reduction of result["losses"].

I’m not entirely sure if this is directly related to the loss in this case, but it could be useful for your debugging. If you have any further questions, we can continue the discussion!

[roskoN]

Thank you for responding so quickly, and pointing out the relevant piece of code!

If I am reading the code correctly, it seems everything is being summed up together. Which I guess should be also in line with the outputs (losses of over) one can seen in the WandB report.

Closing the issue!

[nishitanand]

Hi @roskoN @cwx-worst-one, I am getting this error about the summation of losses when doing pretraining. Their tensor shape is different. Could you let me know if you also faced this error?

I tried printing their shapes and got this (192 for cls loss vs 78720 for IMAGE_regression loss). How to add these? Shouldn't this be handled in the code by itself?

For reference I am sending 10s files at 32000Hz sampling rate as input and task.h5_format is set to False (instead of Default True) in the pretraining bash script

[nishitanand]

Hi, I had manually added loss.sum() in model_criterion.py. But after going through the above thread, I checked that model_criterion.py does not itself have a .sum() on loss. Then I checked the latest fairseq on github and in it, it had .sum() in model_criterion.py.

Which version of fairseq should I use or which particular commit? I uses 0.12.2 release of fairseq and while setting up the codebase, some things were missing in it like SamePad2d, which were present in latest version of fairseq, so I copied those from github and did same for a few more things, but now seeing the same for model_criterion.py, I am wondering if I should use some other version of fairseq? Can you guys let me know which version of commit of fairseq you guys used?

@roskoN @cwx-worst-one

[cwx-worst-one]

This issue has been raised by other users as well. The main cause is using the pip version of Fairseq instead of performing a local installation. Although both versions are labeled as 0.12.2, the pip-installed version is missing some modules required by our project. We recommend following the local installation method provided in the README:

git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./
git clone https://github.com/cwx-worst-one/EAT