Suggestion to Modify Cosine Similarity Loss Function for Improved Consistency

[DAigory]

Hi , I’ve been using the cosine_similarity_loss function from your repository, and I’d like to suggest a small modification to improve its consistency in reflecting loss values.
Current Implementation:

def cosine_similarity_loss(f1, f2):
    f1 = F.normalize(f1, p=2., dim=-1, eps=1e-5)
    f2 = F.normalize(f2, p=2., dim=-1, eps=1e-5)
    return -(f1 * f2).sum(dim=1)

Proposed Modification:

def cosine_similarity_loss(f1, f2):
    f1 = F.normalize(f1, p=2., dim=-1, eps=1e-5)
    f2 = F.normalize(f2, p=2., dim=-1, eps=1e-5)
    return 1.0-(f1 * f2).sum(dim=1)

Rationale: The proposed modification changes the loss calculation to:
1.0 -(f1 * f2).sum(dim=1)
This change ensures that the loss values strive towards zero, making it more intuitive and consistent for gradient-based optimization. In scenarios where f1 and f2 are perfectly aligned, the cosine similarity will be 1, resulting in a zero loss, which is desirable for convergence.

I believe this adjustment will enhance the clarity and effectiveness of the loss function.

Thank you for considering this suggestion. I’m happy to discuss further if needed.

[Shengjiewang-Jason]

That is a good idea. Thank you for your kind suggestion. We will run some experiments to verify it.

[abfariah]

Hi, thank you for open sourcing your project! Have you managed to run these experiments ? I am very interested in this issue raised in this post, in fact, when reproducing the results of https://github.com/YeWR/EfficientZero I noticed that the consistency loss is consistently negative throughout training. Moreover, as training progresses, the consistency loss becomes more and more negative (nothing surprising, we are doing loss minimization). However, this would mean in practice that inner representations of original and data augmented observation are in phase opposition. As the consistency loss has been found to be critical to improve the performance of EfficientZero agents, I would be keen to have your insight on the subject.

[Yyassin]

@abfariah They are in phase, not in opposition. The loss is negated - maximum loss is -1.0 when they are completely in phase; cosine similarity is 1.0 and the loss will then be -1.0