Cross entropy loss over cosine distance

[yl3829]

I'm wondering the reason of using cross entropy loss over the cosine similarity between attractors and embeddings?

I suspect that this would cause inefficient use of the embedding space. Cross entropy loss try to maximize the cosine similarity between the attractors and the embeddings on the positive samples, this is fine. But on the negative samples, it will minimize the cosine similarity, pushing the similarity to negative 1, instead of 0. Negative 1 similarity means they are fully related but in the opposite direction. I believe this would restrict the representation space of attractors and embeddings.

Would love to hear your thoughts and consideration here. Thanks!

[DiLiangWU]

Thank you for your insightful question. You are right that for negative samples, the attractors tend to be pushed toward the opposite direction of the embeddings. Ideally, the cosine similarity between attractors and embeddings for negative samples only needs to lie within the range [-1, 0], which would be sufficient.

In fact, during experiment, we considered using a hinge loss so that for negative samples the loss is defined as

loss=max(cos(𝑒𝑖, 𝑎𝑗), 0), where 𝑒𝑖 denotes the embedding of speaker 𝑖 and 𝑎𝑗 denotes the attractor of speaker 𝑗. However, this approach did not work well. We observed that when the diarization loss is designed such that loss = 0 when the cosine similarity = 0, many attractors ended up satisfying cos(𝑒𝑖, 𝑎𝑗) > 0 on negative samples, which led to classification errors.

We believe this happens because, compared with offline attractors, the online frame-wise attractors exhibit much more flexible and dynamic patterns. A more explicit loss is needed to guide the distribution of the attractors. Therefore, using cos⁡(𝑒𝑖, 𝑎𝑗) = 0 as the threshold between positive and negative samples is reasonable, but using 0 as the learning target for negative samples is not appropriate.