Distilling knowledge with a teacher using sigmoid function loss

[publioelon]

Hi, I'd like to ask what changes are needed to distil knowledge between a teacher with a sigmoid function loss and a student with softmax loss?

[patrickphat]

Hi @publioelon,
I think it's technically possible, since sigmoid is a special case of softmax (where number outputs = 2). But first could you please let me know if the teachers' classes are subset of students classes?
For example:

• Teacher outputs 2 classes: cat, dog
• Student outputs 3 classes: cat, dog and elephant

[publioelon]

Hello, they both have the same classes cat and dog

[publioelon]

Hi, any updates?