SigmoidBinaryCrossEntropyLoss implementation details

[sc12752]

Hi,

SigmoidBinaryCrossEntropyLoss class has two options:

• First option is when NN output has been already passed through sigmoid and it does look like textbook cross entropy loss.
• Second option, where NN output is taken as is without sigmoid applied, and it's the one I have troubles understanding.

Can someone please elaborate on why is there no sigmoid application in that branch (I would have expected that given a class name and logic behind it) or why do we have a relu and later softPlus there? This loss function works for me in practice but I'd like to find out some theory as to why.

Thanks!

[sc12752]

Another thing to note is BinaryAccuracy evaluator along with Loss.sigmoidBinaryCrossEntropyLoss always gives 0.0 accuracy while training. I guess I miss something here, too?

[sc12752]

This one was actually a rather stupid error of mine: I was always calling result.getValidateEvaluation("Accuracy"), even when using BinaryAccuracy. The correct call in the latter case is result.getValidateEvaluation("BinaryAccuracy").

[sc12752]

Also asked this at ai.stackexchange.com.

[zachgk]

This definition was adopted based on the one in MXNet. I looked back through the history there and it seems to come from this PR. That PR links to the definition from TensorFlow where it explains the derivation