How to read detect_anomaly=True output?

[grudloff]

Is there a way to get more informative outputs, or a way to interpret the current output of detect_anomaly when nan are found? Right now I get something like this:

RuntimeError: Function 'PowBackward0' returned nan values in its 0th output.

[rahulvigneswaran]

@grudloff detect_anomaly basically will pinpoint which function starts the nan. As you can see from the error message, the function PowBackward0 is causing the nan.

Do you have a self-implemented function called PowBackward0 ?

[grudloff]

Hello @rahulvigneswaran! Yes, I know that! It's the name for the backward function of the power function. I had to painstakingly look for all the places a pow was present to pinpoint where was the problem. Is there a way for the anomaly detection to print from where the nan is coming from?

[rahulvigneswaran]

@grudloff Am not sure but I guess a line number would be helpful. In the meanwhile, you could use a debugger (I use the Inbuilt one in VScode). The debugger will pause on the exact line where the error message is thrown.