[EZ] Check for stable outputs #69
Conversation
meta-cla
bot
added
the
CLA Signed
There was a problem hiding this comment.
stable/unstable to me means numerically stable or unstable. In this case I believe better terminology would be constant output functions such as full, zeros, ones
And the other problem is degenerate benchmarks where for instance torch.matmul(A,B) has A and B mostly 0 so the output is always 0
Note that it's fine if a single example in the benchmark is degenerate, the problem is if all of them are
|
@msaroufim are you saying its fine if a single test in the op is degenerate, however, if all of the tests are degenerate then we should axe it. I think filtering at a test by test level would end up achieving the same thing, and remove the degenerate tests as well. |
|
Yup maybe warning for degenerate input when benchmarking for now is sufficient. Input includes lots of 0, NaN, constant value for everything |
Fixes: #71 and #70
For checking for ones and zeros and noting that in the dataset, I think there should be a followup PR to #57