Skip to content

[EZ] Check for stable outputs #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

PaliC
Copy link
Contributor

@PaliC PaliC commented Aug 12, 2025

Fixes: #71 and #70

For checking for ones and zeros and noting that in the dataset, I think there should be a followup PR to #57

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 12, 2025
Copy link
Member

@msaroufim msaroufim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stable/unstable to me means numerically stable or unstable. In this case I believe better terminology would be constant output functions such as full, zeros, ones

And the other problem is degenerate benchmarks where for instance torch.matmul(A,B) has A and B mostly 0 so the output is always 0

Note that it's fine if a single example in the benchmark is degenerate, the problem is if all of them are

@PaliC
Copy link
Contributor Author

PaliC commented Aug 13, 2025

@msaroufim are you saying its fine if a single test in the op is degenerate, however, if all of the tests are degenerate then we should axe it. I think filtering at a test by test level would end up achieving the same thing, and remove the degenerate tests as well.

@msaroufim
Copy link
Member

Yup maybe warning for degenerate input when benchmarking for now is sufficient. Input includes lots of 0, NaN, constant value for everything

@PaliC PaliC requested a review from msaroufim August 14, 2025 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Take out tests where the output is almost always all zeros or ones
2 participants