Add Directory STructure for BackendBench #90
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Alright lots of moving parts here and apologies for the lengthy PR but I'm opening this primarily to communicate with @Laurawly and @shaahins
The goal here is to have high confidence that
DirectoryBackend
is correct and that meansTo ensure we're not missing any torchbench operators I have two scripts that generate a fully functional DirectoryBackend with a root
generated_kernels/
and each op name underneath it as a seperate folder. The mapping between a PyTorch operator and folder name can sometimes be tricky. For example what do you expectrelu.default
to default to and so if you're unsure you can generate them with the full result here. In addition as I went through this exercise some ops didn't seem to have much documentation about them despite being in torchbench, so I made a list of ops with no docstrings hereFor any ops that we do implement we're confident they get overridden: The problem here is if we generate a correct operator and rely on logging statement we might still be loading the eager function since a correct function would be hard to distinguish. Instead what I do is purposefully load kernels that are incorrect with watermarked values of 42 that way we know the wrong kernel was loaded by us
Stretch of new features: one of the pitches of BackendBench was that this is a package you could install, I'd rather we tone down that pitch until we figure out details with the IntraKernel dispatcher so I'm deleting most of our code in
__init__py