-
Notifications
You must be signed in to change notification settings - Fork 690
Arm backend: add DeiTTiny evaluator and deterministic shuffled calibration subsets #14579
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14579
Note: Links to docs will display an error until the docs builds have been completed. ❌ 5 New Failures, 1 Cancelled Job, 3 Unrelated FailuresAs of commit cc72152 with merge base c979158 ( NEW FAILURES - The following jobs have failed:
CANCELLED JOB - The following job was cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
@pytorchbot label ciflow/trunk |
@@ -1,5 +1,4 @@ | |||
# Copyright 2024-2025 Arm Limited and/or its affiliates. | |||
# All rights reserved. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unintentional delete?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is intentional since file is anyway modified and there is only an Arm copyright.
top5_accuracy = top5_correct / len(dataset) | ||
return top1_accuracy, top5_accuracy | ||
|
||
def evaluate(self) -> dict[str, Any]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would it better to refactor this and MobileNetV2Evaluator to share much of the code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good suggestion, will do.
d460483
to
fc55696
Compare
Change-Id: I7f61120772906ae0fec5d1f2b9cfcc0aa2c2c7af Signed-off-by: Tirui Wu <[email protected]>
fc55696
to
cc72152
Compare
Change-Id: I7f61120772906ae0fec5d1f2b9cfcc0aa2c2c7af
Summary
Add DeiTTiny evaluator for model accuracy evaluation.
cc @digantdesai @freddan80 @per @zingo @oscarandersson8218