Skip to content

Conversation

tirwu01
Copy link
Collaborator

@tirwu01 tirwu01 commented Sep 25, 2025

Change-Id: I7f61120772906ae0fec5d1f2b9cfcc0aa2c2c7af

Summary

Add DeiTTiny evaluator for model accuracy evaluation.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

Copy link

pytorch-bot bot commented Sep 25, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14579

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 1 Cancelled Job, 3 Unrelated Failures

As of commit cc72152 with merge base c979158 (image):

NEW FAILURES - The following jobs have failed:

CANCELLED JOB - The following job was cancelled. Please retry:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 25, 2025
@tirwu01
Copy link
Collaborator Author

tirwu01 commented Sep 25, 2025

@pytorchbot label ciflow/trunk

@tirwu01 tirwu01 changed the title add DeiTTiny evaluator and deterministic shuffled calibration subsets Arm backend: add DeiTTiny evaluator and deterministic shuffled calibration subsets Sep 25, 2025
@tirwu01 tirwu01 added partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm release notes: none Do not include this in the release notes labels Sep 25, 2025
@@ -1,5 +1,4 @@
# Copyright 2024-2025 Arm Limited and/or its affiliates.
# All rights reserved.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unintentional delete?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is intentional since file is anyway modified and there is only an Arm copyright.

top5_accuracy = top5_correct / len(dataset)
return top1_accuracy, top5_accuracy

def evaluate(self) -> dict[str, Any]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it better to refactor this and MobileNetV2Evaluator to share much of the code?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good suggestion, will do.

@tirwu01 tirwu01 force-pushed the add-deittiny-evaluator branch from d460483 to fc55696 Compare October 7, 2025 10:49
Change-Id: I7f61120772906ae0fec5d1f2b9cfcc0aa2c2c7af
Signed-off-by: Tirui Wu <[email protected]>
@tirwu01 tirwu01 force-pushed the add-deittiny-evaluator branch from fc55696 to cc72152 Compare October 10, 2025 10:19
@tirwu01 tirwu01 merged commit 7395999 into pytorch:main Oct 10, 2025
278 of 287 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm release notes: none Do not include this in the release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants