[Backend Tester] Report delegation statistics #12846
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12846
Note: Links to docs will display an error until the docs builds have been completed. ❌ 3 New FailuresAs of commit bd79ef2 with merge base 3a02146 ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
For reviewers, is there a better way to present the operator counts? Are there other delegation-related stats that we should include? |
ghstack-source-id: cc7e564 ghstack-comment-id: 3115647824 Pull-Request: pytorch#12846
| in the test report. | ||
| """ | ||
| if counter is not None: | ||
| return str(dict(sorted(counter.items()))) |
There was a problem hiding this comment.
we can make this pretty but OK for now
| elapsed = time.perf_counter() - lower_start_time | ||
| extra_stats["lower_time"] = timedelta(seconds=elapsed) | ||
| except Exception as e: | ||
| elapsed = time.perf_counter() - lower_start_time | ||
| extra_stats["lower_time"] = timedelta(seconds=elapsed) | ||
| return build_result(TestResult.LOWER_FAIL, e) | ||
|
|
||
| # Compute delegation statistics. Use the ETRecord to access the edge dialect graph between |
There was a problem hiding this comment.
Curious why do we need to go through ETRecord?
There was a problem hiding this comment.
This is because we need the graph in between to_edge and to_backend when using to_edge_transform_and_lower in order to see the edge-dialect graph before partitioning. In theory, we have the info with the exported program, but it's aten dialect and pre-decompositions, so it's hard to reason about what's delegated.
With to_edge_transform_and_lower, we don't otherwise have a way to inspect the graph between the edge and lowering steps. ETRecord has effectively the same problem (they need the pre-delegation edge graph), so I'm using it here.
Report total number of delegated and undelegated nodes and breakdown by operator count.
Example from CoreML add: