[et] generate debug handle before opeartor decomposition #11997

Gasoonjia · 2025-06-26T07:47:07Z

Stack from ghstack (oldest at bottom):

-> [et] generate debug handle before opeartor decomposition #11997

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph.

Differential Revision: D76860368

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) [ghstack-poisoned]

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) ghstack-source-id: 292801936 Pull Request resolved: #11997

pytorch-bot · 2025-06-26T07:47:12Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11997

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit dba90f5 with merge base b342f83 ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / test-models-linux (mobilebert, portable, linux.2xlarge) / linux-job (gh) (trunk failure)
##[error]The operation was canceled.
pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh) (trunk failure)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-06-26T07:47:19Z

This pull request was exported from Phabricator. Differential Revision: D76860368

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) [ghstack-poisoned]

…ed graph instead of edge program Pull Request resolved: #11997 This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph generated from nn.Module, by using `from_node` as source. N6213836 is an example of how `from_node` infra works. Before this diff: {F1979833525} After this diff: {F1979833595} and if quantized: {F1979833708} ghstack-source-id: 293136004 @exported-using-ghexport Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/)

facebook-github-bot · 2025-06-27T21:46:45Z

This pull request was exported from Phabricator. Differential Revision: D76860368

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) [ghstack-poisoned]

…ed graph instead of edge program Pull Request resolved: #11997 This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph generated from nn.Module, by using `from_node` as source. N6213836 is an example of how `from_node` infra works. After this diff every node sharing the same ancestor node in export graph will have same debug handle. Before this diff: {F1979833525} After this diff: {F1979833595} and if quantized: {F1979833708} ghstack-source-id: 293582394 @exported-using-ghexport Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/)

facebook-github-bot · 2025-07-01T05:55:17Z

This pull request was exported from Phabricator. Differential Revision: D76860368

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) [ghstack-poisoned]

…ed graph instead of edge program Pull Request resolved: #11997 This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph generated from nn.Module, by using `from_node` as source. N6213836 is an example of how `from_node` infra works. After this diff every node sharing the same ancestor node in export graph will have same debug handle. Before this diff: {F1979833525} After this diff: {F1979833595} and if quantized: {F1979833708} ghstack-source-id: 293597605 @exported-using-ghexport Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/)

facebook-github-bot · 2025-07-01T08:10:33Z

This pull request was exported from Phabricator. Differential Revision: D76860368

kimishpatel · 2025-07-02T02:57:31Z

exir/passes/debug_handle_generator_pass.py

+            """
+
+            node_source = node.meta[FROM_NODE_KEY]
+            node_source = node_source[-1]


this is only populated for unlifted graph when .module is called? if so how is it guranteed that this is present

no it is not only populate that .module is explicitly called. In our case it is guarantee generated after run_decompostition so we will have that when generating debug handle.

it is guarantee generated after run_decompostition so we will have that when generating debug handle.

There in my question around to_edge_transform_and_lower. although that specifically is unrelated to this pr

kimishpatel · 2025-07-02T03:00:32Z

exir/passes/debug_handle_generator_pass.py

+            debug_handle = (
+                len(source_node_id_to_debug_handle) + 1
+                if greatest_ancestor_node_id not in source_node_id_to_debug_handle
+                else source_node_id_to_debug_handle[greatest_ancestor_node_id]
+            )


Ok one of my concern is this: is this gonna impact any existing debug utils that rely on debug handle? earlier each node had its own debug handle, but not we are giving a bunch of nodes that might belong to the same original node (e.g. full aten graph nodes?).

Other question is how does this work for delegates that use to_edge_tranform_and_lower, if this pass is run only after to_edge?

For first question, yes like devtool and some other infra or utils are depending on the debgu handle and in the following diffs i will do the migration. There're some testcases rely on debug handle for check and I've talked with the owners and they are ok with the update. Now ci is all clean.

For second question, not sure if I fully understand your concern, but our current debug handle generation pass also locates at the end of to_edge. Our update does not change the location. But we do have concern about whether or not the change of debug handle generation may impact the correctness of the debug handels in the downstream e.g. in delegate, though ci is all clean. And we will pay attention to it when working on the numeric descrepancy dtector on delegated model.

Actually we should discuss this second paragraph further. THe issue is that to_edge_tranform_and_lower happens before we run decomposition, is that right? If so you dont have debug handles generated no? If no debug handles then no debug information. I would like to be either wrong about my assumption or we need to fix it

For the first paragraph, do you forsee any disruption in existing user flows for debugging? That would be one thing that would make me block this PR, so having an answer on that would be nice

The issue is that to_edge_tranform_and_lower happens before we run decomposition, is that right?

No operator decomposition should be happened inside to_edge_transform_and_lower. to_edge stage happened in side to_edge_transform_and_lower and operator decompostition happened inside to_edge stage. https://github.com/pytorch/executorch/blob/main/exir/program/_program.py#L1129

forsee any disruption in existing user flows for debugging

no. One heart break context is no debug tool is actually using debug handle right now. The numerical discrepancy detector will be the first one.

kimishpatel · 2025-07-02T03:03:43Z

devtools/inspector/_inspector_utils.py

+# One debug handle should only be associated with one node. We are in the middle of migrating debug handle generation
+# from graph after to_edge to graph after torch.export, one every debug handle in exported graph may be associated with multiple nodes in to_edge


Put before and after state

i've updated the comment to reflect the function behavior before, during and after the migration.

kimishpatel

For the most part looks good. I left a couple of comments, one specifically around relation of to_edge_lower_and_transform to debug handle (not related ot this diff)

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) [ghstack-poisoned]

…ed graph instead of edge program Pull Request resolved: #11997 This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph generated from nn.Module, by using `from_node` as source. N6213836 is an example of how `from_node` infra works. After this diff every node sharing the same ancestor node in export graph will have same debug handle. Before this diff: {F1979833525} After this diff: {F1979833595} and if quantized: {F1979833708} ghstack-source-id: 293914705 @exported-using-ghexport Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/)

facebook-github-bot · 2025-07-02T18:00:59Z

This pull request was exported from Phabricator. Differential Revision: D76860368

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) [ghstack-poisoned]

…ed graph instead of edge program Pull Request resolved: #11997 This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph generated from nn.Module, by using `from_node` as source. N6213836 is an example of how `from_node` infra works. After this diff every node sharing the same ancestor node in export graph will have same debug handle. Before this diff: {F1979833525} After this diff: {F1979833595} and if quantized: {F1979833708} ghstack-source-id: 294639957 @exported-using-ghexport Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/)

facebook-github-bot · 2025-07-07T17:14:27Z

This pull request was exported from Phabricator. Differential Revision: D76860368

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) [ghstack-poisoned]

…ed graph instead of edge program Pull Request resolved: #11997 This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph generated from nn.Module, by using `from_node` as source. N6213836 is an example of how `from_node` infra works. After this diff every node sharing the same ancestor node in export graph will have same debug handle. Before this diff: {F1979833525} After this diff: {F1979833595} and if quantized: {F1979833708} Beyond that, placeholder and output node will no longer have debug handle. ghstack-source-id: 294751869 @exported-using-ghexport Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/)

This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph. Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/) [ghstack-poisoned]

…ed graph instead of edge program Pull Request resolved: #11997 This diff update the debug handle generation, from each node in the edge program having a individual debug handle, to all nodes having a same ancestor in export graph sharing a same debug handle, which update the start point of tracing our node transformation from edge graph to exported graph generated from nn.Module, by using `from_node` as source. N6213836 is an example of how `from_node` infra works. After this diff every node sharing the same ancestor node in export graph will have same debug handle. Before this diff: {F1979833525} After this diff: {F1979833595} and if quantized: {F1979833708} Beyond that, placeholder and output node will no longer have debug handle. ghstack-source-id: 294784602 @exported-using-ghexport Differential Revision: [D76860368](https://our.internmc.facebook.com/intern/diff/D76860368/)

facebook-github-bot · 2025-07-08T04:09:40Z

This pull request was exported from Phabricator. Differential Revision: D76860368

@Gasoonjia

This PR was created by the merge bot to help merge the original PR into the main branch. ghstack PR number: #11997 by @Gasoonjia ^ Please use this as the source of truth for the PR details, comments, and reviews ghstack PR base: https://github.com/pytorch/executorch/tree/gh/gasoonjia/17/base ghstack PR head: https://github.com/pytorch/executorch/tree/gh/gasoonjia/17/head Merge bot PR base: https://github.com/pytorch/executorch/tree/main Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/gasoonjia/17/orig @diff-train-skip-merge Co-authored-by: gasoonjia <[email protected]>

@Gasoonjia

This PR was created by the merge bot to help merge the original PR into the main branch. ghstack PR number: pytorch#11997 by @Gasoonjia ^ Please use this as the source of truth for the PR details, comments, and reviews ghstack PR base: https://github.com/pytorch/executorch/tree/gh/gasoonjia/17/base ghstack PR head: https://github.com/pytorch/executorch/tree/gh/gasoonjia/17/head Merge bot PR base: https://github.com/pytorch/executorch/tree/main Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/gasoonjia/17/orig @diff-train-skip-merge Co-authored-by: gasoonjia <[email protected]>

Gasoonjia requested review from JacobSzwejbka and larryliu0820 as code owners June 26, 2025 07:47

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 26, 2025

facebook-github-bot added the fb-exported label Jun 26, 2025

kimishpatel reviewed Jul 2, 2025

View reviewed changes

Gasoonjia added the release notes: devtools Changes to dev tooling, for example the debugger & profiler label Jul 2, 2025

kimishpatel approved these changes Jul 3, 2025

View reviewed changes

facebook-github-bot merged commit 9216209 into gh/gasoonjia/17/base Jul 8, 2025
164 of 172 checks passed

facebook-github-bot deleted the gh/gasoonjia/17/head branch July 8, 2025 17:35

facebook-github-bot temporarily deployed to cherry-pick-bot July 8, 2025 17:35 — with GitHub Actions Inactive

pytorchbot mentioned this pull request Jul 8, 2025

[et] generate debug handle before opeartor decomposition #12275

Merged

		# One debug handle should only be associated with one node. We are in the middle of migrating debug handle generation
		# from graph after to_edge to graph after torch.export, one every debug handle in exported graph may be associated with multiple nodes in to_edge

Uh oh!

[et] generate debug handle before opeartor decomposition #11997

[et] generate debug handle before opeartor decomposition #11997

Uh oh!

Conversation

Gasoonjia commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11997

✅ You can merge normally! (2 Unrelated Failures)

Uh oh!

facebook-github-bot commented Jun 26, 2025

Uh oh!

facebook-github-bot commented Jun 27, 2025

Uh oh!

facebook-github-bot commented Jul 1, 2025

Uh oh!

facebook-github-bot commented Jul 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gasoonjia Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gasoonjia Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kimishpatel left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jul 2, 2025

Uh oh!

facebook-github-bot commented Jul 7, 2025

Uh oh!

facebook-github-bot commented Jul 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Gasoonjia commented Jun 26, 2025 •

edited

Loading

pytorch-bot bot commented Jun 26, 2025 •

edited

Loading

Gasoonjia Jul 2, 2025 •

edited

Loading

Gasoonjia Jul 3, 2025 •

edited

Loading