Add Post to backend stage to recipes

[Erik-Lundell]

Add PostToBackend stage to Recipes
After a discussion in #14588, it was decided to
create an additional recipe stage to run passes after partitioning.
This is meant for backends that convert ops directly instead of
partitioning.

Add RequireExportProgram mixin for passes
It is a common pattern that passes require an exported program,
which means that the pass needs to be constructed right before
it is ran, with the correct exported program. Today there is no
unified way this is done, which makes it tricky for general pass handling
to figure out whether to pass an exported program to the constructor or not.
This commit solves this by introducing a RequireExportProgram mixin,
and makes a best effort to migirate passes to this system.

Note that there are passes which require other arguments that
still will break general pass handling.

cc @freddan80 @per @zingo @oscarandersson8218 @digantdesai

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14990

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Long queue for ROCM runners, also B200 and XPU queueing is observed

❌ 22 New Failures

As of commit 090baad with merge base 5cf193a ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner-mypy / linux-job (gh)
  >>> Lint for backends/arm/_passes/unsqueeze_scalar_placeholders_pass.py:
• pull / unittest / linux / linux-job (gh)
  export/tests/test_export_session.py::TestExportSessionCoreFlow::test_default_pipeline_execution_order
• pull / unittest / macos / macos-job (gh)
  export/tests/test_target_recipes.py::TestTargetRecipes::test_resnet18_model
• pull / unittest / windows / windows-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-arm-backend-with-no-fvp (test_pytest_models) / linux-job (gh)
  RuntimeError: Command docker exec -t fe11529e788bf9f925d4632d5aa7a9afeaac6300876581c8974b0c372bb2fb1d /exec failed with exit code 1
• pull / unittest-arm-backend-with-no-fvp (test_pytest_ops) / linux-job (gh)
  RuntimeError: Command docker exec -t 1291f74ed97f5b03171fb96579639dd1f6532873b6e2c27378906e36bec095c5 /exec failed with exit code 1
• pull / unittest-editable / linux / linux-job (gh)
  export/tests/test_export_session.py::TestExportSessionCoreFlow::test_default_pipeline_execution_order
• pull / unittest-editable / macos / macos-job (gh)
  export/tests/test_executorch_export.py::TestExecutorchExport::test_basic_recipe
• pull / unittest-editable / windows / windows-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• trunk / test-arm-backend (test_memory_allocation) / linux-job (gh)
  RuntimeError: Command docker exec -t 6ae2e53e1828377836bc3a3702ec043a559eb57b8370776c17fdd460313fdd34 /exec failed with exit code 1
• trunk / test-arm-backend (test_model_smollm2-135M) / linux-job (gh)
  AttributeError: can't set attribute 'exported_program'
• trunk / test-arm-backend (test_models_ethos-u55) / linux-job (gh)
  AttributeError: can't set attribute 'exported_program'
• trunk / test-arm-backend (test_models_ethos-u85) / linux-job (gh)
  AttributeError: can't set attribute 'exported_program'
• trunk / test-arm-backend (test_models_tosa) / linux-job (gh)
  AttributeError: can't set attribute 'exported_program'
• trunk / test-arm-backend (test_pytest_models_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t f3157acf451f7717aa9564d93fecbe268632e3181f89d11cab2a18944f470804 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t fd4061831dab0f6f3e3cd309cd8048e40387b431d3524e39e4367e1b07b8e29d /exec failed with exit code 1
• trunk / test-arm-backend (test_run_ethosu_fvp) / linux-job (gh)
  AttributeError: can't set attribute 'exported_program'
• trunk / test-arm-backend (test_smaller_stories_llama) / linux-job (gh)
  RuntimeError: Command docker exec -t a71c2bde3864992b1e1319cc7afeeb460dc840ea4e86fe3765d76a3a0a463b74 /exec failed with exit code 1
• trunk / test-arm-ootb-linux / linux-job (gh)
  AttributeError: can't set attribute 'exported_program'
• trunk / unittest-release / linux / linux-job (gh)
  export/tests/test_target_recipes.py::TestTargetRecipes::test_w2l_model
• trunk / unittest-release / macos / macos-job (gh)
  export/tests/test_target_recipes.py::TestTargetRecipes::test_w2l_model
• trunk / unittest-release / windows / windows-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

Hi @abhinaykukkadapu and @digantdesai I think you commented on the original issue leading up to this change, is it OK to merge this?

[abhinaykukkadapu]

@Erik-Lundell @zingo Sorry for the late response and thanks for following up on this change, i've done some "standardization" when passes needs an ExportedProgram. QNN backend has exact issue where some passes wants ExportedProgram, i think we can follow similar pattern here too, here is the code ptr: https://github.com/pytorch/executorch/blob/main/backends/qualcomm/recipes/qnn_recipe_provider.py#L142

Basic idea is that the recipe will get a callable similar to aten_transform_passes and the callable can have a sequence of passes instantiated with or without EP. Let me know if you have any questions.

After you are done, i will probably take a stab at creating a single stage for all the pass related steps and then that stage shall be plugged in at various stages pre/post et stages.