Add aot example with Neutron Backend #10871
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10871
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit be74e00 with merge base 70ea0dd ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
robert-kalmar
force-pushed
the
upstream/release-mcux-25.03-full/aot-example
branch
from
e5ed112 to
46c2a58
Compare
|
@pytorchbot label "module: nxp" "release notes: nxp" |
|
Didn't find following labels among repository labels: ,,label |
pytorch-bot
bot
added
module: nxp
examples/nxp/aot_neutron_compile.py
Outdated
| action="store_true", | ||
| required=False, | ||
| default=False, | ||
| help="Flag for producing ArmBackend delegated model", |
There was a problem hiding this comment.
| help="Flag for producing ArmBackend delegated model", | |
| help="Flag for producing NeutronBackend delegated model", |
examples/nxp/aot_neutron_compile.py
Outdated
| model, example_inputs, strict=True | ||
| ) | ||
|
|
||
| # TODO: Add Neutron ATen Passes, once https://github.com/pytorch/executorch/pull/10579 is merged |
There was a problem hiding this comment.
nit: file a task so we can track and not lose this
examples/nxp/aot_neutron_compile.py
Outdated
| "_portable_lib.cpython* using --portable_lib CLI options. \n" | ||
| "This is required for running quantized models with unquantized input." | ||
| ) | ||
| sys.exit(-1) |
There was a problem hiding this comment.
Can you either: (1) just not sys.exit entirely and let it fail loudly later when it will hit the runtime exception or (2) add a CLI arg to allow skipping this part-- and the part below for the torch.loads
In internal infra, these libraries are loaded a slightly different way and I do not actually pass the .so on command line, and it is not loaded a few lines below.
There was a problem hiding this comment.
✅
Ok, so reverted back to our original solution. There is only a warning raised and normally fails later when exporting to ExecuTorch Program:
# 6. Export to ExecuTorch program
try:
exec_prog = edge_program.to_executorch(
config=ExecutorchBackendConfig(extract_delegate_segments=False)
)
except RuntimeError as e:
if "Missing out variants" in str(e.args[0]):
raise RuntimeError(
e.args[0]
+ ".\nThis likely due to an external so library not being loaded. Supply a path to it with the "
"--portable_lib flag."
).with_traceback(e.__traceback__) from None
else:
raise e
| x = self.conv3(x) | ||
| x = self.pool2(x) | ||
|
|
||
| # The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in TFLite). When running |
There was a problem hiding this comment.
| # The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in TFLite). When running | |
| # The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in Neutron IR). When running |
| x = self.pool2(x) | ||
|
|
||
| # The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in TFLite). When running | ||
| # inference of the `FullyConnected`, TFlite will automatically collapse the channels and spatial dimensions and |
There was a problem hiding this comment.
| # inference of the `FullyConnected`, TFlite will automatically collapse the channels and spatial dimensions and | |
| # inference of the `FullyConnected`, Neutron IR will automatically collapse the channels and spatial dimensions and |
examples/nxp/aot_neutron_compile.py
Outdated
| parser.add_argument( | ||
| "-p", | ||
| "--portable_lib", | ||
| required=True, |
There was a problem hiding this comment.
This probably shouldn't be required because portable library is loaded only when --quantize=True.
There was a problem hiding this comment.
✅ Thanks, fixed in latest push.
examples/nxp/aot_neutron_compile.py
Outdated
|
|
||
| # For quantization we need to build the quantized_ops_aot_lib.so and _portable_lib.*.so | ||
| # Use this CMake options | ||
| # -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON |
There was a problem hiding this comment.
Is this documentation up to date? Is portable lib built just by specifying these two flags?
There was a problem hiding this comment.
The quantized_ops_aot_lib links to portable_lib
$ ldd ./venv3.10/lib/python3.10/site-packages/executorch/kernels/quantized/libquantized_ops_aot_lib.so
_portable_lib.cpython-310d-x86_64-linux-gnu.so => not found
....
For some reason we must load the portable_lib manually prior to libquantized_ops_aot_lib.so, the dlopen does not not find is by its own.
There was a problem hiding this comment.
✅
FYI @skywall , we do not need any custom library loading for the quantized kernels out variants. There are already a python packages for this:
import executorch.extension.pybindings.portable_lib
import executorch.kernels.quantized
Thanks to @digantdesai for the review items which helped me to find it out.
robert-kalmar
force-pushed
the
upstream/release-mcux-25.03-full/aot-example
branch
from
46c2a58 to
2397cb0
Compare
examples/nxp/README.md
Outdated
| 2. After building the ExecuTorch you shall have the `libquantized_ops_aot_lib.so` and `_portable_lib.<python_version>.so` located in the `pip_out/lib` folder. We will need this library when generating the quantized cifarnet ExecuTorch model. So as first step we will find it: | ||
| ```commandline | ||
| $ find . -name "libquantized_ops_aot_lib.so" | ||
| ./pip-out/lib.linux-x86_64-cpython-310-pydebug/executorch/kernels/quantized/libquantized_ops_aot_lib.so |
There was a problem hiding this comment.
FYI I added optimized cortex-M q/dq int8 op if you want to use that , it is still quite early days for that lib
examples/nxp/README.md
Outdated
| ./pip-out/lib.linux-x86_64-cpython-310-pydebug/executorch/kernels/quantized/libquantized_ops_aot_lib.so | ||
|
|
||
| $ find . -name "_portable_lib.cpython-310d-x86_64-linux-gnu.so" | ||
| ./pip-out/lib.linux-x86_64-cpython-310-pydebug/executorch/extension/pybindings/_portable_lib.cpython-310d-x86_64-linux-gnu.so |
There was a problem hiding this comment.
is this using selective build?
There was a problem hiding this comment.
Not sure what you mean.
There was a problem hiding this comment.
Ok, I understand where you are heading. We needed the quantized_aot_lib to get the out variants for quantize/dequantize_per_tensor operators.
I find there are already python bindings and modules to solve it:
import executorch.extension.pybindings.portable_lib
import executorch.kernels.quantized
✅
examples/nxp/aot_neutron_compile.py
Outdated
| torch.ops.load_library(args.portable_lib) | ||
| torch.ops.load_library(args.so_library) |
There was a problem hiding this comment.
why do we need these? just include the python module perhaps?
There was a problem hiding this comment.
You are right (obviously) , we don't. Importing the python modules instead.
import executorch.extension.pybindings.portable_lib
import executorch.kernels.quantized
Thanks for the finding, it helped me to locate these python modules.
|
Ready to merge? Fix linter please? |
Not yet, updating quantizer to the recent changes : moving |
robert-kalmar
force-pushed
the
upstream/release-mcux-25.03-full/aot-example
branch
2 times, most recently
from
3018aea to
2941a74
Compare
|
Linting error - fixed. Now it is ready to merge. |
|
3 checks failed. All with missing the "llm" preset. They were added in a later commit (c256723#diff-fc10486ef573a9c92fe4a135b8a1b20157154af6e83dacfd1ea046bda7814c84). I guess, those failures are unrelated with changes in the PR. Although I wonder, why those tests got even triggered, as they are not in the |
|
Let's re-merge the CI PR, and then we can merge this, so we have some confidence in this and know we won't be regressing. Thanks. |
robert-kalmar
force-pushed
the
upstream/release-mcux-25.03-full/aot-example
branch
from
2941a74 to
c7d4b49
Compare
It is not empty, just it content has not changed - https://github.com/pytorch/executorch/blob/2941a74be7f4d49198087d3983d591911c614260/examples/nxp/setup.sh The WebUI is misleading here. By "empty file" it evidently means empty diff 🙃 |
|
Converting to draft unless the NXP Backend CI is back (#11756) |
StrycekSimon
force-pushed
the
upstream/release-mcux-25.03-full/aot-example
branch
from
c7d4b49 to
7d1fa7f
Compare
|
To add the ciflow label This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows. |
robert-kalmar
force-pushed
the
upstream/release-mcux-25.03-full/aot-example
branch
from
7d1fa7f to
f129aec
Compare
robert-kalmar
force-pushed
the
upstream/release-mcux-25.03-full/aot-example
branch
from
f129aec to
c690d05
Compare
examples/nxp/README.md
Outdated
| @@ -0,0 +1,19 @@ | |||
| # PyTorch Model Delegation to Neutron Backend | |||
|
|
|||
| In this guideline we will show how to use the ExecuTorch AoT part to convert a PyTorch model to ExecuTorch format and delegate the model computation to eIQ Neutron NPU using the eIQ Neutron Backend. | |||
There was a problem hiding this comment.
Nit
| In this guideline we will show how to use the ExecuTorch AoT part to convert a PyTorch model to ExecuTorch format and delegate the model computation to eIQ Neutron NPU using the eIQ Neutron Backend. | |
| In this guide we will show how to use the ExecuTorch AoT flow to convert a PyTorch model to ExecuTorch format and delegate the model computation to eIQ Neutron NPU using the eIQ Neutron Backend. |
examples/nxp/README.md
Outdated
| --delegate --neutron_converter_flavor SDK_25_03 -m cifar10 | ||
| ``` | ||
|
|
||
| 3. It will generate you `cifar10_nxp_delegate.pte` file which can be used with the MXUXpresso SDK `cifarnet_example` project. |
There was a problem hiding this comment.
Add a link for the SDK example?
| 3. It will generate you `cifar10_nxp_delegate.pte` file which can be used with the MXUXpresso SDK `cifarnet_example` project. | |
| 3. It will generate you `cifar10_nxp_delegate.pte` file which can be used with the MCUXpresso SDK `cifarnet_example` project. |
|
Will apply your comments and then rebase. The unit tests failure tracks back to this commit: https://hud.pytorch.org/pytorch/executorch/commit/3419b46912f5c7f675879669e96f64ba11ba4129 What was resolved/mitigated by https://hud.pytorch.org/pytorch/executorch/commit/154065958093e1fcf61c1d29b4a403bff6dc7f47 . |
Co-authored-by: Martin Pavella <[email protected]>
robert-kalmar
force-pushed
the
upstream/release-mcux-25.03-full/aot-example
branch
from
c690d05 to
be74e00
Compare
|
Comment applied. CI passing. The pull / test-eval_llama-mmlu-linux / linux-job (pull_request) job failed with infrastructure problem - could not find the |
### Summary This PR add a AoT example with the eIQ Neutron Backend. The Backend is demonstrated on tiny CNN model named CifarNet, trained on Cifar10 dataset, which is part of the PR. ### Test plan Manual testing, executing the example based on steps in the Readme.md and validating the PTE on i.MX RT700 platform with the Neutron Backend runtime. Resolves #10898 cc @digantdesai @JakeStevens , @JakeStevens , @skywall , @jirioc Co-authored-by: Martin Pavella <[email protected]>
5 participants
Summary
This PR add a AoT example with the eIQ Neutron Backend. The Backend is demonstrated on tiny CNN model named CifarNet, trained on Cifar10 dataset, which is part of the PR.
Test plan
Manual testing, executing the example based on steps in the Readme.md and validating the PTE on i.MX RT700 platform with the Neutron Backend runtime.
Resolves #10898
cc @digantdesai @JakeStevens , @JakeStevens , @skywall , @jirioc