OpenVINO Export Llama Support

Differential Revision: D84544001

Pull Request resolved: #14022

Author: cavusmustafa

backends/openvino/CMakeLists.txt

@@ -53,30 +53,6 @@ target_sources(
 
 executorch_target_link_options_shared_lib(openvino_backend)
 
-if(EXECUTORCH_BUILD_OPENVINO_EXECUTOR_RUNNER)
-  # Build executor runner binary for openvino backend
-  list(APPEND openvino_executor_runner_libs openvino_backend executorch)
-
-  set(_openvino_executor_runner__srcs
-      ${EXECUTORCH_ROOT}/examples/portable/executor_runner/executor_runner.cpp
-      ${EXECUTORCH_ROOT}/extension/data_loader/file_data_loader.cpp
-      ${EXECUTORCH_ROOT}/extension/evalue_util/print_evalue.cpp
-      ${EXECUTORCH_ROOT}/extension/runner_util/inputs.cpp
-      ${EXECUTORCH_ROOT}/extension/runner_util/inputs_portable.cpp
-  )
-  add_executable(openvino_executor_runner ${_openvino_executor_runner__srcs})
-
-  list(APPEND openvino_executor_runner_libs)
-
-  target_link_libraries(
-    openvino_executor_runner gflags portable_ops_lib
-    ${openvino_executor_runner_libs}
-  )
-  target_compile_options(
-    openvino_executor_runner PUBLIC ${_common_compile_options}
-  )
-endif()
-
 # Install OpenVINO backend library to the lib directory
 install(
   TARGETS openvino_backend

backends/openvino/README.md

@@ -18,6 +18,11 @@ For more information on the supported hardware, please refer to [OpenVINO System
 executorch
 ├── backends
 │   └── openvino
+│       ├── quantizer
+│           ├── observers
+│               └── nncf_observers.py
+│           ├── __init__.py
+│           └── quantizer.py
 │       ├── runtime
 │           ├── OpenvinoBackend.cpp
 │           └── OpenvinoBackend.h
@@ -42,11 +47,23 @@ executorch
 
 Before you begin, ensure you have openvino installed and configured on your system.
 
-### Build OpenVINO from Source
+### Use OpenVINO from Release Packages
+
+1. Download the OpenVINO release package from [here](https://docs.openvino.ai/2025/get-started/install-openvino.html). Make sure to select your configuration and click on **OpenVINO Archives** under the distribution section to download the appropriate archive for your platform.
+
+2. Extract the release package from the archive and set the environment variables.
+
+   ```bash
+   tar -zxf openvino_toolkit_<your_release_configuration>.tgz
+   cd openvino_toolkit_<your_release_configuration>
+   source setupvars.sh
+   ```
+
+### (Optional) Build OpenVINO from Source
 
 ```bash
 git clone https://github.com/openvinotoolkit/openvino.git
-cd openvino && git checkout b16b776ac119dafda51f69a80f1e6b7376d02c3b
+cd openvino
 git submodule update --init --recursive
 sudo ./install_build_dependencies.sh
 mkdir build && cd build
@@ -59,44 +76,45 @@ cd <your_preferred_install_location>
 source setupvars.sh
 ```
 
-### Use OpenVINO from Release Packages
-
-1. Download the OpenVINO release package from [here](https://docs.openvino.ai/2025/get-started/install-openvino.html). Make sure to select your configuration and click on **OpenVINO Archives** under the distribution section to download the appropriate archive for your platform.
-
-2. Extract the release package from the archive and set the environment variables.
-
-   ```bash
-   tar -zxf openvino_toolkit_<your_release_configuration>.tgz
-   cd openvino_toolkit_<your_release_configuration>
-   source setupvars.sh
-   ```
-
 For more information about OpenVINO build, refer to the [OpenVINO Build Instructions](https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_linux.md).
 
 ### Setup
 
 Follow the steps below to setup your build environment:
 
-1. **Setup ExecuTorch Environment**: Refer to the [Environment Setup](https://pytorch.org/executorch/main/getting-started-setup#environment-setup) guide for detailed instructions on setting up the ExecuTorch environment.
 
-2. **Setup OpenVINO Backend Environment**
-- Install the dependent libs. Ensure that you are inside `executorch/backends/openvino/` directory
+1. **Create a Virtual Environment**
+- Create a virtual environment and activate it by executing the commands below.
    ```bash
-   pip install -r requirements.txt
+   python -m venv env
+   source env/bin/activate
    ```
-  Note: To achieve optimal performance with NNCF quantization, you should install the latest development version of NNCF (version 2.16.0.dev0+191b53d9 or higher).
-3. Navigate to `scripts/` directory.
-
-4. **Build OpenVINO Backend C++ Libraries and Executor Runner**: Once the prerequisites are in place, run the `openvino_build.sh` script to start the build process. By default, OpenVINO backend will be built under `cmake-out/backends/openvino/` as `libopenvino_backend.a`
-
+2. **Clone ExecuTorch Repository from Github**
+- Clone Executorch repository by executing the command below.
    ```bash
-   ./openvino_build.sh
+   git clone --recurse-submodules https://github.com/pytorch/executorch.git
    ```
-   **Build OpenVINO Backend Python Package with Pybindings**: To build and install the OpenVINO backend Python package with Python bindings, run the `openvino_build.sh` script with the `--enable_python` argument. This will compile and install the ExecuTorch Python package with the OpenVINO backend into your Python environment. This option will also enable python bindings required to execute OpenVINO backend tests and `aot_optimize_and_infer.py` script inside `executorch/examples/openvino` folder.
-
+3. **Build ExecuTorch with OpenVINO Backend**
+- Ensure that you are inside `executorch/backends/openvino/scripts` directory. The following command builds and installs ExecuTorch with the OpenVINO backend, also compiles the C++ runtime libraries and binaries into `<executorch_root>/cmake-out` for quick inference testing.
    ```bash
+   openvino_build.sh
+   ```
+- Optionally, `openvino_build.sh` script can be used to build python package or C++ libraries/binaries seperately.
+
+   **Build OpenVINO Backend Python Package with Pybindings**: To build and install the OpenVINO backend Python package with Python bindings, run the `openvino_build.sh` script with the `--enable_python` argument as shown in the below command. This will compile and install the ExecuTorch Python package with the OpenVINO backend into your Python environment. This option will also enable python bindings required to execute OpenVINO backend tests and `aot_optimize_and_infer.py` script inside `executorch/examples/openvino` folder.
+     ```bash
    ./openvino_build.sh --enable_python
    ```
+   **Build C++ Runtime Libraries for OpenVINO Backend**: Run the `openvino_build.sh` script with the `--cpp_runtime` flag to build the C++ runtime libraries as shown in the below command. The compiled libraries files and binaries can be found in the `<executorch_root>/cmake-out` directory. The binary located at `<executorch_root>/cmake-out/executor_runner` can be used to run inference with vision models.
+     ```bash
+   ./openvino_build.sh --cpp_runtime
+   ```
+   **Build C++ Llama Runner**: First, ensure the C++ runtime libraries are built by following the earlier instructions. Then, run the `openvino_build.sh` script with the `--llama_runner flag` to compile the LlaMA runner as shown the below command, which enables executing inference with models exported using export_llama. The resulting binary is located at: `<executorch_root>/cmake-out/examples/models/llama/llama_main`
+     ```bash
+   ./openvino_build.sh --llama_runner
+   ```
+
+For more information about ExecuTorch environment setup, refer to the [Environment Setup](https://pytorch.org/executorch/main/getting-started-setup#environment-setup) guide.
 
 ### Run
 

backends/openvino/partitioner.py

@@ -26,6 +26,13 @@
 from torch.fx.passes.operator_support import OperatorSupportBase
 
 
+class PatternNode:
+    op_types: dict[str, Optional[list]] = {}
+
+    def __init__(self):
+        self.op_types = {}
+
+
 class OpenvinoOperatorsSupport(OperatorSupportBase):
     extended_support_dict = {
         "torch.ops.dim_order_ops._clone_dim_order.default": None,
@@ -36,6 +43,7 @@ def __init__(
         self,
         op_types_to_skip: Optional[set] = None,
         op_names_to_skip: Optional[set] = None,
+        enabled_ops_by_name: Optional[set] = None,
     ) -> None:
         """
         Initializes the OpenvinoOperatorsSupport class.
@@ -47,9 +55,12 @@ def __init__(
             op_types_to_skip = set()
         if op_names_to_skip is None:
             op_names_to_skip = set()
+        if enabled_ops_by_name is None:
+            enabled_ops_by_name = set()
 
         self._op_types_to_skip = op_types_to_skip
         self._op_names_to_skip = op_names_to_skip
+        self._enabled_ops_by_name = enabled_ops_by_name
 
     def is_node_supported(self, _, node: torch.fx.Node) -> bool:
         """
@@ -66,6 +77,10 @@ def is_node_supported(self, _, node: torch.fx.Node) -> bool:
             op_type = node.target.__name__
         else:
             op_type = str(node.target)
+
+        if node.name in self._enabled_ops_by_name:
+            return True
+
         supported_ops = (
             OperatorSupport(options)._support_dict | self.extended_support_dict
         )
@@ -105,6 +120,7 @@ def __init__(
         self.delegation_spec = DelegationSpec(OpenvinoBackend.__name__, compile_spec)
         self._op_types_to_skip = op_types_to_skip
         self._op_names_to_skip = op_names_to_skip
+        self._enabled_ops_by_name: set = set()
 
     def ops_to_not_decompose(
         self,
@@ -123,19 +139,87 @@ def ops_to_not_decompose(
             torch.ops.aten.upsample_bilinear2d.vec,
             torch.ops.aten.upsample_nearest2d.default,
             torch.ops.aten.upsample_nearest2d.vec,
+            torch.ops.aten.stack.default,
         ]
         return (ops_not_decompose, None)
 
+    def check_pattern(
+        self, node: torch.fx.Node, pattern: type[PatternNode], enabled_ops: list
+    ) -> bool:
+        if node.op == "call_function":
+            if ("call_function" + ":" + str(node.target.__name__)) in pattern.op_types:  # type: ignore[union-attr]
+                pt_input_nodes = node.all_input_nodes
+                pattern_input_ops = pattern.op_types[
+                    "call_function" + ":" + str(node.target.__name__)  # type: ignore[union-attr]
+                ]
+                if pattern_input_ops is None:
+                    enabled_ops.append(node)
+                    return True
+                if len(pt_input_nodes) != len(pattern_input_ops):
+                    return False
+                for i in range(len(pt_input_nodes)):
+                    if not self.check_pattern(
+                        pt_input_nodes[i], pattern_input_ops[i], enabled_ops
+                    ):
+                        return False
+                enabled_ops.append(node)
+                return True
+        elif node.op == "get_attr":
+            if "get_attr" in pattern.op_types:
+                return True
+            else:
+                return False
+        elif node.op == "placeholder":
+            if "placeholder" in pattern.op_types:
+                return True
+            else:
+                return False
+        return False
+
+    def capture_nncf_patterns(self, graph_module: torch.fx.GraphModule):
+        const_node = PatternNode
+        const_node.op_types["get_attr"] = None
+        const_node.op_types["placeholder"] = None
+        bitwise_right_shift_node = PatternNode
+        bitwise_right_shift_node.op_types[
+            "call_function:aten.bitwise_right_shift.Tensor_Scalar"
+        ] = [const_node]
+        bitwise_and_node = PatternNode
+        bitwise_and_node.op_types["call_function:aten.bitwise_and.Scalar"] = [
+            const_node
+        ]
+        stack_node = PatternNode
+        stack_node.op_types["call_function:aten.stack.default"] = [
+            bitwise_and_node,
+            bitwise_right_shift_node,
+        ]
+
+        for node in graph_module.graph.nodes:
+            if (
+                str(node.op) == "call_function"
+                and str(node.target.__name__) == "aten.stack.default"
+            ):
+                enabled_ops: list = []
+                pattern_match = self.check_pattern(node, stack_node, enabled_ops)
+                if pattern_match:
+                    for pattern_op in enabled_ops:
+                        self._enabled_ops_by_name.add(pattern_op.name)
+
     def partition(self, exported_program: ExportedProgram) -> PartitionResult:
         """
         Partitions an exported program into supported and unsupported segments.
 
         :param exported_program: The exported program.
         :return: A PartitionResult containing the partitioned graph and delegation tags.
         """
+        self.capture_nncf_patterns(exported_program.graph_module)
         partitioner = CapabilityBasedPartitioner(
             exported_program.graph_module,
-            OpenvinoOperatorsSupport(self._op_types_to_skip, self._op_names_to_skip),
+            OpenvinoOperatorsSupport(
+                self._op_types_to_skip,
+                self._op_names_to_skip,
+                self._enabled_ops_by_name,
+            ),
             allows_single_node_partition=True,
         )
         partition_list = partitioner.propose_partitions()

backends/openvino/quantizer/__init__.py

@@ -1,3 +1,3 @@
-from .quantizer import OpenVINOQuantizer, quantize_model
+from .quantizer import OpenVINOQuantizer, QuantizationMode, quantize_model
 
-__all__ = ["OpenVINOQuantizer", "quantize_model"]
+__all__ = ["OpenVINOQuantizer", "quantize_model", "QuantizationMode"]