Skip to content

Commit 5915ac2

Browse files
akodankaakodanka
committed
Add command-line flags for Intel OpenVINO options
Implement command-line flag support to configure Intel OpenVINO settings. Users can now set device type, performance mode, and custom properties via flags. Documentation: litert/tools/flags/vendors/README_intel_openvino_flags.md Sample usage: litert/tools/flags/vendors/intel_openvino_flags_example.cc Signed-off-by: Anoob Anto Kodankandath <[email protected]>
1 parent d1daedd commit 5915ac2

File tree

6 files changed

+743
-0
lines changed

6 files changed

+743
-0
lines changed

‎litert/tools/flags/vendors/BUILD‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,3 +185,62 @@ cc_test(
185185
"@com_google_googletest//:gtest_main",
186186
],
187187
)
188+
189+
# INTEL OPENVINO ##############################################################################
190+
191+
cc_library(
192+
name = "intel_openvino_flags",
193+
srcs = ["intel_openvino_flags.cc"],
194+
hdrs = ["intel_openvino_flags.h"],
195+
copts = [
196+
"-DINCLUDE_INTEL_OPENVINO_COMPILE_FLAGS",
197+
"-DINCLUDE_INTEL_OPENVINO_RUNTIME_FLAGS",
198+
],
199+
deps = [
200+
"//litert/c/options:litert_intel_openvino_options",
201+
"//litert/cc:litert_expected",
202+
"//litert/cc:litert_macros",
203+
"//litert/cc/options:litert_intel_openvino_options",
204+
"@com_google_absl//absl/flags:flag",
205+
"@com_google_absl//absl/strings:string_view",
206+
],
207+
alwayslink = 1,
208+
)
209+
210+
cc_library(
211+
name = "intel_openvino_flags_with_dynamic_runtime",
212+
srcs = ["intel_openvino_flags.cc"],
213+
hdrs = ["intel_openvino_flags.h"],
214+
copts = [
215+
"-DINCLUDE_INTEL_OPENVINO_COMPILE_FLAGS",
216+
"-DINCLUDE_INTEL_OPENVINO_RUNTIME_FLAGS",
217+
],
218+
deps = [
219+
"//litert/cc:litert_api_with_dynamic_runtime",
220+
"//litert/cc:litert_expected",
221+
"//litert/cc:litert_macros",
222+
"//litert/cc/options:litert_intel_openvino_options",
223+
"@com_google_absl//absl/flags:flag",
224+
"@com_google_absl//absl/strings:string_view",
225+
],
226+
alwayslink = 1,
227+
)
228+
229+
cc_test(
230+
name = "intel_openvino_flags_test",
231+
srcs = ["intel_openvino_flags_test.cc"],
232+
copts = [
233+
"-DINCLUDE_INTEL_OPENVINO_COMPILE_FLAGS",
234+
"-DINCLUDE_INTEL_OPENVINO_RUNTIME_FLAGS",
235+
],
236+
deps = [
237+
":intel_openvino_flags",
238+
"//litert/c/options:litert_intel_openvino_options",
239+
"//litert/cc:litert_expected",
240+
"//litert/cc/options:litert_intel_openvino_options",
241+
"@com_google_absl//absl/flags:flag",
242+
"@com_google_absl//absl/flags:marshalling",
243+
"@com_google_absl//absl/strings:string_view",
244+
"@com_google_googletest//:gtest_main",
245+
],
246+
)

‎litert/tools/flags/vendors/README_intel_openvino_flags.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# Intel OpenVINO Flags for LiteRT
2+
3+
## Overview
4+
5+
The Intel OpenVINO flags allow users to configure Intel OpenVINO options via command-line arguments, making it easy to tune inference parameters without recompiling code.
6+
7+
## Files Structure
8+
9+
```
10+
litert/tools/flags/vendors/
11+
├── intel_openvino_flags.h # Header with flag declarations
12+
├── intel_openvino_flags.cc # Implementation with flag definitions
13+
├── intel_openvino_flags_test.cc # Unit tests
14+
├── intel_openvino_flags_example.cc # Usage example
15+
└── README_intel_openvino_flags.md # This file
16+
```
17+
18+
## Available Flags
19+
20+
### Device Configuration
21+
22+
| Flag | Type | Default | Description |
23+
|------|------|---------|-------------|
24+
| `--intel_openvino_device_type` | enum | `npu` | Target device (`cpu`, `gpu`, `npu`, `auto`) |
25+
26+
### Performance Configuration
27+
28+
| Flag | Type | Default | Description |
29+
|------|------|---------|-------------|
30+
| `--intel_openvino_performance_mode` | enum | `latency` | Performance mode (`latency`, `throughput`, `cumulative_throughput`) |
31+
32+
### Advanced Configuration
33+
34+
| Flag | Type | Default | Description |
35+
|------|------|---------|-------------|
36+
| `--intel_openvino_configs_map` | string | `""` | Comma-separated key=value pairs for OpenVINO configuration properties (e.g., `INFERENCE_PRECISION_HINT=f16,CACHE_DIR=/tmp/cache`) |
37+
38+
## Usage Examples
39+
40+
### Command Line Usage
41+
42+
```bash
43+
# Basic NPU inference
44+
./your_binary --intel_openvino_device_type=npu
45+
46+
# High throughput NPU inference
47+
./your_binary \
48+
--intel_openvino_device_type=npu \
49+
--intel_openvino_performance_mode=throughput
50+
51+
# Advanced configuration with custom OpenVINO properties
52+
./your_binary \
53+
--intel_openvino_device_type=npu \
54+
--intel_openvino_configs_map="NPU_COMPILATION_MODE_PARAMS=compute-layers-with-higher-precision=Sigmoid,CACHE_DIR=/tmp/ov_cache"
55+
56+
# GPU with low precision inference
57+
./your_binary \
58+
--intel_openvino_device_type=gpu \
59+
--intel_openvino_configs_map="INFERENCE_PRECISION_HINT=f16"
60+
```
61+
62+
### Programmatic Usage
63+
64+
```cpp
65+
#include "litert/tools/flags/vendors/intel_openvino_flags.h"
66+
#include "absl/flags/parse.h"
67+
68+
int main(int argc, char** argv) {
69+
// Parse command line flags
70+
absl::ParseCommandLine(argc, argv);
71+
72+
// Create options from parsed flags
73+
auto options_result = litert::intel_openvino::IntelOpenVinoOptionsFromFlags();
74+
if (options_result) {
75+
auto options = std::move(options_result.Value());
76+
77+
// Use options to configure Intel OpenVINO
78+
if (options.GetDeviceType() == kLiteRtIntelOpenVinoDeviceTypeNPU) {
79+
// Configure NPU-specific settings
80+
}
81+
}
82+
83+
return 0;
84+
}
85+
```
86+
87+
## Flag Value Details
88+
89+
### Device Types
90+
- **`cpu`**: Execute on Intel CPU (best compatibility)
91+
- **`gpu`**: Execute on Intel GPU (good for parallel workloads)
92+
- **`npu`**: Execute on Intel NPU (best efficiency for AI workloads)
93+
- **`auto`**: Let OpenVINO choose the best available device
94+
95+
### Performance Modes
96+
- **`latency`**: Optimize for single inference latency
97+
- **`throughput`**: Optimize for batch processing throughput
98+
- **`cumulative_throughput`**: Optimize for multiple concurrent requests
99+
100+
### Configuration Map
101+
The `--intel_openvino_configs_map` flag allows you to pass arbitrary OpenVINO configuration properties as comma-separated key=value pairs. These properties are passed directly to the OpenVINO Core's `set_property()` method.
102+
103+
**Common Configuration Properties:**
104+
- `INFERENCE_PRECISION_HINT=f16`: Suggest inference precision (f16, f32, bf16)
105+
- `NPU_COMPILATION_MODE_PARAMS=<params>`: NPU-specific compilation parameters
106+
- `CACHE_DIR=/path/to/cache`: Directory for model caching
107+
- `PERFORMANCE_HINT=LATENCY`: Performance hint (alternative to performance_mode flag)
108+
- `NUM_STREAMS=4`: Number of parallel inference streams
109+
- `ENABLE_PROFILING=YES`: Enable performance profiling
110+
111+
**Example:**
112+
```bash
113+
./your_binary --intel_openvino_configs_map="INFERENCE_PRECISION_HINT=f16,CACHE_DIR=/tmp/ov_cache,NUM_STREAMS=2"
114+
```
115+
116+
## Flag Parsing and Validation
117+
118+
The flags support automatic parsing and validation:
119+
120+
```cpp
121+
// Valid device type strings
122+
"cpu", "gpu", "npu", "auto"
123+
124+
// Valid performance mode strings
125+
"latency", "throughput", "cumulative_throughput"
126+
```
127+
128+
Invalid flag values will result in parsing errors with descriptive error messages.
129+
130+
## Build Integration
131+
132+
To use Intel OpenVINO flags in your target:
133+
134+
```bazel
135+
cc_binary(
136+
name = "your_binary",
137+
srcs = ["your_binary.cc"],
138+
deps = [
139+
"//litert/tools/flags/vendors:intel_openvino_flags",
140+
# other dependencies
141+
],
142+
)
143+
```
144+
145+
With dynamic runtime support:
146+
147+
```bazel
148+
cc_binary(
149+
name = "your_binary",
150+
srcs = ["your_binary.cc"],
151+
deps = [
152+
"//litert/tools/flags/vendors:intel_openvino_flags_with_dynamic_runtime",
153+
# other dependencies
154+
],
155+
)
156+
```
157+
158+
## Testing
159+
160+
Run the flag tests to verify functionality:
161+
162+
```bash
163+
bazel test //litert/tools/flags/vendors:intel_openvino_flags_test
164+
```
165+
166+
The tests verify:
167+
- Flag parsing for all enum types
168+
- Default value handling
169+
- Option creation from flags
170+
- Error handling for invalid values
171+
- Flag reset functionality
172+
173+
## Artificial Intelligence
174+
175+
These contents may have been developed with support from one or more Intel-operated generative artificial intelligence solutions.

‎litert/tools/flags/vendors/intel_openvino_flags.cc‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
// Copyright (C) 2025 Intel Corporation
2+
// SPDX-License-Identifier: Apache-2.0
3+
//
4+
// Licensed under the Apache License, Version 2.0 (the "License");
5+
// you may not use this file except in compliance with the License.
6+
// You may obtain a copy of the License at
7+
//
8+
// http://www.apache.org/licenses/LICENSE-2.0
9+
//
10+
// Unless required by applicable law or agreed to in writing, software
11+
// distributed under the License is distributed on an "AS IS" BASIS,
12+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
// See the License for the specific language governing permissions and
14+
// limitations under the License.
15+
16+
#include "litert/tools/flags/vendors/intel_openvino_flags.h"
17+
18+
#include <string>
19+
#include <vector>
20+
21+
#include "absl/flags/flag.h" // from @com_google_absl
22+
#include "absl/strings/str_split.h" // from @com_google_absl
23+
#include "absl/strings/string_view.h" // from @com_google_absl
24+
#include "litert/c/litert_logging.h"
25+
#include "litert/c/options/litert_intel_openvino_options.h"
26+
#include "litert/cc/litert_expected.h"
27+
#include "litert/cc/litert_macros.h"
28+
#include "litert/cc/options/litert_intel_openvino_options.h"
29+
30+
// NOLINTBEGIN(*alien-types*)
31+
// TODO: Move absl parse/unparse function to same file as enum types if
32+
// it becomes an issue.
33+
34+
ABSL_FLAG(LiteRtIntelOpenVinoDeviceType, intel_openvino_device_type,
35+
kLiteRtIntelOpenVinoDeviceTypeNPU,
36+
"Device type for Intel OpenVINO inference (cpu, gpu, npu, auto).");
37+
38+
ABSL_FLAG(LiteRtIntelOpenVinoPerformanceMode, intel_openvino_performance_mode,
39+
kLiteRtIntelOpenVinoPerformanceModeLatency,
40+
"Performance mode for Intel OpenVINO inference (latency, throughput, cumulative_throughput).");
41+
42+
ABSL_FLAG(std::string, intel_openvino_configs_map, "",
43+
"Configuration options for Intel OpenVINO as comma-separated key=value pairs "
44+
"(e.g., 'INFERENCE_PRECISION_HINT=f16,CACHE_DIR=/tmp/cache').");
45+
46+
bool AbslParseFlag(absl::string_view text,
47+
LiteRtIntelOpenVinoDeviceType* options,
48+
std::string* error) {
49+
if (text == "cpu") {
50+
*options = kLiteRtIntelOpenVinoDeviceTypeCPU;
51+
return true;
52+
}
53+
54+
if (text == "gpu") {
55+
*options = kLiteRtIntelOpenVinoDeviceTypeGPU;
56+
return true;
57+
}
58+
59+
if (text == "npu") {
60+
*options = kLiteRtIntelOpenVinoDeviceTypeNPU;
61+
return true;
62+
}
63+
64+
if (text == "auto") {
65+
*options = kLiteRtIntelOpenVinoDeviceTypeAUTO;
66+
return true;
67+
}
68+
69+
*error = "Unknown Intel OpenVINO device type";
70+
return false;
71+
}
72+
73+
std::string AbslUnparseFlag(LiteRtIntelOpenVinoDeviceType options) {
74+
switch (options) {
75+
case kLiteRtIntelOpenVinoDeviceTypeCPU:
76+
return "cpu";
77+
case kLiteRtIntelOpenVinoDeviceTypeGPU:
78+
return "gpu";
79+
case kLiteRtIntelOpenVinoDeviceTypeNPU:
80+
return "npu";
81+
case kLiteRtIntelOpenVinoDeviceTypeAUTO:
82+
return "auto";
83+
}
84+
}
85+
86+
bool AbslParseFlag(absl::string_view text,
87+
LiteRtIntelOpenVinoPerformanceMode* options,
88+
std::string* error) {
89+
if (text == "latency") {
90+
*options = kLiteRtIntelOpenVinoPerformanceModeLatency;
91+
return true;
92+
}
93+
if (text == "throughput") {
94+
*options = kLiteRtIntelOpenVinoPerformanceModeThroughput;
95+
return true;
96+
}
97+
if (text == "cumulative_throughput") {
98+
*options = kLiteRtIntelOpenVinoPerformanceModeCumulativeThroughput;
99+
return true;
100+
}
101+
102+
*error = "Unknown Intel OpenVINO performance mode";
103+
return false;
104+
}
105+
106+
std::string AbslUnparseFlag(LiteRtIntelOpenVinoPerformanceMode options) {
107+
switch (options) {
108+
case kLiteRtIntelOpenVinoPerformanceModeLatency:
109+
return "latency";
110+
case kLiteRtIntelOpenVinoPerformanceModeThroughput:
111+
return "throughput";
112+
case kLiteRtIntelOpenVinoPerformanceModeCumulativeThroughput:
113+
return "cumulative_throughput";
114+
}
115+
}
116+
117+
// NOLINTEND(*alien-types*)
118+
119+
namespace litert::intel_openvino {
120+
121+
Expected<IntelOpenVinoOptions> IntelOpenVinoOptionsFromFlags() {
122+
LITERT_ASSIGN_OR_RETURN(auto options, IntelOpenVinoOptions::Create());
123+
124+
options.SetDeviceType(
125+
absl::GetFlag(FLAGS_intel_openvino_device_type));
126+
options.SetPerformanceMode(
127+
absl::GetFlag(FLAGS_intel_openvino_performance_mode));
128+
129+
// Parse configs map options from the flag (comma-separated key=value pairs)
130+
const std::string configs_map_str = absl::GetFlag(FLAGS_intel_openvino_configs_map);
131+
if (!configs_map_str.empty()) {
132+
std::vector<std::string> config_pairs = absl::StrSplit(configs_map_str, ',');
133+
for (const auto& pair : config_pairs) {
134+
std::vector<std::string> key_value = absl::StrSplit(pair, '=');
135+
if (key_value.size() == 2) {
136+
options.SetConfigsMapOption(key_value[0].c_str(), key_value[1].c_str());
137+
} else {
138+
LITERT_LOG(LITERT_WARNING,
139+
"Ignoring malformed config pair: '%s'. Expected format: key=value",
140+
pair.c_str());
141+
}
142+
}
143+
}
144+
145+
return options;
146+
}
147+
148+
} // namespace litert::intel_openvino

0 commit comments

Comments
 (0)