Commit 860d085
authored
[Plugin EP] Allow EP to provide additional virtual devices (microsoft#26234)
### Description
Adds APIs to allow a plugin EP to create a virtual `OrtHardwareDevice`
that can be used for model cross-compilation. For example, this allows
an EP to create a compiled model for NPU on a device that does not have
an NPU.
#### Application code
An application must explicitly allow registered plugin EPs to create
virtual devices. This is currently done by using a registration name
that ends in the `".virtual"` suffix. Ex:
```c++
#include "onnxruntime_cxx_api.h"
#include "onnxruntime_ep_device_ep_metadata_keys.h"
const char* ep_registration_name = "my_ep_lib.virtual"; // IMPORTANT: ".virtual" suffix is a signal to EP library
ort_env->RegisterExecutionProviderLibrary(ep_registration_name, "my_ep.dll");
std::vector<Ort::ConstEpDevice> ep_devices = ort_env->GetEpDevices();
// ep_devices includes an OrtEpDevice from "my_ep.dll" that uses a virtual OrtHardwareDevice.
Ort::ConstEpDevice virtual_ep_device = std::find_if(ep_devices.begin(), ep_devices.end(),
[](Ort::ConstEpDevice& device) {
return device.EpName() == std::string("MyEpName");
});
// App can look in HW metadata to check if is virtual
Ort::ConstHardwareDevice virtual_hw_device = virtual_ep_device.Device();
std::unordered_map<std::string, std::string> metadata = virtual_hw_device.Metadata().GetKeyValuePairs();
assert(metadata[kOrtHardwareDevice_MetadataKey_IsVirtual] == "1");
// App can use the virtual OrtEpDevice in a session to, for example, compile a model
// ...
```
#### Plugin EP code
This PR introduces a new _optional_ C API function in the `OrtEpFactory`
struct called `SetEnvironmentOptions` that allows ORT to pass options
(as key/value pairs) to an EP factory. Currently, the only key supported
is `"allow_virtual_devices"`, which indicates to the EP factory that
creating virtual devices is allowed.
When the application registers a plugin EP library, ORT creates the
library's EP factories and checks if they implement the
`SetEnvironmentOptions` API function. If so, ORT calls
`ep_factory.SetEnvironmentOptions` with `"allow_virtual_devices"` set to
`"1"` if the EP registration name set by the application ends in the
`".virtual"` suffix (or `"0"` otherwise).
Here's an example implementation of
`OrtEpFactory::SetEnvironmentOptions` taken from a [test plugin EP that
supports a virtual
GPU](https://github.com/microsoft/onnxruntime/tree/adrianl/plugin-ep-specify-ort-hw-device/onnxruntime/test/autoep/library/example_plugin_ep_virt_gpu):
```c++
/*static*/
OrtStatus* ORT_API_CALL EpFactoryVirtualGpu::SetEnvironmentOptionsImpl(OrtEpFactory* this_ptr,
const OrtKeyValuePairs* options) noexcept {
auto* factory = static_cast<EpFactoryVirtualGpu*>(this_ptr);
const char* value = factory->ort_api_.GetKeyValue(options, "allow_virtual_devices");
if (value != nullptr) {
factory->allow_virtual_devices_ = strcmp(value, "1") == 0;
}
return nullptr;
}
```
An EP factory can create a virtual hardware device within
`OrtEpFactory::GetSupportedDevices` by using a new API function called
`CreateHardwareDevice`. The EP factory is expected to own the hardware
device instance, which should be released when the factory is destroyed
via `ReleaseHardwareDevice`.
The [test plugin EP shows an
implementation](https://github.com/microsoft/onnxruntime/blob/d87f8b86406525f5801a7a9933b1ced1eb40940c/onnxruntime/test/autoep/library/example_plugin_ep_virt_gpu/ep_factory.cc#L86)
of `OrtEpFactory::GetSupportedDevices` that creates a virtual GPU
device.
```c++
/*static*/
OrtStatus* ORT_API_CALL EpFactoryVirtualGpu::GetSupportedDevicesImpl(OrtEpFactory* this_ptr,
const OrtHardwareDevice* const* /*devices*/,
size_t /*num_devices*/,
OrtEpDevice** ep_devices,
size_t max_ep_devices,
size_t* p_num_ep_devices) noexcept {
size_t& num_ep_devices = *p_num_ep_devices;
auto* factory = static_cast<EpFactoryVirtualGpu*>(this_ptr);
num_ep_devices = 0;
// Create a virtual OrtHardwareDevice if application indicated it is allowed (e.g., for cross-compiling).
// This example EP creates a virtual GPU OrtHardwareDevice and adds a new OrtEpDevice that uses the virtual GPU.
if (factory->allow_virtual_devices_ && num_ep_devices < max_ep_devices) {
OrtKeyValuePairs* hw_metadata = nullptr;
factory->ort_api_.CreateKeyValuePairs(&hw_metadata);
factory->ort_api_.AddKeyValuePair(hw_metadata, kOrtHardwareDevice_MetadataKey_IsVirtual, "1");
auto* status = factory->ep_api_.CreateHardwareDevice(OrtHardwareDeviceType::OrtHardwareDeviceType_GPU,
factory->vendor_id_,
/*device_id*/ 0,
factory->vendor_.c_str(),
hw_metadata,
&factory->virtual_hw_device_);
// ...
OrtEpDevice* virtual_ep_device = nullptr;
status = factory->ort_api_.GetEpApi()->CreateEpDevice(factory, factory->virtual_hw_device_, ep_metadata,
ep_options, &virtual_ep_device);
// ...
ep_devices[num_ep_devices++] = virtual_ep_device;
```
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->1 parent a958614 commit 860d085
File tree
40 files changed
+1152
-100
lines changed- cmake
- include/onnxruntime/core/session
- onnxruntime
- core/session
- plugin_ep
- test/autoep
- library
- example_plugin_ep_virt_gpu
- example_plugin_ep
40 files changed
+1152
-100
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1982 | 1982 | | |
1983 | 1983 | | |
1984 | 1984 | | |
| 1985 | + | |
| 1986 | + | |
1985 | 1987 | | |
1986 | | - | |
1987 | | - | |
| 1988 | + | |
| 1989 | + | |
| 1990 | + | |
| 1991 | + | |
1988 | 1992 | | |
1989 | 1993 | | |
1990 | 1994 | | |
| |||
1994 | 1998 | | |
1995 | 1999 | | |
1996 | 2000 | | |
1997 | | - | |
| 2001 | + | |
1998 | 2002 | | |
1999 | 2003 | | |
2000 | 2004 | | |
2001 | 2005 | | |
2002 | | - | |
| 2006 | + | |
2003 | 2007 | | |
2004 | 2008 | | |
2005 | 2009 | | |
| |||
2008 | 2012 | | |
2009 | 2013 | | |
2010 | 2014 | | |
| 2015 | + | |
| 2016 | + | |
| 2017 | + | |
| 2018 | + | |
| 2019 | + | |
| 2020 | + | |
| 2021 | + | |
| 2022 | + | |
| 2023 | + | |
| 2024 | + | |
| 2025 | + | |
| 2026 | + | |
| 2027 | + | |
| 2028 | + | |
| 2029 | + | |
| 2030 | + | |
| 2031 | + | |
| 2032 | + | |
| 2033 | + | |
| 2034 | + | |
| 2035 | + | |
| 2036 | + | |
| 2037 | + | |
| 2038 | + | |
| 2039 | + | |
| 2040 | + | |
| 2041 | + | |
| 2042 | + | |
| 2043 | + | |
| 2044 | + | |
| 2045 | + | |
| 2046 | + | |
| 2047 | + | |
| 2048 | + | |
2011 | 2049 | | |
| 2050 | + | |
2012 | 2051 | | |
2013 | 2052 | | |
2014 | 2053 | | |
| |||
2041 | 2080 | | |
2042 | 2081 | | |
2043 | 2082 | | |
2044 | | - | |
| 2083 | + | |
2045 | 2084 | | |
2046 | 2085 | | |
2047 | 2086 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
465 | 465 | | |
466 | 466 | | |
467 | 467 | | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
468 | 495 | | |
469 | 496 | | |
470 | 497 | | |
| |||
981 | 1008 | | |
982 | 1009 | | |
983 | 1010 | | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
984 | 1040 | | |
985 | 1041 | | |
986 | 1042 | | |
| |||
Lines changed: 8 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
746 | 746 | | |
747 | 747 | | |
748 | 748 | | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
749 | 772 | | |
750 | 773 | | |
751 | 774 | | |
| |||
772 | 795 | | |
773 | 796 | | |
774 | 797 | | |
| 798 | + | |
| 799 | + | |
775 | 800 | | |
776 | 801 | | |
777 | 802 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
| 8 | + | |
7 | 9 | | |
8 | 10 | | |
9 | 11 | | |
| |||
205 | 207 | | |
206 | 208 | | |
207 | 209 | | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
208 | 236 | | |
209 | 237 | | |
210 | 238 | | |
| |||
231 | 259 | | |
232 | 260 | | |
233 | 261 | | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
234 | 265 | | |
235 | 266 | | |
236 | 267 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
43 | 50 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
| 7 | + | |
6 | 8 | | |
7 | 9 | | |
8 | 10 | | |
| |||
13 | 15 | | |
14 | 16 | | |
15 | 17 | | |
16 | | - | |
| 18 | + | |
| 19 | + | |
17 | 20 | | |
18 | 21 | | |
19 | 22 | | |
| |||
29 | 32 | | |
30 | 33 | | |
31 | 34 | | |
| 35 | + | |
32 | 36 | | |
33 | 37 | | |
34 | 38 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
90 | 94 | | |
91 | 95 | | |
92 | 96 | | |
| |||
Lines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
86 | 91 | | |
87 | 92 | | |
88 | 93 | | |
| |||
Lines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
85 | 90 | | |
86 | 91 | | |
87 | 92 | | |
| |||
0 commit comments