[EP ABI] Initial support for kernel-based EPs

[adrianlizarraga]

Description

This PR adds an initial set of C APIs necessary to support kernel registration for plugin EPs.

Example use

The example plugin EP implementation now registers MemcpyFromHost and MemcpyToHost operator kernels using the new APIs. New utilities in the example implementation make the process of defining operator kernels very similar to the existing process used by provider-bridge EPs.

First, the operator kernel class is defined:

// File: onnxruntime/test/autoep/library/kernels/memcpy.h
struct Memcpy : public OrtKernelImpl {
  static OrtStatus* Create(const OrtKernelInfo* info, void* state, /*out*/ std::unique_ptr<Memcpy>& kernel);

  Memcpy(const OrtKernelInfo* info, void* state);

  static OrtStatus* ORT_API_CALL ComputeImpl(OrtKernelImpl* this_ptr, OrtKernelContext* kernel_ctx) noexcept;
  static void ORT_API_CALL ReleaseImpl(OrtKernelImpl* this_ptr) noexcept;

  OrtStatus* DoCompute(OrtKernelContext* kernel_ctx) noexcept;

 private:
  const OrtKernelInfo* info_;
  void* state_;  // Custom state passed from OrtEp
};

Then, a macro defines a function that can be called to register the operator with the EP's kernel registry:

// File: onnxruntime/test/autoep/library/kernels/memcpy.cc
ONNX_OPERATOR_KERNEL_EX(
    MemcpyFromHost,
    kOnnxDomain,
    1,
    (Ort::KernelDefBuilder()
         .SetInputMemType(0, OrtMemType::OrtMemTypeCPUInput)
         .AddTypeConstraint("T", MLDataTypes::GetTensorType(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT))),
    Memcpy)

ONNX_OPERATOR_KERNEL_EX(
    MemcpyToHost,
    kOnnxDomain,
    1,
    (Ort::KernelDefBuilder()
         .SetOutputMemType(0, OrtMemType::OrtMemTypeCPUOutput)
         .AddTypeConstraint("T", MLDataTypes::GetTensorType(ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT))),
    Memcpy)

Lastly, the functions defined by the above macro are entered into a table:

// File: onnxruntime/test/autoep/library/ep_kernel_registration.cc

// Include kernel files:
#include "kernels/memcpy.h"

// Forward declarations of kernel classes used as template args for BuildKernelCreateInfo
class ONNX_OPERATOR_KERNEL_CLASS_NAME(kOnnxDomain, 1, MemcpyFromHost);
class ONNX_OPERATOR_KERNEL_CLASS_NAME(kOnnxDomain, 1, MemcpyToHost);

// Table of BuildKernelCreateInfo functions for each operator
static const BuildKernelCreateInfoFn build_kernel_create_info_funcs[] = {
    BuildKernelCreateInfo<void>,  // Dummy to avoid table becoming empty.
    BuildKernelCreateInfo<ONNX_OPERATOR_KERNEL_CLASS_NAME(kOnnxDomain, 1, MemcpyFromHost)>,
    BuildKernelCreateInfo<ONNX_OPERATOR_KERNEL_CLASS_NAME(kOnnxDomain, 1, MemcpyToHost)>,
};

The example EP processes the entries in the above table to add information about the supported operator kernels to the EP's kernel registry (OrtKernelRegistry).

Additionally, during the call to OrtEp::GetCapability, an EP can now lookup registered kernel definitions via the new API EpGraphSupportInfo_LookUpKernel. Note that an EP would not normally lookup kernels for Memcpy**Host, which are inserted by ORT. Instead, it would be used to look up other registered operator kernels like Conv, for example.

static OrtStatus* ORT_API_CALL GetCapabilityImpl(OrtEp* this_ptr, const OrtGraph* graph,
                                                           OrtEpGraphSupportInfo* graph_support_info) noexcept {
  // ...

  for (const OrtNode* node : nodes) {
    const OrtKernelDef* kernel_def = nullptr;
    OrtStatus* status = this_ep->ep_api->EpGraphSupportInfo_LookUpKernel(graph_support_info, node, &kernel_def);

    if (status != nullptr) {
      return status;
    }

    if (kernel_def != nullptr) {  // Take node if this EP has a registered kernel for it.
      if (OrtStatus* st = this_ep->ep_api->EpGraphSupportInfo_AddSingleNode(graph_support_info, node);
          st != nullptr) {
        return st;
      }
    }
  }

  return nullptr;
}

EP implementation details

An EP instance (i.e., OrtEp) that needs to register operator kernels with ONNX Runtime must implement the following OrtEp::GetKernelRegistry() function:

Function Signature	Description
GetKernelRegistry Returns:OrtStatus* Parameters: OrtEp* this_ptr: The OrtEp instance. const OrtKernelRegistry** kernel_registry: Output parameter set to the EP's kernel registry, which must remain valid throughout the lifetime of the EP.	Gets the execution provider's kernel registry, if any. Remarks: A kernel registry contains kernel creation information for operator kernels supported by an EP. Note: Implementation of this function is optional. If set to NULL, ORT assumes the EP compiles nodes.

If defined by the EP, the OrtEp::GetKernelRegistry() function is called by ONNX Runtime after creating an instance of the OrtEp in order to retrieve the EP's kernel registry.

APIs used by EP to add entries to kernel registry

An EP's kernel registry (OrtKernelRegistry) contains information necessary for the (later) creation of operator kernels supported by an EP. Conceptually, a kernel registry contains an array of "kernel creation information" elements, one per operator. Each such element consists of:

• A kernel definition (OrtKernelDef), which specifies operator type, supported versions, type constraints, I/O memory types, etc.
• A function of type OrtKernelCreateFunc that ORT calls to create an instance of the kernel (OrtKernelImpl).
• Custom opaque state (provided by the OrtEp) that is passed to the OrtKernelCreateFunc.

An EP uses the following OrtEpApi::KernelRegistry_AddKernel() function to add an entry for one supported operator.

Function Signature

Description

KernelRegistry_AddKernel Returns:OrtStatus* Parameters: OrtKernelRegistry* kernel_registry: The OrtKernelRegistry instance. const OrtKernelDef* kernel_def: The kernel definition, which includes operator type, version, EP name, type constraints, etc. OrtKernelCreateFunc kernel_create_func: Function that creates an instance of the operator kernel as a OrtKernelImpl instance. void* kernel_create_func_state: Custom state passed to the kernel creation function. Can be null.

Adds kernel creation information for a supported operator kernel to the given kernel registry. Remarks: Refer to OrtEp::GetKernelRegistry, which returns an EP's kernel registry to ORT.

Building a kernel definition

An EP uses a kernel definition builder (OrtKernelDefBuilder) to create a kernel definition (OrtKernelDef). The following table lists some of the C APIs related to building a kernel definition. The above ONNX_OPERATOR_KERNEL_EX macro uses these APIs.

Function Signature	Description
KernelDefBuilder_SetOperatorType Returns:OrtStatus* Parameters: OrtKernelDefBuilder* kernel_def_builder: The OrtKernelDefBuilder instance. const char* op_type: A null-terminated string representing the operator type.	Sets the kernel's operator type.
KernelDefBuilder_SetDomain Returns:OrtStatus* Parameters: OrtKernelDefBuilder* kernel_def_builder: The OrtKernelDefBuilder instance. const char* domain: A null-terminated string representing the operator's domain.	Sets the kernel's domain.
...	...
KernelDefBuilder_Build Returns:OrtStatus* Parameters: OrtKernelDefBuilder* kernel_def_builder: The OrtKernelDefBuilder instance. OrtKernelDef** kernel_def_out: The new OrtKernelDef instance.	Creates a OrtKernelDef instance from the given kernel definition builder.

Defining a kernel implementation

An EP defines a kernel implementation by initializing an instance of OrtKernelImpl (shown below) with function pointers for computation, release, etc.

struct OrtKernelImpl {
  uint32_t ort_version_supported;  ///< Must be initialized to ORT_API_VERSION

  /** \brief Computation function called to execute the kernel on an EP.
   *
   * \param[in] this_ptr The OrtKernelImpl instance.
   * \param[in] context The OrtKernelContext instance that provides access to the inputs and outputs.
   *
   * \snippet{doc} snippets.dox OrtStatus Return Value
   *
   * \since Version 1.24.
   */
  ORT_API2_STATUS(Compute, _In_ OrtKernelImpl* this_ptr, _In_ OrtKernelContext* context);

  /** \brief Called by ORT to release the OrtKernelImpl instance and its resources.
   *
   * \param[in] this_ptr The OrtKernelImpl instance.
   *
   * \since Version 1.24.
   */
  ORT_API_T(void, Release, _In_ OrtKernelImpl* this_ptr);
};

As shown previously, the example EP creates a Memcpy class that inherits from OrtKernelImpl and implements the above functions.

Defining a kernel creation function

An EP must provide a function of type OrtKernelCreateFunc that ORT can later call to create an instance of a kernel (OrtKernelImpl). The signature of the OrtKernelCreateFunc is shown below.

/** \brief Type definition for a function that creates an OrtKernelImpl instance for an operator kernel.
 *
 * \param[in] ctx Unused/reserved for future use.
 * \param[in] kernel_create_func_state Opaque state initially provided by the EP that registered the kernel.
 *                                     Refer to OrtEpApi::KernelRegistry_AddKernel(). May be null.
 * \param[in] info The OrtKernelInfo instance that provides access to the kernel's input and output characteristics.
 * \param[out] kernel_out Output parameter set to the new OrtKernelImpl instance.
 *
 * \snippet{doc} snippets.dox OrtStatus Return Value
 *
 * \since Version 1.24.
 */
typedef OrtStatus*(ORT_API_CALL* OrtKernelCreateFunc)(_In_ OrtKernelCreateContext* ctx,  // unused/reserved as of 1.24
                                                      _In_ void* kernel_create_func_state,
                                                      _In_ const OrtKernelInfo* info,
                                                      _Outptr_result_maybenull_ OrtKernelImpl** kernel_out);

The example EP declares kernel creation functions via use of the previously mentioned ONNX_OPERATOR_KERNEL_EX macro. If one were to expand the macro call, the kernel creation function for MemcpyFromHost would look similar to the following snippet:

OrtStatus* ORT_API_CALL CreateMemcpyKernel(OrtKernelCreateContext* /*ctx*/, void* kernel_create_func_state,
                                           const OrtKernelInfo* info, OrtKernelImpl** kernel_out) {
  *kernel_out = nullptr;

  std::unique_ptr<Memcpy> kernel;
  RETURN_IF_ERROR(Memcpy::Create(info, kernel_create_func_state, kernel));

  *kernel_out = kernel.release();
  return nullptr;
}

Motivation and Context

[Copilot]

Pull Request Overview

This draft PR implements support for kernel-based execution providers (EPs) within the ONNX Runtime EP plugin architecture. The changes enable plugin EPs to register custom kernels directly with the ORT runtime, expanding beyond the current node-based computation model.

• Adds comprehensive kernel registration infrastructure for plugin EPs
• Implements memory copy kernels as examples (MemcpyFromHost/MemcpyToHost)
• Extends the EP API with kernel definition and creation functionality

Reviewed Changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
onnxruntime/test/framework/ep_plugin_provider_test.cc	Updates test to pass kernel registry parameter
onnxruntime/test/autoep/library/kernels/utils.h	Defines kernel creation utilities and macros
onnxruntime/test/autoep/library/kernels/memcpy.h	Declares example Memcpy kernel interface
onnxruntime/test/autoep/library/kernels/memcpy.cc	Implements example Memcpy kernel with registration
onnxruntime/test/autoep/library/kernels/data_types.h	Declares MLDataTypes singleton for type management
onnxruntime/test/autoep/library/kernels/data_types.cc	Implements MLDataTypes for tensor type retrieval
onnxruntime/test/autoep/library/ep_kernel_registration.h	Declares kernel registration functions
onnxruntime/test/autoep/library/ep_kernel_registration.cc	Implements kernel registration logic
onnxruntime/test/autoep/library/ep.h	Adds kernel creation method declarations to EP
onnxruntime/test/autoep/library/ep.cc	Implements kernel creation methods in example EP
onnxruntime/core/session/utils.h	Declares CopyTensors utility function
onnxruntime/core/session/utils.cc	Implements CopyTensors utility function
onnxruntime/core/session/provider_policy_context.cc	Updates EP creation to use new factory method
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.h	Extends PluginExecutionProvider with kernel registry support
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc	Implements kernel registry initialization in plugin EP
onnxruntime/core/session/plugin_ep/ep_kernel_registration.h	Declares kernel registration infrastructure
onnxruntime/core/session/plugin_ep/ep_kernel_registration.cc	Implements plugin EP kernel wrapper and registration
onnxruntime/core/session/plugin_ep/ep_api.h	Declares new EP API functions for kernel support
onnxruntime/core/session/plugin_ep/ep_api.cc	Implements new EP API functions for kernel support
onnxruntime/core/session/onnxruntime_c_api.cc	Refactors CopyTensors to use shared utility
include/onnxruntime/core/session/onnxruntime_ep_c_api.h	Adds kernel-related types and API declarations
include/onnxruntime/core/session/onnxruntime_cxx_inline.h	Implements C++ wrapper methods for kernel APIs
include/onnxruntime/core/session/onnxruntime_cxx_api.h	Declares C++ KernelDefBuilder class
cmake/onnxruntime_unittests.cmake	Updates build to include kernel source files

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[yuslepukhin]

If any of the information for any getters is optional, suggest returning a status

[adrianlizarraga]

The underlying C API function OrtEpApi::KernelDef_GetExecutionProvider returns the const char* directly (doesn't return a status).

Sorry, I don't fully understand the comment. Is the request to make this return a status instead? What is meant by "information for any getters is optional"?

[yuslepukhin]

This was a general comment not to be bound to throwing exceptions by default. In this case, the data is returned without error reporting.

[adrianlizarraga]

Note: moved this into a shared utility function that can be used by the new API KernelInfo_CopyTensors

[adrianlizarraga]

We can probably remove this ctx parameter. It is a stand-in for the FuncManager parameter in the related KernelCreateFn used by provider-bridge EPs:

onnxruntime/include/onnxruntime/core/framework/op_kernel.h

Line 143 in 8af9f58

using KernelCreateFn = std::function<Status(FuncManager& func_mgr, const OpKernelInfo& info, std::unique_ptr<OpKernel>& out)>;

It doesn't look like any EPs in the ORT code base use the FuncManager parameter at all, but I kept it here (with a more generic name) just in case we find a use for it in the future. Would appreciate opinions.

[adrianlizarraga]

Considering moving this to the public C++ API header (but not a singleton). Seems like all kernel-based plugin EPs would benefit from this.

[adrianlizarraga]

I currently only added an API to get tensor data types. We would need to add similar APIs for sequences, maps, etc.

Also, I'm not too sure if we should keep using the term "ML data type". I kept it to remain consistent with the internal names, but perhaps we can rename?

[adrianlizarraga]

This PR does not yet add all KernelDefBuilder functions. It's missing aliasing, "may inplace". However, these things may not be used commonly and could be added later.

[adrianlizarraga]

I also have not added all getters for KernelDef because they are not really used by EPs. An EP retrieves a kernel def during GetCapability to check if a kernel for a node has been registered. Notably, there is only one EP (ACL EP) that actually gets a property from a KernelDef returned by a lookup, and that property is the operator type, which it could instead get from the node.

[edgchen1]

does this EP need to register its own memcpy kernels or can it use the generic ones from the CPU EP (added in #26088)?

[adrianlizarraga]

It doesn't need it. It was a way to test the kernel registration utilities. Perhaps it would be best to create a different kernel-based example EP and leave this one unchanged.

[edgchen1]

if we use this as an example EP implementation for reference, it might be better to show EP authors that they can avoid implementing their own memcpy kernels unless they require some special behavior not provided by the generic ones.

also, the testing of the generic memcpy kernels was relying on this EP not providing its own, but we could update the test set up if needed. on a semi-related note, I don't know how well OpTester will work with an EP that has both a kernel registry and support for compiling nodes.

[edgchen1]

nit: it is one data type with this overload, right?

Suggested change

	KernelDefBuilder& AddTypeConstraint(const char* arg_name, const OrtMLDataType* data_types);
	KernelDefBuilder& AddTypeConstraint(const char* arg_name, const OrtMLDataType* data_type);

[edgchen1]

is it possible to reuse the CopyTensors API or do we need a new one?

[adrianlizarraga]

The existing CopyTensors API takes an OrtEnv as input, which is not available to EPs (if I'm not mistaken)

[edgchen1]

should we check whether the lookup fails to find anything (create_info == nullptr)?

[edgchen1]

when should one use PluginExecutionProvider::PluginExecutionProvider() vs. PluginExecutionProvider::Create()?

[edgchen1]

I think the dummy entry was originally added to support reduced op builds for certain EPs. we probably don't need it in this example.

[edgchen1]

should we add a C++ API type for OrtKernelRegistry?

[edgchen1]

also check that the file was opened? ASSERT_TRUE(ifs)