Skip to content

[EP ABI] Add CreateCustomOpDomains() API for plugin EP to register custom ops - #26759

Closed
chilo-ms wants to merge 52 commits into
mainfrom
chi/custom_op_for_ep
Closed

[EP ABI] Add CreateCustomOpDomains() API for plugin EP to register custom ops #26759
chilo-ms wants to merge 52 commits into
mainfrom
chi/custom_op_for_ep

Conversation

@chilo-ms

@chilo-ms chilo-ms commented Dec 9, 2025

Copy link
Copy Markdown
Contributor

Description

The newly added two APIs, CreateCustomOpDomains() and GetNumCustomOpDomains, are used when running inference on a model that contains EP-specific custom operations.

Workflow:

  1. The EP implements these functions to supply a list of OrtCustomOpDomain instances.
  2. The application either 1) calls SessionOptionsAppendExecutionProvider_V2() with an OrtEpDevice containing
    the plugin EP's factory or 2) enables auto ep selection.
  3. Then ORT either 1) SessionOptionsAppendExecutionProvider_V2() appends the provided OrtCustomOpDomains to the
    session options or 2) registers the OrtCustomOpDomains from the selected EP devices.

As a result, any session created from these session options will have these custom op domains registered
in ORT, ensuring that the custom ops are properly recognized and validated when the model is loaded.

Plugin EPs can provide two types of custom ops:

  1. A full OrtCustomOp with a concrete kernel implementation

    • This Example EP demonstrates this approach.
    • In GetCapability(), it calls EpGraphSupportInfo_AddSingleNode() to inform ORT
      that the custom node should NOT be fused or compiled. Instead, ORT should invoke
      the custom node's Compute() function at runtime.
  2. A "placeholder" OrtCustomOp with an empty kernel implementation

    • A compile-based Plugin EP can supply an OrtCustomOp whose CustomKernel::Compute()
      does nothing. The purpose is to satisfy model validation during model loading by
      registering the custom op as a valid operator in the session.
    • In GetCapability(), the EP should call EpGraphSupportInfo_AddNodesToFuse() to
      notify ORT that this custom node should be fused and compiled by the EP.
    • In Compile(), the EP executes its compiled bits to perform inference for
      the fused custom node.

Motivation and Context

Currently, the provider-bridge TRT RTX EP and TRT EP supports registering custom op domain list in session option so
that it can run model contains TRT specific custom ops.

This PR adds the same feature for plugin EP.

Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h
Comment thread onnxruntime/core/session/utils.cc Outdated
Comment thread onnxruntime/core/session/utils.cc Outdated
Comment thread onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc Outdated
Comment thread onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.h Outdated
Comment thread onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.h Outdated
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
github-actions[bot]

This comment was marked as resolved.

@chilo-ms
chilo-ms marked this pull request as ready for review December 10, 2025 23:12
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated
Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated
adrianlizarraga

This comment was marked as resolved.

github-actions[bot]

This comment was marked as resolved.

Comment thread onnxruntime/test/autoep/library/example_plugin_ep/ep_custom_op.h Fixed
Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated
Comment thread onnxruntime/test/autoep/library/example_plugin_ep/ep_custom_op.h Outdated
Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated
Comment thread onnxruntime/core/session/utils.cc
github-actions[bot]

This comment was marked as resolved.

github-actions[bot]

This comment was marked as resolved.

Comment thread onnxruntime/core/session/utils.cc Outdated
Comment thread onnxruntime/python/onnxruntime_pybind_state.cc Outdated
Comment thread onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc Outdated
Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated
Comment thread onnxruntime/core/session/utils.cc Outdated
Comment thread onnxruntime/core/session/utils.cc Outdated
edgchen1
edgchen1 previously approved these changes Jan 16, 2026
Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated
@chilo-ms
chilo-ms dismissed stale reviews from edgchen1 and adrianlizarraga via 6a571ef January 16, 2026 19:18
edgchen1
edgchen1 previously approved these changes Jan 16, 2026
@chilo-ms
chilo-ms dismissed stale reviews from edgchen1 and adrianlizarraga via 3b2e5a5 January 17, 2026 05:14
@chilo-ms

Copy link
Copy Markdown
Contributor Author

It seems i need to address Copilot comment in order to merge.
@edgchen1 @adrianlizarraga please help sign off again, thanks.

Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
Comment thread onnxruntime/test/testdata/custom_mul.py Fixed
@chilo-ms

chilo-ms commented Jan 17, 2026

Copy link
Copy Markdown
Contributor Author

I think all the comments are resolved and can't find the one that is unresolved (might be outdated and not show), so this PR still can't be merged.

image

i created another identical PR here and can merge that one if we still can't fix this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants