Adding ONNX Runtime C-API for WebGPU EP by fs-eire · Pull Request #21838 · microsoft/onnxruntime

fs-eire · 2024-08-23T07:16:58Z

Description

This PR is for the API change for WebGPU EP.

This PR is for review purpose. The build may fail because of lack implementation of the interface.

Use scenarios

There are 2 scenarios:

users use the "default" device for WebGPU. in this case, they don't need to pass in any pointers. WEBGPU EP will maintain a map internally and the default device will be managed by WEBGPU EP
user want to use a custom device. they are responsible to create the device using the WebGPU API ( this is from the WebGPU public header) and manage the lifecycle of the objects. It is required for users to make sure it's available during the whole life-cycle of InferenceSession instance

design

Introducing this Device ID concept (different from the one that is inside MemoryInfo):

Device ID = 0 means the default context. User no need to pass any value.
Device ID > 0 means a custom context, and the ID is used as a unique key to retrieve the cached context.

There is a concern to pass pointers through string: the integer serialization and deserialization is difficult to enforce using the same set of API, which means it's error prone here. And it is also inefficient. So WebGPU EP options include 2 parts:

struct OrtWebGPUProviderOptions: contains device description. including device ID, handles of WebGPUInstance, WebGPUAdapter and WebGPUDevice. All field being zeroed value for default context.

a string based key-value pair: all other extra information. including:

Key	Possible Values	Default Value
"preferredLayout"	"NHWC" or "NCHW"	"NHWC"
"enableGraphCapture"	"1" or "0"	"0"
"storageBufferCacheMode"	"disabled", "lazyRelease", "simple", "bucket"	"bucket"
"uniformBufferCacheMode"	"disabled", "lazyRelease", "simple", "bucket"	"lazyRelease"
"queryResolveBufferCacheMode"	"disabled", "lazyRelease", "simple", "bucket"	"disabled"
"defaultBufferCacheMode"	"disabled", "lazyRelease", "simple", "bucket"	"disabled"

Consideration

the struct OrtWebGPUProviderOptions may not be modified once released for ABI compatibility considerations. So try to put minimized items in this and use the string based key-value-pair as much as possible. for example, the buffer cache mode may extend in future and this does not need to modify the API.
may want to support creating WebGPU EP via SessionOptionsAppendExecutionProvider() for default device as well (all options are string based in this use case)

Example:

Use default device:

  ...

  OrtWebGPUProviderOptions webgpu_options{};

  ...

  std::vector<std::string> keys { "storageBufferCacheMode" };
  std::vector<std::string> values { "simple" };

  auto status = SessionOptionsAppendExecutionProvider_WebGPU(
                  &session_options,
                  &webgpu_options,
                  keys.data(),
                  values.data(),
                  1);

Use custom device:

  ...

  WGPUInstance instance;
  WGPUAdapter adapter;
  WGPUDevice device;
  CreateWebGpuHandles(..., &instance, &adapter, &device);

  OrtWebGPUProviderOptions webgpu_options{
    1,  //device ID
    instance,
    adapter,
    device
  };

  ...

  std::vector<std::string> keys { "storageBufferCacheMode" };
  std::vector<std::string> values { "simple" };

  auto status = SessionOptionsAppendExecutionProvider_WebGPU(
                  &session_options,
                  &webgpu_options,
                  keys.data(),
                  values.data(),
                  1);

guschmue · 2024-08-23T23:32:23Z

went quickly over it - lgtm in general and will go over it in more detail in a little.
CI is unhappy.

fs-eire · 2024-08-24T03:13:45Z

went quickly over it - lgtm in general and will go over it in more detail in a little. CI is unhappy.

thank you for the review. I am waiting for the answer of whether/how should pass the proc table information before finalizing the interface.

The build is broken - yes because the implementation is not included in the build. I will add a stub implementation.

guschmue · 2024-08-26T15:11:51Z

We need to pass this table:
https://source.chromium.org/chromium/chromium/src/+/main:out/webview-Debug/gen/third_party/dawn/include/dawn/dawn_proc_table.h;drc=c76cca217f4278f5c53a8d90f7870270ee4dd81e;l=26

Knowing about DawnProcTable in onnxruntime is not practical so maybe it would need to be void * and cast.

Not great. We could #ifdef acting on that that field because we need it only in one specific scenario.

gyagp · 2024-08-26T16:13:52Z

+ * \see OrtApi::SessionOptionsAppendExecutionProvider_WebGPU
+ */
+typedef struct OrtWebGPUProviderOptions {
+  int device_id;          // WebGPU device id.


What's this device_id for? Does it only mean the device (and adapter, instance) is not owned by ORT?

It's just a number used as key. when the key is the same, it represents the same set of [WGPUInstance, WGPUAdapter, WGPUDevice], and all resources based on the set, including buffer cache manager, program manager and so on.

a better name suggestion is welcome

gyagp · 2024-08-26T16:15:10Z

+   * | "storageBufferCacheMode"       | "disabled", "lazyRelease", "simple", "bucket"  | "bucket"       |
+   * | "uniformBufferCacheMode"       | "disabled", "lazyRelease", "simple", "bucket"  | "lazyRelease"  |
+   * | "queryResolveBufferCacheMode"  | "disabled", "lazyRelease", "simple", "bucket"  | "disabled"     |
+   * | "defaultBufferCacheMode"       | "disabled", "lazyRelease", "simple", "bucket"  | "disabled"     |


enableGraphCapture is a general session option, not WebGPU specific.
To explain all these caches and different modes could be a little challenging. We should try our best to keep them internal.

enableGraphCapture is a general session option, not WebGPU specific.

I agree. But unfortunately, in current API this is EP specific configuration for:

CUDA/TensorRT: ProviderOptions["enable_cuda_graph"]

ROCM: OrtROCMProviderOptions::enable_hip_graph

considering backward compatibility this may not be easy change... @pranavsharma what do you think?

no need to explain the details of each config entry - these should sit somewhere else in documentation.

however, keep a reference to a full list of available config entries is helpful.

The graph capture functionality referred here is indeed specific to CUDA and TensorRT. Please do not make any backward incompat changes.

gyagp · 2024-08-26T16:20:22Z

+ */
+typedef struct OrtWebGPUProviderOptions {
+  int device_id;          // WebGPU device id.
+  void* instance_handle;  // WebGPU instance handle.


Question: How do we use instance in ORT?

In my understanding, it's the only way to create a WGPUAdapter using a WGPUInstance by calling wgpuInstanceRequestAdapter{2|F}.

We also need to call wgpuInstanceWaitAny with a WGPUInstance.

fs-eire · 2024-08-28T04:49:33Z

Updated with proc table.

We could #ifdef acting on that that field because we need it only in one specific scenario.

It maybe not a good idea to make struct size based on this certain macro considering it's a part of API. But to make users easier to use, I put the proc table member as the last member in the struct. They can still do

OrtWebGPUProviderOptions webgpu_options{};

or

OrtWebGPUProviderOptions webgpu_options{
    1,  //device ID
    instance,
    adapter,
    device
  };

fs-eire · 2024-08-29T21:05:54Z

updated according to comments. PTAL @gyagp @guschmue

do you think there is a better name for this device_id?

snnn · 2024-08-29T21:28:42Z

+ *
+ * \see OrtApi::SessionOptionsAppendExecutionProvider_WGPU
+ */
+typedef struct OrtWGPUProviderOptions {


Move it to an internal place, and use public functions in this file to create/manipulate the struct. Otherwise you cannot maintain backward binary compatibility

I understand why you have this concern. extending an existing struct may cause memory issue. Do you mean to have a set of API like this:

CreateWGPUProviderOptions ReleaseWGPUProviderOptions WGPUProviderOptionsSetDeviceId WGPUProviderOptionsSetAdapter WGPUProviderOptionsSetDevice WGPUProviderOptionsSetInstance WGPUProviderOptionsSetProcTable

I have considered this option. Actually currently WebGPU options are put into 2 places: This WGPUProviderOptions struct and a string-to-string kvp. By carefully inspect the requirement, I am convinced that there will be no requirement for adding a new pointer type options in near future. Other types of possible options can be put in the general kvp. Considering the reduce of complexity I would prefer just define the POD structure in the header file.

I am OK to add a void * unused; to buy us a buffer if you still have concern.

Then people may directly create an object from this struct:

OrtWGPUProviderOptions option;

And, by default the vars in this object are not initialized.

Generally speaking, all our APIs should have a consistent style.

sure will make this change.

Can you clarify the expected usage on Android and iOS? Does the user need to create all these instances/handles in their app (which could be Java/Kotlin/Objective-C/Swift)? If so, what do they need to do that?

It would be better if the user can pass in parameters and the EP creates the instances so we minimize the user code required.

There are 3 scenarios that of I can think may need to pass at least one handle (pointer):

when user's application static link ORT and dawn, they need to pass the proc table pointer so that ORT can call the webgpu API.

when user want to use multiple device instance. This may happen when they have multiple models to run and want to use separate buffer management.

when user want to use the pre-created device. This may happen when user's input is directly from GPU (IOBinding)

User need to figure out how before they use ORT in these scenarios. Usually the handles are created by WebGPU API. User's application may need to dynamic link to dawn, and since onnxruntime.dll dynamic link to dawn as well, they can just include the webgpu header and use the exported functions to create those handles.

For language binding - it's maybe too early stage for users to pass the handles. Anyway, this is optional; they can still use WebGPU EP without passing any handles to have the default behavior

skottmckay · 2024-09-03T02:20:14Z

+ * 2. Use a custom WebGPU device. The user should create their own handles of `WGPUInstance`, `WGPUAdapter`, and
+ *    `WGPUDevice` and use arbitrary number in [1..65536) as the device id. The user should provide the handles
+ *    and the device id in the options.


nit: Do we need to shorten WebGPU to WGPU in the API?

according to #21838 (comment), it seems that the existing webgpu libraries prefer to use wgpu instead of webgpu in native code (C/C++/rust), so making API consistent with what users may be more familiar with.

api

573fca0

fs-eire requested review from guschmue and pranavsharma August 23, 2024 07:17

qjia7 mentioned this pull request Aug 24, 2024

Allow getting adapter information from GPUDevice gpuweb/gpuweb#4810

Closed

impl stub

793ccae

gyagp reviewed Aug 26, 2024

View reviewed changes

fs-eire added 4 commits August 26, 2024 09:58

fix build break

86598d5

Merge remote-tracking branch 'origin/main' into fs-eire/webgpu-ep-api

1e91594

Merge remote-tracking branch 'origin/main' into fs-eire/webgpu-ep-api

0ee3a62

webgpu->wgpu and add proctable

14a71cb

fs-eire mentioned this pull request Aug 29, 2024

[WIP] WebGPU EP [skip ci] #21904

Closed

7 tasks

snnn reviewed Aug 29, 2024

View reviewed changes

skottmckay reviewed Sep 3, 2024

View reviewed changes

snnn closed this Jul 3, 2025

Uh oh!

Conversation

fs-eire commented Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Use scenarios

design

Consideration

Example:

Uh oh!

guschmue commented Aug 23, 2024

Uh oh!

fs-eire commented Aug 24, 2024

Uh oh!

guschmue commented Aug 26, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fs-eire commented Aug 28, 2024

Uh oh!

fs-eire commented Aug 29, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fs-eire Sep 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fs-eire commented Aug 23, 2024 •

edited

Loading

fs-eire Sep 3, 2024 •

edited

Loading