Skip to content

Add weight layout transformation cache for Conv operator - #26595

Closed
jchen10 wants to merge 1 commit into
microsoft:mainfrom
jchen10:conv_weight
Closed

Add weight layout transformation cache for Conv operator#26595
jchen10 wants to merge 1 commit into
microsoft:mainfrom
jchen10:conv_weight

Conversation

@jchen10

@jchen10 jchen10 commented Nov 18, 2025

Copy link
Copy Markdown
Contributor

Implement lazy weight layout transformation for WebGPU Conv kernel to avoid redundant GPU transposes on every inference.

Key changes:

  • Add WeightLayoutTransformCache to cache transformed weights by name and format
  • Implement TransformWeightLayout() helper using existing TransposeKernel for OIHW->HWIO transformation
  • Cache stored in WebGpuExecutionProvider, shared across all kernels

Implement lazy weight layout transformation for WebGPU Conv kernel to
avoid redundant GPU transposes on every inference.

Key changes:
- Add WeightLayoutTransformCache to cache transformed weights by name
and format
- Implement TransformWeightLayout() helper using existing
TransposeKernel for OIHW->HWIO transformation
- Cache stored in WebGpuExecutionProvider, shared across all kernels
@jchen10

jchen10 commented Nov 18, 2025

Copy link
Copy Markdown
Contributor Author

Follow-up for #26554

@jchen10

jchen10 commented Nov 18, 2025

Copy link
Copy Markdown
Contributor Author

@fs-eire PTAL

@jchen10

jchen10 commented Nov 19, 2025

Copy link
Copy Markdown
Contributor Author

I am still looking into the PrePack approach, which seems more appealing as it does release the original tensors.

@fs-eire

fs-eire commented Nov 19, 2025

Copy link
Copy Markdown
Contributor

I am still looking into the PrePack approach, which seems more appealing as it does release the original tensors.

Please take a look at #26602. However I didn't finish all validation yet.

@jchen10

jchen10 commented Nov 19, 2025

Copy link
Copy Markdown
Contributor Author

I am still looking into the PrePack approach, which seems more appealing as it does release the original tensors.

Please take a look at #26602. However I didn't finish all validation yet.

Great. That's exactly what I wanted. One pity of PrePack is that we couldn't know the runtime input/output shapes which may impact how we choose the optimal blocked format for weight. Let's see if this issue will come up in the future. So far so good.

@jchen10 jchen10 closed this Nov 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants