Skip to content

Add python implementation for wgsl-gen - #28355

Merged
guschmue merged 19 commits into
microsoft:mainfrom
danielsongmicrosoft:user/danielsong/templatelibrarypythonport
Jun 16, 2026
Merged

Add python implementation for wgsl-gen#28355
guschmue merged 19 commits into
microsoft:mainfrom
danielsongmicrosoft:user/danielsong/templatelibrarypythonport

Conversation

@danielsongmicrosoft

@danielsongmicrosoft danielsongmicrosoft commented May 4, 2026

Copy link
Copy Markdown
Contributor

Description

Replaces the npm-based @fs-eire/wgsl-template Node.js engine used by the WebGPU EP's static build path with an in-tree, stdlib-only Python implementation under wgsl_templates. The dynamic generator path is removed from the build surface - the code logic remains as unreachable and will be removed in a follow-up.

Motivation and Context

The WebGPU EP's WGSL template-to-C++ codegen is currently driven by the @fs-eire/wgsl-template npm package, so any build that touches a .wgsl.template file needs Node.js installed and has to pull the package over the network. This npm package is also no longer maintained.

@danielsongmicrosoft
danielsongmicrosoft marked this pull request as draft May 4, 2026 23:23
@guschmue guschmue added the ep:WebGPU ort-web webgpu provider label May 5, 2026
@guschmue

guschmue commented May 5, 2026

Copy link
Copy Markdown
Contributor

I think this should move to tools/python/

@danielsongmicrosoft

danielsongmicrosoft commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

Did some manual testing for confirmation:

  • Built ORT with --use_webgpu --wgsl_template static --config RelWithDebInfo
    • I believe this exercises the new py tool against all 27 in-tree .wgsl.template files and c++ compiler accepted it
  • Ran onnxruntime_provider_test.exe with GTX 1060 6GB (what I have on my dev box) for Pad, Transpose, and Conv operator suites.
    • 317/318 tests passed, the 1 failure was due to an fp16 compatibility issue with my GPU

@danielsongmicrosoft

Copy link
Copy Markdown
Contributor Author

@danielsongmicrosoft please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"

Comment thread tools/python/wgsl_template/generator.py Fixed
Comment thread tools/python/wgsl_template/generator.py Fixed
Comment thread tools/python/wgsl_template/generator.py Fixed
@danielsongmicrosoft
danielsongmicrosoft marked this pull request as ready for review May 28, 2026 22:00
Comment thread cmake/onnxruntime_providers_webgpu.cmake Outdated
Comment thread cmake/onnxruntime_providers_webgpu.cmake Outdated
Comment thread tools/python/wgsl_template/test/test_build.py Outdated
Comment thread tools/python/wgsl_template/test/test_build.py Outdated
@danielsongmicrosoft
danielsongmicrosoft force-pushed the user/danielsong/templatelibrarypythonport branch from c6505c0 to d0943f8 Compare June 4, 2026 23:10
@guschmue
guschmue requested a review from Copilot June 5, 2026 15:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR replaces the WebGPU EP’s npm/Node.js-based WGSL template generator (@fs-eire/wgsl-template) with an in-tree, stdlib-only Python implementation (PASS0 loader / PASS1 parser / PASS2 generator + C++ emitters). It also removes the dynamic WGSL generator option from the supported build surface (leaving the old dynamic path in C++ as unreachable with TODOs), updates CMake/build tooling to invoke the Python CLI, and adds a comprehensive Python unit+fixture test suite (including an in-tree smoke test over the real WebGPU templates).

Changes:

  • Add a new tools/python/wgsl_template/ package implementing the WGSL template pipeline and static C++ codegen.
  • Switch WebGPU CMake generation from npm to Python (tools/python/wgsl_gen.py) and remove --wgsl_template build options/plumbing.
  • Add Python tests + golden fixtures for loader/parser/generator/build behavior and an in-tree template smoke test.

Reviewed changes

Copilot reviewed 145 out of 151 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tools/python/wgsl_template/types.py New shared dataclasses for PASS0/1/2 and metadata.
tools/python/wgsl_template/errors.py New error hierarchy for loader/parser/generator/build.
tools/python/wgsl_template/loader.py PASS0: directory scanning, path normalization, security checks.
tools/python/wgsl_template/parser.py PASS1: comment stripping, #include, #define handling.
tools/python/wgsl_template/code_pattern.py PASS2 patterns and #use pattern registry.
tools/python/wgsl_template/generator.py PASS2 generator state machine and validation.
tools/python/wgsl_template/code_generator/init.py Code generator resolver (static-cpp*).
tools/python/wgsl_template/code_generator/static_cpp.py Static C++ and string-table emit implementation.
tools/python/wgsl_template/init.py Public build() orchestrator + safe/idempotent file writes.
tools/python/wgsl_gen.py CLI entry point used by CMake for template generation.
tools/python/wgsl_template/test/run_tests.py Aggregated unittest discovery runner.
tools/python/wgsl_template/test/test_loader.py PASS0-focused unit tests.
tools/python/wgsl_template/test/test_parser.py PASS1-focused + fixture-driven tests.
tools/python/wgsl_template/test/test_generator.py PASS2-focused + fixture-driven tests.
tools/python/wgsl_template/test/test_build.py End-to-end build() tests + fixture runner.
tools/python/wgsl_template/test/test_in_tree_smoke.py Smoke tests over real in-tree WebGPU templates.
tools/python/wgsl_template/test/testcases/.gitattributes Enforce LF for golden/expected files.
tools/python/wgsl_template/test/testcases/parser-basic/test-config.json Parser fixture config.
tools/python/wgsl_template/test/testcases/parser-basic/basic.wgsl.template Parser fixture input.
tools/python/wgsl_template/test/testcases/parser-basic/basic.wgsl.template.pass1 Parser golden output.
tools/python/wgsl_template/test/testcases/parser-comment-removal/test-config.json Comment-stripping fixture config.
tools/python/wgsl_template/test/testcases/parser-comment-removal/comments.wgsl.template Comment-stripping fixture input.
tools/python/wgsl_template/test/testcases/parser-comment-removal/comments.wgsl.template.pass1 Comment-stripping golden output.
tools/python/wgsl_template/test/testcases/parser-empty/test-config.json Empty-dir parser fixture config.
tools/python/wgsl_template/test/testcases/parser-include-basic/test-config.json Include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-basic/main.wgsl.template Include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-basic/main.wgsl.template.pass1 Include golden output.
tools/python/wgsl_template/test/testcases/parser-include-basic/utils.wgsl.template Included fixture input.
tools/python/wgsl_template/test/testcases/parser-include-basic/utils.wgsl.template.pass1 Included golden output.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/test-config.json “Include in comments ignored” fixture config.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/main.wgsl.template Fixture input.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/main.wgsl.template.pass1 Golden output.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/utils.wgsl.template Fixture input.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/utils.wgsl.template.pass1 Golden output.
tools/python/wgsl_template/test/testcases/parser-include-invalid-empty/test-config.json Invalid-include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-empty/main.wgsl.template Invalid-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-invalid-no-quotes/test-config.json Invalid-include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-no-quotes/main.wgsl.template Invalid-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-invalid-nonexist/test-config.json Missing-include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-nonexist/main.wgsl.template Missing-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-invalid-path/test-config.json Invalid-path include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-path/main.wgsl.template Invalid-path include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-invalid-unclosed-quotes/test-config.json Unclosed-quote include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-unclosed-quotes/main.wgsl.template Unclosed-quote include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/test-config.json Nested-include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/main.wgsl.template Nested-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/main.wgsl.template.pass1 Nested-include golden output.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/graphics.wgsl.template Nested-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/graphics.wgsl.template.pass1 Nested-include golden output.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/math.wgsl.template Nested-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/math.wgsl.template.pass1 Nested-include golden output.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/constants.wgsl.template Nested-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/constants.wgsl.template.pass1 Nested-include golden output.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/test-config.json Leading-space include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/main.wgsl.template Leading-space include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/main.wgsl.template.pass1 Leading-space include golden output.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/constants.wgsl.template Included fixture input.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/constants.wgsl.template.pass1 Included golden output.
tools/python/wgsl_template/test/testcases/parser-macro-basic/test-config.json Macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-basic/basic.wgsl.template Macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-basic/basic.wgsl.template.pass1 Macro golden output.
tools/python/wgsl_template/test/testcases/parser-macro-chain/test-config.json Macro chaining fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-chain/chain.wgsl.template Macro chaining fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-chain/chain.wgsl.template.pass1 Macro chaining golden output.
tools/python/wgsl_template/test/testcases/parser-macro-complex-values/test-config.json Complex macro value fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-complex-values/complex.wgsl.template Complex macro value fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-complex-values/complex.wgsl.template.pass1 Complex macro value golden output.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-circular/test-config.json Circular macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-circular/circular.wgsl.template Circular macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-duplicate/test-config.json Duplicate macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-duplicate/duplicate.wgsl.template Duplicate macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-empty-value/test-config.json Empty macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-empty-value/empty.wgsl.template Empty macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-syntax/test-config.json Invalid macro name fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-syntax/invalid.wgsl.template Invalid macro name fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-whitespace-value/test-config.json Whitespace-only macro value fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-whitespace-value/whitespace.wgsl.template Whitespace-only macro value fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/test-config.json Macro name edge-cases fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/names.wgsl.template Macro name edge-cases fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/names.wgsl.template.pass1 Macro name edge-cases golden output.
tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/test-config.json Leading-space macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/main.wgsl.template Leading-space macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/main.wgsl.template.pass1 Leading-space macro golden output.
tools/python/wgsl_template/test/testcases/parser-subfolder/test-config.json Subfolder include fixture config.
tools/python/wgsl_template/test/testcases/parser-subfolder/main.wgsl.template Subfolder fixture input.
tools/python/wgsl_template/test/testcases/parser-subfolder/main.wgsl.template.pass1 Subfolder golden output.
tools/python/wgsl_template/test/testcases/parser-subfolder/shaders/vertex.wgsl.template Subfolder fixture input.
tools/python/wgsl_template/test/testcases/parser-subfolder/shaders/vertex.wgsl.template.pass1 Subfolder golden output.
tools/python/wgsl_template/test/testcases/parser-subfolder/utils/common.wgsl.template Subfolder fixture input.
tools/python/wgsl_template/test/testcases/parser-subfolder/utils/common.wgsl.template.pass1 Subfolder golden output.
tools/python/wgsl_template/test/testcases/generator-basic/test-config.json Generator fixture config.
tools/python/wgsl_template/test/testcases/generator-basic/test.wgsl.template Generator fixture input.
tools/python/wgsl_template/test/testcases/generator-basic/test.wgsl.template.static-cpp-literal.gen Generator golden output.
tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test-config.json Preserve-code-ref generator fixture config.
tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test.wgsl.template Preserve-code-ref fixture input.
tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test.wgsl.template.static-cpp-literal.gen Preserve-code-ref golden output.
tools/python/wgsl_template/test/testcases/generator-param-basic/test-config.json Param substitution generator fixture config.
tools/python/wgsl_template/test/testcases/generator-param-basic/compute-param.wgsl.template Param substitution fixture input.
tools/python/wgsl_template/test/testcases/generator-param-basic/compute-param.wgsl.template.static-cpp-literal.gen Param substitution golden output.
tools/python/wgsl_template/test/testcases/generator-param-basic/compute-no-param.wgsl.template Param substitution fixture input.
tools/python/wgsl_template/test/testcases/generator-param-basic/compute-no-param.wgsl.template.static-cpp-literal.gen Param substitution golden output.
tools/python/wgsl_template/test/testcases/generator-main/test-config.json $MAIN generator fixture config.
tools/python/wgsl_template/test/testcases/generator-main/main.wgsl.template $MAIN fixture input.
tools/python/wgsl_template/test/testcases/generator-main/main.wgsl.template.static-cpp-literal.gen $MAIN golden output.
tools/python/wgsl_template/test/testcases/generator-main-invalid-unclosed/test-config.json Unclosed $MAIN fixture config.
tools/python/wgsl_template/test/testcases/generator-main-invalid-unclosed/test.wgsl.template Unclosed $MAIN fixture input.
tools/python/wgsl_template/test/testcases/generator-main-invalid-syntax/test-config.json Invalid $MAIN syntax fixture config.
tools/python/wgsl_template/test/testcases/generator-main-invalid-syntax/test.wgsl.template Invalid $MAIN syntax fixture input.
tools/python/wgsl_template/test/testcases/generator-main-invalid-nested/test-config.json Nested $MAIN fixture config.
tools/python/wgsl_template/test/testcases/generator-main-invalid-nested/test.wgsl.template Nested $MAIN fixture input.
tools/python/wgsl_template/test/testcases/generator-main-invalid-multiple/test-config.json Multiple $MAIN fixture config.
tools/python/wgsl_template/test/testcases/generator-main-invalid-multiple/test.wgsl.template Multiple $MAIN fixture input.
tools/python/wgsl_template/test/testcases/generator-example-pad/test-config.json Complex generator fixture config.
tools/python/wgsl_template/test/testcases/generator-example-pad/pad.wgsl.template Complex generator fixture input.
tools/python/wgsl_template/test/testcases/generator-example-pad/pad.wgsl.template.static-cpp-literal.gen Complex generator golden output.
tools/python/wgsl_template/test/testcases/build-basic/test-config.json Build fixture config.
tools/python/wgsl_template/test/testcases/build-basic/src/shader/triangle.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/index.h Build fixture expected header index.
tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/index_impl.h Build fixture expected impl index.
tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/string_table.h Build fixture expected string table.
tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/generated/shader/triangle.h Build fixture expected per-template header.
tools/python/wgsl_template/test/testcases/build-basic/expected/dynamic/templates.js Legacy expected artifact (not exercised by new harness).
tools/python/wgsl_template/test/testcases/build-example-pad/test-config.json Build fixture config.
tools/python/wgsl_template/test/testcases/build-example-pad/src/tensor/pad.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/index.h Build fixture expected header index.
tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/index_impl.h Build fixture expected impl index.
tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/generated/tensor/pad.h Build fixture expected per-template header.
tools/python/wgsl_template/test/testcases/build-example-pad/expected/dynamic/templates.js Legacy expected artifact (not exercised by new harness).
tools/python/wgsl_template/test/testcases/build-directories/test-config.json Multi-source-dir build fixture config.
tools/python/wgsl_template/test/testcases/build-directories/src/common/utils.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-directories/src/effects/blur.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-directories/src/effects/lighting.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/index.h Build fixture expected header index.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/index_impl.h Build fixture expected impl index.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/string_table.h Build fixture expected string table.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/utils.h Build fixture expected per-template header.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/@effects/blur.h Build fixture expected per-template header.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/@effects/lighting.h Build fixture expected per-template header.
tools/ci_build/build_args.py Remove --wgsl_template CLI option.
tools/ci_build/build.py Remove WGSL template generator CMake flag/vcpkg feature plumbing.
cmake/CMakeLists.txt Remove onnxruntime_WGSL_TEMPLATE option.
cmake/onnxruntime_providers_webgpu.cmake Switch WGSL generation from npm to Python + add ctest hook.
cmake/external/onnxruntime_external_deps.cmake Note dynamic duktape path is now unreachable.
cmake/deps.txt Note duktape planned removal once dynamic generator is deleted.
onnxruntime/core/providers/webgpu/wgsl_templates/wgsl_gen.h Add TODO noting dynamic path now unreachable.
onnxruntime/core/providers/webgpu/wgsl_templates/wgsl_gen.cc Add TODO noting dynamic path now unreachable.
onnxruntime/core/providers/webgpu/wgsl_templates/README.md Update docs to Python generator + remove dynamic-mode guidance.
onnxruntime/core/providers/webgpu/wgsl_templates/package.json Remove Node.js dependency manifest.
onnxruntime/core/providers/webgpu/wgsl_templates/package-lock.json Remove npm lockfile.
onnxruntime/core/providers/webgpu/wgsl_templates/.gitignore Remove node_modules ignore.
.github/workflows/windows_webgpu.yml Remove wgsl_template matrix/flag usage.
Files not reviewed (1)
  • onnxruntime/core/providers/webgpu/wgsl_templates/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tools/python/wgsl_template/parser.py
Comment thread tools/python/wgsl_template/loader.py
Comment thread tools/python/wgsl_template/generator.py
Comment thread tools/python/wgsl_template/code_generator/static_cpp.py
Comment thread cmake/external/onnxruntime_external_deps.cmake Outdated
Comment thread tools/python/wgsl_template/code_generator/__init__.py
Comment thread tools/python/wgsl_gen.py Outdated
Comment thread tools/python/wgsl_gen.py Outdated
Comment thread tools/python/wgsl_template/parser.py Outdated
Comment thread tools/python/wgsl_template/test/run_tests.py
Comment thread tools/python/wgsl_template/test/test_build.py Outdated
Comment thread tools/python/wgsl_template/test/test_in_tree_smoke.py Outdated
Comment thread tools/python/wgsl_template/test/testcases/generator-main/test-config.json Outdated
@guschmue guschmue closed this Jun 15, 2026
@guschmue guschmue reopened this Jun 15, 2026
guschmue
guschmue previously approved these changes Jun 15, 2026
@danielsongmicrosoft
danielsongmicrosoft force-pushed the user/danielsong/templatelibrarypythonport branch from dca9d9c to 549a11a Compare June 15, 2026 23:48
@guschmue
guschmue enabled auto-merge (squash) June 16, 2026 19:26
@guschmue
guschmue merged commit 9c12e9d into microsoft:main Jun 16, 2026
90 of 187 checks passed
This was referenced Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ep:WebGPU ort-web webgpu provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants