feat:[AutoDeploy] Support Quantized MoE matcher - 1 - #4968
Closed
Fridah-nv wants to merge 12 commits into
Closed
Conversation
Fridah-nv
force-pushed
the
user/fridah/moe-quant-match
branch
from
June 18, 2025 20:52
882fe0b to
122cafe
Compare
* Fixed TP strong scaling if TP > num_kv_heads Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> * Fixed test_graph_sharding. Attention uses simple_shard Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> * head_dim inferred from the graph directly. Fixed test_graph_sharding.py Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> * changed GQA block to fit column_row_shard heuristic requirements Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> * Fixed parameter count Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> --------- Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>
* auto_deploy: rename torch and triton custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename flashinfer custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename moe custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename distribution custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename distribution custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename linear custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename rope custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename mla custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: bug fixes Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename mla custom ops (2) Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename tirton rope custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename torch rope custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename triton attention custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: rename torch quantization custom ops Signed-off-by: Neta Zmora <nzmora@nvidia.com> * auto_deploy: fix failing multi-gpu tests Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * pre-commit formatting fixes Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * pre-commit formatting fixes Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * add a README for the custom ops Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * rename custom_ops/rope.py to custom_ops/triton_rope.py Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * separate moe custom backends to different files Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * pre-commit formatting fixes Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> --------- Signed-off-by: Neta Zmora <nzmora@nvidia.com> Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com>
In #61 I accidentally removed fused_moe.py This PR fixes that and completes my intended change: split fused_moe.py to two files. * add fused_moe.py which I accidentally removed Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * separate torch MOE custom ops to a separate file Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * separate trtllm MOE custom ops to a separate file Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> * remove unused imports Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> --------- Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com>
…64), fixes NVIDIA#4841, fixes NVIDIA#5254 Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
…nit tests Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
…cc issue and e2e mixtral fp4 has kernel error wo moe matching Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Fridah-nv
force-pushed
the
user/fridah/moe-quant-match
branch
from
June 27, 2025 18:16
122cafe to
4f370f7
Compare
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Collaborator
Author
|
Re opened in nv-auto-deploy#68 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Separate from #4361
TODO: update custom op names, update model coverage
Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
Details
run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]Launch build/test pipelines. All previously running jobs will be killed.
--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-[Post-Merge]-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.