[None][fix] AutoDeploy: Update the model yaml file for the correct world_size - #12115
Closed
nvchenghaoz wants to merge 14 commits into
Closed
[None][fix] AutoDeploy: Update the model yaml file for the correct world_size#12115nvchenghaoz wants to merge 14 commits into
nvchenghaoz wants to merge 14 commits into
Conversation
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Balamurugan Marimuthu <246387390+bmarimuthu-nv@users.noreply.github.com>
* build_and_run_ad.py from registry Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * csv generator file Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> --------- Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Balamurugan Marimuthu <246387390+bmarimuthu-nv@users.noreply.github.com>
Signed-off-by: Balamurugan Marimuthu <246387390+bmarimuthu-nv@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Working models: ibm-granite/granite-4.0-micro, ibm-granite/granite-4.0-tiny-preview, ibm-granite/granite-4.0-h-small Signed-off-by: Govind Ramnarayan <105831528+govind-ramnarayan@users.noreply.github.com>
* hunyuan model onboarding Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> * support hunyuan instruct model Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> * update world_size for hunyuan models Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> --------- Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com>
* [None][feat] Add AutoDeploy custom model for Qwen3 (Qwen/Qwen3-0.6B-FP8) Add prefill-only Qwen3 custom model implementation for AutoDeploy export with GQA attention, per-head Q/K normalization, and FP8 dtype handling. - Custom model: modeling_qwen3.py using torch_attention and torch_rope ops - Hierarchical equivalence tests against HF reference (MLP, attention, decoder layer, full model, export with dynamic shapes) - AD config with fuse_finegrained_fp8_linear disabled (CUDA 13.1 NVRTC workaround) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * [None][fix] Address PR feedback: use torch_rmsnorm op, simplify yaml - Replace custom RMSNorm implementation with torch.ops.auto_deploy.torch_rmsnorm - Simplify AD config yaml to only override fuse_finegrained_fp8_linear Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * [None][fix] Add SM check to fuse_finegrained_fp8_linear, remove FP8 config override - Skip fuse_finegrained_fp8_linear on SM < 100 (Hopper and below) since fp8_block_scaling_gemm requires Blackwell (SM 100+) - Remove the no-longer-needed qwen3_0_6b_fp8_no_fuse_fp8.yaml config override Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * [None][fix] Move position slicing to RoPE, assert position_ids, update models.yaml - Move cos/sin position_ids slicing from attention into RoPE forward (avoids redundant slicing per layer) - Assert position_ids is not None in both Model and ForCausalLM forward (no fallback — AD always provides position_ids) - Update Qwen3-14B and Qwen3-32B to world_size_2 in models.yaml Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> --------- Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (30)
📝 WalkthroughWalkthroughIntroduces AutoDeploy model onboarding infrastructure with agent configurations for running, debugging, and reviewing model onboarding; three new custom model implementations (GraniteMoeHybrid, HunYuanDenseV1, Qwen3) for prefill-only inference; model registry with yaml_extra config support; updated default backends and performance parameters; and comprehensive test suites for model equivalence and export validation. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested reviewers
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
--use-registryflag for automatic configuration managementBug Fixes
Configuration
Documentation
Tests