Skip to content

Draft - DO NOT REVIEW - AD host time improvement for large prefill (and more changes that will be removed #11565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MrGeva wants to merge 7 commits into from
Closed

Draft - DO NOT REVIEW - AD host time improvement for large prefill (and more changes that will be removed #11565

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e09e9fa
trtllm attn
Feb 16, 2026
d200052
ms moe super
Feb 15, 2026
05f9a73
set trtllm_attn be in super yaml
Feb 16, 2026
e763def
improved host time
Feb 17, 2026
4662724
cleaned the host optimization to work only with tensors
Feb 17, 2026
e823b0c
Ported CG feature and resolved conflicts but with a mistake in warmup…
Feb 18, 2026
0a909f1
adjustment to make the PW CG work with trtllm attn. and fixed a test
Feb 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions examples/auto_deploy/model_registry/configs/glm-4.7-flash.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,19 @@ max_seq_len: 4096
enable_chunked_prefill: true
cuda_graph_batch_sizes: [1, 2, 4, 8, 16, 32, 64]
transforms:
match_swiglu_pattern:
enabled: true
match_nvfp4_swiglu_pattern:
enabled: true
fuse_nvfp4_moe:
allow_different_input_scales: true
fuse_nvfp4_swiglu:
enabled: true
fuse_swiglu:
enabled: true
multi_stream_moe:
stage: compile
enabled: true
multi_stream_mla_attn:
stage: compile
enabled: true
4 changes: 2 additions & 2 deletions examples/auto_deploy/super_v3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ compile_backend: torch-cudagraph
max_batch_size: 384
max_seq_len: 65536 # tunable
enable_chunked_prefill: true
attn_backend: flashinfer
attn_backend: trtllm
model_factory: AutoModelForCausalLM
skip_loading_weights: false
cuda_graph_batch_sizes: [1, 2, 4, 8, 16, 24, 32, 64, 128, 256, 320, 384]
Expand Down Expand Up @@ -37,7 +37,7 @@ transforms:
"fc2_latent_proj": "gather"
multi_stream_moe:
stage: compile
enabled: false
enabled: true
gather_logits_before_lm_head:
# TODO: fix https://github.com/NVIDIA/TensorRT-LLM/issues/9878 to enable by default
enabled: true
Expand Down
Loading