[#11032][feat] MLA revisited and GLM 4.7 Flash support - #11324
Conversation
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
📝 WalkthroughWalkthroughThe changes reorganize Multi-Head Latent Attention (MLA) implementation by replacing a monolithic module with modular per-backend implementations (FlashInfer, Torch-backend, and vanilla Torch), removing obsolete MLA operations from core attention files, updating configuration to use the flashinfer_mla backend, and introducing two new model architectures (DeepSeekV3 and GLM4 MoE Lite) with extensive test coverage. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant AutoDeploy as AutoDeploy<br/>Pipeline
participant Config as Config<br/>System
participant Backend as MLA<br/>Backend
participant Cache as Cache<br/>Handler
participant CUDA as CUDA<br/>Execution
Client->>AutoDeploy: Load model config
AutoDeploy->>Config: Read default.yaml
Config-->>AutoDeploy: backend=flashinfer_mla
AutoDeploy->>Backend: Initialize FlashInferMLAPlanner
Backend->>Cache: Create paged KV caches
Cache-->>Backend: ckv_cache, kpe_cache ready
Client->>AutoDeploy: Prefill phase (context)
AutoDeploy->>Backend: plan_prefill
Backend->>CUDA: Prepare ragged causal mask
Backend->>Cache: Expand compressed_kv
CUDA-->>Backend: Attention output
Client->>AutoDeploy: Decode phase (single token)
AutoDeploy->>Backend: plan_decode
Backend->>Cache: Append to paged caches
Backend->>CUDA: BatchMLAPagedAttentionWrapper
CUDA-->>Backend: Generated token output
Backend-->>Client: Final model output
sequenceDiagram
participant Source as Source<br/>Attention
participant Descriptor as Attention<br/>Descriptor
participant CachedOp as Cached<br/>Operation
participant Backend as Implementation<br/>Backend
Source->>Descriptor: get_source_attention_op()
Descriptor-->>Source: torch_mla reference op
Source->>Descriptor: Query cache config
Descriptor->>Descriptor: get_cache_initializers()
Descriptor-->>Source: Allocate paged caches
Source->>CachedOp: Provide q_nope, q_pe, compressed_kv, kpe
CachedOp->>Backend: Select backend (FlashInfer/TorchBackend)
alt Prefill Phase
Backend->>Backend: Expand compressed_kv with kv_b_proj_weight
Backend->>Backend: Compute full attention matrix
else Decode Phase
Backend->>Backend: Absorb weights into Q
Backend->>Backend: Cached attention lookup
end
Backend-->>CachedOp: Output [B, S, N, v_head_dim]
CachedOp-->>Source: Attention result
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes 🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
tensorrt_llm/_torch/auto_deploy/models/quant_config_reader.py (1)
1-7:⚠️ Potential issue | 🟠 MajorAdd the NVIDIA copyright header (year 2026).
This file is missing the required header. Please add it at the top of the file with the latest meaningful modification year.
✅ Proposed header placement
+# +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# """ Quantization Config Reader Registry.As per coding guidelines, “All TensorRT-LLM source files (.cpp, .h, .cu, .py, and other source files) should contain an NVIDIA copyright header with the year of latest meaningful modification”.
tests/unittest/_torch/auto_deploy/unit/singlegpu/custom_ops/test_update_kv_cache.py (1)
6-41:⚠️ Potential issue | 🟠 MajorTest lacks assertions to validate correctness.
The test function
test_update_kv_cacheonly uses print statements for debugging output but contains no assertions to verify that_update_kv_cachebehaves correctly. Without assertions, this test will always pass regardless of whether the function works properly.🧪 Proposed fix to add assertions
_update_kv_cache( k.view(batch_size * seq_length, n_heads, K_D_HEAD), v.view(batch_size * seq_length, n_heads, V_D_HEAD), k_cache, v_cache, torch.tensor([3, 1]).long(), torch.tensor([0, 0]), slot_idx=torch.tensor([0, 1]), seq_start=torch.tensor([0, 3]).long(), ) print("k_cache: " + str(k_cache)) print("v_cache: " + str(v_cache)) + + # Verify first sequence (slot 0): 3 tokens starting at position 0 + assert torch.allclose(k_cache[0, 0:3, :, :], torch.ones(3, n_heads, K_D_HEAD)) + assert torch.allclose(v_cache[0, 0:3, :, :], torch.ones(3, n_heads, V_D_HEAD)) + + # Verify second sequence (slot 1): 1 token starting at position 0 + assert torch.allclose(k_cache[1, 0:1, :, :], torch.ones(1, n_heads, K_D_HEAD)) + assert torch.allclose(v_cache[1, 0:1, :, :], torch.ones(1, n_heads, V_D_HEAD)) + + # Verify remaining cache positions are still zeros + assert torch.allclose(k_cache[0, 3:, :, :], torch.zeros(MAX_SEQ_LEN - 3, n_heads, K_D_HEAD)) + assert torch.allclose(k_cache[1, 1:, :, :], torch.zeros(MAX_SEQ_LEN - 1, n_heads, K_D_HEAD))
🤖 Fix all issues with AI agents
In `@examples/auto_deploy/cookbooks/glm_4.7_flash_trtllm_cookbook.ipynb`:
- Around line 39-41: Replace the unpublished TensorRT-LLM container tag used in
the docker run command string "nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc1"
with a published release (for example
"nvcr.io/nvidia/tensorrt-llm/release:1.1.0"); update the notebook cell that
contains the docker run line so it references the published tag instead of
"1.3.0rc1".
In `@examples/auto_deploy/model_registry/models.yaml`:
- Around line 224-225: Update the GLM-4.7-Flash model entry so its yaml_extra
includes the required baseline and world-size files; specifically modify the
yaml_extra for the model named "zai-org/GLM-4.7-Flash" to follow the same
pattern as other entries (e.g. ['dashboard_default.yaml', 'world_size_1.yaml',
'glm-4.7-flash.yaml'] or use world_size_2.yaml if tensor-parallelism >1) so that
dashboard_default.yaml (which provides runtime: trtllm, attn_backend:
flashinfer, model_factory: AutoModelForCausalLM, skip_loading_weights: false) is
applied along with an appropriate world_size_X.yaml and the existing
glm-4.7-flash.yaml.
In `@tensorrt_llm/_torch/auto_deploy/custom_ops/mla/__init__.py`:
- Around line 1-9: Add the standard NVIDIA copyright header (2026) to the top of
the new module tensorrt_llm._torch.auto_deploy.custom_ops.mla.__init__.py:
insert the canonical NVIDIA header block (including year 2026 and appropriate
copyright and license lines) before the module docstring so the file and its
exports (TorchBackendMLAAttention, FlashInferMLAAttention, torch_mla,
torch_backend_mla_with_cache, flashinfer_mla_with_cache) include the required
header.
In `@tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py`:
- Around line 1-18: Add the NVIDIA copyright header (2026) at the top of the
file before the module-level docstring: insert the standard NVIDIA
copyright/SPDX header line(s) including the year 2026 and owner, keeping the
rest of the file intact; apply this change in the file containing
FlashInferMLAAttention and flashinfer_mla_with_cache so the header appears above
the existing triple-quoted module docstring.
In `@tensorrt_llm/_torch/auto_deploy/custom_ops/mla/torch_backend_mla.py`:
- Around line 1-18: This file is missing the required NVIDIA copyright header
for 2026; add the standard NVIDIA copyright/license header at the top of
tensorrt_llm/_torch/auto_deploy/custom_ops/mla/torch_backend_mla.py before the
module docstring, ensuring it covers the year 2026 and the project/license text
used across the repo and preserves the existing symbols
(torch_cached_mla_with_cache, TorchBackendMLAAttention, mla_cache) and
module-level docstring placement.
In `@tensorrt_llm/_torch/auto_deploy/custom_ops/mla/torch_mla.py`:
- Around line 1-8: Add the required NVIDIA copyright header at the very top of
the torch_mla module (above the existing module docstring) following the
project's standard header format and include the year of latest meaningful
modification; ensure the header matches other source files in the repo (use the
same SPDX/license block and wording) and update any file-level metadata if
present so CI/style checks recognize the header for the torch_mla module.
In `@tensorrt_llm/_torch/auto_deploy/custom_ops/torch_backend_attention.py`:
- Around line 26-47: The function _update_kv_cache currently declares a return
type of -> torch.Tensor but performs in-place updates and returns nothing;
change its return type annotation to -> None and update the docstring/signature
accordingly (ensure the def _update_kv_cache(... ) -> None: signature and any
related type hints or callers expecting a tensor are adjusted to reflect no
return value).
In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_glm4_moe_lite.py`:
- Line 1: Update the copyright year in the file header from 2025 to 2026 by
editing the top-of-file copyright comment in modeling_glm4_moe_lite.py (the
existing line starting with "# Copyright (c) 2025, NVIDIA CORPORATION. All
rights reserved.") to read 2026.
- Around line 153-158: The forward method is promoting the final result to
float32 because self.weight is float32; after normalizing in float32 and casting
hidden_states back to input_dtype, multiply with self.weight forces upcast. Fix
by ensuring the multiplication happens in the original input_dtype: cast
self.weight (and any scalar buffers like self.variance_epsilon if needed) to
input_dtype before multiplying (e.g., use self.weight.to(input_dtype) *
hidden_states.to(input_dtype)), mirroring the approach used in NemotronHRMSNorm
so bf16 outputs remain bf16.
In `@tensorrt_llm/_torch/auto_deploy/models/quant_config_reader.py`:
- Line 86: Update the exclusion pattern tuple _ALWAYS_EXCLUDE so mlp gate
variants are matched: replace the entry "*.mlp.gate" with "*.mlp.gate*" to
mirror the existing "*.mixer.gate*" style; this ensures modules like
mlp.gate_proj and mlp.gate_up_proj are excluded by the quantization exclusion
logic in quant_config_reader.py where _ALWAYS_EXCLUDE is defined.
🧹 Nitpick comments (19)
tensorrt_llm/_torch/auto_deploy/custom_ops/mla/torch_mla.py (1)
109-115: Document or warn when causal mask is silently skipped.The causal mask is only applied when
s_q == s_k. If a caller passesis_causal=Truewith differing sequence lengths, no masking occurs without warning. Consider either documenting this behavior in the docstring or emitting a warning.📝 Proposed documentation update
Update the docstring for
is_causalparameter:- is_causal: Whether to apply causal masking (default: True) + is_causal: Whether to apply causal masking (default: True). Note: causal + masking is only applied when query and key sequence lengths match (s_q == s_k).tensorrt_llm/_torch/auto_deploy/models/custom/modeling_deepseek.py (2)
218-242: Minor: Unused unpacked variables.The
bszandseq_lenvariables are unpacked but never used. Consider prefixing with underscore to indicate intentional non-use.📝 Proposed fix
def forward(self, hidden_states: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: """Forward pass returning (selected_experts, routing_weights).""" - bsz, seq_len, hidden_dim = hidden_states.shape + _bsz, _seq_len, hidden_dim = hidden_states.shape hidden_states_flat = hidden_states.view(-1, hidden_dim)
529-534: Minor: Class attributes should useClassVarannotation.Per static analysis, mutable class attributes like
_no_split_modulesshould be annotated withtyping.ClassVarto indicate they are class-level, not instance-level.📝 Proposed fix
+from typing import ClassVar, List, Optional, Tuple + class DeepSeekV3PreTrainedModel(PreTrainedModel): """Base class for DeepSeekV3 models.""" base_model_prefix = "model" - _no_split_modules = ["DeepSeekV3DecoderLayer"] + _no_split_modules: ClassVar[List[str]] = ["DeepSeekV3DecoderLayer"] supports_gradient_checkpointing = Falsetests/unittest/_torch/auto_deploy/unit/singlegpu/custom_ops/test_flashinfer_mla_op.py (3)
19-19: Remove unused noqa directive.The
noqa: F401directive is unnecessary since the import is used to register custom ops as a side effect. However, static analysis indicates this directive is not enabled. Consider removing it or adding a comment explaining the side-effect import.♻️ Proposed fix
-import tensorrt_llm._torch.auto_deploy # noqa: F401 +import tensorrt_llm._torch.auto_deploy # Register custom ops
193-193: Consider addingstrict=Trueto zip().The
zip()call should includestrict=True(Python 3.10+) to catch length mismatches betweeninput_positionsandseq_lengths.♻️ Proposed fix
- kv_lengths = [pos + seq_len for pos, seq_len in zip(input_positions, seq_lengths)] + kv_lengths = [pos + seq_len for pos, seq_len in zip(input_positions, seq_lengths, strict=True)]
2137-2137: Unuseddtypeparameter in test function.The
dtypeparameter is passed to the test but never used within the function body. Either use it or remove it from the parametrization.♻️ Proposed fix - use dtype parameter
The
dtypeparameter should be used when creating buffer tensors to ensure type consistency:def test_flashinfer_mla_init_decode_wrapper_with_buffers(batch_size, dtype, device): """Test that _init_decode_wrapper correctly passes buffer tensors with use_cuda_graph=True. This test directly tests the _init_decode_wrapper method to verify buffer handling. """ # Reset planner _GlobalFlashInferMLAPlanner.workspace_buffer = None _GlobalFlashInferMLAPlanner.cached_cuda_graph_decode_wrappers = {} _GlobalFlashInferMLAPlanner.reset(torch.device(device)) # Create buffer tensors qo_indptr = torch.arange(batch_size + 1, device=device, dtype=torch.int32) kv_indptr = torch.arange(batch_size + 1, device=device, dtype=torch.int32) * 2 kv_indices = torch.arange(batch_size * 2, device=device, dtype=torch.int32) kv_len_arr = torch.ones(batch_size, device=device, dtype=torch.int32) * 64 + + # Verify dtype parameter is valid (even if not used for buffer creation) + assert dtype in [torch.float16, torch.bfloat16, torch.float32], f"Unexpected dtype: {dtype}"Alternatively, remove
dtypefrom parametrization if not needed.tests/unittest/_torch/auto_deploy/unit/singlegpu/custom_ops/test_torch_mla_op.py (3)
33-35: Use explicitOptionaltype hints.PEP 484 prohibits implicit
Optional. Parameters with defaultNoneshould use explicit type hints.♻️ Proposed fix
+from typing import Optional + def numpy_mla_reference_with_expansion( q_nope: np.ndarray, q_pe: np.ndarray, compressed_kv: np.ndarray, kpe: np.ndarray, kv_b_proj_weight: np.ndarray, mla_cache: np.ndarray, seq_len: np.ndarray, input_pos: np.ndarray, cache_loc: np.ndarray, seq_start: np.ndarray, - scale: float = None, - kv_lora_rank: int = None, + scale: Optional[float] = None, + kv_lora_rank: Optional[int] = None, is_generate: bool = False, ):
124-128: Redundant conditional branches.Both branches of the if/else perform identical operations. This can be simplified.
♻️ Proposed fix
# Construct full query and key - if is_generate: - query_full = np.concatenate([q_nope_seq, q_pe_seq], axis=-1) - else: - query_full = np.concatenate([q_nope_seq, q_pe_seq], axis=-1) + query_full = np.concatenate([q_nope_seq, q_pe_seq], axis=-1) key_full = np.concatenate([k_nope, kpe_expanded], axis=-1)
19-19: Remove unused noqa directive.The
noqa: F401directive appears unnecessary. Consider removing it or replacing with an explanatory comment.♻️ Proposed fix
-import tensorrt_llm._torch.auto_deploy # noqa: F401 +import tensorrt_llm._torch.auto_deploy # Register custom opstensorrt_llm/_torch/auto_deploy/custom_ops/mla/__init__.py (2)
11-13: Keep module namespaces for imports.
Coding guidelines require module-level imports; re-export via explicit aliases.♻️ Suggested refactor
-from .flashinfer_mla import FlashInferMLAAttention, flashinfer_mla_with_cache -from .torch_backend_mla import TorchBackendMLAAttention, torch_backend_mla_with_cache -from .torch_mla import torch_mla +from . import flashinfer_mla as _flashinfer_mla +from . import torch_backend_mla as _torch_backend_mla +from . import torch_mla as _torch_mla + +FlashInferMLAAttention = _flashinfer_mla.FlashInferMLAAttention +flashinfer_mla_with_cache = _flashinfer_mla.flashinfer_mla_with_cache +TorchBackendMLAAttention = _torch_backend_mla.TorchBackendMLAAttention +torch_backend_mla_with_cache = _torch_backend_mla.torch_backend_mla_with_cache +torch_mla = _torch_mla.torch_mla
15-21: Sort__all__for deterministic exports.
Ruff flags the current ordering.🔧 Suggested ordering
__all__ = [ - "TorchBackendMLAAttention", - "FlashInferMLAAttention", - "torch_mla", - "torch_backend_mla_with_cache", - "flashinfer_mla_with_cache", + "FlashInferMLAAttention", + "TorchBackendMLAAttention", + "flashinfer_mla_with_cache", + "torch_backend_mla_with_cache", + "torch_mla", ]tests/unittest/_torch/auto_deploy/unit/singlegpu/models/test_deepseek_custom.py (1)
3-17: Prefer module namespace imports per guidelines.
This file uses multiplefrom … import …statements. The guidelines require keeping module namespaces (e.g.,import transformersand referencingtransformers.PretrainedConfig). Consider a mechanical refactor to align with the rule.tensorrt_llm/_torch/auto_deploy/models/custom/modeling_glm4_moe_lite.py (1)
23-34: Prefer module namespace imports per guidelines.
The file relies heavily onfrom … import …. Please consider switching to module-level imports and updating call sites to keep namespaces explicit.tensorrt_llm/_torch/auto_deploy/custom_ops/mla/torch_backend_mla.py (1)
23-36: Prefer module namespace imports per guidelines.
This file uses multiplefrom … import …statements. Please consider switching to module-level imports and updating references to keep namespaces explicit.tests/unittest/_torch/auto_deploy/unit/singlegpu/models/test_glm4_moe_lite_modeling.py (3)
7-18: Prefer module namespace imports per guidelines.
This file uses multiplefrom … import …statements. Please consider switching to module-level imports and updating references to keep namespaces explicit.
20-21: Rename module-level constant withG_prefix.
Global variables should use upper snake_case with aG_prefix.♻️ Suggested rename
-_BATCH_AND_SEQUENCE_TEST_CASES = ((2, 6), (1, 8)) +G_BATCH_AND_SEQUENCE_TEST_CASES = ((2, 6), (1, 8)) -@pytest.mark.parametrize("B,S", _BATCH_AND_SEQUENCE_TEST_CASES) +@pytest.mark.parametrize("B,S", G_BATCH_AND_SEQUENCE_TEST_CASES) -@pytest.mark.parametrize("B,S", _BATCH_AND_SEQUENCE_TEST_CASES) +@pytest.mark.parametrize("B,S", G_BATCH_AND_SEQUENCE_TEST_CASES) -@pytest.mark.parametrize("B,S", _BATCH_AND_SEQUENCE_TEST_CASES) +@pytest.mark.parametrize("B,S", G_BATCH_AND_SEQUENCE_TEST_CASES) -@pytest.mark.parametrize("B,S", _BATCH_AND_SEQUENCE_TEST_CASES) +@pytest.mark.parametrize("B,S", G_BATCH_AND_SEQUENCE_TEST_CASES) -@pytest.mark.parametrize("B,S", _BATCH_AND_SEQUENCE_TEST_CASES) +@pytest.mark.parametrize("B,S", G_BATCH_AND_SEQUENCE_TEST_CASES)Also applies to: 75-77, 103-105, 454-456, 566-568, 603-605
485-523: Remove verbose debug prints from tests.
The current prints dump large state dicts and will clutter CI logs; consider removing or gating behind a debug flag.🧹 Proposed cleanup
- # Debug: print state dict keys and shapes - print("\n=== HF MoE state_dict keys and shapes ===") - for k, v in hf_state_dict.items(): - print(f" {k}: {v.shape}") - - custom_state_dict = _convert_hf_moe_state_dict_to_custom(hf_state_dict, config.n_routed_experts) - - print("\n=== Converted custom state_dict keys and shapes ===") - for k, v in custom_state_dict.items(): - print(f" {k}: {v.shape}") - - print("\n=== Expected custom MoE state_dict keys ===") - for k, v in custom_moe.state_dict().items(): - print(f" {k}: {v.shape}") + custom_state_dict = _convert_hf_moe_state_dict_to_custom(hf_state_dict, config.n_routed_experts) @@ - print(f"\n=== Debug: intermediate_size = {intermediate_size} ===") - print(f"hf_gate_up shape: {hf_gate_up.shape}") - print(f"hf_gate_up[0, :2, :2]: {hf_gate_up[0, :2, :2]}") - - # Get the converted state dict values for comparison - converted_gate_0 = custom_state_dict["experts.0.gate_proj.weight"] - print(f"converted_gate_0 shape: {converted_gate_0.shape}") - print(f"converted_gate_0[:2, :2]: {converted_gate_0[:2, :2]}") - - # After load_state_dict - loaded_gate_0 = custom_moe.experts[0].gate_proj.weight - print(f"loaded_gate_0 shape: {loaded_gate_0.shape}") - print(f"loaded_gate_0[:2, :2]: {loaded_gate_0[:2, :2]}")tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py (2)
31-44: Prefer module namespace imports per guidelines.
This file uses multiplefrom … import …statements; consider switching to module-level imports and updating references to keep namespaces explicit.
389-390: Rename global planner toG_prefix.
Global variables should use upper snake_case with aG_prefix.♻️ Suggested rename
-_GlobalFlashInferMLAPlanner = _FlashInferMLAPlanner() +G_FLASHINFER_MLA_PLANNER = _FlashInferMLAPlanner() @@ - _GlobalFlashInferMLAPlanner.reset(position_ids.device) + G_FLASHINFER_MLA_PLANNER.reset(position_ids.device) @@ - wrapper_chunked = _GlobalFlashInferMLAPlanner.plan_chunked_prefill( + wrapper_chunked = G_FLASHINFER_MLA_PLANNER.plan_chunked_prefill( @@ - wrapper_prefill = _GlobalFlashInferMLAPlanner.plan_prefill( + wrapper_prefill = G_FLASHINFER_MLA_PLANNER.plan_prefill( @@ - wrapper_decode = _GlobalFlashInferMLAPlanner.plan_decode( + wrapper_decode = G_FLASHINFER_MLA_PLANNER.plan_decode( @@ - _GlobalFlashInferMLAPlanner.plan_generate_only( + G_FLASHINFER_MLA_PLANNER.plan_generate_only(Also applies to: 464-466, 676-682, 744-750, 805-810, 504-509
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35023 [ run ] triggered by Bot. Commit: |
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35053 [ run ] triggered by Bot. Commit: |
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
1559295 to
49c7aa6
Compare
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35082 [ run ] triggered by Bot. Commit: |
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35084 [ kill ] triggered by Bot. Commit: |
|
PR_Github #35084 [ kill ] completed with state |
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35110 [ run ] triggered by Bot. Commit: |
cfb1a0f to
cf5661e
Compare
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35118 [ run ] triggered by Bot. Commit: |
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35126 [ run ] triggered by Bot. Commit: |
|
PR_Github #35126 [ run ] completed with state
|
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35157 [ run ] triggered by Bot. Commit: |
|
PR_Github #35157 [ run ] completed with state
|
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Co-authored-by: Balamurugan Marimuthu <246387390+bmarimuthu-nv@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
3a6c1bc to
ca411ab
Compare
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35355 [ run ] triggered by Bot. Commit: |
|
PR_Github #35355 [ run ] completed with state
|
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --disable-fail-fast |
|
PR_Github #35376 [ run ] triggered by Bot. Commit: |
|
PR_Github #35376 [ run ] completed with state |
5 participants
Summary by CodeRabbit
Release Notes
New Features
Tests
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
Details
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.