Skip to content

[https://nvbugs/6546609][fix] Kept the verified reference registration + 4 stale waiver removals; amended… #17210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
trtllm-agent wants to merge 1 commit into from
+7 −4
Closed

[https://nvbugs/6546609][fix] Kept the verified reference registration + 4 stale waiver removals; amended… #17210

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions tests/integration/defs/accuracy/references/gsm8k.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,13 @@ deepseek-ai/DeepSeek-V4-Flash:
# 95.11 reference still holds for the hypothesis test.
- quant_algo: FP8_BLOCK_SCALES
accuracy: 95.11
deepseek-ai/DeepSeek-V4-Flash-Base:
# Base (pretrained, non-instruct) checkpoint, so GSM8K lands well below the
# instruct DeepSeek-V4-Flash above. Measured on 4x B200 183GB at TP=4/EP=4
# via test_fp8_4gpus_static_eplb (WIDEEP, 1319 samples) across three runs:
# 90.90, 91.02, 91.43; reference records the low end for run-to-run margin.
- quant_algo: FP8_BLOCK_SCALES
accuracy: 90.90
Comment on lines +148 to +154

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n -C 8 'GSM8K\(|_run_deepseekv4_eplb|extra_acc_spec|is_integration_test' \
  tests/integration/defs/accuracy || true

rg -n -C 4 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_fp8_4gpus_online_eplb' \
  tests/integration/test_lists || true

Repository: NVIDIA/TensorRT-LLM

Length of output: 50376

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '== changed files =='
git diff --name-status

printf '%s\n' '== reference and lookup implementation =='
rg -n -C 10 'class AccuracyTask|def get_hypothesis_testing_params|hypothesis_testing|quant_algo|backend|eplb|extra_acc_spec' \
  tests/integration/defs/accuracy \
  -g '*.py' -g '*.yaml' | head -n 500

printf '%s\n' '== DeepSeek V4 test methods and exact configuration =='
sed -n '3750,3960p' tests/integration/defs/accuracy/test_llm_api_pytorch.py

printf '%s\n' '== model and test-list entries =='
rg -n -C 5 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' \
  tests/integration/test_lists tests/integration/defs/accuracy \
  -g '*.yaml' -g '*.yml' -g '*.json' -g '*.py' | head -n 500

printf '%s\n' '== target file header =='
head -n 12 tests/integration/defs/accuracy/references/gsm8k.yaml

Repository: NVIDIA/TensorRT-LLM

Length of output: 50375

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '== target reference =='
nl -ba tests/integration/defs/accuracy/references/gsm8k.yaml | sed -n '135,165p'

printf '%s\n' '== matching implementation =='
nl -ba tests/integration/defs/accuracy/accuracy_core.py | sed -n '152,185p;195,232p'

printf '%s\n' '== all DeepSeek V4 declarations and GSM8K calls =='
rg -n 'DeepSeek-V4-Flash-Base|DeepSeek-V4-Flash|GSM8K|def test_.*eplb|_run_deepseekv4_eplb' \
  tests/integration/defs/accuracy/test_llm_api_pytorch.py \
  tests/integration/test_lists -g '*.py' -g '*.yaml' -g '*.yml' -g '*.json' \
  | grep -E 'DeepSeek-V4|GSM8K|eplb|test_' | tail -n 250

printf '%s\n' '== parsed GSM8K entries for the affected model =='
python3 - <<'PY'
import yaml
from pathlib import Path
p = Path("tests/integration/defs/accuracy/references/gsm8k.yaml")
data = yaml.safe_load(p.read_text())
for model, entries in data.items():
    if "DeepSeek-V4" in model:
        print(model)
        for i, entry in enumerate(entries or []):
            print(i, entry)
PY

printf '%s\n' '== test-list files containing the affected model or test names =='
for f in $(git ls-files tests/integration/test_lists); do
  if rg -q 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' "$f"; then
    echo "--- $f"
    rg -n -C 3 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' "$f"
  fi
done

Repository: NVIDIA/TensorRT-LLM

Length of output: 220

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '== target reference =='
sed -n '135,165p' tests/integration/defs/accuracy/references/gsm8k.yaml

printf '%s\n' '== matching implementation =='
sed -n '152,185p;195,232p' tests/integration/defs/accuracy/accuracy_core.py

printf '%s\n' '== all DeepSeek V4 declarations and GSM8K calls =='
rg -n 'DeepSeek-V4-Flash-Base|DeepSeek-V4-Flash|GSM8K|def test_.*eplb|_run_deepseekv4_eplb' \
  tests/integration/defs/accuracy/test_llm_api_pytorch.py \
  tests/integration/test_lists -g '*.py' -g '*.yaml' -g '*.yml' -g '*.json' \
  | grep -E 'DeepSeek-V4|GSM8K|eplb|test_' | tail -n 250

printf '%s\n' '== parsed GSM8K entries for the affected model =='
python3 - <<'PY'
try:
    import yaml
except ImportError:
    print("PyYAML unavailable")
else:
    from pathlib import Path
    data = yaml.safe_load(Path("tests/integration/defs/accuracy/references/gsm8k.yaml").read_text())
    for model, entries in data.items():
        if "DeepSeek-V4" in model:
            print(model)
            for i, entry in enumerate(entries or []):
                print(i, entry)
PY

printf '%s\n' '== test-list files containing the affected model or test names =='
for f in $(git ls-files tests/integration/test_lists); do
  if rg -q 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' "$f"; then
    echo "--- $f"
    rg -n -C 3 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' "$f"
  fi
done

Repository: NVIDIA/TensorRT-LLM

Length of output: 50375

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '== DeepSeek V4 Flash Base class =='
sed -n '4000,4085p' tests/integration/defs/accuracy/test_llm_api_pytorch.py

printf '%s\n' '== shared runner and configuration fields =='
sed -n '3820,3850p' tests/integration/defs/accuracy/test_llm_api_pytorch.py
sed -n '3880,3930p' tests/integration/defs/accuracy/test_llm_api_pytorch.py

printf '%s\n' '== exact test-list coverage for the affected class =='
rg -n -C 2 'TestDeepSeekV4FlashBase' tests/integration/test_lists

printf '%s\n' '== reference matching probe without repository imports =='
python3 - <<'PY'
from pathlib import Path

text = Path("tests/integration/defs/accuracy/references/gsm8k.yaml").read_text()
start = text.index("deepseek-ai/DeepSeek-V4-Flash-Base:")
end = text.find("\n", text.find("accuracy:", start)) + 1
print(text[start:end])

# The matcher compares only the fields passed by evaluate(); absent fields
# default to None except dtype, which defaults to "auto".
entry = {"quant_algo": "FP8_BLOCK_SCALES", "accuracy": 90.90}
for name, specs in {
    "static WIDEEP": {
        "dtype": "auto",
        "quant_algo": "FP8_BLOCK_SCALES",
        "kv_cache_quant_algo": None,
        "spec_dec_algo": None,
        "extra_acc_spec": None,
    },
    "online TRTLLM": {
        "dtype": "auto",
        "quant_algo": "FP8_BLOCK_SCALES",
        "kv_cache_quant_algo": None,
        "spec_dec_algo": None,
        "extra_acc_spec": None,
    },
}.items():
    matched = all(entry.get(k, "auto" if k == "dtype" else None) == v
                  for k, v in specs.items())
    print(name, "matches =", matched, "selected_accuracy =", entry["accuracy"])
PY

Repository: NVIDIA/TensorRT-LLM

Length of output: 11211

Scope the GSM8K reference to the measured configuration.

TestDeepSeekV4FlashBase.test_fp8_4gpus_online_eplb also selects 90.90 because the evaluator passes no EPLB or backend discriminator. Add a discriminator or a separate reference for the online configuration.

Coverage: No test functions changed. The static test is listed in tests/integration/test_lists/qa/llm_function_core.txt; the online test is not listed. Verdict: insufficient.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/accuracy/references/gsm8k.yaml` around lines 148 -
154, Scope the 90.90 GSM8K reference under DeepSeek-V4-Flash-Base to the static
test configuration by adding the appropriate EPLB/backend discriminator used by
test_fp8_4gpus_static_eplb. Ensure
TestDeepSeekV4FlashBase.test_fp8_4gpus_online_eplb no longer matches this
reference, or add a distinct reference for its configuration.

deepseek-ai/DeepSeek-V4-Pro:
# Full GSM8K aggregate gate for the Pro deployment path: TP=8, EP=8,
# attention DP, TRTLLM MoE, FP8 KV cache, MTP max_draft_len=1, padded CUDA
Expand Down
4 changes: 0 additions & 4 deletions tests/integration/test_lists/waives.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,6 @@ full:B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mt
full:B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[tp4-mtp_nextn=0-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=False] SKIP (https://nvbugs/6525007)
full:B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_cute_dsl_nvfp4_4gpus[tp4-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=False] SKIP (https://nvbugs/6526186)
full:B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_cute_dsl_nvfp4_4gpus[tp4-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=True] SKIP (https://nvbugs/6474888)
full:B200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV4FlashBase::test_fp8_4gpus_static_eplb[moe_backend=WIDEEP] SKIP (https://nvbugs/6546609)
full:B200/accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_fp8 SKIP (https://nvbugs/6525011)
full:B200/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=TRTLLM-torch_compile=True] SKIP (https://nvbugs/6473161)
full:B200/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=True] SKIP (https://nvbugs/6473161)
Expand Down Expand Up @@ -187,7 +186,6 @@ full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gp
full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus_chunked_prefill[latency_qsplit] SKIP (https://nvbugs/6423866)
full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True-enable_chunked_prefill=True-v2_kv_cache=True] SKIP (https://nvbugs/6422343)
full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=True] SKIP (https://nvbugs/6474888)
full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV4FlashBase::test_fp8_4gpus_static_eplb[moe_backend=WIDEEP] SKIP (https://nvbugs/6546609)
full:B300/accuracy/test_llm_api_pytorch.py::TestGemma3_1BInstruct::test_fp8_prequantized[torch_compile=True] SKIP (https://nvbugs/6475346)
full:B300/accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_fp8 SKIP (https://nvbugs/6525011)
full:B300/accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_auto_dtype[tp_size=8-ep_size=8] SKIP (https://nvbugs/6445375)
Expand Down Expand Up @@ -215,7 +213,6 @@ full:GB200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[m
full:GB200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True-enable_chunked_prefill=True-v2_kv_cache=True] SKIP (https://nvbugs/6400067)
full:GB200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_cute_dsl_bf16_gemm[cuda_graph=True] SKIP (https://nvbugs/6525897)
full:GB200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[ep4-mtp_nextn=0-fp8kv=False-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False-sampler_async_worker=False] SKIP (https://nvbugs/6547150)
full:GB200/accuracy/test_llm_api_pytorch.py::TestDeepSeekV4FlashBase::test_fp8_4gpus_static_eplb[moe_backend=WIDEEP] SKIP (https://nvbugs/6546609)
full:GB200/accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_mxfp8[use_msa=False] SKIP (https://nvbugs/6479471)
full:GB200/accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_nvfp4[use_msa=False] SKIP (https://nvbugs/6479471)
full:GB200/accuracy/test_llm_api_pytorch.py::TestMinistral8BInstruct::test_auto_dtype SKIP (https://nvbugs/6547151)
Expand Down Expand Up @@ -244,7 +241,6 @@ full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[m
full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[pp4-mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False] SKIP (https://nvbugs/6525057)
full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[pp4-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True-sampler_async_worker=False] SKIP (https://nvbugs/6388153)
full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-pp4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] SKIP (https://nvbugs/6388153)
full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV4FlashBase::test_fp8_4gpus_static_eplb[moe_backend=WIDEEP] SKIP (https://nvbugs/6546609)
full:GB300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v1_kv_cache-dp4-trtllm-fp8] SKIP (https://nvbugs/6474894)
full:GB300/accuracy/test_llm_api_pytorch.py::TestGemma3_1BInstruct::test_fp8_prequantized[torch_compile=True] SKIP (https://nvbugs/6475346)
full:GB300/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=FLASHINFER-torch_compile=False] SKIP (https://nvbugs/6385771)
Expand Down
Loading