[https://nvbugs/6546609][fix] Kept the verified reference registration + 4 stale waiver removals; amended…

[trtllm-agent]

Summary

• Root cause: deepseek-ai/DeepSeek-V4-Flash-Base was absent from references/gsm8k.yaml, so self.reference was empty and the spec lookup fell through its for/else to raise ValueError("Not registered specs").
• Fix: Kept the verified reference registration + 4 stale waiver removals; amended the entry's comment to document all three measurements (90.90/91.02/91.43) and why the built-in ~3.2-point hypothesis-test margin absorbs the drift.
• Automated fix generated by repair-bot

Test plan

• Verify fix on the same GPU type as the original failure
• Check for regressions in related tests

Links

• Bug: https://nvbugs/6546609

Dev Engineer Review

• Added the verified 90.90 GSM8K reference for deepseek-ai/DeepSeek-V4-Flash-Base.
• The reference covers FP8 block-scale quantization with the 4× B200 TP=4/EP=4 WIDEEP static EPLB configuration.
• The entry comment records the observed measurements and explains the hypothesis-test margin.
• Removed four stale waivers for the corresponding B200, B300, GB200, and GB300 configurations.
• The changes are scoped to the intended checkpoint and test configuration.

QA Engineer Review

• Removed the TestDeepSeekV4FlashBase::test_fp8_4gpus_static_eplb[moe_backend=WIDEEP] waiver entries for B200, B300, GB200, and GB300.
• No test functions changed.
• CBTS coverage data is unavailable from the provided changes.

Verdict: needs follow-up

[coderabbitai]

Walkthrough

The PR adds a GSM8K accuracy reference for DeepSeek-V4-Flash-Base and removes its FP8 static EPLB waiver entries for selected GPU configurations.

Changes

DeepSeek V4 Flash Base validation

Layer / File(s)	Summary
Accuracy reference and waiver updates tests/integration/defs/accuracy/references/gsm8k.yaml, tests/integration/test_lists/waives.txt	The GSM8K references record FP8 block-scale accuracy of 90.90. The waiver list removes skips for the DeepSeek V4 Flash Base FP8 static EPLB test on B200, B300, GB200, and GB300 configurations.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#17180: Updates waivers for the same DeepSeek V4 Flash FP8 static EPLB tests.
• NVIDIA/TensorRT-LLM#17186: Changes waivers for the DeepSeek V4 Flash Base test.
• NVIDIA/TensorRT-LLM#17187: Modifies waivers for the same DeepSeek V4 Flash Base test.

Suggested reviewers: qijune

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title identifies the NVBugs fix and accurately summarizes the reference registration and four waiver removals.
Description check	✅ Passed	The description explains the root cause, fix, test plan, and bug link, but it omits the repository checklist section.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/defs/accuracy/references/gsm8k.yaml`:
- Around line 148-154: Scope the 90.90 GSM8K reference under
DeepSeek-V4-Flash-Base to the static test configuration by adding the
appropriate EPLB/backend discriminator used by test_fp8_4gpus_static_eplb.
Ensure TestDeepSeekV4FlashBase.test_fp8_4gpus_online_eplb no longer matches this
reference, or add a distinct reference for its configuration.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2fc8c9ff-b16e-4146-811d-b5d3b26cdb97

📥 Commits

Reviewing files that changed from the base of the PR and between c5427c5 and e4ff11f.

📒 Files selected for processing (2)

• tests/integration/defs/accuracy/references/gsm8k.yaml
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

[coderabbitai]

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n -C 8 'GSM8K\(|_run_deepseekv4_eplb|extra_acc_spec|is_integration_test' \
  tests/integration/defs/accuracy || true

rg -n -C 4 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_fp8_4gpus_online_eplb' \
  tests/integration/test_lists || true

Repository: NVIDIA/TensorRT-LLM

Length of output: 50376

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '== changed files =='
git diff --name-status

printf '%s\n' '== reference and lookup implementation =='
rg -n -C 10 'class AccuracyTask|def get_hypothesis_testing_params|hypothesis_testing|quant_algo|backend|eplb|extra_acc_spec' \
  tests/integration/defs/accuracy \
  -g '*.py' -g '*.yaml' | head -n 500

printf '%s\n' '== DeepSeek V4 test methods and exact configuration =='
sed -n '3750,3960p' tests/integration/defs/accuracy/test_llm_api_pytorch.py

printf '%s\n' '== model and test-list entries =='
rg -n -C 5 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' \
  tests/integration/test_lists tests/integration/defs/accuracy \
  -g '*.yaml' -g '*.yml' -g '*.json' -g '*.py' | head -n 500

printf '%s\n' '== target file header =='
head -n 12 tests/integration/defs/accuracy/references/gsm8k.yaml

Repository: NVIDIA/TensorRT-LLM

Length of output: 50375

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '== target reference =='
nl -ba tests/integration/defs/accuracy/references/gsm8k.yaml | sed -n '135,165p'

printf '%s\n' '== matching implementation =='
nl -ba tests/integration/defs/accuracy/accuracy_core.py | sed -n '152,185p;195,232p'

printf '%s\n' '== all DeepSeek V4 declarations and GSM8K calls =='
rg -n 'DeepSeek-V4-Flash-Base|DeepSeek-V4-Flash|GSM8K|def test_.*eplb|_run_deepseekv4_eplb' \
  tests/integration/defs/accuracy/test_llm_api_pytorch.py \
  tests/integration/test_lists -g '*.py' -g '*.yaml' -g '*.yml' -g '*.json' \
  | grep -E 'DeepSeek-V4|GSM8K|eplb|test_' | tail -n 250

printf '%s\n' '== parsed GSM8K entries for the affected model =='
python3 - <<'PY'
import yaml
from pathlib import Path
p = Path("tests/integration/defs/accuracy/references/gsm8k.yaml")
data = yaml.safe_load(p.read_text())
for model, entries in data.items():
    if "DeepSeek-V4" in model:
        print(model)
        for i, entry in enumerate(entries or []):
            print(i, entry)
PY

printf '%s\n' '== test-list files containing the affected model or test names =='
for f in $(git ls-files tests/integration/test_lists); do
  if rg -q 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' "$f"; then
    echo "--- $f"
    rg -n -C 3 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' "$f"
  fi
done

Repository: NVIDIA/TensorRT-LLM

Length of output: 220

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '== target reference =='
sed -n '135,165p' tests/integration/defs/accuracy/references/gsm8k.yaml

printf '%s\n' '== matching implementation =='
sed -n '152,185p;195,232p' tests/integration/defs/accuracy/accuracy_core.py

printf '%s\n' '== all DeepSeek V4 declarations and GSM8K calls =='
rg -n 'DeepSeek-V4-Flash-Base|DeepSeek-V4-Flash|GSM8K|def test_.*eplb|_run_deepseekv4_eplb' \
  tests/integration/defs/accuracy/test_llm_api_pytorch.py \
  tests/integration/test_lists -g '*.py' -g '*.yaml' -g '*.yml' -g '*.json' \
  | grep -E 'DeepSeek-V4|GSM8K|eplb|test_' | tail -n 250

printf '%s\n' '== parsed GSM8K entries for the affected model =='
python3 - <<'PY'
try:
    import yaml
except ImportError:
    print("PyYAML unavailable")
else:
    from pathlib import Path
    data = yaml.safe_load(Path("tests/integration/defs/accuracy/references/gsm8k.yaml").read_text())
    for model, entries in data.items():
        if "DeepSeek-V4" in model:
            print(model)
            for i, entry in enumerate(entries or []):
                print(i, entry)
PY

printf '%s\n' '== test-list files containing the affected model or test names =='
for f in $(git ls-files tests/integration/test_lists); do
  if rg -q 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' "$f"; then
    echo "--- $f"
    rg -n -C 3 'DeepSeek-V4-Flash-Base|test_fp8_4gpus_static_eplb|test_nvfp4_4gpus_static_eplb|test_nvfp4_4gpus_online_eplb|test_fp8_4gpus_online_eplb' "$f"
  fi
done

Repository: NVIDIA/TensorRT-LLM

Length of output: 50375

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '== DeepSeek V4 Flash Base class =='
sed -n '4000,4085p' tests/integration/defs/accuracy/test_llm_api_pytorch.py

printf '%s\n' '== shared runner and configuration fields =='
sed -n '3820,3850p' tests/integration/defs/accuracy/test_llm_api_pytorch.py
sed -n '3880,3930p' tests/integration/defs/accuracy/test_llm_api_pytorch.py

printf '%s\n' '== exact test-list coverage for the affected class =='
rg -n -C 2 'TestDeepSeekV4FlashBase' tests/integration/test_lists

printf '%s\n' '== reference matching probe without repository imports =='
python3 - <<'PY'
from pathlib import Path

text = Path("tests/integration/defs/accuracy/references/gsm8k.yaml").read_text()
start = text.index("deepseek-ai/DeepSeek-V4-Flash-Base:")
end = text.find("\n", text.find("accuracy:", start)) + 1
print(text[start:end])

# The matcher compares only the fields passed by evaluate(); absent fields
# default to None except dtype, which defaults to "auto".
entry = {"quant_algo": "FP8_BLOCK_SCALES", "accuracy": 90.90}
for name, specs in {
    "static WIDEEP": {
        "dtype": "auto",
        "quant_algo": "FP8_BLOCK_SCALES",
        "kv_cache_quant_algo": None,
        "spec_dec_algo": None,
        "extra_acc_spec": None,
    },
    "online TRTLLM": {
        "dtype": "auto",
        "quant_algo": "FP8_BLOCK_SCALES",
        "kv_cache_quant_algo": None,
        "spec_dec_algo": None,
        "extra_acc_spec": None,
    },
}.items():
    matched = all(entry.get(k, "auto" if k == "dtype" else None) == v
                  for k, v in specs.items())
    print(name, "matches =", matched, "selected_accuracy =", entry["accuracy"])
PY

Repository: NVIDIA/TensorRT-LLM

Length of output: 11211

---

Scope the GSM8K reference to the measured configuration.

TestDeepSeekV4FlashBase.test_fp8_4gpus_online_eplb also selects 90.90 because the evaluator passes no EPLB or backend discriminator. Add a discriminator or a separate reference for the online configuration.

Coverage: No test functions changed. The static test is listed in tests/integration/test_lists/qa/llm_function_core.txt; the online test is not listed. Verdict: insufficient.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/accuracy/references/gsm8k.yaml` around lines 148 -
154, Scope the 90.90 GSM8K reference under DeepSeek-V4-Flash-Base to the static
test configuration by adding the appropriate EPLB/backend discriminator used by
test_fp8_4gpus_static_eplb. Ensure
TestDeepSeekV4FlashBase.test_fp8_4gpus_online_eplb no longer matches this
reference, or add a distinct reference for its configuration.