[https://nvbugs/6529626][fix] Pin mcp<2.0.0 and unwaive the scaffolding tests

[JunyiXu-nv]

Fixes https://nvbugs/6529626

What broke

mcp was unpinned at requirements.txt:80. mcp 2.0.0, published 2026-07-28 13:45 UTC, removed mcp.server.fastmcp:

version	files under mcp/server/fastmcp/
1.29.0	19
2.0.0	0

tests/unittest/scaffolding/test_mcp_worker.py imports it at module level, so any image built after that timestamp fails while pytest is collecting the file.

Why one broken import takes out an entire stage

A collection error is not a test failure. pytest reports Interrupted: 1 error during collection and exits 2 before running anything:

• the other seven files in tests/unittest/scaffolding never execute;
• no JUnit XML is written at all, because the run dies before --periodic-junit-xmlpath is honoured.

That is why this surfaces only as the outer wrapper's assert passed, "failure reported in unittests", with no inner test named anywhere in the archived stage results — the failing import is visible only in the stage's raw stdout.log.

It also cannot be waived. apply_waives (tests/integration/defs/test_list_parser.py) and apply_waives_ut (tests/unittest/conftest.py) both run inside pytest_collection_modifyitems and add skip markers to collected items; a module that fails to import produces none. The only waive that works is the directory-level one against the test_unittests_v2[unittest/scaffolding] wrapper (#16978), which suppresses all eight files to silence one.

Why pin rather than skip the test

tensorrt_llm/scaffolding/worker.py and tensorrt_llm/scaffolding/contrib/mcp/mcp_utils.py both ship against the 1.x API, and test_mcp_worker.py is the only CI coverage that exercises MCPWorker against the installed mcp. Guarding the test with importorskip would have dropped that coverage precisely when the dependency took a major version bump — hiding a library-level risk instead of resolving it.

Note the library's own imports (from mcp import ClientSession, from mcp.client.sse import sse_client) do still resolve under 2.0.0, so this is not an immediate import break in tensorrt_llm/; the point is that the 2.x behaviour is unverified, and the pin keeps the shipped code and its coverage on the version they were written against.

Lifting the pin means migrating the library off the 1.x APIs first, not just the test.

Unwaiving

#16978 waived the whole unittest/scaffolding directory as an interim unblock, because the collection error could not be waived at test granularity. With the pin in place that error is gone, so this change removes the waive in the same commit — leaving it would keep all eight scaffolding files skipped indefinitely for a cause that no longer exists.

-full:DGX_H100/unittest/scaffolding SKIP (https://nvbugs/6529626)

This also means the run on this PR is a direct test of the fix: if the pin did not work, unittest/scaffolding will fail here rather than being silently skipped.

Scope

unittest/scaffolding runs on H100 only — l0_h100.yml:116 is the sole test-db entry, in a block gated on orchestrator: mpi, stage: pre_merge. mcp<2.0.0 resolves to 1.29.0, which ships mcp/server/fastmcp/ intact. This is the only mcp declaration in the repo.

Dev Engineer Review

• Pinned mcp in requirements.txt to mcp<2.0.0 (with an inline note that mcp==2.0.0 removed mcp.server.fastmcp, while the scaffolding code still relies on the 1.x API).
• Change is narrowly scoped to dependency resolution to avoid pytest collection failures caused by the removed import; no other code or behavioral logic was modified.
• The follow-on issue (“migration to standalone fastmcp”) is still not addressed, but this unblocks existing MCP-based scaffolding tests.

QA Engineer Review

• Updated tests/integration/test_lists/waives.txt:
  • Removed the waiver entry for full:DGX_H100/unittest/scaffolding (SKIP reference to nvbugs/6529626).
• Verdict: needs follow-up (CBTS/coverage status for the scaffolding/test list change wasn’t provided).

[JunyiXu-nv]

/bot run

[coderabbitai]

Walkthrough

The PR constrains mcp to versions below 2.0.0 and removes the DGX H100 scaffolding test waiver from the integration waiver list.

Changes

MCP dependency constraint

Layer / File(s)	Summary
Constrain MCP dependency version requirements.txt	Changes mcp from unpinned to <2.0.0 and documents the mcp.server.fastmcp compatibility requirement.

Scaffolding test waiver removal

Layer / File(s)	Summary
Remove obsolete scaffolding waiver tests/integration/test_lists/waives.txt	Removes the DGX H100 scaffolding waiver associated with NVBug 6529626.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#16949: Modifies the same integration waiver list.
• NVIDIA/TensorRT-LLM#16978: Adds the same DGX H100 scaffolding waiver that this PR removes.
• NVIDIA/TensorRT-LLM#16985: Modifies entries in the same integration waiver list.

Suggested reviewers: bowenfu, mlefeb01, mzweilz, qijune, crazydemo

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title matches the change and follows the required [ticket][type] summary format.
Description check	✅ Passed	The description covers the issue, fix, and rationale, but it does not use the template's Test Coverage and PR Checklist sections.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #62371 [ run ] triggered by Bot. Commit: 2ac3f5f Link to invocation

[JunyiXu-nv]

/bot run

[tensorrt-cicd]

PR_Github #62378 [ run ] triggered by Bot. Commit: 963c128 Link to invocation

[tensorrt-cicd]

PR_Github #62371 [ run ] completed with state ABORTED. Commit: 2ac3f5f

Link to invocation

[JunyiXu-nv]

Please also unwaive the scaffolding tests.

[JunyiXu-nv]

/bot run

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_lists/waives.txt`:
- Line 187: Add the required NVIDIA copyright header at the top of the modified
waives.txt file and set its copyright year to the current year, leaving the
existing test entry for test_auto_dtype unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6b2a9b0f-7437-4725-8ed8-56e6ac54eaf7

📥 Commits

Reviewing files that changed from the base of the PR and between 963c128 and 8ff4b7f.

📒 Files selected for processing (2)

• requirements.txt
• tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

• requirements.txt

[tensorrt-cicd]

PR_Github #62404 [ run ] triggered by Bot. Commit: 8ff4b7f Link to invocation

[tensorrt-cicd]

PR_Github #62378 [ run ] completed with state ABORTED. Commit: 963c128

Link to invocation

[tensorrt-cicd]

PR_Github #62404 [ run ] completed with state SUCCESS. Commit: 8ff4b7f
/LLM/main/L0_MergeRequest_PR pipeline #50565 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[BowenFu]

Verified the pin can't regress anything already working: nothing in requirements/setup/docker/constraints or any pyproject requires mcp 2.x (security_scanning already caps >=1.28.1,<2.0.0), and the client APIs scaffolding actually uses — ClientSession, sse_client, initialize, list_tools, call_tool — are all present in 1.29.0. Waiver line matches exactly once on main, removed in the same commit as the cause.

Not blocking on it, but worth knowing: the run on this head was aborted by fail-fast before unittest/scaffolding actually executed, so this PR hasn't yet demonstrated the green scaffolding run your description promises — the two failures it did report (modeling_qwen on A30, test_nvfp4 on B200) are unrelated, and modeling_qwen also failed on #16921's build 50499.

[JunyiXu-nv]

Pipeline #50565 — two unrelated failures, and an honest correction

First, a correction to what I said above. I claimed that removing the waive makes this PR's own pipeline a direct test of the pin. It did not test it. #50565 completed 40 stages and unittest/scaffolding never ran: the DGX_H100 stages that uploaded (PyTorch-2, -4, -5) executed 17–22 unittest cases each, none of them scaffolding. Stage sharding is dynamic, so it landed on a stage that either did not run or did not upload. The pin remains unverified in CI.

Also worth noting for anyone reading the bot comment: it says completed with state SUCCESS while the pipeline line says FAILURE. The pipeline line is the one that matters.

The two actual failures:

[A30-PyTorch-2]       test_unittests_v2[unittest/_torch/modeling -k "modeling_qwen"]
[DGX_B200-PyTorch-7]  test_nvfp4

Ambient rates, from builds that are not this PR's. My scan window covers the last 50 pipelines, which has since moved past #50565 — so the counts below contain no data from this PR, making them independent sightings rather than self-reports:

test	stage-runs	failed	where
unittest/_torch/modeling -k "modeling_qwen"	24	2	#50618, #50639 — both unrelated PRs
test_nvfp4	81	1	#50638 — unrelated PR

One methodological caveat I should state rather than let the table imply more than it supports: test_nvfp4 is matched by test name only, and that name exists in several test classes, so the 1/81 conflates distinct tests that happen to share a name. Treat it as weak evidence. The modeling_qwen figure is a single specific parametrisation and is sound.

Why the change is very unlikely to be the cause: the diff is one dependency pin and one waive line. mcp is imported only by tensorrt_llm/scaffolding/ and tests/unittest/scaffolding/, neither of which is on the path of a Qwen modeling test or an nvfp4 accuracy test. I will not overstate this as strict unreachability, though — pinning a dependency can in principle shift transitive resolution, so it is a strong argument rather than an airtight one.

Status of the waive, measured: across the last 50 pipelines the scaffolding wrapper appears 17 times, and in every case it is skipped with message https://nvbugs/6529626 — i.e. #16978's waive is doing exactly its job, and none of those 17 are passes. That is why this PR needs a run where scaffolding is actually scheduled.

Re-running, with the hope of drawing a stage that schedules unittest/scaffolding so the pin gets a real test.

[JunyiXu-nv]

/bot run

[tensorrt-cicd]

PR_Github #62509 [ run ] triggered by Bot. Commit: 8ff4b7f Link to invocation

[tensorrt-cicd]

PR_Github #62509 [ run ] completed with state SUCCESS. Commit: 8ff4b7f
/LLM/main/L0_MergeRequest_PR pipeline #50657 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation