fix: validate scope checks ran and report overall test status

[pcarleton]

Summary

• Add FAILURE checks to scope scenarios when authorization flow doesn't complete
• Report client timeout and exit errors in test results
• Add OVERALL: PASSED/FAILED summary at end of test output
• Exit with code 1 on any failure (check failures, client timeout, or exit error)

Problem

Previously, if the OAuth authorization flow didn't complete (e.g., client timed out or exited with an error before the authorization request reached the server), the scope checks were never triggered and the test would incorrectly report success like "Passed: 6/6, 0 failed".

Solution

The following scope scenarios now emit a FAILURE if their expected scope check wasn't recorded:

• ScopeFromWwwAuthenticateScenario
• ScopeFromScopesSupportedScenario
• ScopeOmittedWhenUndefinedScenario

Additionally, printClientResults now:

• Accepts optional clientOutput to check for timeout/exit errors
• Reports warnings for client timeout or non-zero exit codes
• Shows an overall PASSED/FAILED summary
• Returns overallFailure boolean for proper exit code

Example output (failure case)

Test Results:
Passed: 6/7, 1 failed, 0 warnings

⚠️  CLIENT EXITED WITH ERROR (code 1) - Test may be incomplete

Failed Checks:
  - Client scope selection from scopes_supported: Client did not complete authorization flow - scope check could not be performed

❌ OVERALL: FAILED

🤖 Generated with Claude Code

[pkg-pr-new]

Open in StackBlitz

npx https://pkg.pr.new/modelcontextprotocol/conformance/@modelcontextprotocol/conformance@80

commit: 1c42539

[felixweinberger]

me parece bien

[felixweinberger]

nit: mutations in a getSomething might be slightly weird, but I deduplication above makes it idempotent at least.

[pcarleton]

good callout. i don't love this pattern in general of throwing in failed checks if they're missing. It feels easy to forget, and I'd rather have you decide on the language wherever you're defining the check. like either the checks are fully "in-line", or they're fully after the fact.

I will have to think if we can swap in something better here, but will leave this as is for now.

[felixweinberger]

more of a suggestion: IIUC we're only showing these output summaries for single scenarios, could consider doing that for test suites as well.