test(dotnet): scenario parity runner — C# server runs the shared conformance corpus - #64
Merged
Conversation
…ormance corpus
Ports python/server/tests/test_scenario_parity.py into the C# server's integration
test suite. ScenarioParityTests discovers spec/conformance/scenarios/*.json and, for
each scenario ([Theory]/[MemberData]), drives the in-process ASP.NET Core WebSocket
host through it: seeds MockChatClient from mockLlmScript, then per step substitutes
{{vars}}, sends the frame, and matches the expect sequence (type, status/statusGte,
assert dot-paths, capture vars, repeat runs, accumulate+assertAccumulated), skipping
keepalive/pong. When all five servers run this corpus green, they are at protocol parity.
Running the corpus surfaced two real C#-server divergences from the shared
spec/events/error.schema.json (which the Python reference server already honors):
- The error frame omitted the envelope-level `error` object. The schema requires the
{code, message} descriptor at BOTH the top level and nested under `data.error` (the
top-level copy "kept for clients that pattern-match on the envelope-level `error`
field"). ProtocolEvents.Error now emits both (and data.requestId), mirroring Python.
- The unknown-session error used code "NOT_FOUND"; the spec/Python use
"SESSION_NOT_FOUND". FrameDispatcher now emits SESSION_NOT_FOUND for both the
get_session and send_message no-such-session paths.
Both seed scenarios (basic-streaming-turn, unknown-session-error) pass, and all
existing dotnet tests stay green (integration 13, server 50, host 2, postgres 33, core 27).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7Mn93HpqhSgEmX6tRdPAv
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the scenario parity runner (
python/server/tests/test_scenario_parity.py) into the C# server's integration test suite, so the C# server runs the sharedspec/conformance/scenarios/*.jsoncorpus — the same corpus the Python server already runs. When all five native servers (Rust · C# · Python · TypeScript · Go) run this corpus green, they are at protocol parity.ScenarioParityTests(dotnet/server/integration-tests/ScenarioParityTests.cs) discovers every scenario via[Theory]/[MemberData]and, for each, drives the in-process ASP.NET Core WebSocket host (reusing the existingMockChatClient+ WS host fromWebSocketProtocolIntegrationTests):MockChatClientfrommockLlmScript({kind:"text"}→PushText,{kind:"toolCall"}→PushToolCall),{{vars}}, sends the frame, then matches theexpectsequence —type,status/statusGte,assertdot-paths,capturevars,repeatruns,accumulate+assertAccumulated— skippingkeepalive/pong.Divergences found + fixed
Running the corpus surfaced two real C#-server divergences from the shared
spec/events/error.schema.json(the Python reference server already honors both):errorobject — the schema requires the{code, message}descriptor at both the top level (error) and nested underdata.error(the top-level copy is "kept for clients that pattern-match on the envelope-levelerrorfield"). The C# server emitted onlydata.error.ProtocolEvents.Errornow emits both (plusdata.requestId), mirroring Python."NOT_FOUND"; the spec and Python server use"SESSION_NOT_FOUND".FrameDispatchernow emitsSESSION_NOT_FOUNDfor both no-such-session paths (get_session+send_message).These are server-conformance fixes (not test or scenario changes); the seed scenarios pass unchanged.
Verify
dotnet test dotnet/server/integration-tests/ -c Release→ 13 passed (both parity scenarios + 11 existing integration tests). No regressions across the other dotnet projects: server 50, host 2, postgres 33, core 27.🤖 Generated with Claude Code