test(go): scenario parity runner — Go server runs the shared conformance corpus - #65
Merged
Conversation
…nce corpus
Port the scenario parity runner (the Python reference at
python/server/tests/test_scenario_parity.py) into the Go server's test suite.
It runs every spec/conformance/scenarios/*.json through the Go server as a
subtest and asserts the normalized outbound event stream — the same corpus the
Python server already runs. When all five native servers run this corpus green,
the polyglot servers are at protocol parity.
The runner drives the server over the raw WebSocket transport (not the typed
client) so it asserts the exact wire frames, faithfully porting the reference
state machine: {{var}} substitution, MockLlmProvider build from mockLlmScript,
per-matcher type / status / statusGte / assert (dot-path) / capture / repeat /
accumulate + assertAccumulated, skipping keepalive/pong.
basic-streaming-turn passes. unknown-session-error reveals a real Go-server
protocol divergence and is skipped via knownGoDivergence with the exact wire
frame documented: the Go error event emits {"data":{"error":{"code":"NOT_FOUND"
…}}} but the corpus (Python reference, Rust server/lambda, error.schema.json
examples) requires the error descriptor ALSO at the event top level
(event.error.{code,message}) and the code SESSION_NOT_FOUND, not NOT_FOUND.
That skip must be deleted when the Go server is fixed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7Mn93HpqhSgEmX6tRdPAv
|
brentrager
added a commit
that referenced
this pull request
Jun 24, 2026
…rgence it caught (#71) New scenario: an unrecognized action must return error/UNSUPPORTED_ACTION with the requestId echoed. It caught a REAL Go-server bug that #65 had masked with a knownGoDivergence SKIP: Go errorEvent() emitted the {code,message} descriptor only under data.error, never at the envelope level error — and used NOT_FOUND instead of canonical SESSION_NOT_FOUND. Both diverge from spec/events/error.schema.json + the Python/Rust/C#/TS servers. Fix (root cause): protocol.go errorEvent now sets the envelope-level error descriptor (kept data.error for back-compat); dispatcher.go uses SESSION_NOT_FOUND; updated the existing TestUnknownSessionErrors to the canonical code; removed the knownGoDivergence skip. Go now passes ALL scenarios (incl. the previously-skipped unknown-session-error) — full suite green -race. All 5 servers green on the new scenario. Claude-Session: https://claude.ai/code/session_01U7Mn93HpqhSgEmX6tRdPAv Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the scenario parity runner (Python reference:
python/server/tests/test_scenario_parity.py) into the Go server's test suite. The newgo/server/scenario_parity_test.goruns everyspec/conformance/scenarios/*.jsonthrough the Go server as a subtest and asserts the normalized outbound event stream — the same shared corpus the Python server already runs. When all five native servers (Rust · C# · Python · TypeScript · Go) run this corpus green, the polyglot servers are at protocol parity.How
spec/conformance/scenarios/*.json(resolved relative to the repo root), onet.Runsubtest per scenario.MockLlmProviderfrommockLlmScript(PushText/PushToolCall), starts the server in local flavor with it as the chat client.WebSocketTransport(not the typedprotocol.Client) so it asserts the exact wire frames, faithfully porting the reference state machine:{{var}}substitution, ordered matcher loop with one-event lookahead forrepeat,status/statusGte/assert(dot-path) /capture/accumulate+assertAccumulated, skippingkeepalive/pong. JSON-decoded corpus values and the server's marshaled events are compared structurally through a normalizing round-trip (sointvsfloat64and[]stringvs[]anycompare equal).Result
basic-streaming-turn— passes unchanged.unknown-session-error— reveals a real Go-server protocol divergence (see below). It's skipped viaknownGoDivergencewith the exact wire frame and canonical sources documented in-code, so the runner lands green and CI-safe. That skip must be deleted the moment the Go server is fixed.go/servertests still pass.go test -race ./...green,go vetclean,gofmtclean.On
send_messageto an unknown session the Go server emits:{"data":{"error":{"code":"NOT_FOUND","message":"Session not found"}},"requestId":"r-orphan","type":"error"}The corpus — and the Python reference, the Rust server (
handler.rs) and Rust lambda (dispatch.rs), and thespec/events/error.schema.jsonexamples — require:event.error.{code,message}); the Go event only nests it underdata.error.SESSION_NOT_FOUND, notNOT_FOUND.Source:
go/server/dispatcher.gohandleSendMessage+go/server/protocol.goerrorEvent. The C# server shares divergence (2) (NOT_FOUND). Fixing the server is left as a follow-up so this PR stays test-only.🤖 Generated with Claude Code