test(ts): scenario parity runner — TS server runs the shared conformance corpus - #63
Merged
Conversation
…nce corpus
Ports the Python reference scenario runner (python/server/tests/
test_scenario_parity.py) into the TS server suite: globs
spec/conformance/scenarios/*.json and, per scenario, drives the TS server through
each step — substituting {{vars}}, building the engine MockLlmProvider from
mockLlmScript, then matching the ordered expect stream (status/statusGte, dot-path
assert, capture, repeat, accumulate+assertAccumulated), skipping keepalive/pong.
Reuses the existing serve() helper + TestClient WS client.
The corpus surfaced two real TS-server divergences from the Python/Rust reference
(unknown-session-error.json failed against the unchanged TS server). Fixed the
server to bring it to parity rather than paper over the corpus:
- protocol.error now duplicates the {code, message} descriptor at the envelope
level (error) in addition to data.error, matching the Python/Rust servers and
error.schema.json. The corpus (and clients) pattern-match on the envelope-level
error; the TS server previously only nested it under data.error.
- session-not-found now uses code SESSION_NOT_FOUND (was NOT_FOUND), matching the
Python dispatcher and the Rust server/lambda handlers, for both get_session and
send_message.
Updated the existing round-trip + conformance tests to assert the corrected,
parity-aligned error shape. Both seed scenarios (basic-streaming-turn,
unknown-session-error) pass; all existing server tests stay green.
Claude-Session: https://claude.ai/code/session_01U7Mn93HpqhSgEmX6tRdPAv
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the Python reference scenario runner (
python/server/tests/test_scenario_parity.py) into the TypeScript server suite astypescript/server/test/scenario-parity.test.ts. It globsspec/conformance/scenarios/*.jsonand, per scenario, drives the TS server through each step — substituting{{vars}}, building the engineMockLlmProviderfrommockLlmScript, then matching the orderedexpectstream (status/statusGte, dot-pathassert,capture,repeat,accumulate+assertAccumulated), skipping non-semantickeepalive/pong. Reuses the existingserve()helper +TestClientWS client.This is the same shared corpus the Python server already runs. When all five native servers run it green, the polyglot servers are at protocol parity.
Parity divergence found + fixed (not papered over)
Running the corpus against the unchanged TS server failed
unknown-session-error.json— it revealed two real TS-server divergences from the Python/Rust reference. Per the runner contract (the corpus is the authoritative parity contract), I fixed the TS server rather than the scenario:errordescriptor was missing.protocol.erroronly nested{ code, message }underdata.error. The Python and Rust reference servers — andspec/events/error.schema.json— duplicate the descriptor at the envelope level (error) too, which is what clients (and the corpus) pattern-match on. Now duplicated at both levels.NOT_FOUND; the Python dispatcher and the Rust server/lambda handlers useSESSION_NOT_FOUND. Aligned bothget_sessionandsend_messagepaths (and the message format).Updated the existing
turn-round-trip+protocol-conformancetests to assert the corrected, parity-aligned shape.Verify
pnpm --filter '@smooai/smooth-operator-server' test— both seed scenarios (basic-streaming-turn,unknown-session-error) pass and all existing server tests stay green (29/29).typecheckclean.🤖 Generated with Claude Code