Skip to content

test(ts): scenario parity runner — TS server runs the shared conformance corpus - #63

Merged
brentrager merged 1 commit into
mainfrom
parity-ts
Jun 24, 2026
Merged

test(ts): scenario parity runner — TS server runs the shared conformance corpus#63
brentrager merged 1 commit into
mainfrom
parity-ts

Conversation

@brentrager

Copy link
Copy Markdown
Contributor

What

Ports the Python reference scenario runner (python/server/tests/test_scenario_parity.py) into the TypeScript server suite as typescript/server/test/scenario-parity.test.ts. It globs spec/conformance/scenarios/*.json and, per scenario, drives the TS server through each step — substituting {{vars}}, building the engine MockLlmProvider from mockLlmScript, then matching the ordered expect stream (status/statusGte, dot-path assert, capture, repeat, accumulate+assertAccumulated), skipping non-semantic keepalive/pong. Reuses the existing serve() helper + TestClient WS client.

This is the same shared corpus the Python server already runs. When all five native servers run it green, the polyglot servers are at protocol parity.

Parity divergence found + fixed (not papered over)

Running the corpus against the unchanged TS server failed unknown-session-error.json — it revealed two real TS-server divergences from the Python/Rust reference. Per the runner contract (the corpus is the authoritative parity contract), I fixed the TS server rather than the scenario:

  1. Envelope-level error descriptor was missing. protocol.error only nested { code, message } under data.error. The Python and Rust reference servers — and spec/events/error.schema.json — duplicate the descriptor at the envelope level (error) too, which is what clients (and the corpus) pattern-match on. Now duplicated at both levels.
  2. Wrong error code. Session-not-found emitted NOT_FOUND; the Python dispatcher and the Rust server/lambda handlers use SESSION_NOT_FOUND. Aligned both get_session and send_message paths (and the message format).

Updated the existing turn-round-trip + protocol-conformance tests to assert the corrected, parity-aligned shape.

Verify

pnpm --filter '@smooai/smooth-operator-server' test — both seed scenarios (basic-streaming-turn, unknown-session-error) pass and all existing server tests stay green (29/29). typecheck clean.

Heads-up for a follow-up (out of scope here): the Go server's commit log mentions it also emits NOT_FOUND for unknown-session — it likely has the same divergence and will fail this corpus when its runner is ported.

🤖 Generated with Claude Code

…nce corpus

Ports the Python reference scenario runner (python/server/tests/
test_scenario_parity.py) into the TS server suite: globs
spec/conformance/scenarios/*.json and, per scenario, drives the TS server through
each step — substituting {{vars}}, building the engine MockLlmProvider from
mockLlmScript, then matching the ordered expect stream (status/statusGte, dot-path
assert, capture, repeat, accumulate+assertAccumulated), skipping keepalive/pong.
Reuses the existing serve() helper + TestClient WS client.

The corpus surfaced two real TS-server divergences from the Python/Rust reference
(unknown-session-error.json failed against the unchanged TS server). Fixed the
server to bring it to parity rather than paper over the corpus:

- protocol.error now duplicates the {code, message} descriptor at the envelope
  level (error) in addition to data.error, matching the Python/Rust servers and
  error.schema.json. The corpus (and clients) pattern-match on the envelope-level
  error; the TS server previously only nested it under data.error.
- session-not-found now uses code SESSION_NOT_FOUND (was NOT_FOUND), matching the
  Python dispatcher and the Rust server/lambda handlers, for both get_session and
  send_message.

Updated the existing round-trip + conformance tests to assert the corrected,
parity-aligned error shape. Both seed scenarios (basic-streaming-turn,
unknown-session-error) pass; all existing server tests stay green.

Claude-Session: https://claude.ai/code/session_01U7Mn93HpqhSgEmX6tRdPAv

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Jun 24, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 70e78b6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant