Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## [Unreleased]

## [1.26.0] - 2026-05-18

### Added

- **Timing-aware recording and replay** — proxy recording captures per-frame
Expand All @@ -13,6 +15,12 @@
sources (recorded timings, streaming profiles, global latency). Per-fixture
`replaySpeed` override. Covers SSE, NDJSON, Bedrock EventStream, and
WebSocket protocols.
- **Context-based fixture routing** — `X-AIMock-Context` header scopes fixtures per integration. Fixtures with `match.context` only match requests carrying that context; fixtures without `context` remain shared. Recorder auto-captures context and routes recorded fixtures into context subdirectories.

## [1.25.0] - 2026-05-18

### Added

- **Gemini `embedContent` endpoint** — `POST /v1beta/models/{model}:embedContent`
with deterministic fallback embeddings and fixture matching
- **`/v1/images/edit` and `/v1/images/variations` endpoints** — multipart
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or
- **[MCP](https://aimock.copilotkit.dev/mcp-mock) / [A2A](https://aimock.copilotkit.dev/a2a-mock) / [AG-UI](https://aimock.copilotkit.dev/agui-mock) / [Vector](https://aimock.copilotkit.dev/vector-mock)** — Mock every protocol your AI agents use
- **[Chaos Testing](https://aimock.copilotkit.dev/chaos-testing)** — 500 errors, malformed JSON, mid-stream disconnects at any probability
- **Per-Request Strict Mode** — `X-AIMock-Strict` header overrides the server-level `--strict` flag per request (`true`/`1` = strict, `false`/`0` = lenient)
- **Context-Based Fixture Routing** — `X-AIMock-Context` header scopes fixtures per integration; fixtures with `match.context` only match requests carrying that context, fixtures without it remain shared
- **[Drift Detection](https://aimock.copilotkit.dev/drift-detection)** — Daily CI validation against real APIs
- **[Streaming Physics](https://aimock.copilotkit.dev/streaming-physics)** — Configurable `ttft`, `tps`, and `jitter`
- **[WebSocket APIs](https://aimock.copilotkit.dev/websocket)** — OpenAI Realtime (GA protocol with models: gpt-realtime, gpt-realtime-2, gpt-realtime-1.5, gpt-realtime-mini; transcription/translation via gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1; image input; commentary phase), Responses WS, Gemini Live
Expand Down
8 changes: 8 additions & 0 deletions docs/docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,14 @@ <h2>What's New</h2>
<p class="highlight-card-desc">One-line CI setup for mock-backed test suites</p>
<a href="/github-action" class="highlight-card-cta">Setup <span>&rarr;</span></a>
</div>

<div class="highlight-card">
<span class="highlight-card-title">Context Routing</span>
<p class="highlight-card-desc">
Scope fixtures per integration with <code>X-AIMock-Context</code> header
</p>
<a href="/fixtures#context" class="highlight-card-cta">Docs <span>&rarr;</span></a>
</div>
</div>

<!-- ─── The Suite (compact table) ───────────────────────────── -->
Expand Down
50 changes: 42 additions & 8 deletions docs/fixtures/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,16 @@ <h2>Match Fields</h2>
their own fixture APIs rather than via this field
</td>
</tr>
<tr>
<td>context</td>
<td>string</td>
<td>
Restrict to a named context via <code>X-AIMock-Context</code> header. Fixtures with
<code>context</code> only match requests carrying that exact value; fixtures without
<code>context</code> match any request. Same opt-in semantics as
<code>endpoint</code>
</td>
</tr>
<tr>
<td>predicate</td>
<td>function</td>
Expand Down Expand Up @@ -242,14 +252,15 @@ <h3>5. Validation warnings surface shadowing at load time</h3>
<code>duplicate userMessage 'hello' — shadows fixture 0</code>, where
<code>'hello'</code> is the duplicated message and <code>0</code> is the zero-based
index of the earlier fixture being shadowed. This is advisory, not a hard error: the
check now factors in <code>turnIndex</code>, <code>hasToolResult</code>, and
<code>sequenceIndex</code> when deciding whether two fixtures truly collide, but it does
<em>not</em> consider <code>toolCallId</code>, <code>model</code>, or
<code>predicate</code>, so the warning may still fire when those discriminators are
present. Treat it as advisory: if a runtime differentiator is in place, the fixtures
won't actually shadow each other at match time. Only fixtures with no differentiator at
all will truly shadow on match &mdash; that's the case where the second is never reached
because the first wins. Safe to ignore in the former case; investigate in the latter.
check now factors in <code>turnIndex</code>, <code>hasToolResult</code>,
<code>context</code>, and <code>sequenceIndex</code> when deciding whether two fixtures
truly collide, but it does <em>not</em> consider <code>toolCallId</code>,
<code>model</code>, or <code>predicate</code>, so the warning may still fire when those
discriminators are present. Treat it as advisory: if a runtime differentiator is in
place, the fixtures won't actually shadow each other at match time. Only fixtures with
no differentiator at all will truly shadow on match &mdash; that's the case where the
second is never reached because the first wins. Safe to ignore in the former case;
investigate in the latter.
</li>
<li>
<strong>Catch-all not last</strong> &mdash; a fixture with an empty <code>match</code>
Expand Down Expand Up @@ -511,6 +522,29 @@ <h3>From a directory</h3>
</p>
</div>

<h3 id="context">Context-scoped fixtures</h3>
<div class="code-block">
<div class="code-block-header">
fixtures/context-example.json <span class="lang-tag">json</span>
</div>
<pre><code>{
<span class="key">"fixtures"</span>: [
{
<span class="key">"match"</span>: { <span class="key">"userMessage"</span>: <span class="str">"hello"</span>, <span class="key">"context"</span>: <span class="str">"langgraph-python"</span> },
<span class="key">"response"</span>: { <span class="key">"content"</span>: <span class="str">"Hi from LangGraph!"</span> }
},
{
<span class="key">"match"</span>: { <span class="key">"userMessage"</span>: <span class="str">"hello"</span> },
<span class="key">"response"</span>: { <span class="key">"content"</span>: <span class="str">"Hi from the shared fallback!"</span> }
}
]
}</code></pre>
</div>
<p>
Requests with <code>X-AIMock-Context: langgraph-python</code> match the first fixture; all
other requests fall through to the shared fixture.
</p>

<h3>Programmatically</h3>
<div class="code-block">
<div class="code-block-header">programmatic.ts <span class="lang-tag">ts</span></div>
Expand Down
23 changes: 22 additions & 1 deletion docs/multi-turn/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,10 @@ <h3>Turn 2 &mdash; client runs the tool, sends the result</h3>
</p>
</div>

<h2>Choosing between sequenceIndex, toolCallId, turnIndex, hasToolResult, and predicate</h2>
<h2>
Choosing between sequenceIndex, toolCallId, turnIndex, hasToolResult, context, and
predicate
</h2>
<p>Five mechanisms handle different shapes of &ldquo;the same prompt twice&rdquo;:</p>

<table class="endpoint-table">
Expand Down Expand Up @@ -341,6 +344,15 @@ <h2>Choosing between sequenceIndex, toolCallId, turnIndex, hasToolResult, and pr
pinning a specific <code>tool_call_id</code>.
</td>
</tr>
<tr>
<td>Same user prompt, different response per integration or caller identity</td>
<td><code>context</code></td>
<td>
Exact match on the <code>X-AIMock-Context</code> header. Fixtures with
<code>context</code> only match requests carrying that value; fixtures without it
remain shared. Stateless.
</td>
</tr>
<tr>
<td>
Arbitrary inspection &mdash; message count, specific content at any position, custom
Expand Down Expand Up @@ -442,6 +454,15 @@ <h2 id="gotchas">Gotchas</h2>
instance. See <a href="/sequential-responses">Sequential Responses</a> for when
<code>sequenceIndex</code> is the right tool.
</li>
<li>
<strong><code>context</code> is an additional discriminator, not a replacement.</strong>
<code>context</code> scopes fixtures by integration identity (<code
>X-AIMock-Context</code
>
header). It combines with all other match fields via AND. Two fixtures with the same
<code>userMessage</code> but different <code>context</code>
values are not duplicates &mdash; the validator accounts for this.
</li>
</ul>
</main>
<aside class="page-toc" id="page-toc"></aside>
Expand Down
48 changes: 47 additions & 1 deletion docs/record-replay/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -556,6 +556,49 @@ <h2 id="model-aware-recording">Model-Aware Recording</h2>
});</code></pre>
</div>

<h2 id="context-aware-recording">Context-Aware Recording</h2>
<p>
When a request carries an <code>X-AIMock-Context</code> header, the recorder automatically
captures the context value in <code>match.context</code>. On replay, fixtures with
<code>context</code> only match requests carrying that exact header value &mdash; fixtures
without <code>context</code> remain shared across all callers.
</p>

<h3>Directory routing</h3>
<p>
Without snapshot-style recording (<code>X-Test-Id</code>), recorded fixtures for a given
context are written to a <code>&lt;fixturePath&gt;/&lt;context&gt;/</code> subdirectory:
</p>

<div class="code-block">
<div class="code-block-header">
Context directory layout <span class="lang-tag">text</span>
</div>
<pre><code>fixtures/recorded/
openai-2026-05-18T10-30-00-000Z-a1b2c3d4.json # no context (shared)
langgraph-python/
openai-2026-05-18T10-30-01-000Z-e5f6a7b8.json # context = langgraph-python
crewai/
openai-2026-05-18T10-30-02-000Z-c9d0e1f2.json # context = crewai</code></pre>
</div>

<p>
When <code>X-Test-Id</code> is also present, snapshot-style paths take precedence and the
context is captured only in <code>match.context</code> within the fixture file, not in the
directory structure.
</p>

<h3>Sending <code>X-AIMock-Context</code></h3>
<div class="code-block">
<div class="code-block-header">Header example <span class="lang-tag">ts</span></div>
<pre><code><span class="cm">// Set as a default header on your LLM client</span>
<span class="kw">const</span> <span class="op">client</span> = <span class="kw">new</span> <span class="fn">OpenAI</span>({
<span class="prop">baseURL</span>: <span class="str">"http://localhost:4010/v1"</span>,
<span class="prop">apiKey</span>: <span class="str">"mock"</span>,
<span class="prop">defaultHeaders</span>: { <span class="str">"X-AIMock-Context"</span>: <span class="str">"langgraph-python"</span> },
});</code></pre>
</div>

<h2 id="upstream-timeouts">Upstream Timeouts</h2>

<p>
Expand Down Expand Up @@ -858,7 +901,10 @@ <h2 id="recording-multi-turn-conversations">Recording Multi-Turn Conversations</
}
<span class="cm">// Chat/multimedia — key on the LAST user message only</span>
<span class="kw">const</span> lastUser = <span class="fn">getLastMessageByRole</span>(request.messages, <span class="str">"user"</span>);
<span class="kw">return</span> { <span class="prop">userMessage</span>: <span class="fn">getTextContent</span>(lastUser.content) };
<span class="kw">const</span> match = { <span class="prop">userMessage</span>: <span class="fn">getTextContent</span>(lastUser.content) };
<span class="cm">// Capture context from X-AIMock-Context header if present</span>
<span class="kw">if</span> (request._context) match.<span class="prop">context</span> = request._context;
<span class="kw">return</span> match;
}</code></pre>
</div>

Expand Down
33 changes: 33 additions & 0 deletions src/__tests__/fixture-loader.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1006,6 +1006,30 @@ describe("validateFixtures", () => {
).toBe(true);
});

// --- match.context type checks ---

it("validateFixtures reports error for non-string context", () => {
const fixtures = [
makeFixture({ match: { userMessage: "test", context: 42 as unknown as string } }),
];
const results = validateFixtures(fixtures);
expect(
results.some((r) => r.severity === "error" && r.message.includes("must be a string")),
).toBe(true);
});

it("context is a valid discriminator", () => {
const fixtures = [
makeFixture({ match: { context: "x" } }),
makeFixture({ match: { userMessage: "hello" } }),
];
const results = validateFixtures(fixtures);
const catchAllWarnings = results.filter(
(r) => r.severity === "warning" && r.message.includes("catch-all"),
);
expect(catchAllWarnings).toHaveLength(0);
});

// --- Warning checks ---

it("warning: duplicate userMessage", () => {
Expand Down Expand Up @@ -1545,6 +1569,15 @@ describe("auto-stringify JSON objects in fixture entries", () => {
expect((fixture.response as TextResponse).content).toBe("Hello, world!");
});

it("entryToFixture preserves context field", () => {
const entry: FixtureFileEntry = {
match: { context: "my-ctx", userMessage: "hi" },
response: { content: "ok" },
};
const fixture = entryToFixture(entry);
expect(fixture.match.context).toBe("my-ctx");
});

it("passes systemMessage through entryToFixture", () => {
const entry: FixtureFileEntry = {
match: { userMessage: "test", systemMessage: "name=Atai" },
Expand Down
28 changes: 28 additions & 0 deletions src/__tests__/helpers.test.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import http from "node:http";
import { describe, it, expect } from "vitest";
import {
generateId,
Expand All @@ -11,6 +12,7 @@ import {
buildToolCallChunks,
buildTextCompletion,
buildToolCallCompletion,
getContext,
} from "../helpers.js";

describe("generateId", () => {
Expand Down Expand Up @@ -292,6 +294,32 @@ describe("buildToolCallChunks", () => {
});
});

describe("getContext", () => {
it("returns header value", () => {
const req = {
headers: { "x-aimock-context": "langgraph-python" },
} as unknown as http.IncomingMessage;
expect(getContext(req)).toBe("langgraph-python");
});

it("returns undefined when header absent", () => {
const req = { headers: {} } as unknown as http.IncomingMessage;
expect(getContext(req)).toBeUndefined();
});

it("returns first value from array header", () => {
const req = {
headers: { "x-aimock-context": ["first", "second"] },
} as unknown as http.IncomingMessage;
expect(getContext(req)).toBe("first");
});

it("returns undefined for empty string", () => {
const req = { headers: { "x-aimock-context": "" } } as unknown as http.IncomingMessage;
expect(getContext(req)).toBeUndefined();
});
});

describe("buildTextCompletion", () => {
it("returns a valid chat.completion object", () => {
const result = buildTextCompletion("Hello!", "gpt-4");
Expand Down
25 changes: 24 additions & 1 deletion src/__tests__/recorder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import * as os from "node:os";
import * as path from "node:path";
import type { Fixture, FixtureFile } from "../types.js";
import { createServer, type ServerInstance } from "../server.js";
import { proxyAndRecord, type ProxyCapturedResponse } from "../recorder.js";
import { proxyAndRecord, buildFixtureMatch, type ProxyCapturedResponse } from "../recorder.js";
import type { RecordConfig } from "../types.js";
import { Logger } from "../logger.js";
import { LLMock } from "../llmock.js";
Expand Down Expand Up @@ -4105,6 +4105,29 @@ describe("buildFixtureMatch model recording", () => {
});
});

// ---------------------------------------------------------------------------
// buildFixtureMatch context
// ---------------------------------------------------------------------------

describe("buildFixtureMatch context", () => {
it("captures _context in match criteria", () => {
const match = buildFixtureMatch({
model: "gpt-4o",
messages: [{ role: "user", content: "hello" }],
_context: "langgraph-python",
});
expect(match.context).toBe("langgraph-python");
});

it("omits context when _context is absent", () => {
const match = buildFixtureMatch({
model: "gpt-4o",
messages: [{ role: "user", content: "hello" }],
});
expect(match.context).toBeUndefined();
});
});

async function setupUpstreamAndRecorder(
upstreamFixtures: Fixture[],
providerKey: string = "openai",
Expand Down
37 changes: 37 additions & 0 deletions src/__tests__/router.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1107,6 +1107,43 @@ describe("matchFixture — hasToolResult", () => {
});
});

// ---------------------------------------------------------------------------
// matchFixture — context matching
// ---------------------------------------------------------------------------

describe("matchFixture — context matching", () => {
it("matches fixture with matching context", () => {
const fixture = makeFixture({ context: "foo" });
const req = makeReq({ _context: "foo" });
expect(matchFixture([fixture], req)).toBe(fixture);
});

it("skips fixture with non-matching context", () => {
const fixture = makeFixture({ context: "foo" });
const req = makeReq({ _context: "bar" });
expect(matchFixture([fixture], req)).toBeNull();
});

it("matches fixture without context regardless of request context", () => {
const fixture = makeFixture({});
const req = makeReq({ _context: "bar" });
expect(matchFixture([fixture], req)).toBe(fixture);
});

it("skips context fixture when request has no context", () => {
const fixture = makeFixture({ context: "foo" });
const req = makeReq();
expect(matchFixture([fixture], req)).toBeNull();
});

it("context fixture wins over shared when listed first", () => {
const contextual = makeFixture({ context: "foo" }, { content: "contextual" });
const shared = makeFixture({}, { content: "shared" });
const req = makeReq({ _context: "foo" });
expect(matchFixture([contextual, shared], req)).toBe(contextual);
});
});

// ---------------------------------------------------------------------------
// matchFixture — first-match-wins
// ---------------------------------------------------------------------------
Expand Down
Loading
Loading