Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 153 additions & 101 deletions CHANGELOG.md

Large diffs are not rendered by default.

13 changes: 10 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,20 @@ npm install @copilotkit/aimock
```

```typescript
// The class is still named `LLMock` for back-compat after the v1.7.0 package
// rename from `@copilotkit/llmock` to `@copilotkit/aimock`.
import { LLMock } from "@copilotkit/aimock";

const mock = new LLMock({ port: 0 });
mock.onMessage("hello", { content: "Hi there!" });
await mock.start();

// Set env BEFORE importing/constructing the OpenAI (or other provider) client.
// Many SDKs cache the base URL at construction time — if the client is built
// before these are set, it will talk to the real API (surprise bills) instead
// of aimock.
process.env.OPENAI_BASE_URL = `${mock.url}/v1`;
process.env.OPENAI_API_KEY = "mock"; // SDK requires a value, even when base URL is mocked

// ... run your tests ...

Expand All @@ -44,8 +51,8 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or
- **[Record & Replay](https://aimock.copilotkit.dev/record-replay)** — Proxy real APIs, save as fixtures, replay deterministically forever
- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `toolCallId`, `sequenceIndex`, or custom predicates
- **[11 LLM Providers](https://aimock.copilotkit.dev/docs)** — OpenAI Chat, OpenAI Responses, OpenAI Realtime, Claude, Gemini, Gemini Live, Azure, Bedrock, Vertex AI, Ollama, Cohere — full streaming support
- **[Multimedia APIs](https://aimock.copilotkit.dev/images)** — Image generation (DALL-E, Imagen), text-to-speech, audio transcription, video generation
- **[MCP / A2A / AG-UI / Vector](https://aimock.copilotkit.dev/mcp-mock)** — Mock every protocol your AI agents use
- **Multimedia APIs** — [image generation](https://aimock.copilotkit.dev/images) (DALL-E, Imagen), [text-to-speech](https://aimock.copilotkit.dev/speech), [audio transcription](https://aimock.copilotkit.dev/transcription), [video generation](https://aimock.copilotkit.dev/video)
- **[MCP](https://aimock.copilotkit.dev/mcp-mock) / [A2A](https://aimock.copilotkit.dev/a2a-mock) / [AG-UI](https://aimock.copilotkit.dev/agui-mock) / [Vector](https://aimock.copilotkit.dev/vector-mock)** — Mock every protocol your AI agents use
- **[Chaos Testing](https://aimock.copilotkit.dev/chaos-testing)** — 500 errors, malformed JSON, mid-stream disconnects at any probability
- **[Drift Detection](https://aimock.copilotkit.dev/drift-detection)** — Daily CI validation against real APIs
- **[Streaming Physics](https://aimock.copilotkit.dev/streaming-physics)** — Configurable `ttft`, `tps`, and `jitter`
Expand Down Expand Up @@ -87,7 +94,7 @@ npx @copilotkit/aimock convert vidaimock ./templates/ ./fixtures/
npx @copilotkit/aimock convert mockllm ./config.yaml ./fixtures/

# Docker
docker run -d -p 4010:4010 -v ./fixtures:/fixtures ghcr.io/copilotkit/aimock -f /fixtures -h 0.0.0.0
docker run -d -p 4010:4010 -v "$(pwd)/fixtures:/fixtures" ghcr.io/copilotkit/aimock -f /fixtures -h 0.0.0.0
```

> **Note on `llmock` vs `aimock` CLIs.** The `llmock` bin is retained as a compat alias for users of the pre-1.7.0 `@copilotkit/llmock` package. It runs a narrower flag-driven CLI without `--config` or the `convert` subcommand. New projects should use `aimock` (or `npx @copilotkit/aimock`) for full feature support.
Expand Down
68 changes: 45 additions & 23 deletions docs/fixtures/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -105,13 +105,17 @@ <h2>Match Fields</h2>
<td>toolCallId</td>
<td>string</td>
<td>
Match on <code>tool_call_id</code> of the last <code>role: "tool"</code> message
Match on <code>tool_call_id</code> of the last <code>role: "tool"</code> message.
The <code>onToolResult(id, response)</code> helper is sugar over this field
</td>
</tr>
<tr>
<td>toolName</td>
<td>string</td>
<td>Match on tool function name</td>
<td>
Match on tool function name &mdash; compared against the names of tool definitions
in the request&rsquo;s <code>tools:</code> array
</td>
</tr>
<tr>
<td>model</td>
Expand All @@ -132,7 +136,9 @@ <h2>Match Fields</h2>
<td>endpoint</td>
<td>string</td>
<td>
Restrict to endpoint type: chat, image, speech, transcription, video, embedding
Restrict to endpoint type: chat, image, speech, transcription, video, embedding.
Search, rerank, and moderation services (added in 1.7.0) are registered through
their own fixture APIs rather than via this field
</td>
</tr>
<tr>
Expand Down Expand Up @@ -213,23 +219,25 @@ <h3>5. Validation warnings surface shadowing at load time</h3>
<ul>
<li>
<strong>Duplicate <code>userMessage</code></strong> &mdash; two fixtures with the same
string <code>userMessage</code> produces:
<code>duplicate userMessage '&lt;value&gt;' — shadows fixture &lt;N&gt;</code>. This is
advisory, not a hard error: the check looks at <code>userMessage</code> alone, so if
you've added a runtime differentiator (<code>sequenceIndex</code>,
<code>toolCallId</code>, <code>model</code>, or a <code>predicate</code>) the fixtures
won't actually shadow each other at match time. Only fixtures with no differentiator at
all truly shadow on match &mdash; that's the case where the second is never reached
because the first wins. Safe to ignore in the former case; investigate in the latter.
string <code>userMessage</code> produce a warning of the form
<code>duplicate userMessage 'hello' — shadows fixture 0</code>, where
<code>'hello'</code> is the duplicated message and <code>0</code> is the zero-based
index of the earlier fixture being shadowed. This is advisory, not a hard error: the
check looks at <code>userMessage</code> alone, so if you've added a runtime
differentiator (<code>sequenceIndex</code>, <code>toolCallId</code>, <code>model</code>,
or a <code>predicate</code>) the fixtures won't actually shadow each other at match
time. Only fixtures with no differentiator at all truly shadow on match &mdash; that's
the case where the second is never reached because the first wins. Safe to ignore in the
former case; investigate in the latter.
</li>
<li>
<strong>Catch-all not last</strong> &mdash; a fixture with an empty <code>match</code>
(no discriminator fields) matches everything. If it is not the final fixture, every
fixture after it is unreachable:
fixture after it is unreachable. The warning is of the form
<code
>empty match acts as catch-all but is not the last fixture — shadows fixtures
&lt;N&gt;+</code
>.
>empty match acts as catch-all but is not the last fixture — shadows fixtures 3+</code
>, where <code>3</code> is the zero-based index of the first shadowed fixture (i.e.
every fixture from that index onward).
</li>
</ul>

Expand Down Expand Up @@ -374,12 +382,13 @@ <h2>Response Override Fields</h2>
<td>string</td>
<td>
Override finish reason (default: "stop" or "tool_calls"). Provider mappings:
<code>stop</code> &rarr; <code>end_turn</code> (Claude), <code>STOP</code> (Gemini);
<code>tool_calls</code> &rarr; <code>tool_use</code> (Claude),
<code>FUNCTION_CALL</code> (Gemini); <code>length</code> &rarr;
<code>max_tokens</code> (Claude), <code>MAX_TOKENS</code> (Gemini);
<code>content_filter</code> &rarr; <code>SAFETY</code> (Gemini),
<code>failed</code> (Responses API)
<code>stop</code> &rarr; <code>end_turn</code> (Claude), <code>STOP</code> (Gemini),
<code>completed</code> (Responses API); <code>tool_calls</code> &rarr;
<code>tool_use</code> (Claude), <code>FUNCTION_CALL</code> (Gemini),
<code>completed</code> (Responses API); <code>length</code> &rarr;
<code>max_tokens</code> (Claude), <code>MAX_TOKENS</code> (Gemini),
<code>incomplete</code> (Responses API); <code>content_filter</code> &rarr;
<code>SAFETY</code> (Gemini), <code>failed</code> (Responses API)
</td>
</tr>
<tr>
Expand Down Expand Up @@ -499,6 +508,15 @@ <h3>Programmatically</h3>
</p>
</div>

<div class="info-box">
<p>
<code>onTranscription</code> takes the response object directly &mdash; there is no
user-provided input to match against, unlike <code>onMessage</code> /
<code>onToolCall</code> / <code>onEmbedding</code>. Every transcription request matches
the same fixture.
</p>
</div>

<h2>Provider Support Matrix</h2>
<table class="endpoint-table">
<thead>
Expand Down Expand Up @@ -573,7 +591,7 @@ <h2>Provider Support Matrix</h2>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>&mdash;</td>
<td>Yes</td>
<td>&mdash;</td>
</tr>
<tr>
Expand All @@ -596,12 +614,16 @@ <h2>Provider Support Matrix</h2>
<td>Yes</td>
<td>Yes</td>
<td>&mdash;</td>
<td>Yes</td>
<td>Yes<sup>*</sup></td>
<td>&mdash;</td>
<td>&mdash;</td>
</tr>
</tbody>
</table>
<p class="footnote">
<sup>*</sup> Azure inherits OpenAI&rsquo;s override support because Azure OpenAI routes
through the OpenAI Chat Completions response format internally.
</p>
</main>
<aside class="page-toc" id="page-toc"></aside>
</div>
Expand Down
40 changes: 24 additions & 16 deletions docs/record-replay/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ <h2>Proxy-Only Mode</h2>
Proxy-only mode <span class="lang-tag">shell</span>
</div>
<pre><code>$ docker run -d -p 4010:4010 \
-v ./fixtures:/fixtures \
-v $(pwd)/fixtures:/fixtures \
ghcr.io/copilotkit/aimock \
-f /fixtures -h 0.0.0.0 \
--proxy-only \
Expand Down Expand Up @@ -150,7 +150,7 @@ <h2>Quick Start</h2>
<div class="code-block">
<div class="code-block-header">CLI usage <span class="lang-tag">shell</span></div>
<pre><code>$ docker run -d -p 4010:4010 \
-v ./fixtures:/fixtures \
-v $(pwd)/fixtures:/fixtures \
ghcr.io/copilotkit/aimock \
-f /fixtures -h 0.0.0.0 \
--record \
Expand Down Expand Up @@ -325,20 +325,28 @@ <h2>Programmatic API</h2>

<p>
To proxy unmatched requests without writing fixtures to disk, set
<code>proxyOnly: true</code> and omit <code>fixturePath</code>:
<code>proxyOnly: true</code> and omit <code>fixturePath</code>. Despite the method name,
<code>enableRecording({ proxyOnly: true })</code> does <strong>not</strong> write fixtures
&mdash; omit or set <code>proxyOnly: false</code> to actually record. Remember to
<code>mock.stop()</code> when done:
</p>

<div class="code-block">
<div class="code-block-header">
Proxy-only (no recording) <span class="lang-tag">ts</span>
</div>
<pre><code><span class="op">mock</span>.<span class="fn">enableRecording</span>({
<span class="prop">providers</span>: {
<span class="prop">openai</span>: <span class="str">"https://api.openai.com"</span>,
<span class="prop">anthropic</span>: <span class="str">"https://api.anthropic.com"</span>,
},
<span class="prop">proxyOnly</span>: <span class="kw">true</span>,
});</code></pre>
<pre><code><span class="kw">try</span> {
<span class="op">mock</span>.<span class="fn">enableRecording</span>({
<span class="prop">providers</span>: {
<span class="prop">openai</span>: <span class="str">"https://api.openai.com"</span>,
<span class="prop">anthropic</span>: <span class="str">"https://api.anthropic.com"</span>,
},
<span class="prop">proxyOnly</span>: <span class="kw">true</span>,
});
<span class="cm">// ...make requests; unmatched ones proxy through without being saved</span>
} <span class="kw">finally</span> {
<span class="kw">await</span> <span class="op">mock</span>.<span class="fn">stop</span>();
}</code></pre>
</div>

<h2>Stream Collapsing</h2>
Expand Down Expand Up @@ -438,7 +446,7 @@ <h2>Fixture Auto-Generation</h2>
<div class="code-block-header">
Recorded fixture file <span class="lang-tag">json</span>
</div>
<pre><code><span class="cm">// fixtures/recorded/openai-2025-01-15T10-30-00-000Z-0.json</span>
<pre><code><span class="cm">// fixtures/recorded/openai-&lt;YYYY-MM-DD&gt;T&lt;HH-MM-SS&gt;-000Z-0.json</span>
{
"fixtures": [
{
Expand Down Expand Up @@ -496,13 +504,13 @@ <h2>Local Development Workflow</h2>
</div>
<pre><code># First run: record real API responses
$ docker run -d -p 4010:4010 \
-v ./fixtures:/fixtures \
-v $(pwd)/fixtures:/fixtures \
ghcr.io/copilotkit/aimock \
--record --provider-openai https://api.openai.com -f /fixtures -h 0.0.0.0

# Subsequent runs: replay from recorded fixtures
$ docker run -d -p 4010:4010 \
-v ./fixtures:/fixtures \
-v $(pwd)/fixtures:/fixtures \
ghcr.io/copilotkit/aimock \
-f /fixtures -h 0.0.0.0</code></pre>
</div>
Expand All @@ -522,7 +530,7 @@ <h2>CI Pipeline Workflow</h2>
<pre><code>- name: Start aimock
run: |
docker run -d --rm --name aimock \
-v ./fixtures:/fixtures \
-v $(pwd)/fixtures:/fixtures \
-p 4010:4010 \
ghcr.io/copilotkit/aimock \
--strict -f /fixtures -h 0.0.0.0
Expand Down Expand Up @@ -619,7 +627,7 @@ <h2 id="recording-multi-turn-conversations">Recording Multi-Turn Conversations</
<div class="code-block-header">
Recorder match derivation <span class="lang-tag">ts</span>
</div>
<pre><code><span class="cm">// src/recorder.ts (simplified)</span>
<pre><code><span class="cm">// src/recorder.ts (simplified; real implementation guards against a null last-user-message)</span>
<span class="kw">function</span> <span class="fn">buildFixtureMatch</span>(request) {
<span class="kw">if</span> (request.embeddingInput) {
<span class="kw">return</span> { <span class="prop">inputText</span>: request.embeddingInput };
Expand Down Expand Up @@ -686,7 +694,7 @@ <h2>Cross-Language Testing</h2>
Any language, one server <span class="lang-tag">bash</span>
</div>
<pre><code># Docker image serves all languages
docker run -d -p 4010:4010 -v ./fixtures:/fixtures ghcr.io/copilotkit/aimock -f /fixtures -h 0.0.0.0
docker run -d -p 4010:4010 -v $(pwd)/fixtures:/fixtures ghcr.io/copilotkit/aimock -f /fixtures -h 0.0.0.0

# Python
import openai
Expand Down
6 changes: 4 additions & 2 deletions docs/sidebar.js
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,8 @@
if (!content) return;

var headings = content.querySelectorAll("h2, h3");
if (headings.length < 4) return;

// Ensure each heading has an id for anchor links
// Always assign ids so cross-page anchors resolve, even on short pages.
for (var i = 0; i < headings.length; i++) {
var h = headings[i];
if (!h.id) {
Expand All @@ -151,6 +150,9 @@
}
}

// TOC rendering requires at least 4 headings to be worth the space.
if (headings.length < 4) return;

// Build TOC HTML
var html = '<div class="page-toc-label">On this page</div>';
for (var j = 0; j < headings.length; j++) {
Expand Down
Loading