feat: add chaos() API for HTTP-layer fault injection by Copilot · Pull Request #2119 · counterfact/api-simulator

Copilot · 2026-05-25T00:55:25Z

Summary

Implements the chaos() API for HTTP-layer fault injection and updates runtime wiring so chaos rules are managed by a single server-level ChaosRegistry shared across all API runners (including multi-API mode). The REPL chaos() global now affects all configured API groups consistently.

Follow-up feedback updates were also applied:

Content-Type is explicitly protected from chaos header mutation APIs (header() / removeHeader() ignore it).
probability(value) now throws a RangeError when the value is outside 0..1 (including NaN).
timeout() was removed from the chaos API to avoid long-running timeout behavior; use delay(ms) for timeout-like scenarios instead.
Chaos docs now include:
- a dedicated fault simulation pattern in the API reference, and
- a new usage pattern page under docs/patterns explaining how to test fault scenarios with chaos().

Original Prompt

Implement a chaos() API for injecting HTTP-layer faults into simulated API responses, expose it in the REPL, and support rule matching/lifecycle behavior (next/always/probability/status/delay/timeout/header/removeHeader/body/transformBody/stop/start) with deterministic rule selection and comprehensive test coverage.

Manual acceptance tests

Start the REPL with yarn go:example, run chaos("/pets").next(3).status(503), and verify the next 3 /pets* requests return HTTP 503.
In REPL, run chaos().always().delay(2000) and verify responses are delayed by about 2 seconds.
In REPL, create const f = chaos("/pets").always().status(500), verify /pets* returns 500, then run f.stop() and verify /pets* returns normal status again.
In a multi-API setup, run chaos().always().status(503) once in REPL and verify requests across different API groups all return 503 (shared server-level registry behavior).
In REPL, run chaos("/pets").always().probability(1.5) (and chaos("/pets").always().probability(NaN)) and verify an out-of-range probability error is thrown.
Open docs/patterns/index.md and verify it links to Test Fault Scenarios with Chaos Rules, then open that page and verify it documents intermittent failures (probability), bounded failures (next(count)), and stopping rules (stop()).

Tasks

- Add ChaosRule and ChaosRegistry classes in src/server/chaos.ts - Integrate ChaosRegistry into Dispatcher (applies rules post-response) - Create ChaosRegistry in ApiRunner and pass it to Dispatcher - Expose chaos() global in REPL via startRepl parameter - Thread chaosRegistry from app.ts -> startReplServer - Add 48 tests covering all required scenarios in test/server/chaos.test.ts - Document the chaos API in docs/reference.md Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/1b9bc7aa-b2e1-4090-9a24-b1165b72d855

- Improve CHAOS_TIMEOUT_DELAY_MS comment explaining Node.js 32-bit int constraint - Replace `any` body type in ChaosRule.tryApply with `unknown` + explicit cast - Extract PROBABILITY_TEST_ITERATIONS constant in test file Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/1b9bc7aa-b2e1-4090-9a24-b1165b72d855

Copilot

Pull request overview

Adds a new HTTP-layer fault-injection facility (“chaos rules”) to Counterfact, wiring it through the runtime dispatcher and exposing it as a REPL global for interactive testing of failure scenarios.

Changes:

Added ChaosRule / ChaosRegistry with fluent configuration (status, delay/timeout, headers, body transforms) and deterministic “most recently updated” precedence.
Integrated chaos rule application into Dispatcher.request() and plumbed a registry from ApiRunner through to the REPL.
Added comprehensive unit/integration tests and documented the new Chaos API in the reference docs.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/server/chaos.test.ts	New test suite covering rule behavior, registry selection, and dispatcher integration.
src/server/dispatcher.ts	Accepts an optional ChaosRegistry and applies a selected rule to responses (with optional delay).
src/server/chaos.ts	New implementation of ChaosRule + ChaosRegistry and exported timeout delay constant.
src/repl/repl.ts	Adds optional ChaosRegistry to REPL startup and exposes `chaos()` global when present.
src/app.ts	Passes the runner’s ChaosRegistry into REPL startup.
src/api-runner.ts	Instantiates a ChaosRegistry and injects it into the Dispatcher.
docs/reference.md	Adds a Chaos API reference section, usage, semantics, and examples.

pmcelhaney · 2026-05-25T01:35:38Z

+    // Apply chaos rules after normal response processing.
+    if (this.chaosRegistry !== undefined) {
+      const rule = this.chaosRegistry.findBestMatch(path);
+
+      if (rule !== undefined) {
+        const chaosResult = rule.tryApply(normalizedResponse);
+
+        if (chaosResult !== null) {
+          if (chaosResult.delayMs !== undefined && chaosResult.delayMs > 0) {
+            await new Promise<void>((resolve) => {
+              setTimeout(resolve, chaosResult.delayMs);
+            });
+          }
+
+          return chaosResult.response;
+        }
+      }
+    }


@copilot to be safe let's just have a special case that contentType cannot be modified or removed.

Implemented: chaos().header("content-type", ...) and chaos().removeHeader("content-type") are now ignored, with tests and docs updated accordingly. Addressed in commits 20fc1df and 831a623.

+    // Apply header modifications.
+    const headers: CounterfactResponseObject["headers"] = {
+      ...(response.headers ?? {}),
+    };
+
+    for (const [name, value] of this._headers) {
+      headers[name] = value;
+    }
+
+    for (const name of this._removedHeaders) {
+      delete headers[name];
+    }
+
+    // Apply body modifications.
+    // The body is typed as unknown here because chaos rules may replace it
+    // with any value (including plain objects that Koa will serialize to JSON).
+    // The cast to CounterfactResponseObject['body'] is applied below when
+    // setting the result property.
+    let body: unknown = response.body;
+
+    if (this._body !== UNSET) {
+      body = this._body;
+    } else if (this._transformBody !== undefined) {
+      body = this._transformBody(body);
+    }
+
+    const result: CounterfactResponseObject = {
+      ...response,
+      // Cast is safe: Koa serializes object bodies to JSON at the middleware level.
+      body: body as CounterfactResponseObject["body"],
+      headers,
+    };
+
+    if (this._status !== undefined) {
+      result.status = this._status;
+    }
+
+    const delayMs = this._isTimeout ? CHAOS_TIMEOUT_DELAY_MS : this._delay;
+
+    return { response: result, delayMs };
+  }


pmcelhaney · 2026-05-25T01:33:22Z

    ? Object.fromEntries(groupedBindings.map((binding) => [binding.key, {}]))
    : {};

+  if (chaosRegistry !== undefined) {


@copilot, again there should be a single chaosRegistry at the server level.

Confirmed and already wired this way: there is a single server-level ChaosRegistry shared across all runners and the REPL. Addressed in commit 395d262, with cross-group behavior covered in 81e0ece.

pmcelhaney · 2026-05-25T01:32:22Z

        primaryRunner.scenarioRegistry,
        runners.map((runner) => ({
          contextRegistry: runner.contextRegistry,
          group: runner.group,
          openApiDocument: runner.openApiDocument,
          registry: runner.registry,
          scenarioRegistry: runner.scenarioRegistry,
        })),
+        primaryRunner.chaosRegistry,
      ),


@copilot the chaosRegistry shouldn't be tied to a runner. There should be one registry at the server level.

Implemented this by moving chaos registry ownership to the server level in counterfact() and sharing a single instance across all ApiRunner dispatchers plus the REPL binding. Also added a multi-API test proving REPL chaos() affects all groups. Commits: 395d262 and 81e0ece.

+| `.header(name, value)` | Set or replace a response header. |
+| `.removeHeader(name)` | Remove a response header if present. |
+| `.body(value)` | Replace the response body. |
+| `.transformBody(fn)` | Transform the response body: `fn` receives the current body and returns the new one. |
+


pmcelhaney

See copilot's comments and my responses.

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/9d655172-35a8-434f-8b66-d08e066cbd9c

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/0de918d1-768c-432d-9ae0-e7275cbcd1ab

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

+    this._removedHeaders.add(name);
+    this.touch();
+    return this;
+  }


pmcelhaney · 2026-05-25T15:06:01Z

+  public probability(value: number): this {
+    this._probability = value;
+    this.touch();
+    return this;
+  }


@copilot throw an error if it's outside of range

Implemented in 8e7af9d: chaos().probability(value) now throws a RangeError when value is outside [0, 1] (including NaN), and I added unit coverage for invalid inputs in test/server/chaos.test.ts.

+    const cr = new ChaosRegistry();
+    cr.createRule("/orders").always().delay(2_000);
+    const dispatcher = makeDispatcher(cr);
+
+    const response = await get(dispatcher, "/orders");
+    expect(response.status).toBe(200);
+    expect(delays).toContain(2_000);
+
+    jest.restoreAllMocks();


Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/6fbc6713-6552-410c-afb5-e32137d66448

pmcelhaney

@copilot get rid of timeout(). It could leave CI processes hanging for days. Users can use delay() with a large enough value to simulate timeout scenarios.

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/578539c1-45a7-4108-be44-b2d8742b90b2

Copilot · 2026-05-25T15:22:15Z

@copilot get rid of timeout(). It could leave CI processes hanging for days. Users can use delay() with a large enough value to simulate timeout scenarios.

Removed in 4f2a2d6: ChaosRule.timeout() and its internal max-delay behavior are gone, related tests were removed, and the reference docs now describe using delay(ms) only. No UI changes in this update.

pmcelhaney

@copilot Add a pattern to the docs for simulating faults using chaos().

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/29a1c8e4-af2b-414e-b266-0b4dfcefcfc5 Co-authored-by: pmcelhaney <51504+pmcelhaney@users.noreply.github.com>

Copilot · 2026-05-25T22:15:07Z

@copilot Add a pattern to the docs for simulating faults using chaos().

Added in 08e3556: the Chaos API docs now include a dedicated “Fault simulation pattern” section showing how to simulate intermittent faults with chaos() using always(), probability(0.2), status(503), and Retry-After. No UI changes in this update.

pmcelhaney · 2026-05-25T22:18:54Z

@copilot add a new pattern under docs/patterns explained how to test fault scenarios using the chaos() API

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/c96b832d-e961-413d-82f5-ced242d37e6e

Copilot · 2026-05-25T22:21:58Z

@copilot add a new pattern under docs/patterns explained how to test fault scenarios using the chaos() API

Added in e24f997: I created a new pattern at docs/patterns/test-fault-scenarios-with-chaos.md explaining how to test fault scenarios with chaos() (intermittent failures, bounded outages, and stopping rules), and linked it from docs/patterns/index.md. No UI changes in this update.

Copilot AI added 2 commits May 25, 2026 00:51

Copilot AI assigned Copilot and pmcelhaney May 25, 2026

Copilot AI requested review from Copilot and removed request for Copilot May 25, 2026 00:55

Copilot created this pull request from a session on behalf of pmcelhaney May 25, 2026 00:56 View session

Copilot finished work on behalf of pmcelhaney May 25, 2026 00:56

Copilot AI requested a review from pmcelhaney May 25, 2026 00:56

pmcelhaney marked this pull request as ready for review May 25, 2026 00:59

Copilot AI review requested due to automatic review settings May 25, 2026 00:59

Copilot started reviewing on behalf of pmcelhaney May 25, 2026 00:59 View session

Copilot AI reviewed May 25, 2026

View reviewed changes

Copilot started work on behalf of pmcelhaney May 25, 2026 01:32 View session

pmcelhaney requested changes May 25, 2026

View reviewed changes

Copilot AI added 3 commits May 25, 2026 01:36

refactor: share chaos registry across all api runners

395d262

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/9d655172-35a8-434f-8b66-d08e066cbd9c

test: verify repl chaos affects all api groups

81e0ece

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/9d655172-35a8-434f-8b66-d08e066cbd9c

test: tidy repl chaos shared-registry coverage

3b9d87b

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/9d655172-35a8-434f-8b66-d08e066cbd9c

Copilot finished work on behalf of pmcelhaney May 25, 2026 01:44

Copilot AI requested a review from pmcelhaney May 25, 2026 01:44

Copilot started work on behalf of pmcelhaney May 25, 2026 01:44 View session

Copilot AI added 2 commits May 25, 2026 01:48

fix: block content-type header mutations in chaos rules

20fc1df

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/0de918d1-768c-432d-9ae0-e7275cbcd1ab

test: simplify content-type removal guard case

831a623

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/0de918d1-768c-432d-9ae0-e7275cbcd1ab

Copilot finished work on behalf of pmcelhaney May 25, 2026 01:51

pmcelhaney approved these changes May 25, 2026

View reviewed changes

pmcelhaney requested review from Copilot and dethell May 25, 2026 14:17

Copilot started reviewing on behalf of pmcelhaney May 25, 2026 14:17 View session

Copilot AI reviewed May 25, 2026

View reviewed changes

Copilot started work on behalf of pmcelhaney May 25, 2026 15:06 View session

fix: validate chaos probability range

8e7af9d

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/6fbc6713-6552-410c-afb5-e32137d66448

Copilot finished work on behalf of pmcelhaney May 25, 2026 15:12

Copilot AI requested a review from pmcelhaney May 25, 2026 15:12

pmcelhaney requested changes May 25, 2026

View reviewed changes

Copilot started work on behalf of pmcelhaney May 25, 2026 15:17 View session

refactor: remove chaos timeout API

4f2a2d6

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/578539c1-45a7-4108-be44-b2d8742b90b2

Copilot finished work on behalf of pmcelhaney May 25, 2026 15:23

Copilot AI requested a review from pmcelhaney May 25, 2026 15:23

pmcelhaney requested changes May 25, 2026

View reviewed changes

Copilot started work on behalf of pmcelhaney May 25, 2026 22:12 View session

Copilot AI and others added 4 commits May 25, 2026 22:13

docs: add chaos fault simulation pattern

cb6318d

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/29a1c8e4-af2b-414e-b266-0b4dfcefcfc5 Co-authored-by: pmcelhaney <51504+pmcelhaney@users.noreply.github.com>

docs: clarify chaos API in fault pattern section

08d171b

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/29a1c8e4-af2b-414e-b266-0b4dfcefcfc5 Co-authored-by: pmcelhaney <51504+pmcelhaney@users.noreply.github.com>

docs: clarify always plus probability chaos pattern

56d46ba

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/29a1c8e4-af2b-414e-b266-0b4dfcefcfc5 Co-authored-by: pmcelhaney <51504+pmcelhaney@users.noreply.github.com>

docs: tighten chaos fault pattern wording

08e3556

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/29a1c8e4-af2b-414e-b266-0b4dfcefcfc5 Co-authored-by: pmcelhaney <51504+pmcelhaney@users.noreply.github.com>

Copilot finished work on behalf of pmcelhaney May 25, 2026 22:15

Copilot AI requested a review from pmcelhaney May 25, 2026 22:15

Copilot started work on behalf of pmcelhaney May 25, 2026 22:19 View session

docs: add chaos fault-testing usage pattern

e24f997

Agent-Logs-Url: https://github.com/counterfact/api-simulator/sessions/c96b832d-e961-413d-82f5-ced242d37e6e

Copilot finished work on behalf of pmcelhaney May 25, 2026 22:22

Uh oh!

Conversation

Copilot AI commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Manual acceptance tests

Tasks

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

pmcelhaney May 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2026

Choose a reason for hiding this comment

Uh oh!

pmcelhaney May 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2026

Choose a reason for hiding this comment

Uh oh!

pmcelhaney May 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2026

Choose a reason for hiding this comment

Uh oh!

pmcelhaney left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

pmcelhaney May 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2026

Choose a reason for hiding this comment

Uh oh!

pmcelhaney left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented May 25, 2026

Uh oh!

pmcelhaney left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented May 25, 2026

Uh oh!

pmcelhaney commented May 25, 2026

Uh oh!

Copilot AI commented May 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented May 25, 2026 •

edited

Loading