diff --git a/AGENTS.md b/AGENTS.md index 50c632ea..d7a63a43 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -174,30 +174,7 @@ Exceptions: - `interactive-sigint.test.ts` — needs a longer duration for SIGINT testing - Help output checks — testing that `--help` mentions `--duration` is fine -**Test structure:** -- `test/unit/` — Fast, mocked tests. Auth via `MockConfigManager` (automatic). Only set `ABLY_API_KEY` env var when testing env var override behavior. -- `test/integration/` — Integration tests (e.g., interactive mode). Mocked external services but tests multi-component interaction. -- `test/tty/` — Interactive mode tests requiring a real pseudo-TTY (e.g., SIGINT/Ctrl+C with readline). Uses `node-pty`. Local only — cannot run in CI. -- `test/e2e/` — Full scenarios against real Ably. Auth via env vars (`ABLY_API_KEY`, `ABLY_ACCESS_TOKEN`). -- Helpers in `test/helpers/` — `runCommand()`, `runLongRunningBackgroundProcess()`, `e2e-test-helper.ts`, `mock-config-manager.ts`. - -**Required test describe blocks** (exact names — every unit test file must have all 5): -1. `"help"` — verify `--help` shows USAGE -2. `"argument validation"` — test required args or unknown flag rejection -3. `"functionality"` — core happy-path behavior -4. `"flags"` — verify flags exist and work -5. `"error handling"` — API errors, network failures - -Do NOT use variants like `"command arguments and flags"`, `"command flags"`, `"flag options"`, or `"parameter validation"`. Exempt: `interactive.test.ts`, `interactive-sigint.test.ts`, `bench/*.test.ts`. - -**Running tests:** -```bash -pnpm test:unit # All unit tests -pnpm test:integration # Integration tests -pnpm test:tty # TTY tests (local only, needs real terminal) -pnpm test:e2e # All E2E tests -pnpm test test/unit/commands/foo.test.ts # Specific test -``` +See [`docs/Testing.md`](../docs/Testing.md) for test layers, directory layout, required describe blocks, running tests, helpers reference, and conventions. ## CLI Output & Flag Conventions diff --git a/docs/Testing.md b/docs/Testing.md index ea347755..d5148a23 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -1,400 +1,249 @@ -# Testing Strategy & Policy +# Testing Guide -
-

📘 ESSENTIALS FIRST 📘

-
+## Test layers -> **💡 QUICK START:** Run `pnpm test` for all tests or `pnpm test:unit` for faster unit tests. -> **📋 MANDATORY:** All code changes require related tests. See [AGENTS.md](../AGENTS.md). -> **🐛 DEBUGGING:** See [Debugging Guide](Debugging.md) for troubleshooting tips and the [Debug Test Execution](#-debug-test-execution) section below. -> **🔍 TROUBLESHOOTING:** See [Troubleshooting Guide](Troubleshooting.md) for common errors. +| Layer | Directory | Purpose | Speed | Mocking | When to use | +|-------|-----------|---------|-------|---------|-------------| +| **Unit** | `test/unit/` | Command logic, flag parsing, output format, error handling | Fast (~ms) | Everything mocked (SDK, REST, config) | Every command, every PR | +| **E2E** | `test/e2e/` | Real workflows against live Ably | Slow (~seconds) | Nothing mocked | Core workflows, new integrations | +| **TTY** | `test/tty/` | Interactive mode with real pseudo-terminal | Fast (~2s) | No Ably, real PTY | SIGINT/readline behavior | +| **Subprocess** | `test/integration/` | Interactive mode via spawned process | Fast | No Ably, real process | Interactive command flows | ---- +> **Note on "integration" tests:** The `test/integration/` directory contains 4 files, all testing interactive mode subprocess behavior. It is not a general-purpose integration layer. A future PR may rename it to `test/subprocess/` or fold it into `test/tty/`. -## 🚀 Testing Goals & Guiding Principles +### Unit tests -1. **Confidence:** Ensure each command works as intended and avoid regressions. -2. **Speed & Developer Experience:** Most tests should be quick to run, easy to debug, and not require a live environment. -3. **Real Integration Coverage (where needed):** Some commands may need to be tested against real APIs (e.g., Ably's pub/sub product APIs and Control APIs) to verify end-to-end flows—especially for mission-critical commands. -4. **Scalability:** The test setup should scale as commands grow in complexity. -5. **Mandatory Coverage:** Adding or updating relevant tests is a **required** step for all feature additions or bug fixes. +The workhorse. Every command gets unit tests. They should cover every code branch and every conditional — if there's an `if`, both paths get tested. We mock at the Ably library level (SDK, Control API, config), so tests run in milliseconds with no network. Auth is provided automatically by `MockConfigManager` — no env vars needed. ---- +### E2E tests -## 🏃‍♂️ Running Tests +Run against real Ably services with real credentials (via env vars). These cover the entire journey — the CLI's interaction with the actual Ably service, end to end. Every command must have an E2E test. E2E tests should cover the happy path and major sad paths (e.g., invalid capabilities, nonexistent resources). They are slow and can incur costs, so use them deliberately. -Refer to [AGENTS.md](../AGENTS.md) for the mandatory requirement to run tests. +### TTY tests -| Test Type | Command | Description | -|-----------|---------|-------------| -| **All Tests** | `pnpm test` | Run all test types except Playwright | -| **Unit Tests** | `pnpm test:unit` | Fast tests with mocked dependencies | -| **Integration Tests** | `pnpm test:integration` | Tests with mocked Ably services | -| **E2E Tests** | `pnpm test:e2e` | Tests against real Ably services | -| **TTY Tests** | `pnpm run test:tty` | Interactive mode SIGINT tests (requires real TTY, local only) | -| **Playwright Tests** | `pnpm test:playwright` | Web CLI browser tests | +Use `node-pty` to create a real pseudo-terminal. This is the only way to test readline SIGINT handling, which doesn't work with piped stdio. Local only — cannot run in CI (no TTY in GitHub Actions runners). Rebuild `node-pty` with `pnpm rebuild node-pty` if it fails to load. -**Run Specific Files:** -```bash -# CLI Tests - Run a specific test file -pnpm test test/unit/commands/bench/bench.test.ts +### Subprocess (integration) tests -# CLI Tests - Run all tests in a directory -pnpm test test/unit/commands/auth/**/*.test.ts -``` +Spawn the CLI as a child process to test interactive mode flows. These verify multi-step interactive prompts, terminal feedback, and process lifecycle without needing a real TTY. ---- +### Test directory layout -## 🐛 Debug Test Execution +| Directory | Contents | +|-----------|----------| +| `test/helpers/` | Shared utilities — `runCommand()`, mock SDKs, mock config, standard test generators, nock helpers | +| `test/fixtures/` | Factory functions for mock API responses (`mockApp()`, `mockKey()`, etc.) | +| `test/unit/` | Mirrors `src/` structure — e.g., `src/commands/channels/publish.ts` → `test/unit/commands/channels/publish.test.ts` | +| `test/integration/` | Interactive mode subprocess tests (4 files) | +| `test/tty/` | TTY tests with `node-pty` helpers | +| `test/e2e/` | E2E tests organized by feature (`channels/`, `rooms/`, `spaces/`, etc.) | -The test runner includes built-in debugging support to help diagnose test failures, especially for E2E tests that interact with real services. +--- -### Environment Variables +## What tests does my PR need? -| Variable | Description | -|----------|-------------| -| `E2E_DEBUG=true` | Enable detailed test debugging output | -| `ABLY_CLI_TEST_SHOW_OUTPUT=true` | Show detailed CLI output during tests | -| `TEST_DEBUG=true` | Alias for E2E_DEBUG | +### Always required -### Examples +- **Unit tests** for any new or changed command, with all 5 required describe blocks (see [Test structure](#test-structure) below). +- **E2E tests** for any new or changed command — happy path and major sad paths. -```bash -# Debug E2E tests with verbose output -E2E_DEBUG=true ABLY_CLI_TEST_SHOW_OUTPUT=true pnpm test:e2e +### Sometimes required -# Debug specific failing tests -E2E_DEBUG=true pnpm run test 'test/e2e/rooms/*.test.ts' +- **TTY tests** when: changing SIGINT/Ctrl+C handling or readline behavior in interactive mode. +- **Subprocess tests** when: changing interactive mode prompt flows or subprocess lifecycle. -# Debug specific test file with grep filter -pnpm test test/e2e/spaces/*.test.ts --t "should have properly structured spaces member commands" -``` +### Output mode coverage -### Debug Output Features +Unit tests for commands with `--json` support should test all three output modes: +- Default (human-readable) +- `--json` (compact NDJSON) +- `--pretty-json` (indented JSON) -When debugging is enabled (`E2E_DEBUG=true` and/or `ABLY_CLI_TEST_SHOW_OUTPUT=true`), you'll see: -- ✅ **Detailed console output** from the CLI commands being tested -- ✅ **Ably SDK logs** showing connection and API interactions -- ✅ **Process cleanup information** from the test setup -- ✅ **Enhanced error reporting** with full stack traces +### Not needed ---- +- E2E tests that only cover things unit tests already cover: flag parsing, help text, error messages, output formatting. +- Duplicate coverage across layers — one layer per behavior is enough. -### 🔧 Pre-Push Validation +--- -The `scripts/pre-push-validation.sh` script runs a comprehensive test suite: +## Choosing the right layer -```bash -# Run the full pre-push validation -./scripts/pre-push-validation.sh +``` +What are you testing? +│ +├─ Flag parsing, help output, error messages, output format? +│ → Unit test +│ +├─ Real pub/sub, CRUD against Ably, multi-step workflow with real services? +│ → E2E test +│ +├─ SIGINT / Ctrl+C with readline in a real terminal? +│ → TTY test +│ +└─ Interactive mode subprocess behavior (prompts, process lifecycle)? + → Subprocess (integration) test ``` -The script will: -- Build and prepare the project -- Run linter checks -- Run all unit, integration, and E2E tests -- Clean up automatically after completion +Explicit rules: +- Flag parsing, help text, error messages, output formatting → **Unit**. Always. +- SDK method calls with mocked responses → **Unit**. Mock the SDK, assert the command calls it correctly. +- Real network round-trip to Ably (pub/sub, presence, history, Control API CRUD) → **E2E**. +- SIGINT/Ctrl+C with readline → **TTY**. Piped stdio cannot test this. +- Interactive mode prompts and subprocess lifecycle → **Subprocess (integration)**. --- -
-

📊 Testing Approach - Expand for Details

 +## Running tests -### 🧪 Unit Tests (`test/unit`) +| Command | What it runs | +|---------|-------------| +| `pnpm test:unit` | All unit tests | +| `pnpm test:integration` | Subprocess/interactive tests | +| `pnpm test:e2e` | E2E tests (needs `ABLY_API_KEY` etc.) | +| `pnpm test:tty` | TTY tests (local only, needs real terminal) | +| `pnpm test` | Unit + integration + E2E | +| `pnpm test test/unit/commands/foo.test.ts` | Specific test file | +| `pnpm test test/unit/commands/auth/**/*.test.ts` | All tests in a directory | +| `pnpm test:playwright` | Web CLI browser tests (Playwright, separate from Vitest) | -* **Primary Purpose:** Quickly verify command logic, flag parsing, input validation, error handling, and basic output formatting **in isolation**. Focus on testing individual functions or methods within a command class. -* **Dependencies:** **MUST** stub/mock all external dependencies (Ably SDK calls, Control API requests, filesystem access, `ConfigManager`, etc.). Use libraries like `vitest` and `nock`. -* **Speed:** Very fast; no network or filesystem dependency. -* **Value:** Useful for testing complex parsing, conditional logic, and edge cases within a command, but **less effective** at verifying core interactions with Ably services compared to Integration/E2E tests. +### Debugging E2E failures -**CLI Core and Commands:** -* **Tools:** Vitest, `@oclif/test`. -* **Location:** Primarily within the `test/unit/` directory, mirroring the `src/` structure. -* **Execution:** Run all unit tests with `pnpm test:unit` or target specific files, e.g., `pnpm vitest --project unit test/unit/commands/bench/bench.test.ts`. +Set `E2E_DEBUG=true` and/or `ABLY_CLI_TEST_SHOW_OUTPUT=true` for verbose output: -**Example (Vitest):** -```typescript -// Example unit test with proper mocking -import {describe, it, expect, beforeEach, vi} from 'vitest' -import {AblyCommand} from '../../src/base/ably-command' - -describe('MyCommand', () => { - let mockClient: any - - beforeEach(() => { - // Set up mocks - mockClient = { - channels: { - get: vi.fn().mockReturnedValue(...) - }, - close: vi.fn(), - } - vi.spyOn(AblyCommand.prototype, 'getAblyClient').mockResolvedValue(mockClient) - }) - - it('publishes a message to the specified channel', async () => { - // Test implementation - }) -}) +```bash +E2E_DEBUG=true ABLY_CLI_TEST_SHOW_OUTPUT=true pnpm test:e2e ``` -**React Web CLI Components (`@ably/react-web-cli`):** -* **Frameworks:** [Vitest](https://vitest.dev/) and [React Testing Library](https://testing-library.com/docs/react-testing-library/intro/). Vitest provides a Jest-compatible API for running tests, assertions, and mocking. React Testing Library is used to interact with components like a user would. -* **Location:** Test files are co-located with the components they test (e.g., `packages/react-web-cli/src/AblyCliTerminal.test.tsx`). -* **Execution:** - * Run all tests for `@ably/react-web-cli`: `pnpm --filter @ably/react-web-cli test`. - * Individual files via Vitest CLI: `pnpm exec vitest packages/react-web-cli/src/AblyCliTerminal.test.tsx`. -* **Mocking:** Dependencies (e.g., `@xterm/xterm`, WebSockets) are mocked using Vitest's capabilities (`vi.mock`, `vi.fn`). - -#### 🏗️ Testing Pyramid for React Web CLI Components - -While developing the browser-based **Web CLI** we have found that an "inverted" test pyramid (many end-to-end Playwright tests, few unit tests) quickly becomes brittle and slows the feedback loop. We therefore adopt a **pyramid approach** for this part of the codebase: +--- -1. **Unit tests (_broad base_) –** Exhaustive coverage of core logic that can execute **in isolation**: - * `global-reconnect` timing & state machine. - * React hooks and helpers inside `AblyCliTerminal` (without a real browser). - * Mock **all** browser APIs (`WebSocket`, `xterm.js`, timers). +## Test structure -2. **Focused E2E / Playwright tests (_narrow top_) –** Only verify **user-visible** flows: - * Automatic reconnect succeeds when the server is restarted. - * Users can cancel the reconnect countdown and later trigger a manual reconnect. +### Required describe blocks -Everything else (exact countdown rendering, every internal state transition, console noise) is left to the unit layer. This greatly reduces flake due to timing variance and Docker start-up times. +Every unit test file for a command must include all 5 of these describe blocks (exact names, in this order): -> **Tip for contributors:** If you find yourself mocking several browser APIs in a Playwright test, it probably belongs in a unit test instead. +1. **`"help"`** — verify `--help` shows USAGE +2. **`"argument validation"`** — test required args or unknown flag rejection +3. **`"functionality"`** — core happy-path behavior +4. **`"flags"`** — verify flags exist and work +5. **`"error handling"`** — API errors, network failures -### 🔄 Integration Tests (`test/integration`) +Do NOT use variants like `"command arguments and flags"`, `"command flags"`, `"flag options"`, or `"parameter validation"`. -* **Primary Purpose:** Verify the interaction between multiple commands or components, including interactions with *mocked* Ably SDKs or Control API services. Test the CLI execution flow. -* **Dependencies:** Primarily stub/mock network calls (`nock` for Control API, `vi` stubs for SDK methods), but may interact with the local filesystem for config management (ensure isolation). Use `ConfigManager` mocks. -* **Speed:** Relatively fast; generally avoids real network latency. -* **Value:** Good for testing command sequences (e.g., `config set` then `config get`), authentication flow logic (with mocked credentials), and ensuring different parts of the CLI work together correctly without relying on live Ably infrastructure. -* **Tools:** Vitest, `@oclif/test`, `nock`, `execa` (to run the CLI as a subprocess). +Exempt: `interactive.test.ts`, `interactive-sigint.test.ts`, `bench/*.test.ts`. -Refer to the [Debugging Guide](Debugging.md) for tips on debugging failed tests, including Playwright and Vitest tests. +### Auth in tests -### 🖥️ TTY Tests (`test/tty`) +**Unit tests** — `MockConfigManager` provides auth automatically. No env vars or flags needed: -* **Primary Purpose:** Verify interactive mode behavior that depends on a real terminal (pseudo-TTY), such as SIGINT/Ctrl+C handling with readline. -* **Dependencies:** Requires `node-pty` (already in devDependencies) to create real pseudo-terminals. Cannot run in CI (GitHub Actions runners have no TTY). -* **Speed:** Fast (~2 seconds), but requires native module compilation. -* **Value:** Tests SIGINT handling that is fundamentally untestable with piped stdio — readline's signal handling only works in real TTY environments. -* **Tools:** Vitest, `node-pty`. -* **Location:** `test/tty/` directory. -* **Execution:** Run locally with `pnpm run test:tty`. Not included in `pnpm test:unit` or CI pipelines. -* **Helpers:** `test/tty/tty-test-helper.ts` provides `spawnTty()`, `waitForOutput()`, `writeTty()`, `sendCtrlC()`, `killTty()` (async), and constants `PROMPT_PATTERN` (`"ably>"`) and `DEFAULT_WAIT_TIMEOUT` (8000ms). `ABLY_CLI_DEFAULT_DURATION` is intentionally omitted from the TTY vitest config — TTY tests manage their own timing via explicit `--duration` flags and real PTY I/O. +```typescript +// WRONG — don't pass auth flags +runCommand(["channels", "publish", "my-channel", "hello", "--api-key", key]); -> **Note:** If `node-pty` fails to load, rebuild it with `pnpm rebuild node-pty`. +// CORRECT — MockConfigManager handles it +runCommand(["channels", "publish", "my-channel", "hello"]); -### 🌐 End-to-End (E2E) Tests (`test/e2e`) +// CORRECT — access mock auth values when needed +import { getMockConfigManager } from "../../helpers/mock-config-manager.js"; +const mockConfig = getMockConfigManager(); +const apiKey = mockConfig.getApiKey()!; +``` -* **Primary Purpose:** Verify critical user flows work correctly against **real Ably services** using actual credentials (provided via environment variables). -* **Dependencies:** Requires a live Ably account and network connectivity. Uses real Ably SDKs and Control API interactions. -* **Scope:** Focus on essential commands and common workflows (login, app/key management basics, channel publish/subscribe/presence/history, logs subscribe). -* **Speed:** Slowest test type due to network latency and real API interactions. -* **Value:** Provides the highest confidence that the CLI works correctly for end-users in a real environment. **Preferred** over unit tests for verifying core Ably interactions. -* **Tools:** Vitest, `@oclif/test`, `execa`, environment variables (`E2E_ABLY_API_KEY`, etc.). -* **Frequency:** Run automatically in CI (GitHub Actions) on PRs and merges. Can be run locally but may incur costs. +**E2E tests** — commands run as real subprocesses, so auth must go via env vars: -**Example:** ```typescript -// Example E2E test with real services -import {describe, it, expect} from 'vitest' -import {execSync} from 'child_process' - -describe('channels commands', () => { - const testChannel = `test-${Date.now()}` - const testMessage = 'Hello E2E test' - - it('can publish and then retrieve history from a channel', async () => { - // Publish a message - execSync(`ABLY_API_KEY=${process.env.E2E_ABLY_API_KEY} ably channels publish ${testChannel} "${testMessage}"`) - - // Wait a moment for message to be stored - await new Promise(resolve => setTimeout(resolve, 1000)) - - // Get message from history - const result = execSync( - `ABLY_API_KEY=${process.env.E2E_ABLY_API_KEY} ably channels history ${testChannel} --json` - ).toString() - - const history = JSON.parse(result) - expect(history).toBeInstanceOf(Array) - expect(history.length).toBeGreaterThanOrEqual(1) - expect(history[0].data).toBe(testMessage) - }) -}, {timeout: 10000}) +runCommand(["channels", "publish", "my-channel", "hello"], { + env: { ABLY_API_KEY: key }, +}); ``` -### 🎭 Playwright Tests (`test/e2e/web-cli`) +### Duration in tests -* **Primary Purpose:** Verify the functionality of the Web CLI example application (`examples/web-cli`) running in a real browser. -* **Dependencies:** Requires Node.js, a browser (installed via Playwright), and the Web CLI example app to be built. -* **Speed:** Slow; involves browser automation and WebSocket connections. -* **Value:** Ensures the embeddable React component works correctly with the hosted terminal server. -* **Tools:** Playwright Test runner (`@playwright/test`). -* **Frequency:** Run automatically in CI, separate from Vitest tests. +Unit and integration tests set `ABLY_CLI_DEFAULT_DURATION: "0.25"` in `vitest.config.ts`, so subscribe/long-running commands auto-exit after 250ms. Do NOT pass `--duration` to `runCommand()` — it overrides the fast default. -
+Exceptions: `test:wait` command tests (required flag), `interactive-sigint.test.ts` (needs longer for SIGINT), and help output checks. ---- +### Resource cleanup -
-

🔧 Advanced Testing Guidance - Expand for Details

 +**Unit tests:** Mock SDK init and cleanup is handled globally by `test/unit/setup.ts` — no per-test setup needed. If your test uses nock, call `nock.cleanAll()` in `afterEach` (or use `controlApiCleanup()`). -## 📝 Test Coverage and Considerations - -* **Adding/Updating Tests:** When adding features or fixing bugs, add or update tests in the appropriate category (Unit, Integration, E2E, Playwright). -* **Focus:** Prioritize **Integration and E2E tests** for verifying core functionality involving Ably APIs/SDKs, as unit tests with extensive mocking provide less confidence in these areas. -* **Output Modes:** Tests should cover different output modes where relevant: - * Default (Human-readable) - * JSON (`--json`) - * Pretty JSON (`--pretty-json`) -* **Web CLI Mode:** Integration/E2E tests for commands with different behavior in Web CLI mode should simulate this using `ABLY_WEB_CLI_MODE=true` environment variable. The Playwright tests cover the actual Web CLI environment. -* **Test Output:** Test output (stdout/stderr) should be clean. Avoid polluting test logs with unnecessary debug output from the CLI itself. Failures should provide clear error messages. -* **Asynchronous Operations:** Use `async/await` properly. Avoid brittle `setTimeout` calls where possible; use event listeners or promise-based waits. -* **Resource Cleanup:** Ensure tests clean up resources (e.g., close Ably clients, kill subprocesses, delete temp files). Use the `afterEach` or `afterAll` hooks and helpers like `trackAblyClient`. -* **Realtime SDK Stubbing:** For Unit/Integration tests involving the Realtime SDK, stub the SDK methods directly (`vi.spyOn(ably.channels.get('...'), 'subscribe')`) rather than trying to mock the underlying WebSocket, which is complex and brittle. -* **Credentials:** E2E tests rely on `E2E_ABLY_API_KEY` (and potentially others) being set in the environment (locally via `.env` or in CI via secrets). **Never** hardcode credentials in tests. - -## 🗂️ Codebase Integration & Structure - -### Folder Structure - -``` -. -├── src/ -│ └── commands/ -├── packages/ -│ └── react-web-cli/ # @ably/react-web-cli (tests co-located with components) -├── test/ -│ ├── setup.ts # Global test setup (runs in Vitest context) -│ ├── root-hooks.ts # Root hooks for E2E test lifecycle -│ ├── helpers/ # Shared test utilities -│ │ ├── cli-runner.ts # CliRunner class for E2E process management -│ │ ├── cli-runner-store.ts # Per-test runner tracking -│ │ ├── command-helpers.ts # High-level E2E helpers -│ │ ├── e2e-test-helper.ts # E2E setup and teardown -│ │ ├── mock-ably-*.ts # Mock SDKs (chat, realtime, rest, spaces) -│ │ └── mock-config-manager.ts # MockConfigManager (provides test auth) -│ ├── unit/ # Fast, mocked tests -│ │ ├── base/ # Base command class tests -│ │ ├── base-command/ # AblyBaseCommand tests -│ │ ├── commands/ # Command unit tests (mirrors src/commands/) -│ │ ├── core/ # Core CLI functionality tests -│ │ ├── help/ # Help system tests -│ │ ├── hooks/ # Hook tests -│ │ ├── services/ # Service tests -│ │ └── utils/ # Utility tests -│ ├── integration/ # Multi-component tests (mocked external services) -│ │ ├── commands/ # Command flow integration tests -│ │ └── interactive-mode.test.ts -│ ├── e2e/ # End-to-End tests (runs against real Ably) -│ │ ├── auth/ # Auth E2E tests -│ │ ├── bench/ # Benchmark E2E tests -│ │ ├── channels/ # Channel E2E tests -│ │ ├── connections/ # Connection E2E tests -│ │ ├── control/ # Control API E2E tests -│ │ ├── core/ # Core CLI E2E tests -│ │ ├── interactive/ # Interactive mode E2E tests -│ │ ├── rooms/ # Chat rooms E2E tests -│ │ ├── spaces/ # Spaces E2E tests -│ │ ├── stats/ # Stats E2E tests -│ │ └── web-cli/ # Playwright browser tests for Web CLI -│ └── manual/ # Manual test scripts -└── ... -``` - -### E2E Test Organization - -E2E tests are organized by feature/topic (e.g., `channels-e2e.test.ts`, `presence-e2e.test.ts`) to improve maintainability and allow targeted runs. They use shared helpers from `test/helpers/e2e-test-helper.ts`. - -
+**E2E tests:** Use `trackAblyClient(client)` to register clients for automatic cleanup, and call helpers from `e2e-test-helper.ts` in `afterAll` to tear down test apps and connections. --- -## 🧩 Shared Test Helpers & Conventions - -### Required Describe Block Order - -Every unit test file for a command MUST include all 5 of these describe blocks in this canonical order (exact names): - -1. **`"help"`** — verify `--help` shows USAGE -2. **`"argument validation"`** — test required args or unknown flag rejection -3. **`"functionality"`** — core happy-path behavior -4. **`"flags"`** — verify flags exist and work -5. **`"error handling"`** — API errors, network failures - -Do NOT use variants like `"command arguments and flags"`, `"command flags"`, `"flag options"`, or `"parameter validation"`. Exempt: `interactive.test.ts`, `interactive-sigint.test.ts`, `bench/*.test.ts`. - -### Standard Test Generators - -The file `test/helpers/standard-tests.ts` provides generator functions that produce the boilerplate tests for the required describe blocks: +## Standard test generators -- **`standardHelpTests(command, importMetaUrl)`** — generates the `"help"` describe block, verifying `--help` output contains USAGE -- **`standardArgValidationTests(command, importMetaUrl, options?)`** — generates the `"argument validation"` block, testing unknown flag rejection. If `options.requiredArgs` is provided, also tests that missing args produce an error. -- **`standardFlagTests(command, importMetaUrl, flags)`** — generates the `"flags"` block, verifying each flag in the array appears in `--help` output -- **`standardControlApiErrorTests(opts)`** — generates 401/500/network error tests for Control API commands. Call **inside** a `describe("error handling", ...)` block (does NOT create the describe block itself). Takes `{ commandArgs, importMetaUrl, setupNock }` where `setupNock(scenario)` receives `"401"`, `"500"`, or `"network"`. +`test/helpers/standard-tests.ts` provides generators for boilerplate describe blocks: -Call the generators at describe-block level (not inside nested describes). You still need to write `"functionality"` and `"error handling"` blocks manually since those are command-specific. For Control API commands, combine `standardControlApiErrorTests()` with command-specific error tests inside the same `describe("error handling", ...)` block. +| Generator | Creates | +|-----------|---------| +| `standardHelpTests(command, importMetaUrl)` | `"help"` block — verifies `--help` contains USAGE | +| `standardArgValidationTests(command, importMetaUrl, options?)` | `"argument validation"` block — tests unknown flag rejection and optionally missing required args | +| `standardFlagTests(command, importMetaUrl, flags)` | `"flags"` block — verifies each flag appears in `--help` output | +| `standardControlApiErrorTests(opts)` | 401/500/network error tests — call **inside** `describe("error handling", ...)` | -### Control API Test Helpers +Call generators at describe-block level. Write `"functionality"` and `"error handling"` blocks manually (they're command-specific). -The file `test/helpers/control-api-test-helpers.ts` provides shared helpers for testing commands that use the Control API with nock: - -- **`nockControl()`** — returns a `nock` scope pre-configured for `https://control.ably.net` -- **`getControlApiContext()`** — returns `{ appId, accountId, mock }` from `MockConfigManager` -- **`controlApiCleanup()`** — calls `nock.cleanAll()` for use in `afterEach` hooks -- **`CONTROL_HOST`** — the default Control API host constant (`"https://control.ably.net"`) +--- -### Mock Factory Functions +## Helpers reference -The file `test/fixtures/control-api.ts` provides factory functions for building realistic Control API response bodies. Each accepts an optional `Partial` to override any field: +### Core test helpers (`test/helpers/command-helpers.ts`) -- **`mockApp(overrides?)`** — mock app object (id, name, status, tlsOnly, etc.) -- **`mockKey(overrides?)`** — mock API key object (id, key, capability, etc.) -- **`mockRule(overrides?)`** — mock integration rule object (ruleType, source, target, etc.) -- **`mockQueue(overrides?)`** — mock queue object (name, region, state, messages, stats, amqp, stomp, etc.) -- **`mockNamespace(overrides?)`** — mock namespace object (id, persisted, pushEnabled, etc.) -- **`mockStats(overrides?)`** — mock stats object (intervalId, unit, all.messages, etc.) +| Helper | Purpose | +|--------|---------| +| `runCommand(args, opts?)` | Run a CLI command in-process. Returns `{ stdout, stderr, error? }`. Used in unit and integration tests. | +| `runLongRunningBackgroundProcess(args, opts?)` | Spawn a long-running command (subscribe, etc.) as a background process. Returns a handle with `stdout`, `stderr`, and `kill()`. | -```typescript -import { mockApp, mockQueue } from "../../../fixtures/control-api.js"; +### Mock SDKs (`test/helpers/mock-ably-*.ts`) -// Use defaults -nockControl().get(`/v1/apps/${appId}`).reply(200, mockApp()); +| File | Provides | +|------|----------| +| `mock-ably-realtime.ts` | Mock `Ably.Realtime` — channels, presence, connection events | +| `mock-ably-rest.ts` | Mock `Ably.Rest` — REST channel operations, request() | +| `mock-ably-chat.ts` | Mock Chat SDK — rooms, messages, typing, reactions | +| `mock-ably-spaces.ts` | Mock Spaces SDK — members, cursors, locations, locks | -// Override specific fields -nockControl().get(`/v1/apps/${appId}/queues`).reply(200, [ - mockQueue({ id: "q-1", appId, name: "my-queue" }), -]); -``` +These are initialized and cleaned up automatically in `test/unit/setup.ts` — no per-test setup needed. -### NDJSON Test Helpers +### Control API test helpers (`test/helpers/control-api-test-helpers.ts`) -The file `test/helpers/ndjson.ts` provides helpers for testing JSON output: +| Helper | Purpose | +|--------|---------| +| `nockControl()` | Returns a `nock` scope for `https://control.ably.net` | +| `getControlApiContext()` | Returns `{ appId, accountId, mock }` from MockConfigManager | +| `controlApiCleanup()` | Calls `nock.cleanAll()` — use in `afterEach` | +| `CONTROL_HOST` | `"https://control.ably.net"` constant | -- **`parseNdjsonLines(stdout)`** — parse stdout containing one JSON object per line into an array of records -- **`parseLogLines(lines)`** — parse an array of log lines into JSON records (skips non-JSON) -- **`captureJsonLogs(fn)`** — capture all `console.log` output from an async function and parse as JSON records. Use to verify JSON envelope structure in `--json` mode. +### Mock factories (`test/fixtures/control-api.ts`) ---- +Each accepts an optional `Partial` to override fields: -## 🎯 Best Practices Quick Reference +| Factory | Creates | +|---------|---------| +| `mockApp()` | Mock app object | +| `mockKey()` | Mock API key object | +| `mockRule()` | Mock integration rule object | +| `mockQueue()` | Mock queue object | +| `mockNamespace()` | Mock namespace object | +| `mockStats()` | Mock stats object | -1. **✅ DO** prioritize Integration and E2E tests for core Ably functionality -2. **✅ DO** clean up all resources in tests (clients, connections, mocks) -3. **✅ DO** use proper mocking (`vitest`, `nock`) for Unit/Integration tests -4. **✅ DO** avoid testing implementation details when possible (test behavior) -5. **✅ DO** use path-based test execution for faster development workflow +### NDJSON helpers (`test/helpers/ndjson.ts`) -6. **❌ DON'T** rely solely on unit tests for Ably API interactions -7. **❌ DON'T** leave resources unclosed (memory leaks) -8. **❌ DON'T** use brittle `setTimeout` when avoidable -9. **❌ DON'T** hardcode credentials or API keys in tests +| Helper | Purpose | +|--------|---------| +| `parseNdjsonLines(stdout)` | Parse stdout with one JSON object per line into an array | +| `parseLogLines(lines)` | Parse log line array into JSON records (skips non-JSON) | +| `captureJsonLogs(fn)` | Capture `console.log` output from async function, parse as JSON | ---- +### TTY helpers (`test/tty/tty-test-helper.ts`) -
-🔍 For detailed troubleshooting help, see the Troubleshooting Guide. -
+| Helper | Purpose | +|--------|---------| +| `spawnTty()` | Spawn CLI in a real pseudo-terminal | +| `waitForOutput()` | Wait for specific output text | +| `writeTty()` | Send input to the terminal | +| `sendCtrlC()` | Send SIGINT | +| `killTty()` | Kill the PTY process (async) | +| `PROMPT_PATTERN` | `"ably>"` | +| `DEFAULT_WAIT_TIMEOUT` | 8000ms |