Skip to content

[DX-967] Adds more happy path E2E tests#250

Open
umair-ably wants to merge 1 commit intomainfrom
e2eTests
Open

[DX-967] Adds more happy path E2E tests#250
umair-ably wants to merge 1 commit intomainfrom
e2eTests

Conversation

@umair-ably
Copy link
Copy Markdown
Collaborator

@umair-ably umair-ably commented Apr 1, 2026

Summary

Adds 26 new E2E test files and updates the E2E test helper, covering commands across channels, rooms, spaces, auth, logs, config, integrations, and more. These are all happy-path tests verifying commands succeed and produce expected output against real Ably infrastructure.

New test files (26)

  • Channels: annotations CRUD, batch-publish, message-ops (update/append/delete), occupancy get, presence subscribe
  • Rooms: list, messages (send/history/update/delete), messages subscribe, message reactions (send/subscribe/remove), reactions (send/subscribe), typing, presence (enter/get/subscribe), occupancy (get/subscribe)
  • Spaces: CRUD (create/list/get), locations/cursors/locks, occupancy, subscribe
  • Auth: keys (list/create/get/update), tokens (issue ably/jwt, revoke)
  • Other: accounts, integrations, logs (history + subscribe variants), config (show), status, support

Current E2E coverage after this PR

Category Covered Total Leaf Commands Coverage
Accounts 2 5 40%
Apps 8 10 80%
Auth 7 10 70%
Bench 2 2 100%
Channels 15 16 94%
Config 2 2 100%
Connections 1 1 100%
Integrations 5 5 100%
Logs 5 7 71%
Push 13 17 76%
Queues 3 3 100%
Rooms 18 18 100%
Spaces 12 17 71%
Stats 2 2 100%
Other 5 8 63%
Total 100 123 ~81%

Remaining gaps are mostly commands requiring interactive OAuth / account config (accounts login/switch, apps current/switch, auth keys current/switch, channels inspect), subscribe variants for spaces sub-resources, and some push channel subscription CRUD.

Note: These are happy-path tests only. Error paths, flag variations, and edge cases are not yet covered at the E2E level (they are covered in unit tests).

Test plan

  • pnpm exec eslint test/e2e/ passes with 0 errors
  • New test files follow existing E2E patterns (env var auth, unique resource names, proper cleanup)
  • pnpm test:e2e passes against real Ably infrastructure

🤖 Generated with Claude Code

@vercel
Copy link
Copy Markdown

vercel bot commented Apr 1, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
cli-web-cli Ready Ready Preview, Comment Apr 1, 2026 2:15pm

Request Review

@umair-ably umair-ably requested a review from AndyTWF April 1, 2026 00:35
@umair-ably umair-ably changed the title Adds more happy path E2E tests [DX-967] Adds more happy path E2E tests Apr 1, 2026
@umair-ably umair-ably force-pushed the jsonFixes branch 3 times, most recently from 4a75d10 to 572225b Compare April 1, 2026 12:06
Base automatically changed from jsonFixes to main April 1, 2026 12:26
@umair-ably umair-ably marked this pull request as ready for review April 1, 2026 14:04
@claude-code-ably-assistant
Copy link
Copy Markdown

Walkthrough

This PR adds 26 new E2E test files covering happy-path scenarios for channels, rooms, spaces, auth, logs, config, integrations, and more — bringing overall E2E coverage from ~60% to ~81% of leaf commands. Alongside the tests, three small production fixes were made to unblock the new test scenarios.

Changes

Area Files Summary
Commands src/commands/logs/subscribe.ts Removed redundant ensureAppAndKey call that was blocking log subscription in E2E env
Commands src/commands/rooms/typing/keystroke.ts Fixed hold-loop to only run when --auto-type flag is set, preventing premature exit
Services src/base-command.ts Added ABLY_API_KEY env var fallback in ensureAppAndKey so CI/E2E tests resolve app config without interactive prompts
Tests test/unit/base/base-command.test.ts Updated unit test to match new direct env-var path (no longer expects interactive key selection)
Tests test/helpers/e2e-test-helper.ts Added E2E_ACCESS_TOKEN / SHOULD_SKIP_CONTROL_E2E exports to gate Control API E2E tests on a separate env var
Tests test/helpers/e2e-mutable-messages.ts New helper for mutable message (update/delete/append) E2E scenarios
Tests test/e2e/accounts/, test/e2e/auth/ E2E coverage for accounts list, auth keys CRUD, token issue/revoke
Tests test/e2e/channels/ (5 files) E2E coverage for annotations, batch publish, message ops, occupancy get, presence subscribe
Tests test/e2e/rooms/ (8 files) E2E coverage for rooms list, messages CRUD+subscribe, reactions, typing, presence, occupancy
Tests test/e2e/spaces/ (4 files) E2E coverage for spaces CRUD, locations, occupancy, subscribe
Tests test/e2e/config/, test/e2e/integrations/, test/e2e/logs/, test/e2e/status/, test/e2e/support/ E2E coverage for config show, integrations CRUD, logs history+subscribe, status, support info

Review Notes

  • Behavioral change in ensureAppAndKey: The new ABLY_API_KEY env var fallback is inserted before the Control API interactive-selection path. This means any environment with ABLY_API_KEY set will now short-circuit and skip interactive app/key selection — confirm this is the intended precedence order.
  • logs/subscribe.ts dead-code removal: The ensureAppAndKey call was removed without any replacement. The logs channel name [meta]log doesn't require app config at that point, but reviewers should verify the downstream createAblyRealtimeClient still picks up auth correctly via the new env var fallback.
  • SHOULD_SKIP_CONTROL_E2E gate: Control API tests now require both E2E_ABLY_API_KEY and E2E_ABLY_ACCESS_TOKEN to be set. CI pipelines that only set the API key will silently skip these tests — ensure the CI environment exports both variables.
  • Happy-path only: All 26 new test files cover success paths only. Error cases, flag variations, and edge cases remain at the unit-test level.

Copy link
Copy Markdown

@claude-code-ably-assistant claude-code-ably-assistant bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

File Status Issues
src/base-command.ts OK
src/commands/logs/subscribe.ts OK
src/commands/rooms/typing/keystroke.ts 1 issue Unhandled promise rejection
test/helpers/e2e-test-helper.ts OK
test/helpers/e2e-mutable-messages.ts OK
test/unit/base/base-command.test.ts OK
26 new E2E test files OK

src/commands/rooms/typing/keystroke.ts

[Bug] Unhandled promise rejection when --auto-type is false

setupRoomStatusHandler creates a failurePromise that rejects if the room transitions to RoomStatus.Failed. When --auto-type is not set, failurePromise is created (line 96) but never awaited — only the auto-type branch awaits it (line 175). If the room happens to fail between the keystroke call and process exit, the rejection has no handler and Node.js will emit an unhandled rejection warning (or error, depending on Node version).

The fix: either move setupRoomStatusHandler inside the if (flags["auto-type"]) block (it only serves the hold path), or attach a no-op catch in the else branch:

const { failurePromise } = this.setupRoomStatusHandler(room, flags, { roomName });

// ... send keystroke ...

if (flags["auto-type"]) {
  await Promise.race([
    this.waitAndTrackCleanup(flags, "typing", flags.duration),
    failurePromise,
  ]);
} else {
  // Normal cleanup (detach/close) won't trigger RoomStatus.Failed, but
  // suppress the rejection to avoid an unhandled rejection warning.
  failurePromise.catch(() => {});
}

Other changes look correct

  • base-command.ts: The new ABLY_API_KEY env-var fallback in ensureAppAndKey() (before the interactive/Control-API path) mirrors the existing web-CLI mode behaviour and unblocks E2E scenarios where no config file is present. The unit test update accurately reflects the new behaviour.

  • logs/subscribe.ts: The removed block fetched appConfig but never used the result — dead code. Correct removal; createAblyRealtimeClient() handles auth independently.

  • E2E tests: Auth via env vars, --duration on subscribe runners, and --json + parseNdjsonLines for structured assertions are all correct patterns. The checkMutableMessagesSupport probe handles eventual consistency and the 93002 feature-flag error sensibly.

One non-blocking observation: every new beforeEach block calls resetTestTracking() then immediately repeats testOutputFiles.clear(); testCommands.length = 0 — but those two lines are exactly what resetTestTracking() already does internally. Harmless, but worth a follow-up cleanup.

@sacOO7
Copy link
Copy Markdown
Contributor

sacOO7 commented Apr 2, 2026

Reviewed PR with following points in mind

  1. Missing assertions on output data and corresponding fields in both json and non-json mode with zero validation and incomplete validation
  2. Missing stderr validation
  3. Missing command coverage

Shared review for the same

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants