Add PostHog telemetry for site, CLI, and MCP with opt-out controls by willwashburn · Pull Request #2 · AgentWorkforce/relaycast

willwashburn · 2026-02-08T18:30:29Z

Summary

add anonymous PostHog telemetry for the static site, CLI, and MCP server
instrument funnel and usage events for: site views -> install intent, workspace creation, message sends, inbox checks, pricing CTA clicks, and CLI vs MCP usage
add CLI telemetry controls (relaycast telemetry status|enable|disable) with opt-out precedence via DO_NOT_TRACK=1 and RELAYCAST_TELEMETRY_DISABLED=1
support local/staging telemetry overrides using Relay-compatible env vars (POSTHOG_API_KEY, POSTHOG_HOST) plus Relaycast aliases
add focused telemetry documentation and update README/.env examples

Validation

npm --prefix packages/cli test -- src/__tests__/index.test.ts src/__tests__/telemetry.test.ts
npm --prefix packages/mcp test -- src/__tests__/piggyback.test.ts
node --check site/script.js

Introduce anonymous telemetry instrumentation using PostHog for CLI, MCP server, and website. Adds telemetry modules (packages/cli/src/telemetry.ts, packages/mcp/src/telemetry.ts), CLI telemetry commands (telemetry enable/disable/status), and hooks to capture CLI command start/complete/fail events, derived events, and first-run. MCP server piggybacks telemetry on tool invocations, sessions, and message-related tools; HTTP/stdio transports now surface telemetry context. Site now captures page views, CTA clicks, pricing clicks, section views, and SDK tab selections (site meta tags and script updates). Include docs (docs/TELEMETRY.md), README telemetry section, tests, .env.example variables, and .gitignore update for .npm-cache. Telemetry respects opt-out via DO_NOT_TRACK or RELAYCAST_TELEMETRY_DISABLED and stores anonymous id in ~/.relay/telemetry.json.

devin-ai-integration

Devin Review found 1 potential issue.

View 8 additional findings in Devin Review.

Copilot

Pull request overview

Adds PostHog-based anonymous telemetry across the static marketing site, the relaycast CLI, and the MCP server, including funnel/usage events, opt-out controls, and documentation updates.

Changes:

Site: adds lightweight PostHog capture + click/section/SDK-tab instrumentation.
CLI: adds a telemetry module, command lifecycle events, and relaycast telemetry status|enable|disable opt-out controls.
MCP: adds server/tool/session telemetry hooks and updates piggyback wrapper to emit usage events.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
site/script.js	Implements site-side PostHog capture and instruments views/CTAs/sections/SDK tab selections.
site/index.html	Adds PostHog meta config and telemetry CTA attributes on links/buttons.
packages/mcp/src/transports.ts	Adds telemetry capture for HTTP session start and passes transport info into server creation.
packages/mcp/src/telemetry.ts	Introduces MCP telemetry client (state, opt-out env handling, PostHog posting, flush).
packages/mcp/src/server.ts	Creates telemetry instance and captures server/session authentication events; passes telemetry into piggyback.
packages/mcp/src/piggyback.ts	Wraps tool handlers to capture invoked/completed/failed + derived usage events; guards inbox piggybacking.
packages/cli/src/telemetry.ts	Introduces CLI telemetry client (state, opt-out precedence, first-run event, PostHog posting, flush).
packages/cli/src/index.ts	Wires telemetry into CLI lifecycle hooks, captures command events, adds derived usage events, and flushes on exit/errors.
packages/cli/src/commands/telemetry.ts	Adds `relaycast telemetry status
packages/cli/src/tests/telemetry.test.ts	Adds Vitest coverage for CLI telemetry config/opt-out behavior.
packages/cli/src/tests/index.test.ts	Updates CLI entrypoint mocks for commander hooks/parseAsync + telemetry wiring.
docs/TELEMETRY.md	Adds focused telemetry documentation and opt-out instructions.
README.md	Adds a Telemetry section linking to docs and describing opt-out options.
.gitignore	Ignores `.npm-cache/`.
.env.example	Adds optional telemetry env var examples/overrides.

Comments suppressed due to low confidence (2)

packages/cli/src/index.ts:86

The CLI binary is installed as relaycast (per packages/cli/package.json), but commander is configured with .name('relay'), which will show relay in help output and in usage strings. Align the displayed name with the published binary (relaycast) to avoid user confusion, especially now that docs reference relaycast telemetry ....

program
  .name('relay')
  .description('Relay — agent-to-agent messaging CLI')
  .version(CLI_VERSION);

packages/mcp/src/server.ts:29

MCP versions are hard-coded as 0.1.0 (server version, createMcpTelemetry('0.1.0')), but packages/mcp/package.json is 0.1.2. This will emit incorrect mcp_version in telemetry and advertise the wrong MCP server version. Consider sourcing the version from package metadata or a single shared constant.

  const telemetry = createMcpTelemetry('0.1.0');

  const mcpServer = new McpServer(
    { name: 'agent-relay', version: '0.1.0' },
    {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-08T19:56:59Z

+export function createMcpTelemetry(version = 'unknown'): McpTelemetry {
+  const state = loadState();
+  const sessionId = randomUUID();
+  const pending = new Set<Promise<void>>();
+  const distinctId = process.env.RELAYCAST_TELEMETRY_DISTINCT_ID || state.anonymousId;


packages/mcp has Vitest coverage (e.g., __tests__/piggyback.test.ts), but this new telemetry module isn’t covered. Consider adding tests for opt-out precedence (env vs persisted flag) and host trimming / error-tolerant state persistence, since this code runs during server startup.

Copilot · 2026-02-08T19:56:59Z

+  const capture = (event, properties = {}) => {
+    const payload = {
+      api_key: posthogKey,
+      event,
+      distinct_id: distinctId,


Site telemetry currently has no opt-out / Do Not Track handling (e.g., navigator.doNotTrack, a localStorage flag, or a query param for consent testing). If users enable DNT, events will still be sent and a distinct ID persisted in localStorage; add a guard so capture() becomes a no-op when tracking is disabled.

Copilot · 2026-02-08T19:56:59Z

+function saveState(state: TelemetryState): void {
+  const dir = path.dirname(TELEMETRY_PATH);
+  fs.mkdirSync(dir, { recursive: true });
+  fs.writeFileSync(TELEMETRY_PATH, JSON.stringify(state, null, 2) + '\n', 'utf8');
+}


saveState() uses synchronous mkdirSync/writeFileSync without a try/catch. If $HOME is read-only or missing (common in some containers/CI), CLI startup can throw before command parsing. Telemetry should be best-effort: catch filesystem errors and fall back to an in-memory state so the CLI still works.

Copilot · 2026-02-08T19:57:00Z

+function trackSiteView() {
+  telemetry.capture('relaycast_site_viewed', {
+    referrer: document.referrer || null,
+    query: window.location.search || null,
+  });


trackSiteView sends the full window.location.search and document.referrer to PostHog. Query strings and referrers can contain sensitive data (e.g., auth codes, emails, internal URLs). Consider either omitting these fields, stripping values (keep only whitelisted params like UTM), or hashing/redacting before sending.

Copilot · 2026-02-08T19:57:00Z

+  const state: TelemetryState = { anonymousId: randomUUID() };
+  const dir = path.dirname(TELEMETRY_PATH);
+  fs.mkdirSync(dir, { recursive: true });
+  fs.writeFileSync(TELEMETRY_PATH, JSON.stringify(state, null, 2) + '\n', 'utf8');
+  return state;


loadState() always tries to create ~/.relay/telemetry.json on first run and does not catch errors from mkdirSync/writeFileSync. In environments where the home directory is not writable, this can crash the MCP server even if telemetry would otherwise be disabled. Wrap the write path in a try/catch and fall back to an ephemeral anonymous ID when persistence fails.

Copilot · 2026-02-08T19:57:00Z

+        telemetry.capture('relaycast_mcp_http_session_started', {
+          source_surface: 'mcp',
+          transport: 'http',
+          session_id: transport.sessionId,


This event sets session_id: transport.sessionId, which overrides the base session_id added by createMcpTelemetry (because ...properties comes last). That changes the meaning of session_id for this event vs others, making analytics inconsistent. Use a different property name (e.g., mcp_transport_session_id) to avoid clobbering the telemetry session id.

Suggested change

session_id: transport.sessionId,

mcp_transport_session_id: transport.sessionId,

Copilot · 2026-02-08T19:57:00Z

 export const CLI_VERSION = '0.1.0' as const;

 const program = new Command();
+const telemetry = createCliTelemetry(CLI_VERSION);
+let activeCommandPath: string | null = null;


CLI_VERSION is hard-coded to 0.1.0 but packages/cli/package.json is 0.1.2. This will report an incorrect cli_version in telemetry and also advertise the wrong version in --version. Consider deriving this from package.json at build time, or updating the constant to match the package version.

Copilot · 2026-02-08T19:57:01Z

+  const telemetry = createMcpTelemetry('0.1.0');
+


createHttpHandler instantiates a telemetry client here, but createRelayMcpServer also instantiates its own telemetry client for the same HTTP session. This splits events across different telemetry session_ids/queues and makes correlation harder. Consider creating one telemetry instance per HTTP session and passing it into the MCP server (and piggyback wrapper) instead of creating two.

willwashburn · 2026-02-08T22:49:28Z

@copilot open a new pull request to apply changes based on the comments in this thread. also resolve the conflicts with main

Copilot · 2026-02-08T22:49:36Z

@willwashburn I've opened a new pull request, #4, to work on those changes. Once the pull request is ready, I'll request review from you.

…port, and fix issues Co-authored-by: willwashburn <957608+willwashburn@users.noreply.github.com>

Co-authored-by: willwashburn <957608+willwashburn@users.noreply.github.com>

Fix telemetry security and reliability issues from PR #2 review

Copilot

Pull request overview

Copilot reviewed 15 out of 17 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-08T23:26:16Z

+  telemetry.capture('relaycast_site_cta_clicked', {
+    cta_name: ctaName,
+    section,
+    destination,
+    is_external: anchor.target === '_blank' || destination.startsWith('http'),
+  });


is_external is computed as target === '_blank' || destination.startsWith('http'), which will incorrectly mark non-HTTP external links (e.g. mailto: in the footer/pricing CTA) as not external. Consider treating any absolute URL with a non-empty scheme (e.g. mailto:, tel:, https:) as external, or explicitly include mailto:/tel: in the check so this property is accurate.

Copilot · 2026-02-08T23:26:17Z

+  const captureFirstRunIfNeeded = () => {
+    if (state.firstRunCaptured) return;
+    state.firstRunCaptured = true;
+    saveState(state);
+
+    capture('relaycast_cli_first_run', {
+      install_source: process.env.RELAYCAST_INSTALL_SOURCE ?? 'unknown',
+      install_ref: process.env.RELAYCAST_INSTALL_REF ?? null,
+      command_count: Math.max(process.argv.length - 2, 0),
+    });
+  };


captureFirstRunIfNeeded() sets state.firstRunCaptured = true and persists it before checking whether telemetry is actually enabled. If the user has DO_NOT_TRACK / RELAYCAST_TELEMETRY_DISABLED set (or has opted out), this permanently suppresses the first-run event even if telemetry is later enabled. Consider checking getStatus(state).enabled (or similar) before setting/persisting firstRunCaptured, so first-run is only marked captured when it is actually sent.

Copilot · 2026-02-08T23:26:17Z

 export function createRelayMcpServer(options: McpServerOptions): McpServer {
  const relay = new Relay({ apiKey: options.apiKey, baseUrl: options.baseUrl });
  const session: SessionState = createInitialSession();
+  const telemetry = options.telemetry ?? createMcpTelemetry('0.1.0');



Telemetry is being initialized with a hard-coded version string (createMcpTelemetry('0.1.0')), but packages/mcp/package.json is 0.1.2. This will misreport mcp_version in analytics and can drift again on future releases. Consider importing and using the existing MCP_VERSION constant (or deriving from package metadata at build time) so telemetry stays in sync with the published package version.

Copilot · 2026-02-08T23:26:17Z

+      // New session — create fresh MCP server + transport + telemetry
+      const telemetry = createMcpTelemetry('0.1.0');
+
      const transport = new StreamableHTTPServerTransport({
        sessionIdGenerator: () => randomUUID(),
      });

-      const mcpServer = createRelayMcpServer(baseOptions);
+      const mcpServer = createRelayMcpServer({
+        ...baseOptions,
+        telemetryTransport: 'http',
+        telemetry,
+      });


createHttpHandler creates per-session telemetry using createMcpTelemetry('0.1.0'), but @relaycast/mcp is currently version 0.1.2 (per packages/mcp/package.json). This will cause mcp_version to be reported incorrectly. Consider using a shared version constant (e.g. MCP_VERSION) rather than a hard-coded string.

devin-ai-integration

Devin Review found 1 new potential issue.

View 12 additional findings in Devin Review.

willwashburn requested review from Copilot and khaliqgant February 8, 2026 19:52

Copilot started reviewing on behalf of willwashburn February 8, 2026 19:52 View session

devin-ai-integration Bot reviewed Feb 8, 2026

View reviewed changes

Comment thread packages/cli/src/telemetry.ts

Copilot AI reviewed Feb 8, 2026

View reviewed changes

Initial plan

0fc7154

Copilot AI mentioned this pull request Feb 8, 2026

Fix telemetry security and reliability issues from PR #2 review #4

Merged

Copilot AI and others added 3 commits February 8, 2026 22:54

Address telemetry review comments: add tests, error handling, DNT sup…

cd90129

…port, and fix issues Co-authored-by: willwashburn <957608+willwashburn@users.noreply.github.com>

Address code review feedback: improve test clarity and variable naming

83859a5

Co-authored-by: willwashburn <957608+willwashburn@users.noreply.github.com>

Merge pull request #4 from AgentWorkforce/copilot/sub-pr-2

be6fe19

Fix telemetry security and reliability issues from PR #2 review

willwashburn requested a review from Copilot February 8, 2026 23:20

Copilot started reviewing on behalf of willwashburn February 8, 2026 23:20 View session

Copilot AI reviewed Feb 8, 2026

View reviewed changes

Merge branch 'main' into telemtry

98e2775

devin-ai-integration Bot reviewed Feb 9, 2026

View reviewed changes

Comment thread packages/cli/src/index.ts

pr feedback

e040146

khaliqgant merged commit 13e6b1e into main Feb 9, 2026
1 check passed

khaliqgant deleted the telemtry branch February 9, 2026 07:52

	session_id: transport.sessionId,
	mcp_transport_session_id: transport.sessionId,

Uh oh!

Conversation

willwashburn commented Feb 8, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

willwashburn commented Feb 8, 2026

Uh oh!

Copilot AI commented Feb 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

willwashburn commented Feb 8, 2026 •

edited by devin-ai-integration Bot

Loading