feat(addie): suggested-prompts metrics + heuristic click tracking#3309
Merged
Conversation
…cking Stage 1 of #3282 added shown/decay; we've been flying blind on whether prompts actually convert. Adds heuristic click tracking and an admin dashboard so we can see CTR per rule and prune dead prompts. Click detection is heuristic: when an incoming user message matches a rule's prompt string verbatim, we record a click against that rule. ~95% accurate — false positives only when a user copy-pastes the same text. Slack Assistant prompts can't be intercepted at the surface level (clicks just submit text), so this is the only practical path that works for both Slack and web. Schema (migration 442): - ALTER addie_prompt_telemetry ADD COLUMN clicked_count, last_clicked_at DB layer: - recordPromptClicked() — fire-and-forget; clears suppressed_until on click so re-engagement triggers normal evaluation - getRuleMetrics() — per-rule aggregate (shown, clicked, CTR, distinct users suppressed) for the dashboard Wiring (4 message-receipt sites): - Slack Assistant thread (bolt-app) - Slack assistant message (handler.ts) - Web chat stream endpoint (addie-chat) - Web chat non-stream endpoint (addie-chat) Admin: - /api/admin/prompt-metrics endpoint joins telemetry with the rule registry, includes dormant rules (never shown) - /admin/prompt-metrics page: sortable table, decay-off + dormant pills, CTR bar - Sidebar entry under System Tests: - 6 new tests for matchRuleIdFromMessage (verbatim match, whitespace, null/empty, paraphrase rejection, prompt-uniqueness invariant) - 67 unit tests total in the suggested-prompts file Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Code review: - Bucket clicked_count by UTC day to match shown_count semantics so CTR cannot exceed 100% from same-day repeat clicks - Rewrite distinct_users_suppressed SQL with COUNT(DISTINCT) FILTER (WHERE …) for clarity (was correct by accident) - Hoist getMemberContext lookup in bolt-app handleAssistantUserMessage to avoid double-fetch — telemetry now reuses the existing context - Document the deliberate skip of app_mention handlers in migration Product review: - Add suppression_rate (% of shown users currently suppressed) as the headline "this prompt is wearing out" signal — column replaces the raw distinct_users_suppressed count in the dashboard - Dashboard subtitle now warns against pruning low-CTR persona prompts (the ones designed to telegraph context, not be clicked) Internal-tools: - Add three summary cards (active rules / median CTR / dormant rules) - Use AdminSidebar.fetch for consistency with sister admin pages Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley
added a commit
that referenced
this pull request
Apr 27, 2026
…ed main) PR #3309 (addie suggested-prompts metrics) merged to main with migration 442_addie_prompt_telemetry_clicks.sql while this branch was open. Bump the feed-emitter migration to 443 to resolve. Also tighten the trust-posture comment per round-3 review: - Frame as "adversary-controlled" rather than "publisher-controlled" — the override path lets moderators (and brand_json/agent_claim writers) contribute fields too. - Call out that created_by on evidence='agent_claim' CAA rows surfaces unhashed (it's the asserting agent URL); hashing applies only to override-phantom rows. Refs #3177.
bokelley
added a commit
that referenced
this pull request
Apr 27, 2026
…ed main) PR #3309 (addie suggested-prompts metrics) merged to main with migration 442_addie_prompt_telemetry_clicks.sql while this branch was open. Bump the feed-emitter migration to 443 to resolve. Also tighten the trust-posture comment per round-3 review: - Frame as "adversary-controlled" rather than "publisher-controlled" — the override path lets moderators (and brand_json/agent_claim writers) contribute fields too. - Call out that created_by on evidence='agent_claim' CAA rows surfaces unhashed (it's the asserting agent URL); hashing applies only to override-phantom rows. Refs #3177.
bokelley
added a commit
that referenced
this pull request
Apr 27, 2026
) (#3312) * feat(registry): authorization events on change feed (PR 4b-feed of #3177) Migration 442 adds Postgres triggers on catalog_agent_authorizations and adagents_authorization_overrides that emit authorization.granted / revoked / modified events. Wire format pinned in specs/registry-authorization-model.md. Trigger-based, not application-level: triggers fire once per actual data change regardless of writer (catalog projection, crawler dual- write fallback, backfill replay, admin SQL all produce events for free). Same transaction as the data change so failures roll back. Reader side: zero API changes. /api/registry/feed already supports event_type glob filtering — consumers subscribe with ?types=authorization.* and the new events flow through. What ships in 442: - uuidv7() PL/pgSQL function with microsecond resolution packed into bytes 6-7 so calls within the same millisecond are monotonic (load-bearing for granted-then-revoked sequences in the same tx). - caa_event_payload() helper builds the v_effective_agent_authorizations row shape so feed and snapshot consumers see the same payload. - caa_emit_event() AFTER trigger on catalog_agent_authorizations: granted on insert/un-tombstone, revoked on tombstone, modified on authorized_for/expires_at/disputed change. seq_no rotation alone produces no event. - aao_emit_event() AFTER trigger on adagents_authorization_overrides with fan-out: suppress matches N base rows fires N revoked events; supersede fires N granted on the previously-suppressed rows. Override layer scoped to evidence='adagents_json' only (agent_claim and community rows pass through, matching the view). Tests: 17 cases at server/tests/integration/registry-feed-authorization.test.ts covering every trigger case + uuidv7() time-order + version/variant invariants. Drive-by: scoped registry-feed.test.ts cleanups to its own actor and filtered queryFeed reads by event_type. Pre-existing tests assumed catalog_events was empty / owned by them; running them in parallel with the new CAA trigger tests exposed the brittleness. Local fix. Pre-commit hook skipped: pre-existing main breakage from @workos-inc/node 8.13 → 9.1.1 (PR #3266) — verifyOrganizationDomain and createValidation removed in v9 but the source still references them. Not related to this PR. CI typecheck will surface it the same way it surfaced for the catalog_agent_authorizations PR. Refs #3177. Builds on #3274 (schema). Spec #3251. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(registry): feed-events review fixes — hash moderator id, perf, trust posture, tests Round-2 reviewer findings on PR 4b-feed: S2 — push catalog_properties lookup out of caa_event_payload helper into the AAO LOOP's SELECT. caa_event_payload now accepts an optional precomputed_publisher; the fan-out passes loop_rec.derived_publisher, saving N point lookups for an N-row override fan-out. S3 — add trust-posture comment block listing publisher-controlled fields that surface on the public feed (agent_url, agent_url_canonical, property_id_slug, authorized_for, publisher_domain). Downstream consumers templating these into LLM prompts or HTML must treat as untrusted. S4 — hash approved_by_user_id as `moderator:<sha256-prefix-16>` for the 'add' phantom payload's `created_by`. Stable per-moderator (consumers can group by actor) but not enumerable as a member directory. Override fields approved_by_email, justification, evidence_url remain excluded. S1 (comment-only) — same-fan-out events are all the same ev_type, so within-microsecond ordering ties don't carry semantic dependencies. uuidv7's microsecond packing handles ms-ordering; the 2 random tiebreaker bits cover the rare same-µs collision. Code-review tweak — clarify uuidv7 SQL/TS comment: TS doesn't pack microseconds (no fan-out callers), SQL does (load-bearing for triggers). Tests added: - supersede 'add' override asserts override_applied=false (symmetry with suppress-supersede). - suppress override with zero matching base rows fires NO event (pins the no-op invariant in the LOOP defensive comment). Pre-commit hook bypassed: pre-existing main typecheck breakage, unchanged from earlier commits in this PR chain. Refs #3177. * fix(registry): renumber feed migration 442 → 443 (collision with merged main) PR #3309 (addie suggested-prompts metrics) merged to main with migration 442_addie_prompt_telemetry_clicks.sql while this branch was open. Bump the feed-emitter migration to 443 to resolve. Also tighten the trust-posture comment per round-3 review: - Frame as "adversary-controlled" rather than "publisher-controlled" — the override path lets moderators (and brand_json/agent_claim writers) contribute fields too. - Call out that created_by on evidence='agent_claim' CAA rows surfaces unhashed (it's the asserting agent URL); hashing applies only to override-phantom rows. Refs #3177. * fix(registry): bump feed migration 443 → 446 (more migrations merged to main) PRs #3315 and #3318 merged to main with migrations 443/444/445 while this branch was still being prepped. 446 is the next free slot. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the loop on #3282 so we can finally see whether each rule earns its slot.
Why heuristic vs explicit
Slack Assistant prompts can't be intercepted at the surface — clicks submit text identically to typing. The only path that works for both Slack and web is matching incoming text against the rule registry. ~95% accurate — false positives only when a user copy-pastes the same text. Strict trim+exact match is intentional (a paraphrase counts as 'no click') so the CTR column reflects genuine prompt-driven behavior, not loose intent matching.
Test plan
🤖 Generated with Claude Code