Skip to content

feat(dashboard): Human Intervention metric (#34) - #70

Merged
jeff-r2026 merged 2 commits into
mainfrom
feat/human-intervention-metric
Jun 30, 2026
Merged

feat(dashboard): Human Intervention metric (#34)#70
jeff-r2026 merged 2 commits into
mainfrom
feat/human-intervention-metric

Conversation

@jeff-r2026

Copy link
Copy Markdown
Collaborator

Summary

Implements Human Intervention metric for the team dashboard (closes #34) — tracking how often a human has to step in during a Claude Code / Cursor / CodeBuddy session. Three signals are detected from the session transcript and hook stream, then rolled up per-session, per-user and into the team digest:

  • interrupt — user pressed ESC mid-turn ([Request interrupted by user] in the transcript).
  • toolReject — user denied a tool/permission prompt (The tool use was rejected / The user doesn't want to proceed with this tool use).
  • correction — a re-prompt within 60s of a Stop that matches a course-correction keyword (不对/重新/redo/undo/…) rather than starting a fresh task.

Wiring (no new dependencies):

  • src/dashboard-collector.tscountInterventions() streams the transcript (memory-bounded, cheap pre-filter), idempotent Stop snapshot, rebuildSessions() exposes per-session badge data, aggregateSessionInterventions().
  • src/types.ts — intervention types + detection constants (TRANSCRIPT_INTERRUPT_PREFIX, TRANSCRIPT_REJECT_MARKERS, CORRECTION_KEYWORDS, CORRECTION_WINDOW_MS, scan cap).
  • src/team-push.tscomputeInterventionDelta() / mergeInterventionStats() (idempotent deltas into stats/<user>.yaml).
  • src/digest.tssummarizeInterventions() (totals, avg/session, ranked).
  • src/dashboard-html.ts — per-session intervention badge.
  • docs/usage-guide.md — metric documentation.

Test Plan

All commands run on this branch rebased onto main (a4a344e).

Type checknpx tsc --noEmit → ✅ pass (exit 0)

Unit suitenpx vitest run

Test Files  120 passed (120)
     Tests  1617 passed (1617)

Intervention-specific coverage (4 new unit files + amended collector test, 72 cases):
dashboard-collector.test.ts (countInterventions + rebuildSessions), digest-interventions.test.ts, intervention-report.test.ts, team-push-interventions.test.ts (delta + merge idempotency).

E2E suitenpm run test:e2e (runs under vitest.e2e.config.ts)

Test Files  6 passed (6)
     Tests  44 passed | 23 skipped (67)

Includes intervention-e2e.test.ts (2 tests): full hook STDIN → events.jsonlrebuildSessions → delta → digest pipeline.

Real end-to-end with the smallest model (claude-haiku-4-5) — beyond the in-process tests, the full chain was exercised against a real Claude Code session and the real compiled CLI binary:

  1. Ran a real headless haiku session (claude --model claude-haiku-4-5 -p …) that used the Write tool → captured the real transcript JSONL. Confirmed our parser handles the production transcript schema (user/assistant/tool_use/tool_result + queue-operation/attachment/last-prompt) and reports 0 interventions on a clean session (no false positives).
  2. Verified our detection markers are byte-for-byte present in the real Claude Code binary (2.1.196): Request interrupted by user, The tool use was rejected, doesn't want to proceed with this tool use — all FOUND.
  3. Confirmed a headless sandbox block ("…was blocked. For security…") is correctly not counted as a human intervention (only genuine user actions count).
  4. Drove the real built binary as the hook (node dist/index.js dashboard-report --stdin --tool claude) over SessionStart → PostToolUse → Stop → UserPromptSubmit, against the real transcript with the genuine interrupt + reject markers appended:
Stop event snapshot      : { interrupt: 1, toolReject: 1 }
session card             : { interrupt: 1, toolReject: 1, correction: 1 }  count=3  status=running
team delta               : { sessions: 1, interrupt: 1, toolReject: 1, correction: 1 }
digest summary           : totalSessions=1  totalInterventions=3  avgPerSession=3  ranked[0]=jeff

Notes

  • No package.json / version bump (release is tag-triggered).
  • Branch rebased onto current main; full suite is 100% green (the older agent-status test-isolation flakiness is not present here).

🤖 Generated with Claude Code

jeff-r2026 and others added 2 commits June 30, 2026 12:10
Add unit + e2e coverage for the interrupt / tool-reject / correction
signals and the team-stats roll-up:
- dashboard-collector.test.ts: countInterventions + rebuildSessions
- digest-interventions.test.ts / intervention-report.test.ts
- team-push-interventions.test.ts: delta + merge idempotency
- intervention-e2e.test.ts: full hook -> events -> digest pipeline

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jeff-r2026
jeff-r2026 merged commit 2f81aea into main Jun 30, 2026
6 checks passed
jeff-r2026 added a commit that referenced this pull request Jun 30, 2026
Post-merge quality cleanups on the HTTP team-repo + status-reporter
code (PRs #68/#70/#71), surfaced by /simplify. No behavior change.

- path-safety: extract assertWithinRoot() and replace the hand-rolled
  containment check duplicated in source-http.ts and skill-command.ts.
  It compares resolved-but-not-symlink-followed paths, so it stays
  correct for not-yet-created targets and under symlinked roots
  (e.g. macOS /var -> /private/var) — unlike assertSafePath, which
  would false-reject legitimate writes there.
- status-report: drop the no-op `await import('fs-extra')` in flushQueue
  (fs-extra is already loaded via utils/fs) and use the project
  remove/writeFile helpers; parallelize scanReportableSkills with
  Promise.all instead of awaiting per-skill reads in a loop.
- team-push: reuse readJson/writeJson for the reported-interventions
  snapshot instead of hand-rolled JSON.parse/stringify.

Verified: `tsc --noEmit` clean; full unit suite green (1662 passed).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@jeff-r2026
jeff-r2026 deleted the feat/human-intervention-metric branch July 2, 2026 08:03
@hsuchifeng hsuchifeng mentioned this pull request Jul 3, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feat] 统计每个对话的人工干预次数(Human Intervention 指标)

1 participant