Add parser corpus audit tooling - #81
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
More reviews will be available in 1 minute and 8 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. 📝 WalkthroughWalkthroughThis PR introduces a new audit system that scans agent output stored in JSONL history and markdown context directories to detect suspicious patterns: raw blocks, status-leak responses, and actionable prompts from responses. It includes a Node.js CLI script with command-line options for directory selection, maximum findings, and JSON output, plus comprehensive test coverage. ChangesAgent Output Parser Audit System
Sequence DiagramsequenceDiagram
participant CLI as audit:parser CLI
participant AuditFn as auditAgentOutputParserCorpus
participant HistoryScan as historyCandidates
participant ContextScan as contextCandidates
participant Parser as parseAgentOutput
participant Detector as Block inspector
CLI->>AuditFn: invoke with historyDirs, contextDirs, maxFindings
AuditFn->>HistoryScan: scan history directories
HistoryScan->>AuditFn: yield candidate records
AuditFn->>ContextScan: scan context directories
ContextScan->>AuditFn: yield candidate files
AuditFn->>Parser: parse candidate content
Parser->>Detector: return parsed blocks
Detector->>AuditFn: identify raw blocks, status-leak patterns, actionable prompts
AuditFn->>AuditFn: accumulate findings, maintain counts by flag
AuditFn->>CLI: return summary with scanned count, findings, countsByFlag
Estimated Code Review Effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly Related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/agent-output-parser-audit.test.ts (1)
22-81: ⚡ Quick winConsider adding test coverage for context directory scanning.
The current tests validate
historyCandidates(JSONL files) but do not exercisecontextCandidates(live.md/summary.md files). Since context scanning is a parallel code path with its own recursive directory logic, adding at least one test case would improve confidence in that behavior.📋 Example test case for context directory scanning
it("scans context markdown files for suspicious patterns", () => { const dir = makeTempDir(); const contextSubdir = join(dir, "context-subdir"); mkdirSync(contextSubdir); writeFileSync( join(contextSubdir, "live.md"), "Some content with terminal-notifier status leak", ); const summary = auditAgentOutputParserCorpus({ contextDirs: [dir] }); expect(summary.scanned).toBe(1); expect(summary.findings.length).toBeGreaterThan(0); });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/agent-output-parser-audit.test.ts` around lines 22 - 81, Add a test exercising the contextDirs path of auditAgentOutputParserCorpus: create a temporary dir and a nested subdir, write a live.md or summary.md containing a suspicious marker (e.g., "terminal-notifier"), call auditAgentOutputParserCorpus({ contextDirs: [dir] }) and assert that summary.scanned increments and summary.findings contains the expected flag(s); this ensures the contextCandidates branch (the recursive markdown scanning logic) is covered alongside the existing historyCandidates JSONL tests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@package.json`:
- Line 39: The "audit:parser" npm script currently runs Node directly against
compiled files and can execute stale or missing dist output; update the
"audit:parser" script so it prepends "yarn build &&" before the Node invocation
to ensure TypeScript is compiled to dist (the script that imports
dist/agent-output-parser-audit.js) before running; modify the package.json
"audit:parser" entry to include this build step.
---
Nitpick comments:
In `@src/agent-output-parser-audit.test.ts`:
- Around line 22-81: Add a test exercising the contextDirs path of
auditAgentOutputParserCorpus: create a temporary dir and a nested subdir, write
a live.md or summary.md containing a suspicious marker (e.g.,
"terminal-notifier"), call auditAgentOutputParserCorpus({ contextDirs: [dir] })
and assert that summary.scanned increments and summary.findings contains the
expected flag(s); this ensures the contextCandidates branch (the recursive
markdown scanning logic) is covered alongside the existing historyCandidates
JSONL tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c129fb35-d146-46e1-bd15-0a32b3f2b014
📒 Files selected for processing (4)
package.jsonscripts/audit-agent-output-parser.mjssrc/agent-output-parser-audit.test.tssrc/agent-output-parser-audit.ts
1 participant
Summary
Verification
Note
Summary by CodeRabbit
New Features
Tests
Chores
audit:parsernpm script for convenient command-line access to the audit tool.