Skip to content

Test parser live mined actions#71

Merged
TraderSamwise merged 1 commit into
masterfrom
test/chat-parser-live-mining
Jun 6, 2026
Merged

Test parser live mined actions#71
TraderSamwise merged 1 commit into
masterfrom
test/chat-parser-live-mining

Conversation

@TraderSamwise

@TraderSamwise TraderSamwise commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • Classify Claude/Codex tool action rows such as Bash, Read, Update, and Ran command output as status instead of assistant chat.
  • Add live-mined fixtures for Codex startup suggestion loops without box borders and Claude tool/action rows.
  • Extend deterministic parser fuzzing and dummy runtime harness coverage with action/status rows.

Verification

  • /Users/sam/cs/aimux/node_modules/.bin/vitest run src/agent-output-parser-fixtures.test.ts src/agent-output-parser-harness.test.ts src/agent-output-parser-fuzz.test.ts src/agent-output-parser.test.ts
  • /Users/sam/cs/aimux/node_modules/.bin/tsc --noEmit
  • PATH=/Users/sam/cs/aimux/node_modules/.bin:$PATH yarn lint
  • PATH=/Users/sam/cs/aimux/node_modules/.bin:$PATH yarn build
  • PATH=/Users/sam/cs/aimux/node_modules/.bin:$PATH yarn test

Summary by CodeRabbit

  • Tests

    • Added comprehensive test fixtures for Codex and Claude agent output scenarios
    • Expanded test coverage for tool action detection and status line recognition
    • Enhanced fuzz testing with additional action row variants
  • Improvements

    • Strengthened status line detection logic for improved output parsing accuracy

@vercel

vercel Bot commented Jun 6, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
app Ready Ready Preview, Comment Jun 6, 2026 9:21am

@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown

Ready to act? Review this PR in Change Stack to turn feedback into patch suggestions you can inspect and refine.

Review Change Stack

📝 Walkthrough

Walkthrough

This PR extends the agent output parser to recognize tool/action-like text in status lines and adds comprehensive test coverage. The parser now detects common command patterns (bash, read, update, git, yarn) and completion phrases, classifying bullet-prefixed lines containing these patterns as status blocks. Two new fixtures and updated fuzz/integration tests validate the behavior.

Changes

Tool/Action Row Parsing and Validation

Layer / File(s) Summary
Tool/action text detection in status classification
src/agent-output-parser.ts
New looksLikeToolActionText helper regex patterns recognize command/tool names and execution phrases. Updated isStatusLine to treat bullet lines (•/⏺) containing tool/action text as status blocks instead of other line types.
Test fixtures for tool/action parsing
src/agent-output-parser-fixtures.ts
Added codex-live-startup-suggestion-loop fixture verifying MCP startup and suggestion blocks with invariant that "Explain this codebase" does not appear in parsed prompts. Added claude-live-tool-action-rows fixture demonstrating action row parsing into status blocks while separating action phrases from response text.
Fuzz generation and integration test updates
src/agent-output-parser-fuzz.test.ts, src/agent-output-parser-harness.test.ts
Fuzz test now generates toolActionRow fragments with tool-specific sentinel tokens (claude: bash/read/update/background; codex: git/yarn/bash/diff/status variants). Fragment makers updated for both tools. Integration test replaced with edge-case coverage validating readAll(-200) parsing of new fixtures with expected block sequences and specific text substrings.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • TraderSamwise/aimux#67: Both modify src/agent-output-parser.ts status-line classification logic to detect activity/progress/tool-like patterns and add fixtures/tests to lock in status block behavior.
  • TraderSamwise/aimux#69: Extends agent-output-parser-harness.test.ts with new runtime readAll coverage using fixtures that directly build on the harness infrastructure.
  • TraderSamwise/aimux#66: Directly modifies src/agent-output-parser-fixtures.ts by updating the shared AGENT_OUTPUT_PARSER_FIXTURES set.

Poem

🐰 Tool actions grouped with status cheer,
Bullet points now crystal clear!
Codex startup, Claude's action rows—
Parser logic safely grows.
Fuzz and fixtures seal the test,
Status lines now know the rest! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Test parser live mined actions' directly aligns with the PR's main objective of adding test coverage for live-mined parser actions and classifying tool action rows.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/chat-parser-live-mining

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/agent-output-parser.ts (1)

34-48: 💤 Low value

Consider tightening the \b.*\b pattern in the tool-name regex.

Lines 38-40 use \b.*\b(?:ctrl\+o|to expand|Running in the background|exit code) which matches word-boundary + any content + word-boundary before the specific phrases. While this works correctly for the tested fixtures, it's quite permissive and could theoretically match unintended text like Bash arbitrary content ctrl+o.

A more constrained pattern might reduce false-positive risk, for example limiting the middle portion to common patterns like \d+\s+\w+.* for file-count constructs. However, given the regex is anchored with specific tool names and the fixtures validate the intended behavior, the current approach is acceptable.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agent-output-parser.ts` around lines 34 - 48, The regex in
looksLikeToolActionText uses \b.*\b before the tail phrases which is overly
permissive; update that portion of the first big regexp in the
looksLikeToolActionText function to a tighter pattern (for example restrict the
middle to common constructs like file/count patterns or limit length) such as
replacing \b.*\b with a constrained alternative like \d+\s+\w+.* or a bounded
character class (e.g. .{0,50}) so it only matches expected short phrases (file
counts, short descriptions) before the group (ctrl+o|to expand|Running in the
background|exit code), then run tests to ensure no desired fixtures break.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/agent-output-parser.ts`:
- Around line 34-48: The regex in looksLikeToolActionText uses \b.*\b before the
tail phrases which is overly permissive; update that portion of the first big
regexp in the looksLikeToolActionText function to a tighter pattern (for example
restrict the middle to common constructs like file/count patterns or limit
length) such as replacing \b.*\b with a constrained alternative like \d+\s+\w+.*
or a bounded character class (e.g. .{0,50}) so it only matches expected short
phrases (file counts, short descriptions) before the group (ctrl+o|to
expand|Running in the background|exit code), then run tests to ensure no desired
fixtures break.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a4153b42-218b-405b-9cd5-4275930168a8

📥 Commits

Reviewing files that changed from the base of the PR and between 7052a66 and 83758cf.

📒 Files selected for processing (4)
  • src/agent-output-parser-fixtures.ts
  • src/agent-output-parser-fuzz.test.ts
  • src/agent-output-parser-harness.test.ts
  • src/agent-output-parser.ts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant