Skip to content

Fix parser action status leaks - #88

Merged
TraderSamwise merged 2 commits into
masterfrom
test/chat-parser-next-live-pass
Jun 6, 2026
Merged

Fix parser action status leaks#88
TraderSamwise merged 2 commits into
masterfrom
test/chat-parser-next-live-pass

Conversation

@TraderSamwise

@TraderSamwise TraderSamwise commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • parse plain Codex command rows like Ran git worktree list --porcelain as status instead of assistant text
  • keep compact Claude terminal-notifier rows out of assistant chat when the command wraps before its closing parenthesis
  • add an action-status-leak parser corpus audit flag for action/activity rows leaking into response blocks

Verification

  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin /Users/sam/cs/aimux/node_modules/.bin/vitest run src/agent-output-parser-audit.test.ts src/agent-output-parser.test.ts src/agent-output-parser-fixtures.test.ts
  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/cs/aimux/node_modules/.bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin yarn build
  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin node scripts/audit-agent-output-parser.mjs --history /Users/sam/cs/aimux/.aimux/history --context /Users/sam/cs/aimux/.aimux/context --flag action-status-leak --max 5
  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin node scripts/audit-agent-output-parser.mjs --history /Users/sam/cs/aimux/.aimux/history --context /Users/sam/cs/aimux/.aimux/context --flag status-leak-response --max 5
  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin node scripts/audit-agent-output-parser.mjs --history /Users/sam/cs/aimux/.aimux/history --context /Users/sam/cs/aimux/.aimux/context --flag raw-block --max 5
  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin node scripts/audit-agent-output-parser.mjs --history /Users/sam/cs/aimux/.aimux/history --context /Users/sam/cs/aimux/.aimux/context --flag prompt-from-response-record --max 5
  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin /Users/sam/cs/aimux/node_modules/.bin/tsc --noEmit
  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin /Users/sam/cs/aimux/node_modules/.bin/eslint src/ --max-warnings=0
  • PATH=/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/.local/volta-shims:/Users/sam/.local/bin:/Users/sam/.bun/bin:/Users/sam/.codex/tmp/arg0/codex-arg0MLlwy8:/Users/sam/.volta/tools/image/packages/@openai/codex/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path:/opt/homebrew/opt/mysql-client/bin:/Users/sam/opt/anaconda3/envs/py39/bin:/Users/sam/.npm-global/bin:/Users/sam/pathother:/Users/sam/apache-ant/bin:/usr/local/mysql/bin:/Users/sam/.codex/tmp/arg0/codex-arg0FiIdYh:/Applications/cmux.app/Contents/Resources/bin:/Users/sam/.nvm/versions/node/v24.16.0/bin:/Users/sam/opt/anaconda3/condabin:/Users/Sam/.npm-global/bin:/Users/Sam/pathother:/Users/Sam/apache-ant/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:/Users/sam/.cargo/bin:/Users/sam/bin /Users/sam/cs/aimux/node_modules/.bin/vitest run $(rg --color never --files src -g '*.test.ts' | grep -v '^src/daemon.test.ts$')

Note: src/daemon.test.ts remains excluded locally because a live aimux-dev project service owns its fixed test port in this environment.

Summary by CodeRabbit

  • New Features

    • Added a new action-status-leak audit finding to flag action/status leakage in responses
  • Bug Fixes

    • Improved detection and classification of command/action vs status lines, including compact terminal-notifier rows
    • Refined status vs response separation to avoid swallowing prose
  • Tests

    • Expanded tests and fixtures for action-status-leak behavior and compact notifier/command parsing

@vercel

vercel Bot commented Jun 6, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
app Ready Ready Preview, Comment Jun 6, 2026 1:52pm

@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f309d1e1-a634-4530-b387-36da43d1e475

📥 Commits

Reviewing files that changed from the base of the PR and between 76d1e6d and 7d38415.

📒 Files selected for processing (2)
  • src/agent-output-parser-audit.test.ts
  • src/agent-output-parser.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/agent-output-parser.ts
  • src/agent-output-parser-audit.test.ts

📝 Walkthrough

Walkthrough

This PR adds an action-status-leak audit finding flag, refines parser heuristics to detect tool action / terminal-notifier status lines, updates audit detection patterns and counters, wires CLI support for the flag, and expands tests and fixtures to validate the behavior.

Changes

Action-status-leak audit detection

Layer / File(s) Summary
Parser heuristics for tool actions and status output
src/agent-output-parser.ts
Added looksLikeRanCommandText and refactored looksLikeToolActionText; isStatusLine now treats -prefixed Bash(...terminal-notifier) lines as status.
Parser tests for new classification behaviors
src/agent-output-parser.test.ts
Two new Vitest cases validate Codex-style command bullets and compact Claude terminal-notifier rows are routed into response vs status blocks correctly.
Audit flag type and detection patterns
src/agent-output-parser-audit.ts
ParserAuditFindingFlag includes "action-status-leak"; ACTION_STATUS_LEAK_PATTERNS, emptyCounts() update, and audit loop additions detect and record matches in response blocks.
CLI support and audit test fixtures
scripts/audit-agent-output-parser.mjs, src/agent-output-parser-audit.test.ts
CLI --flag parsing accepts action-status-leak. Test fixtures were reformatted and expanded to assert that action/command rows are flagged when requested and that ordinary prose is not flagged; truncation/count-aggregation fixtures updated accordingly.

Sequence Diagram

sequenceDiagram
  participant Input as Input response block
  participant Parser as isStatusLine & looksLikeToolActionText
  participant Patterns as ACTION_STATUS_LEAK_PATTERNS
  participant Audit as auditAgentOutputParserCorpus
  participant Findings as Findings array

  Input->>Parser: classification checks (Ran <cmd>, tool-header, ⏺ terminal-notifier)
  Parser-->>Audit: classification result (status/response)
  Input->>Patterns: test response text against patterns
  Patterns-->>Audit: match detected?
  Audit->>Findings: record "action-status-leak"
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • TraderSamwise/aimux#81: Extends the same audit corpus tooling; this PR adds the action-status-leak flag and patterns.
  • TraderSamwise/aimux#79: Overlapping changes to src/agent-output-parser.ts status-line classification for terminal-notifier/runtime rows.
  • TraderSamwise/aimux#77: Related parser test coverage for spinner/terminal-notifier and tool/action row classification.

Poem

🐰 I sniff the lines where actions hide,
Patterns and flags at my side,
A tiny hop, a careful peek —
Status leaks found, the parser speaks! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Fix parser action status leaks' clearly and concisely summarizes the main change—addressing action/status information leaking into response blocks in the parser.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/chat-parser-next-live-pass

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/agent-output-parser.ts`:
- Around line 34-40: The helper looksLikeToolActionText is too broad and is
being applied to every non-empty line (causing normal assistant prose to be
misclassified); restrict its use by narrowing the regex and its application:
update looksLikeToolActionText to only match explicit tool-action prefixes at
the start of a line (e.g., require a verb like "Ran", "Searched", "Read"
followed by a command token or numeric file/count pattern and anchor to ^), and
change the call site that currently applies it to all non-empty lines (in
src/agent-output-parser.ts) so it only runs against short single-line status
candidates (e.g., lines under a small length threshold or lines composed mostly
of a verb + token), thereby avoiding reclassifying ordinary prose. Ensure you
modify the function name references unchanged and keep behavior for genuine
tool-action lines intact.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: cbb724ff-5890-411a-9e68-205c06c69a7f

📥 Commits

Reviewing files that changed from the base of the PR and between 29e8ac5 and 76d1e6d.

📒 Files selected for processing (5)
  • scripts/audit-agent-output-parser.mjs
  • src/agent-output-parser-audit.test.ts
  • src/agent-output-parser-audit.ts
  • src/agent-output-parser.test.ts
  • src/agent-output-parser.ts

Comment thread src/agent-output-parser.ts Outdated
@TraderSamwise
TraderSamwise merged commit a5d78b3 into master Jun 6, 2026
3 checks passed
@TraderSamwise
TraderSamwise deleted the test/chat-parser-next-live-pass branch June 6, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant