Skip to content

Test parser fuzz inference invariants - #76

Merged
TraderSamwise merged 1 commit into
masterfrom
test/chat-parser-fuzz-expansion
Jun 6, 2026
Merged

Test parser fuzz inference invariants#76
TraderSamwise merged 1 commit into
masterfrom
test/chat-parser-fuzz-expansion

Conversation

@TraderSamwise

@TraderSamwise TraderSamwise commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • add generated assistant prose that casually mentions Claude Code and OpenAI Codex
  • assert omitted-tool parser mode matches explicit mode when transcript chrome is present
  • add no-chrome fuzz cases proving casual tool mentions do not infer parser mode

Verification

  • vitest run src/agent-output-parser-fuzz.test.ts
  • eslint src/ --max-warnings=0
  • tsc --noEmit
  • yarn build
  • vitest run

Summary by CodeRabbit

  • Tests
    • Enhanced test coverage for the agent output parser with improved inference validation and additional test scenarios.

@vercel

vercel Bot commented Jun 6, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
app Ready Ready Preview, Comment Jun 6, 2026 10:38am

@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown

Too many files changed? Review this PR in Change Stack to see how the pieces fit before you dive in.

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3d71b52a-1220-47f2-a52a-3ef8387b666f

📥 Commits

Reviewing files that changed from the base of the PR and between b52fd74 and 4f6b29b.

📒 Files selected for processing (1)
  • src/agent-output-parser-fuzz.test.ts

📝 Walkthrough

Walkthrough

This PR enhances test coverage for parseAgentOutput's tool inference behavior by adding a new fragment generator that produces assistant responses mentioning both "Claude Code" and "OpenAI Codex", wiring it into the fuzz test infrastructure, and verifying both that inference works correctly and that casual mentions alone do not trigger mode inference.

Changes

Tool mention inference testing

Layer / File(s) Summary
Tool mention fragment and wiring
src/agent-output-parser-fuzz.test.ts
New assistantToolMention fragment generator produces assistant responses with tool-specific markers and sentinel strings referencing both "Claude Code" and "OpenAI Codex". Fragment is added to both claude and codex entries in fragmentMakersForTool fuzz test mapping.
Inference and non-inference test coverage
src/agent-output-parser-fuzz.test.ts
Main fuzz test now validates inferred parsing (parseAgentOutput(raw)) against explicit parsing (parseAgentOutput(raw, { tool })), checking that inferred.parser.tool matches expected tool and blocks are equal. New deterministic test confirms that assistantToolMention-only transcripts leave parser tool as "unknown" with a single response block.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

  • TraderSamwise/aimux#75: Adds the inference helper logic that these new fuzz and deterministic tests exercise for both success (tool is inferred) and boundary (only mentions, no inference) cases.
  • TraderSamwise/aimux#68: Prior evolution of tool-mode fuzz coverage in the same test file, adding/adjusting transcript fragment makers and parsing assertions.
  • TraderSamwise/aimux#71: Overlaps at the same fragmentMakersForTool wiring layer, extending tool-specific fragment lists in the fuzz test infrastructure.

Poem

🐰 A fragment for mentions, both Claude and Codex alike,
Now fuzzed and deterministic, the parser's inference in sight.
Inference works when we mean it, stays unknown when we're just casual,
Test coverage blessed—the tool detection's no longer accidental! 🔍

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Test parser fuzz inference invariants' directly describes the main purpose of the changes: adding fuzz tests that verify parser inference behavior doesn't incorrectly infer tool mode from casual tool mentions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/chat-parser-fuzz-expansion

Comment @coderabbitai help to get the list of available commands and usage tips.

@TraderSamwise
TraderSamwise merged commit e8de9bd into master Jun 6, 2026
3 checks passed
@TraderSamwise
TraderSamwise deleted the test/chat-parser-fuzz-expansion branch June 6, 2026 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant