Skip to content

pearl th-7fdfa9: tmux load-buffer for newlines + ban slash commands in driver LLM - #57

Merged
brentrager merged 1 commit into
mainfrom
th-7fdfa9-tmux-newlines
May 20, 2026
Merged

pearl th-7fdfa9: tmux load-buffer for newlines + ban slash commands in driver LLM#57
brentrager merged 1 commit into
mainfrom
th-7fdfa9-tmux-newlines

Conversation

@brentrager

Copy link
Copy Markdown
Contributor

Summary

Fix two bench harness bugs surfaced by a single-task score-tui --debug smoke run against a healthy Big Smooth.

  • Bug 1: newlines mangled into j characters. TmuxDriver::send used tmux send-keys -l TEXT, which interprets every embedded \n as the C-j keysym; in literal mode (-l) C-j degrades to the bare letter j. The pearl debug log showed the affine-cipher task prompt rendering as affine-cipher (python).jjWorking directory:…jFiles present:j - INSTRUCTIONS.mdj…. Switched to tmux load-buffer + tmux paste-buffer, which inserts the payload as raw bytes — newlines preserved verbatim.
  • Bug 2: driver LLM uses Claude-Code-style slash commands. The default driver model (smooth-summarize) was emitting /open, /read, /help instead of plain English, and the TUI rejected them as unknown commands (and in two cases accidentally fired skills). Hardened the system + user prompts with explicit "no slash commands; you have no shell/file access" directives, and added a slash-command guard in run_human_loop that drops /-prefixed turns, logs them to the pane-debug log, re-asks with a reinforcement prompt, and after 3 consecutive bad turns bails with TASK_STUCK.

Pearl: th-7fdfa9.

Tests

  • tmux_driver::send_preserves_newlines_no_j_leakage — 3-line payload through cat > tmpfile asserts exactly 3 lines and no stray j.
  • human_driver::run_human_loop_marks_stuck_after_three_slash_commands — fake driver returning /open 3x asserts LoopExit::Stuck after exactly 2 retries.
  • human_driver::run_human_loop_accepts_plain_english_message — fake driver returning plain English asserts zero retries fired.
  • is_slash_command_detects_leading_slash, build_driver_prompt_warns_against_slash_commands, build_slash_retry_prompt_contains_offending_text_and_attempts.

All 125 smooth-bench tests pass. cargo fmt --check, cargo clippy, cargo build clean.

Verification

Re-ran the smoke that originally caught these bugs:

cargo run --release -p smooai-smooth-bench -- score-tui --pr \
  --th-binary $(which th) --task-limit 1 --debug \
  --output /tmp/bench/smoke2.json --budget-usd 2

New debug log: ~/.smooth/bench-runs/80c092b0/python-affine-cipher.pane.log. Initial prompt now renders with real newlines:

You:
You are solving an Aider Polyglot coding benchmark task: `affine-cipher` (python).

You:
Working directory: the current directory.

You:
Files present:

You:
  - INSTRUCTIONS.md

(Zero j artifacts.) Driver follow-ups are all plain English — sample turns from the log:

Please read INSTRUCTIONS.md and affine_cipher_test.py.
Once you've finished reading, please run the tests.
This is a Python project. Please read INSTRUCTIONS.md and affine_cipher_test.py.
Please run the tests to see what needs fixing.
Please read the contents of affine_cipher.py.

(Zero /-prefixed turns.) The single affine-cipher task itself FAILed — expected, it's a hard task on a single attempt. Harness is healthy.

Test plan

  • CI green (cargo fmt, clippy, test, build)
  • Reviewer can run a multi-task --pr sweep when ready

🤖 Generated with Claude Code

…n driver LLM

Fix two bench harness bugs surfaced by a single-task --debug smoke run.

1. TmuxDriver::send used `send-keys -l TEXT` which interprets every `\n`
   as the C-j keysym; in literal mode C-j degrades to the bare letter `j`.
   Multi-line task prompts rendered with `j` where the newlines should
   have been. Switched to `load-buffer` + `paste-buffer`, which inserts
   the payload as raw bytes — newlines preserved verbatim.

2. The default driver LLM (smooth-summarize) was emitting Claude-Code-
   style slash commands (/open, /read, /help) instead of plain English.
   Hardened the system + user prompts to forbid slash commands, and
   added a slash-command guard in run_human_loop that drops bad turns,
   logs them to the pane-debug log, re-asks the model with a
   reinforcement prompt, and after 3 consecutive bad turns bails with
   TASK_STUCK.

Tests:
- send_preserves_newlines_no_j_leakage: 3-line payload through cat>file
  asserts no stray `j` and exactly 3 lines.
- run_human_loop_marks_stuck_after_three_slash_commands.
- run_human_loop_accepts_plain_english_message.

Verified with smooth-bench score-tui --pr --task-limit 1 --debug
against a healthy Big Smooth: zero `j` newline artifacts, zero slash
commands in the new pane log.
@changeset-bot

changeset-bot Bot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 7ac910c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@smooai/smooth Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@brentrager
brentrager merged commit 484ee20 into main May 20, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant