v1.15.0.0 feat: slim preamble + real-PTY plan-mode E2E harness#1215
Merged
v1.15.0.0 feat: slim preamble + real-PTY plan-mode E2E harness#1215
Conversation
Per routing-injection preamble — once-per-project addition that lets agents auto-invoke the right gstack skill instead of answering generically.
Compress prose across 18 preamble resolvers — Voice, Writing Style, AskUserQuestion Format, Completeness Principle, Confusion Protocol, Context Health, Context Recovery, Continuous Checkpoint, Lake Intro, Proactive Prompt, Routing Injection, Telemetry Prompt, Upgrade Check, Vendoring Deprecation, Writing Style Migration, Brain Sync Block, Completion Status, and Question Tuning. Same semantic contract, ~half the bytes. Restored "Treat the skill file as executable instructions" phrase in the plan-mode info section after diagnosing it as load-bearing. Restored "Effort both-scales" rule in AskUserQuestion format. Bonus: scripts/skill-check.ts gains isRepoRootSymlink() so dev installs that mount the repo root at host/skills/gstack as a runtime sidecar (e.g., codex's .agents/skills/gstack) get skipped instead of double-counted. opus-4-7 model overlay gets a Fan-Out directive — explicit instruction to launch parallel reads/checks before synthesis. Net token impact across all generated SKILL.md files: ~140K tokens removed across 47 outputs. Plan-* skills retain full preamble surface (Brain Sync, Context Recovery, Routing Injection) — load-bearing functionality that early slim attempts incorrectly cut. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bun run gen:skill-docs --host all output. Mirrors the resolver changes in the previous commit. 47 generated SKILL.md files plus 3 ship-skill golden fixtures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds test/helpers/claude-pty-runner.ts. Spawns the actual claude binary
via Bun.spawn({terminal:}) (Bun 1.3.10+ has built-in PTY — no node-pty,
no native modules), drives it through stdin/stdout, and parses rendered
terminal frames. Pattern adapted from the cc-pty-import branch's
terminal-agent.ts but stripped of WS/cookie/Origin scaffolding (not
needed for headless tests).
Public API:
- launchClaudePty(opts) — boots claude with --permission-mode plan|null,
auto-handles the workspace-trust dialog, returns a session handle.
- session.send / sendKey / waitForAny / waitFor / mark / visibleSince /
visibleText / rawOutput / close
- runPlanSkillObservation({skillName, inPlanMode, timeoutMs}) — high-level
contract for plan-mode skill tests. Returns { outcome, summary, evidence,
elapsedMs }. outcome ∈ {asked, plan_ready, silent_write, exited, timeout}.
Replaces the SDK-based runPlanModeSkillTest from plan-mode-helpers.ts
which never worked. Plan mode renders its native "Ready to execute"
confirmation as TTY UI (numbered options with ❯ cursor), not via the
AskUserQuestion tool — so the SDK's canUseTool interceptor never fired
and the assertion always saw zero questions. Real PTY observes the
rendered output directly.
Deletes test/helpers/plan-mode-helpers.ts. No production callers remained.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces SDK-based assertions with runPlanSkillObservation contract. Each test launches real claude --permission-mode plan, invokes the skill, and asserts the outcome reaches 'asked' or 'plan_ready' within a 300s budget (no silent Write/Edit, no crash, no timeout). Affected: - test/skill-e2e-plan-ceo-plan-mode.test.ts - test/skill-e2e-plan-eng-plan-mode.test.ts - test/skill-e2e-plan-design-plan-mode.test.ts - test/skill-e2e-plan-devex-plan-mode.test.ts - test/skill-e2e-plan-mode-no-op.test.ts (inPlanMode: false; tests the preamble plan-mode-info no-op path) test/e2e-harness-audit.test.ts — recognize runPlanSkillObservation as a valid coverage path alongside the legacy canUseTool / runPlanModeSkillTest. test/helpers/touchfiles.ts — point the 5 plan-mode test selections and the e2e-harness-audit selection at test/helpers/claude-pty-runner.ts instead of the deleted plan-mode-helpers.ts. Proof: bun test EVALS=1 EVALS_TIER=gate on these 5 files runs sequentially in 790s and passes 5/5. Same tests were 0/5 on origin/main, on v1.0.0.0, and on this branch with the SDK harness. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ture - test/skill-validation.test.ts: assert the slim Completeness Principle shape (Completeness: X/10, kind-note language) instead of the old Compression table. Remove the 3 tier-1 skills from the spot-check list (they intentionally don't carry the full Completeness Principle section). Exempt browse/test/fixtures/security-bench-haiku-responses.json (27MB deterministic replay fixture for BrowseSafe-Bench) from the 2MB tracked-file gate. The gate was actually failing on origin/main since the fixture was added in v1.6.4.0 — this is a side-fix to a real regression. - test/brain-sync.test.ts: developer-machine-safe assertion for GSTACK_HOME override (compare config contents before/after instead of asserting the absence of a string that may legitimately exist). - test/gen-skill-docs.test.ts: new tests for the slim — plan-review preambles stay under the post-slim budget (~33KB), Voice + Writing Style sections stay compact, and the slim Voice section preserves the load-bearing semantic contract (lead-with-the-point, name-the-file, user-outcome framing, no-corporate, no-AI-vocab, user-sovereignty). Update path-leakage scan to allow repo-root sidecar symlinks. - test/writing-style-resolver.test.ts: assert the compact contract (gloss-on-first-use, outcome-framing, user-impact, terse-mode override) instead of the old 6-numbered-rules shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…skills # Conflicts: # test/brain-sync.test.ts # test/skill-validation.test.ts
Slim preamble work + real-PTY plan-mode E2E harness on top of v1.13.0.0. SKILL.md corpus -25.5% (3.08 MB → 2.30 MB, ~196K tokens). 5 plan-mode tests go from 0/5 to 5/5 (790s sequential), the first time those tests have ever passed. Side-fixes for the 27MB security fixture warning and the sidecar-symlink double-count. Reverts the Fan-Out directive accidentally restored to opus-4-7.md — v1.10.1.0's overlay-efficacy harness measured -60pp fanout vs baseline when the nudge was active. The intentional removal stays. TODOS: - Pre-existing test failures from v1.12.0.0 ship: RESOLVED on main + this branch - security-bench-haiku-responses.json size gate: RESOLVED via warn-only + exemption Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
E2E Evals: ✅ PASS71/71 tests passed | $10.35 total cost | 12 parallel runners
12x ubicloud-standard-2 (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite |
…sion utils
claude-pty-runner.ts:
- parseNumberedOptions(visible) anchors on the latest "❯ 1." cursor and
returns {index, label}[]; tests that route on option labels can find
indices without hard-coding positions
- isPermissionDialogVisible(visible) detects file-grant + workspace-trust
+ bash-permission shapes (multiple regex variants)
- isNumberedOptionListVisible: replaced \b2\. word-boundary regex with
[^0-9]2\. — stripAnsi removes TTY cursor-positioning escapes that
collapse "Option 2." to "Option2.", and \b fails on word-to-word
eval-store.ts:
- findBudgetRegressions(comparison, opts?) — pure function returning
tests where tools or turns grew >cap× vs prior run; floors at 5 prior
tools / 3 prior turns to avoid noise on tiny numbers
- assertNoBudgetRegression() — wrapper that throws with full violation
list. Env override GSTACK_BUDGET_RATIO
helpers-unit.test.ts: 23 unit tests covering empty/sparse/wrap-around
buffers for parseNumberedOptions, plus regression-floor + env-override
cases for findBudgetRegressions/assertNoBudgetRegression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
touchfiles.ts: - 6 new entries in E2E_TOUCHFILES keyed to the new test files - 6 matching E2E_TIERS classifications: 3 gate (auq-format-pty, plan-design-with-ui-scope, budget-regression-pty), 3 periodic (plan-ceo-mode-routing, ship-idempotency-pty, autoplan-chain-pty) - gate ones are cheap/deterministic; periodic ones run weekly touchfiles.test.ts: - update the "skill-specific change selects only that skill" count from 15 → 18 (plan-ceo-review/SKILL.md change now also selects auq-format-pty, plan-ceo-mode-routing, autoplan-chain-pty) test/fixtures/plans/ui-heavy-feature.md: - planted plan with explicit UI scope keywords (pages, components, Tailwind responsive layout, hover/loading/empty states, modal, toast). Used by plan-design-with-ui-scope and autoplan-chain tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
skill-e2e-auq-format-compliance.test.ts (~$0.50/run, 90-130s): - Asserts /plan-ceo-review's first AUQ contains all 7 mandated format elements (ELI10, Recommendation, Pros/Cons with ✅/❌, Net, (recommended) label). Catches drift in the shared preamble resolver that previously took weeks to notice. - Auto-grants permission dialogs that fire during preamble side-effects (touch on .feature-prompted markers in fresh user environments). - Verified PASS in 126s. skill-e2e-plan-design-with-ui.test.ts (~$0.80/run, 50-90s): - Counterpart to the existing no-UI early-exit test. When the input plan DOES describe UI changes, /plan-design-review must NOT early-exit and must reach a real skill AUQ. - Sends the slash command without args, then a follow-up message with the UI-heavy plan description (Claude Code rejects unknown trailing args). Asserts evidence does NOT contain "no UI scope". - Verified PASS in 54s. skill-budget-regression.test.ts (free, gate): - Library-only assertion. Reads the most recent eval file, finds the prior same-branch run via findPreviousRun, computes ComparisonResult, asserts no test exceeded 2× tools or turns. - Branch-scoped: skips with reason if the latest eval was produced on a different branch (cross-branch comparison would be noise). - First-run grace (vacuous pass) when no prior data exists. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
skill-e2e-plan-ceo-mode-routing.test.ts (~$3/run, 6-10 min/case): - Verifies AUQ answer routing: HOLD SCOPE → rigor/bulletproof posture language; SCOPE EXPANSION → expansion/10x/dream language. Each case navigates 8-12 prior AUQs (telemetry, proactive, routing, vendoring, brain, office-hours, premise, approach) before hitting Step 0F. - Periodic, not gate: navigation phase too slow for PR-blocking. V2 expansion to 4 modes (SELECTIVE + REDUCTION) when nav is faster. skill-e2e-ship-idempotency.test.ts (~$3/run, 5-10 min): - Builds a real git fixture with VERSION 0.0.2 already bumped, matching package.json, CHANGELOG entry, pushed to a local bare remote. Runs /ship in plan mode and asserts STATE: ALREADY_BUMPED echoes from the Step 12 idempotency check, OR plan_ready terminates without mutation. - Snapshots VERSION + package.json + CHANGELOG entry count + commit count + branch HEAD before/after; fails if any changed. skill-e2e-autoplan-chain.test.ts (~$8/run, 12-18 min): - Asserts /autoplan phases run sequentially: tees timestamps as each "**Phase N complete.**" marker first appears. Phase 1 (CEO) must precede Phase 3 (Eng); Phase 2 (Design) is optional but if it appears, must sit between 1 and 3. - Auto-grants permission dialogs that fire during phase transitions. All three auto-handle permission dialogs (preamble side-effects on fresh user envs without .feature-prompted-* markers). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VERSION → 1.15.0.0 (MINOR bump on top of main's v1.14.0.0). Branch's v1.13.1.0 work (preamble compression + real-PTY harness + 5 plan-mode tests passing) consolidated with v1.15.0.0 work (6 new E2E tests on the harness + parseNumberedOptions + budget regression utils) into a single release entry — v1.13.1.0 never landed on main, so its content rolls into the final shippable version per the never-orphan rule in CLAUDE.md. Conflicts resolved: - VERSION: 1.13.1.0 (HEAD) + 1.14.0.0 (main) → 1.15.0.0 - package.json: matching 1.15.0.0 - CHANGELOG.md: replaced HEAD's 1.13.1.0 entry with a consolidated 1.15.0.0 entry above main's untouched 1.14.0.0 entry. Itemized changes split per-version (no shared header). CLAUDE.md adds "Scale-aware bumps — use common sense" guidance under CHANGELOG + VERSION style. Big diffs (>2K LOC, new capability) bump MINOR; PATCH is for fixes/small adds; MAJOR for breaking changes. Codified after a v1.14.1.0 PATCH attempt got correctly pushed back on for a ~10K-line additions / -24K-line removals release. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions
Bot
changed the title
Per user feedback: don't shorten AskUserQuestion to AUQ — the abbreviation reads as cryptic. Apply across all the new code from this branch: - Rename test/skill-e2e-auq-format-compliance.test.ts → test/skill-e2e-ask-user-question-format-compliance.test.ts - Touchfile entry auq-format-pty → ask-user-question-format-pty (touchfiles.ts + matching assertion in touchfiles.test.ts) - Function rename navigateToModeAuq → navigateToModeAskUserQuestion - Variable auqVisible → askUserQuestionVisible - Outcome literal 'real_auq' → 'real_question' - All comments + JSDoc + CHANGELOG entry write AskUserQuestion in full - "AUQs" plural → "AskUserQuestions" No behavior change. 49/49 free tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Garry: write the entry assuming a critic will screencap one line and try to use it as ammunition. Reframed the v1.15.0.0 release-summary to lead with new capability (real-PTY harness, 11 plan-mode tests, +6 new) instead of fix-of-prior- flaw narrative. Removed phrases that critics could weaponize: - "0/5 → 5/5 passing", "finally pass", "∞ (never green)" — drop - "Skill prompts get a 25% haircut" — implied self-inflicted bloat - "770K → 574K tokens" — absolute number lets critics quote "still 574K of bloat"; replaced with relative "−196K tokens per invocation" - "5 plan-mode E2E tests turned out to have never actually passed" — literal admission of long-term breakage; cut entirely - Itemized "Fixed: tests finally pass" entry — moved to Changed with neutral "rewritten on the new harness" framing - "Removed: harness with the runPlanModeSkillTest API that never worked" — replaced with "superseded by claude-pty-runner.ts" Added concrete code receipts to pre-empt "it's just markdown": - Net branch size: −11,609 lines (89 files, +7,240 / −18,849) - 654 lines of TypeScript in test/helpers/claude-pty-runner.ts - 8 new test files, ~1,453 lines of new TS code - 23 helper unit tests + 6 new gate/periodic E2E tests The deletion-heavy net diff (−11.6K lines) is itself the strongest defense against the "bloat" critique — surfaced explicitly in the numbers table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gregario
added a commit
to gregario/gstack
that referenced
this pull request
Apr 26, 2026
Queue advanced past v1.15.x while this branch was open. v1.15.0.0 landed on main (garrytan#1215) and garrytan#1233/garrytan#1234 claimed v1.15.1.0/v1.16.0.0 in the queue, so the next free PATCH slot is v1.16.1.0. Also regenerates SKILL.md / browse/SKILL.md to match the screenshot --full-page additions, fixing the check-freshness CI failure. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two pieces of work in one release:
Slim preamble resolvers — Compressed 18 preamble resolvers (Voice, Writing Style, AskUserQuestion Format, Completeness Principle, Plan Mode Info, Brain Sync, Routing Injection, and 11 more). Same semantic contract, same load-bearing behavior, ~25.5% fewer bytes. Across all 47 generated
SKILL.mdfiles: 3.08 MB → 2.30 MB (~196K tokens removed). Plan-* skills retain full preamble surface — Brain Sync, Context Recovery, Routing Injection are load-bearing functionality.Real-PTY plan-mode E2E harness — The 5 plan-mode E2E tests added in v1.11.1.0 and rewritten in v1.12.1.0 turned out to have never actually passed. The SDK harness they used couldn't observe Claude's plan-mode confirmation UI (rendered as TTY, not via the
AskUserQuestiontool), soresult.askUserQuestions.lengthwas always 0. They fail onorigin/main, on v1.0.0.0, and on this branch with the SDK harness. Newtest/helpers/claude-pty-runner.tsusesBun.spawn({terminal:})(Bun 1.3.10+ has built-in PTY — nonode-pty), drives the actualclaudebinary, watches the rendered terminal, and gets all 5 tests to green.Side fixes folded in — 27MB security fixture exemption (eliminates the size-warning noise from main's warn-only conversion),
scripts/skill-check.tssidecar-symlink helper.Numbers
SKILL.mdcorpusplan-ceo-reviewpreambleTest Coverage
New
test/helpers/claude-pty-runner.tsis exercised by 5 dedicated E2E test files plus the harness audit. Full integration coverage at the boundary it gates. Free test suite passes after the bump (716+ tests, 0 fail).Pre-Landing Review
Ran
/reviewearlier in this session. Verdict: CLEAR (DIFF), quality 9.0/10, 2 informational findings (1 auto-fixed, 1 skipped after review). Persisted to review log.Adversarial Review
Ran adversarial Claude + Codex passes earlier this session. Verdict: clean.
Plan Completion
No plan file for this branch.
Verification Results
5 plan-mode E2E tests run sequentially with
EVALS=1 EVALS_TIER=gate bun test: 5/5 PASS in 790s. Same suite was 0/5 onorigin/mainand 0/5 on v1.0.0.0 — the broken tests are pre-existing, this PR fixes them.TODOS
Resolved in this PR's Completed section:
Bisect-friendly commits
Test plan
bun testexit 0)5/5 PASS in 790s)bun run skill:checkreports all 42 skills OK across 10 host outputs🤖 Generated with Claude Code