test(spec-loop): add runner fixture coverage#735
Conversation
Generated-by: Claude (Opus 4.7)
431e0ab to
988524b
Compare
potiuk
left a comment
There was a problem hiding this comment.
Re-reviewed after I rebased this onto current main and force-pushed — a clean, content-preserving replay onto the commits that landed since (no changes to your diff).
The one concern from the earlier review — reconciling the --model handling against main's launch code — resolved automatically on rebase: loop.sh now delegates to lib.sh's spec_loop_launch_agent, which uses the safe conditional [ -n "$model" ] && model_args=(--model "$model"), and the MODEL defaulting in loop.sh (empty for non-claude harnesses → agent default) feeds it correctly. So there's no unconditional --model "" path.
Verified: the runner fixture suite passes (spec-loop runner fixtures: OK), spec-validator pytest + spec-validate green, CI 35/35 including shellcheck. Faithful refactor with genuine test coverage. LGTM.
Summary
Add deterministic fixture coverage for the spec-loop runner
Extracts the spec-loop runner's deterministic logic into a testable library and adds a fixture suite that exercises it, closing the runner's long-standing "no fixture test" gap.
tools/spec-loop/lib.sh: pulls prompt assembly, harness command rendering, agent launch, and.last-syncmarker helpers out ofloop.shinto standalone, sourceable functions.loop.shnow sources them (net ~85 fewer lines there), so the same code paths the runner uses are the ones under test.tools/spec-loop/tests/test_runner_fixtures.sh: deterministic Bash fixtures covering prompt assembly per beat (build vs update scoping), harness argv construction across agents, and the.last-syncmarker write/branch-name helpers. No network, no agent, nouv.spec-validatorwiring: adds check feat(release-manager-handoff): explicit hand-off + publication-ready comments #10,validate_spec_loop_runner_fixtures, which runs the fixture suite viabashwhen the file is present and surfaces any non-zero exit as a validation violation.spec-validatenow executes the runner tests as part of normal validation, with a pytest added for the new validator branch.specs/spec-loop-runner.md): recordslib.shand the test file under Where it lives and source, adds them to the Validation block (shellcheck), and removes the now-resolved Known gap about missing fixture tests.The runner previously had no deterministic test for prompt assembly, harness command construction, or marker handling, so refactors risked silently breaking how prompts and agent invocations are built. Splitting the logic into
lib.shmakes it unit-testable, and routing the suite throughspec-validatekeeps it running automatically rather than by hand.The bash suite passes (
spec-loop runner fixtures: OK); the pytest covers the new validator hook.Generated-by: Claude (Opus 4.7)
Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)