Skip to content

test(spec-loop): add runner fixture coverage#735

Merged
potiuk merged 1 commit into
apache:mainfrom
justinmclean:spec-loop-runner-fixture-tests
Jul 4, 2026
Merged

test(spec-loop): add runner fixture coverage#735
potiuk merged 1 commit into
apache:mainfrom
justinmclean:spec-loop-runner-fixture-tests

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

Add deterministic fixture coverage for the spec-loop runner

Extracts the spec-loop runner's deterministic logic into a testable library and adds a fixture suite that exercises it, closing the runner's long-standing "no fixture test" gap.

  • New tools/spec-loop/lib.sh: pulls prompt assembly, harness command rendering, agent launch, and .last-sync marker helpers out of loop.sh into standalone, sourceable functions. loop.sh now sources them (net ~85 fewer lines there), so the same code paths the runner uses are the ones under test.
  • New tools/spec-loop/tests/test_runner_fixtures.sh: deterministic Bash fixtures covering prompt assembly per beat (build vs update scoping), harness argv construction across agents, and the .last-sync marker write/branch-name helpers. No network, no agent, no uv.
  • spec-validator wiring: adds check feat(release-manager-handoff): explicit hand-off + publication-ready comments #10, validate_spec_loop_runner_fixtures, which runs the fixture suite via bash when the file is present and surfaces any non-zero exit as a validation violation. spec-validate now executes the runner tests as part of normal validation, with a pytest added for the new validator branch.
  • Spec update (specs/spec-loop-runner.md): records lib.sh and the test file under Where it lives and source, adds them to the Validation block (shellcheck), and removes the now-resolved Known gap about missing fixture tests.

The runner previously had no deterministic test for prompt assembly, harness command construction, or marker handling, so refactors risked silently breaking how prompts and agent invocations are built. Splitting the logic into lib.sh makes it unit-testable, and routing the suite through spec-validate keeps it running automatically rather than by hand.

The bash suite passes (spec-loop runner fixtures: OK); the pytest covers the new validator hook.

Generated-by: Claude (Opus 4.7)

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other: bash suite passes

@justinmclean justinmclean self-assigned this Jul 4, 2026
Generated-by: Claude (Opus 4.7)
@potiuk potiuk force-pushed the spec-loop-runner-fixture-tests branch from 431e0ab to 988524b Compare July 4, 2026 18:53

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed after I rebased this onto current main and force-pushed — a clean, content-preserving replay onto the commits that landed since (no changes to your diff).

The one concern from the earlier review — reconciling the --model handling against main's launch code — resolved automatically on rebase: loop.sh now delegates to lib.sh's spec_loop_launch_agent, which uses the safe conditional [ -n "$model" ] && model_args=(--model "$model"), and the MODEL defaulting in loop.sh (empty for non-claude harnesses → agent default) feeds it correctly. So there's no unconditional --model "" path.

Verified: the runner fixture suite passes (spec-loop runner fixtures: OK), spec-validator pytest + spec-validate green, CI 35/35 including shellcheck. Faithful refactor with genuine test coverage. LGTM.

@potiuk potiuk merged commit 8182353 into apache:main Jul 4, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants