Approve stdout and stderr separately in every approval test - #58
Closed
MPV wants to merge 1 commit into
Closed
Conversation
This was referenced Jul 23, 2026
The approval tests only captured stdout (the image list), so a change to the error text a failing manifest produces was invisible to the suite. Behavior changes in later PRs — e.g. treating an unsupported kind as skippable rather than an error — should show up as a reviewable golden-file diff, which requires capturing stderr too. Verify stdout and stderr as separate approved files using go-approval-tests' WithAdditionalInformation (which needs v1.5.0). The existing Service golden becomes the .stdout file (still empty) and a new .stderr golden captures the current error message. Received files are gitignored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pc6NAURAqjU4LYJx93tgSC
MPV
force-pushed
the
claude/approve-stdout-stderr
branch
from
July 23, 2026 08:17
929d8a3 to
a0611cd
Compare
MPV
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Give the approval suite a shared
verify(t, file)helper that approves what the tool prints to stdout (the images) and to stderr (the error message, if any) as two separate golden files, and route every test through it.Requires bumping
go-approval-testsv1.2.0 → v1.5.0 forOptions().WithAdditionalInformation(...)..received.txtfiles are gitignored.Why
The suite only captured stdout, so a change to what a manifest sends to stderr — an error appearing or disappearing — was invisible. Later PRs change exactly that (treating an unsupported kind as skippable; the CLI's exit/stderr behavior), and those changes should land as reviewable golden-file diffs rather than silent shifts.
What's in the diff
Nine
verify()call sites, each gaining a.stdout+.stderrgolden (the old single-stream.approved.txtis renamed to.stdout.approved.txt, and a.stderr.approved.txtis added):TestKindkinds andTestMultiple— carry an empty.stderrgolden. That empty golden is the point: a regression that leaked to stderr on a passing manifest would now fail the suite.TestError.Service— carries a non-empty.stderrgolden holding the current text,error processing document: unsupported kind Service. This is the baseline that Skip non-workload documents instead of aborting the stream #54 later flips to empty when a Service becomes skippable.Behavior change
None — test-infrastructure only (test code +
go.mod/go.sum+.gitignore). Supersedes the draft #46 (which converted only the error test and wasn'tgofmt-clean).Stack
Base of both stacks: parsing (#49 → #54 → #56) and CLI (#50 → #55 → #57).
🤖 Generated with Claude Code
https://claude.ai/code/session_01Pc6NAURAqjU4LYJx93tgSC