Skip to content

make the E2E test more explicit so agents decide not to do just 2 instead of 3 commits - #642

Merged
Soph merged 2 commits into
mainfrom
soph/copilot-do-what-I-told-ya
Mar 6, 2026
Merged

make the E2E test more explicit so agents decide not to do just 2 instead of 3 commits#642
Soph merged 2 commits into
mainfrom
soph/copilot-do-what-I-told-ya

Conversation

@Soph

@Soph Soph commented Mar 6, 2026

Copy link
Copy Markdown
Collaborator

Turns out Copilot CLI (with gpt-4.1) sometimes stuck to making 3 commits and sometimes decided 2 are fine too. This should make our ask more explicit.

…tead of 3 commits

Entire-Checkpoint: 3d240d908ecb
@Soph
Soph requested a review from a team as a code owner March 6, 2026 14:02
Copilot AI review requested due to automatic review settings March 6, 2026 14:02
@cursor

cursor Bot commented Mar 6, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Low risk: only changes an E2E test prompt string, affecting test determinism rather than production behavior.

Overview
Updates TestRapidSequentialCommits to use a stricter, step-by-step prompt that explicitly instructs running git add and git commit (with specific messages) after creating each of the three markdown files.

This is intended to reduce flaky agent behavior where fewer than three commits are produced, while keeping the test assertions the same (3 new commits, each with a checkpoint trailer).

Written by Cursor Bugbot for commit 8eec694. Configure here.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates an E2E edge-case prompt to be more deterministic so agents reliably create three separate commits (one per file) rather than occasionally collapsing the work into fewer commits.

Changes:

  • Replaced a high-level instruction (“commit each file separately”) with an explicit ordered 3-step sequence.
  • Included concrete git add … && git commit -m … commands for each file to reduce agent ambiguity.

pjbgf
pjbgf previously approved these changes Mar 6, 2026
Entire-Checkpoint: 521936526b23
@Soph
Soph merged commit 1d44b11 into main Mar 6, 2026
11 checks passed
@Soph
Soph deleted the soph/copilot-do-what-I-told-ya branch March 6, 2026 16:47
toothbrush added a commit that referenced this pull request Jun 24, 2026
org/project/repo delete echoed the user-supplied arg, so a name-based
delete confirmed a name while the operation used the resolved ULID —
ambiguous where names are reused. Add resolvedRefLabel to render
"name (ULID)" for names and the bare ULID otherwise.

Addresses trail #642 finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: c14749e12a9d
toothbrush added a commit that referenced this pull request Jun 24, 2026
org/project/repo delete called the destructive API immediately after
resolving the ref — a name typo that resolves to a real resource (or a
wrong-but-valid ULID) deleted silently. Gate each behind a confirmation
prompt (the existing trail-delete pattern) with a --force/-f bypass and
--yes/-y alias; non-interactive runs refuse without --force rather than
deleting unprompted. Shared body extracted to runControlPlaneDelete.

Addresses trail #642 finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 5201fa5dc67c
toothbrush added a commit that referenced this pull request Jun 24, 2026
org/project/repo delete echoed the user-supplied arg, so a name-based
delete confirmed a name while the operation used the resolved ULID —
ambiguous where names are reused. Add resolvedRefLabel to render
"name (ULID)" for names and the bare ULID otherwise.

Addresses trail #642 finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: c14749e12a9d
toothbrush added a commit that referenced this pull request Jun 24, 2026
org/project/repo delete called the destructive API immediately after
resolving the ref — a name typo that resolves to a real resource (or a
wrong-but-valid ULID) deleted silently. Gate each behind a confirmation
prompt (the existing trail-delete pattern) with a --force/-f bypass and
--yes/-y alias; non-interactive runs refuse without --force rather than
deleting unprompted. Shared body extracted to runControlPlaneDelete.

Addresses trail #642 finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 5201fa5dc67c
toothbrush added a commit that referenced this pull request Jun 24, 2026
- org/project/repo delete now treat a 404 from the delete call as the
  desired end state (already gone), printing "<noun> <label> not found;
  nothing to delete" instead of surfacing a raw 404.
- Add command-level tests driving each delete command through cobra
  against a fake control plane: --force issues DELETE on the right path,
  an already-gone resource is idempotent, and a non-interactive run
  without --force refuses rather than deleting unprompted.

Addresses trail #642 findings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 416181183067
toothbrush added a commit that referenced this pull request Jun 24, 2026
org/project/repo delete echoed the user-supplied arg, so a name-based
delete confirmed a name while the operation used the resolved ULID —
ambiguous where names are reused. Add resolvedRefLabel to render
"name (ULID)" for names and the bare ULID otherwise.

Addresses trail #642 finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: c14749e12a9d
toothbrush added a commit that referenced this pull request Jun 24, 2026
org/project/repo delete called the destructive API immediately after
resolving the ref — a name typo that resolves to a real resource (or a
wrong-but-valid ULID) deleted silently. Gate each behind a confirmation
prompt (the existing trail-delete pattern) with a --force/-f bypass and
--yes/-y alias; non-interactive runs refuse without --force rather than
deleting unprompted. Shared body extracted to runControlPlaneDelete.

Addresses trail #642 finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 5201fa5dc67c
toothbrush added a commit that referenced this pull request Jun 24, 2026
- org/project/repo delete now treat a 404 from the delete call as the
  desired end state (already gone), printing "<noun> <label> not found;
  nothing to delete" instead of surfacing a raw 404.
- Add command-level tests driving each delete command through cobra
  against a fake control plane: --force issues DELETE on the right path,
  an already-gone resource is idempotent, and a non-interactive run
  without --force refuses rather than deleting unprompted.

Addresses trail #642 findings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 416181183067
toothbrush added a commit that referenced this pull request Jun 25, 2026
org/project/repo delete echoed the user-supplied arg, so a name-based
delete confirmed a name while the operation used the resolved ULID —
ambiguous where names are reused. Add resolvedRefLabel to render
"name (ULID)" for names and the bare ULID otherwise.

Addresses trail #642 finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: c14749e12a9d
toothbrush added a commit that referenced this pull request Jun 25, 2026
org/project/repo delete called the destructive API immediately after
resolving the ref — a name typo that resolves to a real resource (or a
wrong-but-valid ULID) deleted silently. Gate each behind a confirmation
prompt (the existing trail-delete pattern) with a --force/-f bypass and
--yes/-y alias; non-interactive runs refuse without --force rather than
deleting unprompted. Shared body extracted to runControlPlaneDelete.

Addresses trail #642 finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 5201fa5dc67c
toothbrush added a commit that referenced this pull request Jun 25, 2026
- org/project/repo delete now treat a 404 from the delete call as the
  desired end state (already gone), printing "<noun> <label> not found;
  nothing to delete" instead of surfacing a raw 404.
- Add command-level tests driving each delete command through cobra
  against a fake control plane: --force issues DELETE on the right path,
  an already-gone resource is idempotent, and a non-interactive run
  without --force refuses rather than deleting unprompted.

Addresses trail #642 findings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 416181183067
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants