add_reviewer reports success for Copilot bot even when GraphQL requestReviews produces no observable change

[dsyme]

Summary

add_reviewer handler reports Successfully added copilot as reviewer to PR #N even when the GraphQL requestReviews(botIds:[…]) mutation accepts the request but produces no observable change on the PR (the resulting requested_reviewers / GraphQL reviewRequests collections remain empty).

The handler should either:

1. Verify after the mutation that the bot actually appears in reviewRequests and warn/fail if not, or
2. Surface the GraphQL response and detect a no-op outcome, or
3. Confirm at startup that the Copilot reviewer bot is enabled for the target repository and fail-fast with a clear error if not.

Currently the safe-output reports success and the manifest links the PR, but downstream tools (and humans) that re-check the PR see no reviewer added.

Reproduction

Tested in githubnext/gh-aw-test against the included test-copilot-add-reviewer.md workflow (samples mode, hidden --use-samples flag).

• Sample: reviewers: ["copilot"]
• allowed-reviewers: [copilot]
• gh-aw ref: feat/use-samples-hidden-flag (e262eb3)
• Affected handler: actions/setup/js/add_reviewer.cjs around line 239

Observed safe_outputs job logs

Successfully added copilot as reviewer to PR #2236
📝 Manifest: logged add_reviewer → https://github.com/githubnext/gh-aw-test/pull/2236

Job succeeds; the run is marked successful.

Post-run PR state on PR #2236

$ gh api repos/githubnext/gh-aw-test/pulls/2236/requested_reviewers
{"users":[],"teams":[]}

$ gh api graphql -f query='query { repository(owner:"githubnext",name:"gh-aw-test") { pullRequest(number:2236) { reviewRequests(first:20) { nodes { requestedReviewer { __typename ... on User { login } ... on Bot { login } } } } } } }'
{"data":{"repository":{"pullRequest":{"reviewRequests":{"nodes":[]}}}}}

The bot is not in the requested reviewers list despite the success log.

Root cause hypothesis

In actions/setup/js/add_reviewer.cjs:

await githubClient.graphql(requestReviewsMutation, {
  pullRequestId,
  botIds: [await resolveCopilotBotNodeId()],
});
const response = await githubClient.rest.pulls.get({ … });
latestPullRequest = response?.data || latestPullRequest;
core.info(`Successfully added copilot as reviewer to PR #${prNumber}`);

The mutation result is awaited but not inspected. If the requestReviews mutation returns a successful response without actually attaching the bot (e.g. because the Copilot code-review bot isn't enabled for the repo, or the botIds: union: true semantics no-op for this resolved bot ID), there is no log line indicating that nothing happened.

Suggested fix

After the mutation, re-query reviewRequests on the PR and verify the bot ID appears. If it doesn't:

• emit core.warning("requestReviews returned success but Copilot reviewer is not present on PR #N — is Copilot code review enabled for this repo?"),
• and consider returning success: false from the handler so the safe-outputs manifest doesn't claim a successful add.

This would make this category of silent failure visible in CI without requiring callers to re-poll the PR themselves.

Impact

Blocks meaningful e2e verification of add-reviewer for the Copilot bot in githubnext/gh-aw-test (test test-copilot-add-reviewer polls reviewRequests for 240s and times out). Same handler path is used by the other engines (claude, codex) for their copilot-bot reviewer cases.

Related issues filed in this batch: #37526 (apply_samples JSON parse), #37527 (assign_milestone milestone_title rejected).

[dsyme]

Hi — verifying this in githubnext/gh-aw-test against gh-aw main (v0.78.3-50-g57027f2fa3) shows the bug is still present for the safe-outputs.add-reviewer + allowed-reviewers: [copilot] path.

Reproduction: https://github.com/githubnext/gh-aw-test/actions/runs/27091539627

The handler log reports apparent success:

2026-06-07T11:44:57.8831797Z Adding reviewers to PR #2270: reviewers=["copilot"], team_reviewers=[]
2026-06-07T11:44:59.0300037Z Successfully added copilot as reviewer to PR #2270

But the PR has no observable change:

$ gh pr view 2270 --repo githubnext/gh-aw-test --json reviewRequests,latestReviews
{"requests": [], "reviews": []}

The GraphQL requestReviews(input: { botIds: [...] union: true }) mutation in add_reviewer.cjs:219-232 succeeds (no exception thrown) but does not actually add Copilot as a reviewer. The downstream pulls.get confirms the unchanged state but the handler still logs Successfully added copilot as reviewer because the success path doesn't validate the post-mutation reviewer list.

Could you re-open or link to the fix PR if there is one? Suggested instrumentation: after the GraphQL mutation, re-query reviewRequests / latestReviews and warn if Copilot is absent (instead of logging unconditional success).

[dsyme]

Correction: the reproduction above was using copilot as the reviewer, which requires special CCR tokens and is not a valid e2e test scenario. I've updated the test in githubnext/gh-aw-test to use a real GitHub user (pelikhan) instead. I'll re-verify against that path and reopen this issue if the bug genuinely persists with a normal user reviewer.