Skip to content

Address Holistic Review Workflow Feedback#131031

Draft
jeffhandley wants to merge 5 commits into
mainfrom
jeffhandley/improve-holistic-review
Draft

Address Holistic Review Workflow Feedback#131031
jeffhandley wants to merge 5 commits into
mainfrom
jeffhandley/improve-holistic-review

Conversation

@jeffhandley

@jeffhandley jeffhandley commented Jul 19, 2026

Copy link
Copy Markdown
Member

This draft collects and addresses feedback from the recently enabled Holistic Review automation. Maintainers are encouraged to push refinements directly to this draft while reviewing its results.

Holistic Review feedback addressed

  • Calibrates complexity-versus-maintainability guidance and restores concise, evidence-focused review output.
  • Centralizes per-PR orchestration state in a dedicated issue rather than adding workflow noise to reviewed PRs.
  • Orders state creation before worker dispatch, recovers callback outcomes, and suppresses no-op report issues.
  • Posts PR reviews only for new actionable feedback; routine LGTM, no-findings, and unchanged assessments complete without adding a review or being retried.

Other feedback addressed in merged PRs

Note

This PR description was generated by GitHub Copilot.

Copilot AI review requested due to automatic review settings July 19, 2026 08:18
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
15 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refines the Holistic Review workflow guidance to more explicitly evaluate when additional complexity in a PR is justified by its goals, and updates the compiled lockfile accordingly.

Changes:

  • Add explicit complexity-vs-maintainability tradeoff assessment guidance (including when to escalate to ⚠️ Needs Human Review) in the Holistic Review instructions.
  • Regenerate the holistic-review.lock.yml metadata to reflect the updated workflow source.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.github/workflows/holistic-review.md Adds guidance instructing reviewers to explicitly assess whether added complexity is proportionate and well-justified, and when to escalate for maintainer judgment.
.github/workflows/holistic-review.lock.yml Updates the generated lock metadata (hash) corresponding to the workflow text change.

Comment thread .github/workflows/holistic-review.md Outdated
Comment on lines +394 to +395
Explicitly assess whether the PR's added complexity is necessary and proportionate to its validated
goal. Do not treat size, low-level code, or specialized algorithms as concerns by themselves when
@jeffhandley jeffhandley added area-skills Agent Skills and removed area-Infrastructure labels Jul 19, 2026
@jeffhandley jeffhandley changed the title Calibrate holistic review complexity guidance Address Holistic Review Workflow Feedback Jul 20, 2026
@MihaZupan

Copy link
Copy Markdown
Member

Is there consideration for having it not post a comment at all if it's deemed not actionable by the LLM itself? Or would you still want it to always post, but perhaps with less verbosity?

E.g. in #130802 (review) there's a whole page of text for what boils down to Detailed Findings: None.
I also don't see much value in having it restate what the PR is doing, assuming that it matches what the description says, because the Copilot reviewer already posts a similar summary as a comment anyway, e.g. #130802 (review)

jeffhandley and others added 4 commits July 20, 2026 04:59
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 04:11
@jeffhandley
jeffhandley force-pushed the jeffhandley/improve-holistic-review branch from 82c5aba to 029bbef Compare July 21, 2026 04:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Comment on lines +58 to +62
# Reusable workflows retain the worker's event context. Bind state to that
# context rather than trusting the agent-controlled callback payload.
PR_NUMBER: ${{ github.event.inputs.pr_number }}
HEAD_SHA: ${{ github.event.inputs.pr_head_sha }}
OUTCOME: ${{ inputs.outcome }}
Comment on lines +4 to +7
state_issue_comment_prefix='<!-- holistic-review-orchestrator-state:pr-'
state_issue_comment_pattern='^<!-- holistic-review-orchestrator-state:pr-[1-9][0-9]* -->\n'

state_issue_number="$STATE_ISSUE_NUMBER"
Comment on lines +163 to +165
pr_title="$(jq -er '.title' <<< "$entry")"
pr_title_html="$(html_escape "$pr_title")"
base_ref="$(jq -er '.base_ref' <<< "$entry")"
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 21, 2026 04:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/workflows/holistic-review-completed.yml:62

  • The workflow_call inputs are defined as pr_number and head_sha, but this job reads github.event.inputs.pr_head_sha, which doesn’t exist for this reusable workflow. That makes HEAD_SHA empty and causes orchestrator-reconcile-state.sh to fail the head SHA check, preventing state from being reconciled/upserted.
          # Reusable workflows retain the worker's event context. Bind state to that
          # context rather than trusting the agent-controlled callback payload.
          PR_NUMBER: ${{ github.event.inputs.pr_number }}
          HEAD_SHA: ${{ github.event.inputs.pr_head_sha }}
          OUTCOME: ${{ inputs.outcome }}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-skills Agent Skills

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants