feat: add migration eval hooks for security, quality, and audit - #13
feat: add migration eval hooks for security, quality, and audit#13AlexDeMichieli wants to merge 3 commits into
Conversation
Add Copilot coding agent hooks that run at every lifecycle phase of a migration agent session: - sessionStart: verify-ci-sources.sh - fail fast if no CI files exist - preToolUse: security-guard.sh - block dangerous commands, enforce policies - postToolUse: audit-log.sh - JSONL audit trail of all tool calls - sessionEnd: eval-migration.sh - 11-check quality scorecard Tested against agbqwebqebqt/jenkins-migration-test PR #4: 9 passed, 0 failed, 2 warnings
There was a problem hiding this comment.
Pull request overview
This PR adds a set of Copilot migration-agent hook scripts and a hook configuration file to enforce migration guardrails (CI source presence, security restrictions, audit logging) and to generate a post-session migration quality scorecard.
Changes:
- Add a
sessionStarthook to detect whether the repo contains migratable CI/CD source configs. - Add a
preToolUsesecurity guard to restrict dangerous commands, restrict writable paths, and enforce workflow hardening. - Add
postToolUseaudit logging and asessionEndevaluation script that scores migration output against standards.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/hooks/verify-ci-sources.sh | Session start validator for presence of CI/CD source files. |
| .github/hooks/security-guard.sh | Pre-tool-call policy enforcement for commands, file edits, secrets, and workflows. |
| .github/hooks/migration-eval.json | Hook wiring for sessionStart/preToolUse/postToolUse/sessionEnd. |
| .github/hooks/eval-migration.sh | Post-session migration scorecard checks (workflows, archive, pinning, permissions, docs). |
| .github/hooks/audit-log.sh | JSONL audit trail for each tool invocation. |
Comments suppressed due to low confidence (1)
.github/hooks/eval-migration.sh:347
grep -rn '^\s*-\?\s*uses:.*@[a-f0-9]\{40\}'uses\swhich isn’t portable in ERE; this will likely undercount SHA-pinneduses:lines and make the “Version comments” check unreliable. Use[[:space:]](orgrep -Pif available) for whitespace matching.
while IFS= read -r line_num; do
total=$((total + 1))
done < <(grep -rn '^\s*-\?\s*uses:.*@[a-f0-9]\{40\}' "$WORKFLOWS_DIR"/ 2>/dev/null)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| else | ||
| unpinned+=("$(echo "$line" | sed 's/.*uses: *//')") | ||
| fi | ||
| done < <(grep -rh '^\s*-\?\s*uses:' "$WORKFLOWS_DIR"/ 2>/dev/null | grep -v '#') |
| FILE_PATH=$(echo "$INPUT" | jq -r '.toolArgs' | jq -r '.path // .filePath // empty' 2>/dev/null) | ||
| ;; | ||
| bash) | ||
| # Try to extract meaningful context from bash commands (first 100 chars) | ||
| FILE_PATH=$(echo "$INPUT" | jq -r '.toolArgs' | jq -r '.command // empty' 2>/dev/null | head -c 100) |
| set -euo pipefail | ||
|
|
||
| INPUT=$(cat) | ||
|
|
||
| TOOL_NAME=$(echo "$INPUT" | jq -r '.toolName') | ||
| TOOL_ARGS=$(echo "$INPUT" | jq -r '.toolArgs') | ||
|
|
| # CI source file patterns — at least one must exist for migration to proceed | ||
| CI_PATTERNS=( | ||
| "Jenkinsfile" | ||
| "*.jenkinsfile" | ||
| ".gitlab-ci.yml" | ||
| ".circleci/config.yml" | ||
| ".travis.yml" | ||
| "azure-pipelines.yml" | ||
| "azure-pipelines/*.yml" | ||
| ".drone.yml" |
| echo "Per migration guardrails: DO NOT create GitHub Actions" >&2 | ||
| echo "workflows without a source CI/CD configuration file." >&2 | ||
| echo "================================================================" >&2 | ||
| exit 1 |
| # Search everywhere except .github/ci-archive/ and .git/ | ||
| while IFS= read -r f; do | ||
| # Skip files inside the archive or .git | ||
| if [[ "$f" != *"$ARCHIVE_DIR"* && "$f" != *".git/"* ]]; then | ||
| found+=("$f") | ||
| fi | ||
| done < <(find . -path "./$ARCHIVE_DIR" -prune -o -path "./.git" -prune -o -name "$pattern" -print 2>/dev/null) |
| local output | ||
| output=$(actionlint "$WORKFLOWS_DIR"/*.yml "$WORKFLOWS_DIR"/*.yaml 2>&1 || true) | ||
| local errors | ||
| errors=$(echo "$output" | grep -c "error" 2>/dev/null || echo "0") |
| done < <(grep -roh 'secrets\.[A-Z_]*' "$WORKFLOWS_DIR"/ 2>/dev/null | sed 's/secrets\.//' | sort -u) | ||
|
|
||
| if [[ ${#undocumented[@]} -eq 0 ]]; then | ||
| local total | ||
| total=$(grep -roh 'secrets\.[A-Z_]*' "$WORKFLOWS_DIR"/ 2>/dev/null | sed 's/secrets\.//' | sort -u | wc -l | tr -d ' ') |
| # Block destructive system commands | ||
| if echo "$COMMAND" | grep -qE '(^|\s)(rm -rf /|sudo |mkfs |dd if=|chmod 777|chown )'; then | ||
| deny "Destructive system command blocked: $COMMAND" | ||
| fi | ||
|
|
||
| # Block external network calls to unknown hosts | ||
| # Allow: github.com, api.github.com (for MCP), localhost | ||
| if echo "$COMMAND" | grep -qE '(curl|wget|nc |ncat )\s' ; then | ||
| if ! echo "$COMMAND" | grep -qE '(github\.com|githubusercontent\.com|localhost|127\.0\.0\.1|actionlint)'; then | ||
| deny "External network call not permitted during migration. Only github.com and localhost are allowed." | ||
| fi |
|
|
||
| if [ -n "$CONTENT" ]; then | ||
| # Check for actions not pinned to SHAs (uses: org/action@v4 instead of @sha) | ||
| UNPINNED=$(echo "$CONTENT" | grep -E '^\s*-?\s*uses:' | grep -vE '@[a-f0-9]{40}' | grep -vE '^\s*#' || true) |
|
Warning This is an internal experiment to assess Copilot's ability to auto-approve PRs. Please 👍 this comment if the assessment below is correct and 👎 if not. Feedback in #f-ccr-auto-approve is appreciated! Copilot thinks this PR is not ready to approve — see review comments for details. |
|
@AlexDeMichieli, I like many of the ideas we discussed, and I see those reflected here. This PR touches on a few different concerns, and I think the best path forward is to break them into separate, focused features while being deliberate about the technical direction. cc: @ssulei7 |
|
Closing this PR. While the hooks work well and were validated end-to-end in a live Copilot coding agent session (see test results in description), the current approach isn't ideal from a customer perspective — it requires pushing hook files into each target repository, which would mean creating a PR per repo before migrations can begin. Key takeaways:
A better path forward may involve working with product on enterprise-level hook support, or exploring alternative approaches that don't require per-repo setup. |
Summary
The current
migration-guardrails.mdprovides excellent guidance for Copilot agents, but relies entirely on prompt-based enforcement. As migrations scale to hundreds of repos, prompt-only guardrails face limitations:This PR adds Copilot coding agent hooks that mechanically enforce guardrails at runtime. Hooks fire at lifecycle events (session start/end, before/after each tool call) and can block dangerous operations, log all actions, and generate quality scorecards automatically.
Hooks Added
verify-ci-sources.shsessionStartsecurity-guard.shpreToolUserm -rf, external curl, source code edits, unpinned actions, hardcoded secrets, custom actions creationaudit-log.shpostToolUse.github/ci-archive/migration-audit.jsonleval-migration.shsessionEnd.github/ci-archive/migration-scorecard.mdCloud Test Results
Tested on EMU enterprise (volcano-coffee) with repo
alexdemichieli-migrations/jenkins-migration-test:PR #5 — All 4 hooks confirmed working
verify-ci-sources.shsecurity-guard.shDENYfor dangerous ops (rm -rf, external curl, source edits, unpinned actions).create/edittool calls, agent fell back to bashaudit-log.shmigration-audit.jsonlcommitted with 20+ entrieseval-migration.shmigration-scorecard.mdcommitted with 9 pass, 2 warnScorecard Output
Location:
.github/ci-archive/migration-scorecard.md(automatically generated)Security Guard Blocks (from audit log)
{"tool":"create","result":"denied","file":".github/workflows/ci.yml"} {"tool":"edit","result":"denied","file":".github/workflows/ci.yml"}The agent correctly fell back to
bashcommands after direct create/edit was blocked.Guardrails Coverage
Enforced mechanically by hooks (14 guardrails):
rm -rfcommands.github/actions/)Must remain as prompt guidance (6 guardrails):
Files
Test Artifacts
.github/ci-archive/migration-scorecard.md.github/ci-archive/migration-audit.jsonl