Replace skill-failure noop with manual-analysis fallback#134
Merged
Conversation
Reverses the strict abort rule from #133 in the openings, applies-to, and style sweeps. Real-run evidence (elastic/docs-content runs 25377518194 + 25378183549, both openings) showed the hardened "two failures = noop" rule firing prematurely: - 13:09:34 ✗ skill(docs-page-opening-optimizer) Skill not found - 13:09:37 ✗ skill(docs-page-opening-optimizer) Skill not found - 13:09:41 ● noop "skill unavailable — confirmed after exact-form attempts" - 13:14:36 ● create_issue "shard 20/28 — 20 pages" ← agent eventually produced an issue, but suppressed because noop already fired The agent's tool serialization keeps logging the skill call as `skill(docs-X)` — without the `skill:` prefix — which always returns "Skill not found" from Copilot CLI. Whether the agent's underlying invocation was actually reformatted, or whether it's a log-rendering quirk, doesn't matter: my hardening committed the agent to noop after two of these "failures", before it had a chance to do useful work. Replace with frontmatter's softer pattern: try the skill once; if "Skill not found" comes back, fall back to manual analysis using bash + the agent's own judgment, and note the skill failure once in the issue body's Notes section. Only noop if even manual analysis produces no high-confidence findings. Applied uniformly to: - gh-aw-docs-openings-sweep.md (the sweep that's been noop'ing) - gh-aw-docs-applies-to-sweep.md - gh-aw-docs-style-sweep.md Frontmatter, coherence, staleness, typos already had softer or deterministic-only fallback paths and don't need this change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reverses #133's strict abort rule for the openings, applies-to, and style sweeps. Real-run evidence showed it firing prematurely and suppressing legitimate output.
From elastic/docs-content openings run 25378183549:
The agent's tool serialization keeps logging the skill call as
skill(docs-X)(noskill:prefix), and that form always returns "Skill not found" from Copilot CLI. Whether the agent really sends that form or it's a log-rendering quirk doesn't matter — my hardening committed the agent to noop after two of these "failures", which then locked the safe-outputs system out of accepting the create_issue call the agent emitted 5 minutes later after manual analysis.Fix
Lift the frontmatter sweep's softer pattern across the three strict-abort sweeps:
Each sweep gets a category-specific manual-analysis recipe:
applies_toagainst allowed valuesSweeps unchanged
frontmatteralready had this softer pattern (if one fails, report only the other and note the skill failure in Notes).coherenceandstalenessuse different mechanisms.typosdoesn't import any skill.Test plan
v1.docs-openings-sweep. Confirm: even if the skill returns "Skill not found", the agent produces a fix-issue with manual-analysis findings rather than noop'ing.Xdid not produce output" Notes line appears in the resulting issue body.🤖 Generated with Claude Code