Stabilize daily-doc-healer model selection#37505
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
Hey This is currently a draft WIP with no file changes committed yet (0 additions, 0 deletions). A few things to keep in mind as you land the implementation:
When you're ready to push the fixes, here's a prompt you can use to self-review before marking the PR ready:
|
| env: | ||
| COPILOT_GITHUB_TOKEN: "${{ secrets.COPILOT_GITHUB_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}" |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Updates the Daily Documentation Healer experiment configuration to use simplified Claude model identifiers for the model_size variants.
Changes:
- Renamed
model_sizeexperiment variants in the workflow frontmatter. - Updated the locked
GH_AW_EXPERIMENT_SPECJSON to match the new variant names.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/daily-doc-healer.md | Adjusts experiment variant identifiers for model selection. |
| .github/workflows/daily-doc-healer.lock.yml | Propagates the updated model variant identifiers into the locked experiment spec. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 2
| experiments: | ||
| model_size: | ||
| variants: [claude-sonnet-4-6, claude-haiku-4-5-20251001] | ||
| variants: [claude-sonnet-4.6, claude-haiku-4.5] |
| uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 | ||
| env: | ||
| GH_AW_EXPERIMENT_SPEC: '{"model_size":{"variants":["claude-sonnet-4-6","claude-haiku-4-5-20251001"],"description":"Tests whether Claude Haiku detects and corrects documentation gaps with equivalent quality at lower token cost versus Claude Sonnet.","hypothesis":"H0: no change in issue/PR creation rate or run success rate. H1: Claude Haiku reduces effective token usage \u003e=30% with equivalent run success rate (\u003e=0.90).","metric":"effective_tokens_total","secondary_metrics":["run_success_rate","run_duration_ms"],"guardrail_metrics":[{"name":"run_success_rate","threshold":"\u003e=0.90"},{"name":"empty_output_rate","threshold":"\u003c=0.10"}],"min_samples":20,"weight":[50,50],"start_date":"2026-06-04"}}' | ||
| GH_AW_EXPERIMENT_SPEC: '{"model_size":{"variants":["claude-sonnet-4.6","claude-haiku-4.5"],"description":"Tests whether Claude Haiku detects and corrects documentation gaps with equivalent quality at lower token cost versus Claude Sonnet.","hypothesis":"H0: no change in issue/PR creation rate or run success rate. H1: Claude Haiku reduces effective token usage \u003e=30% with equivalent run success rate (\u003e=0.90).","metric":"effective_tokens_total","secondary_metrics":["run_success_rate","run_duration_ms"],"guardrail_metrics":[{"name":"run_success_rate","threshold":"\u003e=0.90"},{"name":"empty_output_rate","threshold":"\u003c=0.10"}],"min_samples":20,"weight":[50,50],"start_date":"2026-06-04"}}' |
Daily Documentation Healer was failing due to config-level regression: unsupported Claude variant identifiers (triggering
effort-related 400s).This PR applies a targeted workflow-frontmatter fix so the daily job can execute with valid model inputs again.
experiments.model_size.variantsin.github/workflows/daily-doc-healer.md:claude-sonnet-4-6→claude-sonnet-4.6claude-haiku-4-5-20251001→claude-haiku-4.5.github/workflows/daily-doc-healer.lock.ymlto align compiled workflow output.