Skip to content

Stabilize daily-doc-healer model selection#37505

Merged
pelikhan merged 3 commits into
mainfrom
copilot/fix-daily-documentation-healer-model-inventory-che
Jun 7, 2026
Merged

Stabilize daily-doc-healer model selection#37505
pelikhan merged 3 commits into
mainfrom
copilot/fix-daily-documentation-healer-model-inventory-che

Conversation

Copilot AI commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Daily Documentation Healer was failing due to config-level regression: unsupported Claude variant identifiers (triggering effort-related 400s).
This PR applies a targeted workflow-frontmatter fix so the daily job can execute with valid model inputs again.

  • Documentation Healer: normalize experiment model variants
    • Updated experiments.model_size.variants in .github/workflows/daily-doc-healer.md:
      • claude-sonnet-4-6claude-sonnet-4.6
      • claude-haiku-4-5-20251001claude-haiku-4.5
    • Regenerated .github/workflows/daily-doc-healer.lock.yml to align compiled workflow output.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix daily documentation healer and model inventory checker failures Stabilize daily-doc-healer model selection and harden daily-model-inventory Copilot auth fallback Jun 7, 2026
Copilot AI requested a review from pelikhan June 7, 2026 06:22
@github-actions

github-actions Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Hey @app/copilot-swe-agent 👋 — thanks for picking up these daily workflow failures! Fixing the Documentation Healer and Model Inventory Checker is exactly the kind of maintenance work that keeps the project healthy.

This is currently a draft WIP with no file changes committed yet (0 additions, 0 deletions). A few things to keep in mind as you land the implementation:

  • Add tests — once the workflow config fixes are applied, include any relevant test or validation coverage (e.g. schema validation, dry-run checks, or unit tests for any Go logic touched).
  • Fill out the description — the checklist body is a good start, but once the work is done, replace the TODO items with a summary of what was broken, why it was broken, and what was changed to fix it.
  • Keep the diff surgical — the title says "minimal config fixes"; please make sure the final diff stays tightly scoped to the two failing workflows and avoids unrelated changes.

When you're ready to push the fixes, here's a prompt you can use to self-review before marking the PR ready:

Review the changes in this branch and verify:
1. Only the Documentation Healer and Model Inventory Checker workflow definitions have been modified.
2. The root-cause errors identified in the failing runs are directly addressed by the changes.
3. Any Go or script code touched has corresponding test coverage added or updated.
4. The PR body is updated to describe: what was broken, why it was broken, and what was changed.
5. No unrelated files or formatting changes are included in the diff.
If any of these are missing, fix them before marking the PR ready for review.

Generated by ✅ Contribution Check · 366 AIC · ⌖ 13 AIC · ⊞ 24.3K ·

Comment on lines +19 to +20
env:
COPILOT_GITHUB_TOKEN: "${{ secrets.COPILOT_GITHUB_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot remove this line. The error is something else.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2d0df94 — I removed that Copilot token fallback line and reverted the corresponding generated lockfile entries.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan pelikhan marked this pull request as ready for review June 7, 2026 06:35
Copilot AI review requested due to automatic review settings June 7, 2026 06:35
@pelikhan pelikhan merged commit b9741ea into main Jun 7, 2026
@pelikhan pelikhan deleted the copilot/fix-daily-documentation-healer-model-inventory-che branch June 7, 2026 06:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates the Daily Documentation Healer experiment configuration to use simplified Claude model identifiers for the model_size variants.

Changes:

  • Renamed model_size experiment variants in the workflow frontmatter.
  • Updated the locked GH_AW_EXPERIMENT_SPEC JSON to match the new variant names.
Show a summary per file
File Description
.github/workflows/daily-doc-healer.md Adjusts experiment variant identifiers for model selection.
.github/workflows/daily-doc-healer.lock.yml Propagates the updated model variant identifiers into the locked experiment spec.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 2

experiments:
model_size:
variants: [claude-sonnet-4-6, claude-haiku-4-5-20251001]
variants: [claude-sonnet-4.6, claude-haiku-4.5]
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
GH_AW_EXPERIMENT_SPEC: '{"model_size":{"variants":["claude-sonnet-4-6","claude-haiku-4-5-20251001"],"description":"Tests whether Claude Haiku detects and corrects documentation gaps with equivalent quality at lower token cost versus Claude Sonnet.","hypothesis":"H0: no change in issue/PR creation rate or run success rate. H1: Claude Haiku reduces effective token usage \u003e=30% with equivalent run success rate (\u003e=0.90).","metric":"effective_tokens_total","secondary_metrics":["run_success_rate","run_duration_ms"],"guardrail_metrics":[{"name":"run_success_rate","threshold":"\u003e=0.90"},{"name":"empty_output_rate","threshold":"\u003c=0.10"}],"min_samples":20,"weight":[50,50],"start_date":"2026-06-04"}}'
GH_AW_EXPERIMENT_SPEC: '{"model_size":{"variants":["claude-sonnet-4.6","claude-haiku-4.5"],"description":"Tests whether Claude Haiku detects and corrects documentation gaps with equivalent quality at lower token cost versus Claude Sonnet.","hypothesis":"H0: no change in issue/PR creation rate or run success rate. H1: Claude Haiku reduces effective token usage \u003e=30% with equivalent run success rate (\u003e=0.90).","metric":"effective_tokens_total","secondary_metrics":["run_success_rate","run_duration_ms"],"guardrail_metrics":[{"name":"run_success_rate","threshold":"\u003e=0.90"},{"name":"empty_output_rate","threshold":"\u003c=0.10"}],"min_samples":20,"weight":[50,50],"start_date":"2026-06-04"}}'
Copilot AI restored the copilot/fix-daily-documentation-healer-model-inventory-che branch June 7, 2026 06:37
Copilot AI changed the title Stabilize daily-doc-healer model selection and harden daily-model-inventory Copilot auth fallback Stabilize daily-doc-healer model selection Jun 7, 2026
Copilot AI requested a review from pelikhan June 7, 2026 06:37
@github-actions github-actions Bot deleted the copilot/fix-daily-documentation-healer-model-inventory-che branch June 15, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P1] Daily Documentation Healer + Model Inventory Checker: 4th consecutive failure, #37271 fix insufficient

3 participants