Remove experimental state from forecast command and simplify output - #48911
Merged
Conversation
Simplify the forecast command's console output by centralizing all explanations in the footer, removing user-facing Monte Carlo wording, and adding a note that all forecasts are estimates and may be inaccurate. Remove the experimental status entirely: drop the stderr experimental warning and [EXPERIMENTAL] help labels, and promote the specification from Experimental Draft (v0.1.0) to Draft (v1.0.0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c8996a3a-0647-4237-9072-96b70c0baff4
Contributor
There was a problem hiding this comment.
Pull request overview
Promotes gh aw forecast from experimental status and simplifies user-facing forecast messaging.
Changes:
- Removes experimental warnings and labels.
- Consolidates explanatory output in the footer.
- Updates documentation and the forecast specification.
Show a summary per file
| File | Description |
|---|---|
pkg/cli/README.md |
Removes experimental designation. |
pkg/cli/forecast.go |
Removes experimental stderr warning. |
pkg/cli/forecast_test.go |
Removes experimental-warning test. |
pkg/cli/forecast_render.go |
Revises footer and accuracy messaging. |
pkg/cli/forecast_command.go |
Updates command help. |
docs/src/content/docs/specs/forecast-specification.md |
Promotes and revises specification. |
docs/src/content/docs/setup/cli.md |
Updates CLI documentation. |
docs/src/content/docs/reference/glossary.md |
Updates forecast definition. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 8/8 changed files
- Comments generated: 2
- Review effort level: Medium
Comment on lines
+115
to
+116
| fmt.Fprintln(os.Stderr, console.FormatWarningMessage( | ||
| "All forecasts are estimates derived from historical samples and may be inaccurate.")) |
Comment on lines
+115
to
+116
| fmt.Fprintln(os.Stderr, console.FormatWarningMessage( | ||
| "All forecasts are estimates derived from historical samples and may be inaccurate.")) |
Collaborator
Author
…nstead of CopilotBYOKDefaultModel
Three tests in pkg/workflow were using constants.CopilotBYOKDefaultModel ("auto")
as the expected fallback default for Claude, but claude_engine.go uses
constants.SonnetDefaultModel ("claude-sonnet-5"). Fix:
- model_env_vars_test.go: update expectedDefault/expectedFallbackVal for the three
Claude test cases in TestModelEnvVarInjectionForAgentJob,
TestModelEnvVarInjectionForDetectionJob, and TestExpressionModelUsesEnvVar
- wasm_golden_test.go: add SonnetDefaultModel to normalizeOutput() so golden files
remain stable across Claude default model updates (mirrors the existing handling
for CopilotBYOKDefaultModel and CodexDefaultModel)
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Fixed in commit
|
Collaborator
Author
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Fixed in commit
|
Contributor
|
🎉 This pull request is included in a new release. Release: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Promotes the
gh aw forecastcommand out of experimental status. Removes the[EXPERIMENTAL]command-line label and the experimental-status stderr warning, replacing them with a permanent accuracy-disclosure note stating that forecasts are estimates derived from historical samples and may be inaccurate. The forecast specification document is updated from "Experimental Draft" (v0.1.0) to "Draft" (v1.0.0). Two unrelated maintenance changes are bundled in this PR (see below).Behavior changes
RunForecast(pkg/cli/forecast.go) no longer emits"forecast is an experimental command and may change without notice"to stderr.renderForecastTable(pkg/cli/forecast_render.go) now emits"All forecasts are estimates derived from historical samples and may be inaccurate."viaconsole.FormatWarningMessageon every non-JSON invocation (suppressed for--json, per R-IMPL-040). Footer message ordering was also adjusted, and "Monte Carlo" wording was replaced with generic "usage"/"statistical simulation" wording in console output.NewForecastCommand(pkg/cli/forecast_command.go) Short/Long help text no longer carries[EXPERIMENTAL]; Long help now includes the accuracy disclaimer inline and refers to "statistical simulation" instead of "Monte Carlo simulation".Files changed — forecast experimental-status removal
docs/src/content/docs/reference/glossary.mddocs/src/content/docs/setup/cli.md[EXPERIMENTAL]label from CLI reference; description now notes forecasts may be inaccurate.docs/src/content/docs/specs/forecast-specification.mdpkg/cli/README.mdgh aw forecastcommand table andForecastConfigtype table.pkg/cli/forecast.goRunForecast.pkg/cli/forecast_command.go[EXPERIMENTAL]labels from help text; replaced "Monte Carlo" wording; added inline accuracy disclaimer.pkg/cli/forecast_render.gopkg/cli/forecast_test.goTestRunForecast_R_IMPL_040_ExperimentalWarning, which asserted the old experimental warning appeared/was suppressed on stderr. No replacement test asserting the new accuracy-disclosure note was added in this diff.Files changed — unrelated bundled fixes
Commit
2db6e0cb9— Claude model fallback constant fix:pkg/workflow/model_env_vars_test.goconstants.SonnetDefaultModelinstead ofconstants.CopilotBYOKDefaultModelfor Claude engine fallback defaults.Commit
c9e263dc1— wasm golden test resilience:pkg/workflow/testdata/TestWasmGolden_CompileFixtures/playwright-cli-mode.goldenAWF_CONFIG_PAYLOAD.pkg/workflow/testdata/TestWasmGolden_CompileFixtures/smoke-copilot.goldenThis makes the golden snapshots independent of AWF config payload contents (domain allowlists, model lists, container image tag), so future AWF config changes won't require golden-file updates.
Notable test impact
TestRunForecast_R_IMPL_040_ExperimentalWarning) was deleted rather than updated.