ci(runtime): make live-inference smoke honestly non-blocking#1900
Merged
Conversation
The nightly Live Inference Smoke job hard-failed on main because the 'Run inference validation' step had no continue-on-error, so its exit 1 reddened the job even though the adjacent annotation claimed the result 'does not block releases'. The documented intent and the actual gate disagreed. The live-inference hero scenarios exercise 'apm run' against GitHub Models and depend on a Copilot-entitled credential plus a GitHub Models-compatible runtime, neither guaranteed in CI (codex >= v0.116 is incompatible with GitHub Models -- apm#605). They are a smoke signal, not a release gate. Mark the step continue-on-error with an id, and rewrite 'Annotate result' to read steps.inference.outcome (not job.status) so the annotation stays truthful while the job goes green-with-warning. The real gate (Run smoke tests / pytest) stays blocking. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Makes the ci-runtime.yml live-inference validation step honestly non-blocking (green job with an explicit warning/notice), aligning workflow behavior with the stated intent of treating live inference as a smoke signal rather than a release gate.
Changes:
- Marks the
Run inference validationstep ascontinue-on-error: trueand gives it anidfor outcome inspection. - Updates the annotation logic to report the inference step’s own outcome (instead of
job.status) so failures are still surfaced even when non-blocking.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/ci-runtime.yml | Makes live-inference validation non-blocking and updates result annotation to key off the inference step outcome. |
Copilot's findings
- Files reviewed: 1/1 changed files
- Comments generated: 1
Comment on lines
+96
to
100
| if [[ "${{ steps.inference.outcome }}" == "success" ]]; then | ||
| echo "::notice::Runtime inference smoke passed" | ||
| else | ||
| echo "::warning::⚠️ Runtime inference tests failed — this does not block releases" | ||
| echo "::warning::Runtime inference smoke failed -- non-blocking. Known causes: the CI token is not a Copilot-entitled credential for live inference, and codex >= v0.116 is incompatible with GitHub Models (apm#605). This does NOT gate releases." | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
The nightly
Live Inference Smokejob inci-runtime.ymlhard-failed onmainevery night even though its own annotation says the result "does not block releases." This makes the step honestly non-blocking so the daily false-red stops, while keeping the real gate intact.Problem (WHY)
ci-runtime.yml'sRun inference validationstep had nocontinue-on-error, so itsexit 1reddened the whole job. The very next step (Annotate result) emitted"... this does not block releases"— purely cosmetic. The documented intent and the actual gate disagreed, producing a persistent, misleading red onmain.The live-inference hero scenarios behind that step exercise
apm runagainst GitHub Models and require:>= v0.116is incompatible with GitHub Models (/responses404s), see apm#605.Neither is guaranteed in CI, so these tests are a smoke signal, not a release gate.
Approach (WHAT)
id: inference+continue-on-error: trueto theRun inference validationstep.Annotate resultto readsteps.inference.outcome(notjob.status) so the annotation stays truthful (it still reports the real failure) while the job goes green-with-warning.The real gate —
Run smoke tests(pytest,tests/integration/test_runtime_smoke.py) — stays blocking and is unaffected.Scope notes
apm packnow defaults--archiveto.zip, which the GH-AW Compatibility job'smicrosoft/apm-actionconsumer didn't detect) is fixed independently in fix: detect .zip bundles in archive mode (apm pack default changed to zip) apm-action#49.How to test
ci-runtime.ymlviaworkflow_dispatch. With a non-entitled CI token the inference step will fail, but the job concludes green and surfaces a::warning::annotation naming the known causes. The pytest smoke step still gates as before.