Summary
samples: blocks under safe-outputs: are emitted into the lock file as a literal JSON string at compile time, with no opportunity to inject runtime values (${{ github.event.* }}, ${{ inputs.* }}, etc.). For workflow_dispatch-triggered tests that target a runtime-supplied issue or PR, this makes samples unusable: the schema requires an item_number / issue_number field, but the only sensible value comes from a workflow input.
Two suggestions, either of which would unblock these tests:
- Substitute expressions before validation. Treat sample property values as templated strings; only enforce the JSON-Schema validation against post-substitution values, or relax the regex validators for fields like
item_number to additionally accept ${{ ... }} placeholders that will be resolved at runtime.
- Move sample injection from compile time to runtime. Emit samples as a multi-line
GH_AW_SAMPLES env value containing live ${{ ... }} expressions, so GitHub Actions can substitute on the runner before apply_samples.cjs reads it.
Reproduction
Tested in githubnext/gh-aw-test against feat/use-samples-hidden-flag (e262eb3).
Example workflow test-copilot-siderepo-add-labels.md:
on:
workflow_dispatch:
inputs:
issue_number:
description: 'Issue number'
required: true
type: number
safe-outputs:
add-labels:
target-repo: 'githubnext/gh-aw-side-repo'
allowed-repos: ['githubnext/gh-aw-side-repo']
samples:
- item_number: ${{ github.event.inputs.issue_number }}
labels: ["copilot-safe-output-label-test"]
Compile error:
test-copilot-siderepo-add-labels.md:1:1: error: safe-outputs.add-labels.samples[0]:
jsonschema validation failed with 'inmem://safe-outputs-tools/add_labels.json#'
- at '/item_number': '${{ github.event.inputs.issue_number }}' does not match pattern '^(\d+|#?aw_[A-Za-z0-9_]{3,12})$'
If the sample omits item_number, compilation succeeds but the safe-outputs job fails at runtime with:
✗ Message 1 (add_labels) failed: No issue/PR number available
because the test trigger is workflow_dispatch (no implicit triggering issue) and the handler cannot infer the issue number from any other source.
Impact
Blocks --use-samples mode for any workflow_dispatch-triggered safe-output that operates on a runtime-supplied issue/PR/comment. In githubnext/gh-aw-test this includes the siderepo variants (add-labels, add-comment, update-issue), plus any test that needs dispatch-worker-style runtime parameters. Without runtime templating, these tests can never be made deterministic for CI.
Suggested resolution
Pick whichever of the two strategies above fits the architecture. Option 2 (runtime substitution) has the bonus that operator-supplied secrets and dynamic context (PR number, branch, etc.) become naturally usable inside samples, without further schema relaxation.
Related issues filed in this batch: #37526 (apply_samples JSON parse), #37527 (assign_milestone milestone_title rejected), #37530 (add_reviewer false-success for Copilot bot).
Summary
samples:blocks undersafe-outputs:are emitted into the lock file as a literal JSON string at compile time, with no opportunity to inject runtime values (${{ github.event.* }},${{ inputs.* }}, etc.). Forworkflow_dispatch-triggered tests that target a runtime-supplied issue or PR, this makes samples unusable: the schema requires anitem_number/issue_numberfield, but the only sensible value comes from a workflow input.Two suggestions, either of which would unblock these tests:
item_numberto additionally accept${{ ... }}placeholders that will be resolved at runtime.GH_AW_SAMPLESenv value containing live${{ ... }}expressions, so GitHub Actions can substitute on the runner beforeapply_samples.cjsreads it.Reproduction
Tested in
githubnext/gh-aw-testagainstfeat/use-samples-hidden-flag(e262eb3).Example workflow
test-copilot-siderepo-add-labels.md:Compile error:
If the sample omits
item_number, compilation succeeds but the safe-outputs job fails at runtime with:because the test trigger is
workflow_dispatch(no implicit triggering issue) and the handler cannot infer the issue number from any other source.Impact
Blocks
--use-samplesmode for anyworkflow_dispatch-triggered safe-output that operates on a runtime-supplied issue/PR/comment. Ingithubnext/gh-aw-testthis includes the siderepo variants (add-labels,add-comment,update-issue), plus any test that needsdispatch-worker-style runtime parameters. Without runtime templating, these tests can never be made deterministic for CI.Suggested resolution
Pick whichever of the two strategies above fits the architecture. Option 2 (runtime substitution) has the bonus that operator-supplied secrets and dynamic context (PR number, branch, etc.) become naturally usable inside samples, without further schema relaxation.
Related issues filed in this batch: #37526 (apply_samples JSON parse), #37527 (assign_milestone milestone_title rejected), #37530 (add_reviewer false-success for Copilot bot).