Skip to content

samples: blocks cannot reference runtime expressions (github.event.inputs.*) because schema validation runs before substitution #37532

Description

@dsyme

Summary

samples: blocks under safe-outputs: are emitted into the lock file as a literal JSON string at compile time, with no opportunity to inject runtime values (${{ github.event.* }}, ${{ inputs.* }}, etc.). For workflow_dispatch-triggered tests that target a runtime-supplied issue or PR, this makes samples unusable: the schema requires an item_number / issue_number field, but the only sensible value comes from a workflow input.

Two suggestions, either of which would unblock these tests:

  1. Substitute expressions before validation. Treat sample property values as templated strings; only enforce the JSON-Schema validation against post-substitution values, or relax the regex validators for fields like item_number to additionally accept ${{ ... }} placeholders that will be resolved at runtime.
  2. Move sample injection from compile time to runtime. Emit samples as a multi-line GH_AW_SAMPLES env value containing live ${{ ... }} expressions, so GitHub Actions can substitute on the runner before apply_samples.cjs reads it.

Reproduction

Tested in githubnext/gh-aw-test against feat/use-samples-hidden-flag (e262eb3).

Example workflow test-copilot-siderepo-add-labels.md:

on:
  workflow_dispatch:
    inputs:
      issue_number:
        description: 'Issue number'
        required: true
        type: number

safe-outputs:
  add-labels:
    target-repo: 'githubnext/gh-aw-side-repo'
    allowed-repos: ['githubnext/gh-aw-side-repo']
    samples:
      - item_number: ${{ github.event.inputs.issue_number }}
        labels: ["copilot-safe-output-label-test"]

Compile error:

test-copilot-siderepo-add-labels.md:1:1: error: safe-outputs.add-labels.samples[0]:
  jsonschema validation failed with 'inmem://safe-outputs-tools/add_labels.json#'
- at '/item_number': '${{ github.event.inputs.issue_number }}' does not match pattern '^(\d+|#?aw_[A-Za-z0-9_]{3,12})$'

If the sample omits item_number, compilation succeeds but the safe-outputs job fails at runtime with:

✗ Message 1 (add_labels) failed: No issue/PR number available

because the test trigger is workflow_dispatch (no implicit triggering issue) and the handler cannot infer the issue number from any other source.

Impact

Blocks --use-samples mode for any workflow_dispatch-triggered safe-output that operates on a runtime-supplied issue/PR/comment. In githubnext/gh-aw-test this includes the siderepo variants (add-labels, add-comment, update-issue), plus any test that needs dispatch-worker-style runtime parameters. Without runtime templating, these tests can never be made deterministic for CI.

Suggested resolution

Pick whichever of the two strategies above fits the architecture. Option 2 (runtime substitution) has the bonus that operator-supplied secrets and dynamic context (PR number, branch, etc.) become naturally usable inside samples, without further schema relaxation.

Related issues filed in this batch: #37526 (apply_samples JSON parse), #37527 (assign_milestone milestone_title rejected), #37530 (add_reviewer false-success for Copilot bot).

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions