Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion pkg/cli/forecast_compliance_fixtures_formal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ package cli
// - FC-P3: run_summary_zero_et.json has total_effective_tokens == 0 (T-FC-022)
// - FC-P4: run_summary_high_et.json has total_effective_tokens >= 1,000,000 (T-ET-006)
// - FC-P6: run_summary_failed.json has conclusion == "failure" (T-FC-035)
// - FC-P7: run_summary_cancelled.json has conclusion == "cancelled" (T-FC-036)
// - FC-P8: RunSummary JSON round-trip serialization is lossless
// - FC-P9: run_started_at <= updated_at in every fixture
// - FC-P10: all four fixtures have required Monte Carlo input fields
// - FC-P10: all five fixtures have required Monte Carlo input fields

import (
"encoding/json"
Expand Down Expand Up @@ -483,6 +484,25 @@ func TestFormal_FC_P6_FailedRunFixture(t *testing.T) {
"so it is not counted as a Bernoulli success")
}

// TestFormal_FC_P7_CancelledRunFixture verifies that run_summary_cancelled.json
// has conclusion == "cancelled", confirming it is included in the Bernoulli sample
// (status == "completed" and conclusion != "skipped") but is not counted as a success.
//
// Formal predicate (FC-P7): fixture["run"]["conclusion"] = "cancelled"
// Specification reference: T-FC-036; specs/forecast-compliance-fixtures/README.md
func TestFormal_FC_P7_CancelledRunFixture(t *testing.T) {
fixture := loadFixture(t, "run_summary_cancelled.json")

run, ok := fixture["run"].(map[string]any)
require.True(t, ok, "FC-P7: 'run' must be a JSON object")

conclusion, ok := run["conclusion"].(string)
require.True(t, ok, "FC-P7: run.conclusion must be a string")
assert.Equal(t, "cancelled", conclusion,
"FC-P7 (T-FC-036): run_summary_cancelled.json must have conclusion == \"cancelled\" "+
"so it is included in the sample but not counted as a Bernoulli success")
}

// TestFormal_FC_P8_RunSummaryRoundTrip verifies that marshalling a RunSummary to
// JSON and unmarshalling it back produces an equal value (cache-hit determinism).
//
Expand Down Expand Up @@ -539,6 +559,7 @@ func TestFormal_FC_P9_TimestampOrdering(t *testing.T) {
"run_summary_zero_et.json",
"run_summary_failed.json",
"run_summary_high_et.json",
"run_summary_cancelled.json",
}

for _, name := range fixtures {
Expand Down Expand Up @@ -584,6 +605,7 @@ func TestFormal_FC_P10_MonteCarloInputCompleteness(t *testing.T) {
{name: "run_summary_zero_et.json", wantConclusion: "success", aicMustBeGT0: false},
{name: "run_summary_failed.json", wantConclusion: "failure", aicMustBeGT0: false},
{name: "run_summary_high_et.json", wantConclusion: "success", aicMustBeGT0: true},
{name: "run_summary_cancelled.json", wantConclusion: "cancelled", aicMustBeGT0: false},
}

for _, tc := range cases {
Expand Down
32 changes: 16 additions & 16 deletions specs/awf-config-sources-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,21 +104,21 @@ When updating AWF config generation, schema sync, or validation in gh-aw, agents

The following fields previously existed in schema but were missed in spec CLI mapping checks:

| Config path | CLI flag |
|---|---|
| `apiProxy.anthropicAutoCache` | `--anthropic-auto-cache` |
| `apiProxy.anthropicCacheTailTtl` | `--anthropic-cache-tail-ttl` |
| `apiProxy.models` | config-only (model alias rewriting) |
| `apiProxy.modelMultipliers` | config-only (effective-token accounting) |
| `apiProxy.modelFallback` | config-only (model fallback policy; set `sandbox.agent.model-fallback: false` to prevent deployment-name rewriting for BYOK Azure) |
| `apiProxy.maxRuns` | config-only (LLM invocation hard cap) |
| `apiProxy.auth.*` | config-only (maps to `AWF_AUTH_*` env vars) |
| `apiProxy.targets.openai.authHeader` | `--openai-api-auth-header` (frontmatter: `sandbox.agent.targets.openai.authHeader`) |
| `apiProxy.targets.anthropic.authHeader` | `--anthropic-api-auth-header` (frontmatter: `sandbox.agent.targets.anthropic.authHeader`) |
| `apiProxy.targets.copilot.extraHeaders` | config-only (frontmatter: `sandbox.agent.targets.copilot.extraHeaders`; maps to `AWF_BYOK_EXTRA_HEADERS`) |
| `apiProxy.targets.copilot.extraBodyFields` | config-only (frontmatter: `sandbox.agent.targets.copilot.extraBodyFields`; maps to `AWF_BYOK_EXTRA_BODY_FIELDS`) |
| `apiProxy.targets.copilot.sessionId` | config-only (frontmatter: `sandbox.agent.targets.copilot.sessionId`; maps to `AWF_PROVIDER_SESSION_ID`) |
| `container.dockerHostPathPrefix` | `--docker-host-path-prefix` |
| Config path | CLI flag | Test reference |
|---|---|---|
| `apiProxy.anthropicAutoCache` | `--anthropic-auto-cache` | `pkg/workflow/awf_config_test.go` |
| `apiProxy.anthropicCacheTailTtl` | `--anthropic-cache-tail-ttl` | `pkg/workflow/awf_config_test.go` |
| `apiProxy.models` | config-only (model alias rewriting) | `pkg/workflow/awf_config_test.go` |
| `apiProxy.modelMultipliers` | config-only (effective-token accounting) | `pkg/workflow/awf_config_test.go` |
Comment on lines +109 to +112
| `apiProxy.modelFallback` | config-only (model fallback policy; set `sandbox.agent.model-fallback: false` to prevent deployment-name rewriting for BYOK Azure) | `pkg/workflow/awf_config_test.go` (`TestAWFConfig_ModelFallback*`) |
| `apiProxy.maxRuns` | config-only (LLM invocation hard cap) | `pkg/workflow/awf_config_test.go` |
| `apiProxy.auth.*` | config-only (maps to `AWF_AUTH_*` env vars) | `pkg/workflow/awf_config_test.go` |
| `apiProxy.targets.openai.authHeader` | `--openai-api-auth-header` (frontmatter: `sandbox.agent.targets.openai.authHeader`) | `pkg/workflow/awf_config_test.go` |
| `apiProxy.targets.anthropic.authHeader` | `--anthropic-api-auth-header` (frontmatter: `sandbox.agent.targets.anthropic.authHeader`) | `pkg/workflow/awf_config_test.go` |
| `apiProxy.targets.copilot.extraHeaders` | config-only (frontmatter: `sandbox.agent.targets.copilot.extraHeaders`; maps to `AWF_BYOK_EXTRA_HEADERS`) | `pkg/workflow/copilot_byok_extra_fields_compilation_test.go` (`TestCopilotBYOKExtraFieldsInCompiledWorkflow`) |
| `apiProxy.targets.copilot.extraBodyFields` | config-only (frontmatter: `sandbox.agent.targets.copilot.extraBodyFields`; maps to `AWF_BYOK_EXTRA_BODY_FIELDS`) | `pkg/workflow/copilot_byok_extra_fields_compilation_test.go` (`TestCopilotBYOKExtraFieldsInCompiledWorkflow`) |
| `apiProxy.targets.copilot.sessionId` | config-only (frontmatter: `sandbox.agent.targets.copilot.sessionId`; maps to `AWF_PROVIDER_SESSION_ID`) | `pkg/workflow/copilot_byok_extra_fields_compilation_test.go` (`TestCopilotBYOKExtraFieldsInCompiledWorkflow`) |
| `container.dockerHostPathPrefix` | `--docker-host-path-prefix` | `pkg/workflow/awf_config_test.go` |

Agents SHOULD treat this class of mismatch as a regression signal and open a corrective PR when detected.

Expand Down Expand Up @@ -275,7 +275,7 @@ The drift detection procedure (Section 7.2, Step 5) **MUST** produce a list of z

When canonical sources in `github/gh-aw-firewall` are unavailable (GitHub outage, auth failure, transient fetch errors), agents and automation MUST apply the following safeguards:

1. The workflow **MUST** attempt to use the last-known validated local snapshot (for example cached schema/spec artifacts from the previous successful run) to keep checks deterministic.
1. The workflow **MUST** attempt to use the last-known validated local snapshot (for example cached schema/spec artifacts from the previous successful run) to keep checks deterministic. The snapshot **MUST** be stored at a stable, well-known path: `~/.cache/gh-aw/schema-consistency/last-known-snapshot/` on self-hosted runners or `/tmp/gh-aw/agent/schema-consistency/last-known-snapshot/` when the runner is ephemeral. Snapshots older than **7 days** (168 hours from the `detected_at` timestamp of the last successful refresh) **MUST** be treated as expired and **MUST NOT** be used to suppress drift warnings; when a snapshot is expired, the run **MUST** be marked degraded even if the snapshot files are physically present. Implementations **SHOULD** delete snapshots older than 14 days to prevent unbounded disk use.
2. The workflow **SHOULD** emit a warning that canonical source retrieval failed, including the failing source path(s) and timestamp.
3. The workflow **MUST** skip destructive validation actions (for example failing required checks, auto-opening corrective PRs, or auto-creating drift issues from stale snapshots) when canonical data cannot be refreshed, and mark the run as degraded instead of silently passing.
4. The workflow **SHOULD** open or update a tracking issue when canonical source unavailability persists for more than one consecutive scheduled run.
12 changes: 10 additions & 2 deletions specs/compiler-threat-detection-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar:

# GitHub Actions Compiler Threat Detection Specification

**Version**: 1.0.18
**Version**: 1.0.19
**Status**: Candidate Recommendation
**Latest Version**: https://github.com/github/gh-aw/blob/main/specs/compiler-threat-detection-spec.md
**Editors**: GitHub Next (GitHub, Inc.)
Expand Down Expand Up @@ -78,6 +78,7 @@ This section anchors the specification version to the minimum gh-aw binary versi

| Spec version | Minimum gh-aw binary version | Lock-file compatibility notes |
|--------------|------------------------------|-------------------------------|
| `1.0.19` | `v0.72.1` (or newer) | Threat-detection behavior must remain compatible with current `.lock.yml` compilation semantics, including manifest drift enforcement (`gh-aw-manifest` checks for CTR-016), update-check validation (`check-for-updates` handling for CTR-018), cache-memory integrity enforcement (`update_cache_memory` gating for CTR-019), conditional import rejection (`imports.if` rejection for CTR-020), and `workflow_run` trigger branch scope enforcement (CTR-021). The `docker-sbx` runtime enforcement (CTR-004 scope) requires `sudo: true`, compatible runner topology, and a minimum AWF version; the credential refresh step emitted before agent execution is a security improvement with no new constraint on `.lock.yml` semantics. Playwright CLI mode (`tools.playwright.mode: cli`) is compiler-generated infrastructure with no new constraint on `.lock.yml` semantics. |
| `1.0.18` | `v0.72.1` (or newer) | Threat-detection behavior must remain compatible with current `.lock.yml` compilation semantics, including manifest drift enforcement (`gh-aw-manifest` checks for CTR-016), update-check validation (`check-for-updates` handling for CTR-018), cache-memory integrity enforcement (`update_cache_memory` gating for CTR-019), conditional import rejection (`imports.if` rejection for CTR-020), and `workflow_run` trigger branch scope enforcement (CTR-021). The `docker-sbx` runtime enforcement (CTR-004 scope) requires `sudo: true`, compatible runner topology, and a minimum AWF version; the credential refresh step emitted before agent execution is a security improvement with no new constraint on `.lock.yml` semantics. Playwright CLI mode (`tools.playwright.mode: cli`) is compiler-generated infrastructure with no new constraint on `.lock.yml` semantics. |
| `1.0.17` | `v0.72.1` (or newer) | Threat-detection behavior must remain compatible with current `.lock.yml` compilation semantics, including manifest drift enforcement (`gh-aw-manifest` checks for CTR-016), update-check validation (`check-for-updates` handling for CTR-018), cache-memory integrity enforcement (`update_cache_memory` gating for CTR-019), conditional import rejection (`imports.if` rejection for CTR-020), and `workflow_run` trigger branch scope enforcement (CTR-021). The `docker-sbx` runtime enforcement (CTR-004 scope) requires `sudo: true`, compatible runner topology, and a minimum AWF version; the credential refresh step emitted before agent execution is a security improvement with no new constraint on `.lock.yml` semantics. |
| `1.0.16` | `v0.72.1` (or newer) | Threat-detection behavior must remain compatible with current `.lock.yml` compilation semantics, including manifest drift enforcement (`gh-aw-manifest` checks for CTR-016), update-check validation (`check-for-updates` handling for CTR-018), cache-memory integrity enforcement (`update_cache_memory` gating for CTR-019), conditional import rejection (`imports.if` rejection for CTR-020), and `workflow_run` trigger branch scope enforcement (CTR-021). The `docker-sbx` runtime enforcement (CTR-004 scope) requires `sudo: true`, compatible runner topology, and a minimum AWF version; the credential refresh step emitted before agent execution is a security improvement with no new constraint on `.lock.yml` semantics. |
Expand Down Expand Up @@ -302,7 +303,9 @@ The mappings above are pattern-based references and MUST be validated against co

When mappings change, this table MUST be updated in the same change set as the implementation update.

### 7.2 Mapping Audit (2026-07-27)
### 7.2 Mapping Audit (2026-07-31)

Audit result: ✅ all listed `CTR-001` through `CTR-021` rows currently include non-empty implementation references and non-empty test coverage targets; no `TODO` placeholders were found in the mapping table. Review window: SPDD daily spec review cycle 2026-07-31 (rotation index 5–9 of 18, covering `specs/compiler-threat-detection-spec.md` among others). Security-relevant items evaluated: (1) **CTR-016/018/019/020/021 sync references**: each of these five rules was individually verified against current `pkg/workflow/` source locations — `safe_update_enforcement.go` (CTR-016), `strict_mode_update_check_validation.go` (CTR-018), `cache.go` + `expression_builder.go` (CTR-019), `pkg/parser/import_bfs.go` (CTR-020), `agent_validation.go` (CTR-021); all implementation references and test coverage targets in Section 7.1 are current and accurate; no drift detected. (2) **Section 6 Optimizer Failure Safeguards** (§6.6): the three failure modes (API unavailability, runner timeout, rate-limit exhaustion) are specified normatively but are not currently covered by a dedicated unit or integration test; flagged as a coverage gap for the next implementation cycle — a future PR should add tests in `pkg/workflow/` or an integration harness that exercises the `OPTIMIZER_DEGRADED`, `OPTIMIZER_TIMEOUT`, and `OPTIMIZER_RATE_LIMITED` diagnostic paths. No new threat class; no new CTR rule required this cycle.

Audit result: ✅ all listed `CTR-001` through `CTR-021` rows currently include non-empty implementation references and non-empty test coverage targets; no `TODO` placeholders were found in the mapping table. Review window: commit d4872c2 (fix: disable Chromium sandbox for playwright-cli mode in CI containers), merged 2026-07-26. Security-relevant items evaluated: (1) **Playwright CLI mode** (`pkg/workflow/playwright_cli.go`): compiler adds `tools.playwright.mode: cli` support; in CLI mode, `@playwright/cli` is installed via npm and `playwright-cli install --skills` runs before the agent; the npm install step uses `RunInstallScripts: true` internally, but this is a compiler-controlled invocation for a single trusted package (`@playwright/cli`) rather than a user-controlled `runtimes.node.run-install-scripts: true` frontmatter flag — CTR-014 validates the latter only; no new threat class; no new CTR rule required. (2) **Playwright MCP deprecation warning** (`pkg/workflow/playwright_validation.go`): compiler emits a non-blocking deprecation warning when `tools.playwright` is in MCP mode; no new permissions or trust surface introduced; no new threat class; no new CTR rule required. (3) **Playwright Chromium `--no-sandbox` flag** (MCP mode, `pkg/workflow/mcp_config_playwright_renderer.go`): disabling Chromium's process sandbox is required for Chromium to reach `localhost` inside CI containers; this is a browser-process-level flag, not a workflow sandbox bypass; threat class is distinct from CTR-004 (workflow sandbox bypass) and is an expected and documented operational necessity for containerized Playwright; no new CTR rule required.

Expand Down Expand Up @@ -382,6 +385,11 @@ The following test IDs map one-to-one to the CTR rules in Section 5.1. Each test

## 10. Change Log

### 1.0.19 (2026-07-31)

- Updated Section 7.2 mapping audit to 2026-07-31 covering SPDD daily spec review cycle (rotation index 5–9 of 18): re-validated CTR-016/018/019/020/021 sync references against current `pkg/workflow/` implementations — all Section 7.1 references confirmed accurate, no drift detected; identified Section 6.6 Optimizer Failure Safeguards (`OPTIMIZER_DEGRADED`, `OPTIMIZER_TIMEOUT`, `OPTIMIZER_RATE_LIMITED` diagnostic paths) as untested — flagged as a coverage gap for the next implementation cycle; no new threat class; no new CTR rules required
- Updated Section 2 spec-to-implementation sync table with version 1.0.19 entry

### 1.0.18 (2026-07-27)

- Updated Section 7.2 mapping audit to 2026-07-27 covering commit d4872c2 (fix: disable Chromium sandbox for playwright-cli mode in CI containers, 2026-07-26): evaluated Playwright CLI mode (`playwright_cli.go`: compiler-generated npm install for `@playwright/cli`, not user-frontmatter-controlled `run-install-scripts`), Playwright MCP deprecation warning (`playwright_validation.go`: non-blocking deprecation warning, no new trust surface), and Playwright Chromium `--no-sandbox` flag (`mcp_config_playwright_renderer.go`: browser-process-level flag distinct from CTR-004 workflow sandbox bypass); no new threat class; no new CTR rules required
Expand Down
1 change: 1 addition & 0 deletions specs/forecast-compliance-fixtures/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,4 @@ To add a fixture covering a specific compliance scenario:
| `run_summary_zero_et.json` | Run with missing/zero ET (artifact not downloaded) | T-FC-022 |
| `run_summary_failed.json` | Run with `conclusion: "failure"` for Bernoulli sampling | T-FC-035 |
| `run_summary_high_et.json` | Run with very high ET (≥ 1,000,000) for overflow checks | T-ET-006 |
| `run_summary_cancelled.json` | Run with `conclusion: "cancelled"` (included in sample but not a Bernoulli success; ET is zero because the run did not complete) | T-FC-036 |
56 changes: 56 additions & 0 deletions specs/forecast-compliance-fixtures/run_summary_cancelled.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
{
"cli_version": "0.0.0-test",
"run_id": 12345682,
"processed_at": "2026-05-01T12:00:00Z",
"run": {
"id": 12345682,
"name": "daily-report",
"workflow_id": 98765,
"head_branch": "main",
"head_sha": "abc123def456abc123def456abc123def456abc123",
"status": "completed",
"conclusion": "cancelled",
"created_at": "2026-05-01T11:00:00Z",
"updated_at": "2026-05-01T11:02:10Z",
"run_started_at": "2026-05-01T11:00:05Z"
},
"metrics": {
"total_steps": 2,
"failed_steps": 0,
"tool_calls": 3
},
"access_analysis": null,
"firewall_analysis": null,
"redacted_domains_analysis": null,
"missing_tools": [],
"missing_data": [],
"noops": [],
"mcp_failures": [],
"artifacts_list": [],
"job_details": [],
"token_usage_summary": {
"total_input_tokens": 900,
"total_output_tokens": 200,
"total_cache_read_tokens": 0,
"total_cache_write_tokens": 0,
"total_requests": 1,
"total_duration_ms": 4200,
"total_response_bytes": 3800,
"cache_efficiency": 0.0,
"total_effective_tokens": 0,
"total_aic": 0.0,
"by_model": {
"claude-3-7-sonnet": {
"provider": "anthropic",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sampling dead-zone: total_aic: 0.0 causes forecastWorkflow to skip this fixture entirely — runs with runAIC <= 0 are filtered before Bernoulli sampling. The fixture cannot exercise the claimed "included in sample but not a Bernoulli success" path while total_aic is zero. Set a small positive value (e.g. 0.001) so the fixture enters the sample path and the spec claim is actually testable.

@copilot please address this.

"input_tokens": 900,
"output_tokens": 200,
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"requests": 1,
"duration_ms": 4200,
"response_bytes": 3800,
"effective_tokens": 0
}
}
}
}
8 changes: 8 additions & 0 deletions specs/intent-attribution-agent-governance.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,14 @@ label-to-intent selectors derived from `.github/objective-mapping.json` — duri
or CI. Keys present in one source but not the other SHOULD surface as a sync warning or
compliance failure so attribution and authorization stay aligned.

**Escalation norm**: A sync warning that persists across **3 or more consecutive CI runs**
without a corresponding corrective PR or explicit waiver **MUST** be escalated to a compliance
failure. When a sync warning escalates, the CI check responsible for drift detection **MUST**
Comment on lines +131 to +133
fail with a non-zero exit code and open (or update) a tracking issue recording the affected
keys, the first-detected date, and the on-call maintainer as the default assignee. This norm
ensures that aspirational migration language does not mask persistent, unaddressed
configuration drift.

## Product boundary

The system can establish:
Expand Down
Loading