Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/daily-observability-report.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 5 additions & 3 deletions .github/workflows/daily-observability-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,14 +141,16 @@ The AWF Firewall uses Squid proxy for egress control. The key log file is `acces
For each firewall-enabled workflow run, check:

1. **access.log existence**: Search recursively inside each run folder for firewall access logs
- Canonical path: `/tmp/gh-aw/aw-mcp/logs/run-<id>/sandbox/firewall/logs/access.log`
- Canonical path (current AWF layout): `run-<id>/sandbox/firewall/logs/squid-logs/access.log`
- Also accept the legacy path: `run-<id>/sandbox/firewall/logs/access.log` (older AWF layout)
- Also accept equivalent paths nested under artifact-prefixed directories (workflow_call)
- Do not assume a fixed top-level location; use recursive discovery

2. **access.log content quality**:
- Are there log entries present?
- Do entries follow squid format: `timestamp duration client status size method url user hierarchy type`
- Are both allowed and blocked requests logged?
- Do entries follow AWF custom format: `timestamp client_ip:port domain dest_ip:port proto method status decision url user_agent`
- Example entry: `1761332530.474 172.30.0.20:35288 api.github.com:443 140.82.112.22:443 1.1 CONNECT 200 TCP_TUNNEL:HIER_DIRECT api.github.com:443 "-"`
- Are both allowed (TCP_TUNNEL) and blocked (TCP_DENIED) requests logged?

3. **Firewall configuration**:
- Check `aw_info.json` for firewall settings:
Expand Down
21 changes: 20 additions & 1 deletion actions/setup/js/generate_observability_summary.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ const AGENT_OUTPUT_PATH = "/tmp/gh-aw/agent_output.json";
const OTLP_EXPORT_ERRORS_PATH = "/tmp/gh-aw/otlp-export-errors.count";
const OTLP_EXPORT_ERROR_DETAILS_PATH = "/tmp/gh-aw/otlp-export-errors.jsonl";
const gatewayEventPaths = ["/tmp/gh-aw/mcp-logs/gateway.jsonl", "/tmp/gh-aw/mcp-logs/rpc-messages.jsonl"];
// Squid access log paths: current AWF layout (squid-logs/ subdirectory) and legacy layout (directly under logs/).
const squidAccessLogPaths = ["/tmp/gh-aw/sandbox/firewall/logs/squid-logs/access.log", "/tmp/gh-aw/sandbox/firewall/logs/access.log"];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] The hardcoded /tmp/gh-aw base path will silently return false on ARC/DinD runners where the Go compiler writes logs under RUNNER_TEMP/gh-aw/squidAccessLogPresent will always be wrong there even when the log file exists. This is the same root cause as the existing comment at this line but the fix direction is worth making explicit.

💡 Suggested approach

Mirror how print_firewall_logs.sh already derives the path: the compiler exports AWF_LOGS_DIR; the JS side can use RUNNER_TEMP to construct the same base, or a new GH_AW_BASE_DIR env var:

const GH_AW_BASE = (process.env.RUNNER_TEMP && process.env.RUNNER_TEMP !== '')
  ? path.join(process.env.RUNNER_TEMP, 'gh-aw')
  : '/tmp/gh-aw';
const squidAccessLogPaths = [
  GH_AW_BASE + '/sandbox/firewall/logs/squid-logs/access.log',
  GH_AW_BASE + '/sandbox/firewall/logs/access.log',
];

Add a unit-test case that sets RUNNER_TEMP to confirm the path is derived dynamically.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoded /tmp/gh-aw/... paths make squidAccessLogPresent always false on ARC/DinD runners, producing a false "log missing" warning on every such run.

💡 Details

The Go compiler deliberately redirects firewall logs to ${{ runner.temp }}/gh-aw/sandbox/firewall/logs on ARC/DinD topologies (see pkg/workflow/engine_firewall_support.go isArcDindTopology branches using constants.AWFProxyLogsDirExpr), because /tmp/gh-aw is not daemon-visible in that mode. print_firewall_logs.sh correctly receives this via the AWF_LOGS_DIR env var and checks the right location. This new .cjs script instead hardcodes /tmp/gh-aw/sandbox/firewall/logs/... with no environment override, so on ARC/DinD it will always report squidAccessLogPresent: false and emit the "cannot be audited" warning even when the log is present — exactly the false-positive class of bug this PR set out to fix.

const squidAccessLogPaths = [
  path.join(process.env.GH_AW_FIREWALL_LOGS_DIR || "/tmp/gh-aw/sandbox/firewall/logs", "squid-logs", "access.log"),
  path.join(process.env.GH_AW_FIREWALL_LOGS_DIR || "/tmp/gh-aw/sandbox/firewall/logs", "access.log"),
];

The generator step would need to pass the same resolved directory as an env var, mirroring what generateFirewallLogParsingStep already does for the shell script.


function readJSONIfExists(path) {
if (!fs.existsSync(path)) {
Expand Down Expand Up @@ -63,6 +65,15 @@ function uniqueCreatedItemTypes(items) {
return [...types].sort();
}

function checkSquidAccessLogPresent() {
for (const path of squidAccessLogPaths) {
if (fs.existsSync(path)) {
return true;
}
}
return false;
}

function readOTLPExportErrorCount() {
if (!fs.existsSync(OTLP_EXPORT_ERRORS_PATH)) {
return 0;
Expand Down Expand Up @@ -117,12 +128,14 @@ function collectObservabilityData() {
// Do NOT fall back to workflow_call_id — it is not a valid OTLP trace ID.
const traceId = process.env.GITHUB_AW_OTEL_TRACE_ID || (awInfo.context ? awInfo.context.otel_trace_id || "" : "");

const firewallEnabled = awInfo.firewall_enabled === true;
return {
workflowName: awInfo.workflow_name || "",
engineId: awInfo.engine_id || "",
traceId,
staged: awInfo.staged === true,
firewallEnabled: awInfo.firewall_enabled === true,
firewallEnabled,
squidAccessLogPresent: firewallEnabled ? checkSquidAccessLogPresent() : null,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new step-summary field never fires for firewall-enabled workflows that lack OTLP, since generateObservabilitySummary bails out early unless isOTLPEnabled(data).

💡 Details

In pkg/workflow/compiler_yaml_ai_execution.go, generateObservabilitySummary returns immediately when !isOTLPEnabled(data), before this script is even wired into the workflow. Firewall can be enabled independently of OTLP tracing, so the majority of firewall-only workflows (no telemetry export configured) will never emit the squidAccessLogPresent line or the missing-log warning — silently defeating the stated goal of "surfaces a warning line when missing so operators see the gap without waiting for the daily report" for that common configuration.

// pkg/workflow/compiler_yaml_ai_execution.go
func (c *Compiler) generateObservabilitySummary(yaml *strings.Builder, data *WorkflowData) {
	if !isOTLPEnabled(data) {
		return // <- firewall-only workflows never reach the squid-log check below
	}
	...

Either decouple the squid-log presence check into its own step gated only on isFirewallEnabled(data), or relax the early return so firewall-only runs still get this diagnostic.

createdItemCount: items.length,
createdItemTypes: uniqueCreatedItemTypes(items),
outputErrorCount: errors.length,
Expand Down Expand Up @@ -156,6 +169,12 @@ function buildObservabilitySummary(data) {
lines.push(`- **agent output errors**: ${data.outputErrorCount}`);
lines.push(`- **otlp export errors**: ${data.otlpExportErrors}`);
lines.push(`- **firewall enabled**: ${data.firewallEnabled}`);
if (data.firewallEnabled && data.squidAccessLogPresent !== null) {
lines.push(`- **squid access.log present**: ${data.squidAccessLogPresent}`);
if (!data.squidAccessLogPresent) {
lines.push("- Squid access.log not found; egress traffic for this run cannot be audited.");
}
}
lines.push(`- **staged**: ${data.staged}`);

if (data.otlpExportErrors > 0) {
Expand Down
46 changes: 46 additions & 0 deletions actions/setup/js/generate_observability_summary.test.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ describe("generate_observability_summary.cjs", () => {
"/tmp/gh-aw/otlp-export-errors.jsonl",
"/tmp/gh-aw/mcp-logs/gateway.jsonl",
"/tmp/gh-aw/mcp-logs/rpc-messages.jsonl",
"/tmp/gh-aw/sandbox/firewall/logs/squid-logs/access.log",
"/tmp/gh-aw/sandbox/firewall/logs/access.log",
]) {
if (fs.existsSync(path)) {
fs.unlinkSync(path);
Expand Down Expand Up @@ -133,4 +135,48 @@ describe("generate_observability_summary.cjs", () => {
expect(mockCore.summary.addRaw).toHaveBeenCalledTimes(1);
expect(mockCore.summary.write).toHaveBeenCalledTimes(1);
});

it("reports squid access.log present when firewall enabled and file exists at squid-logs path", async () => {
fs.mkdirSync("/tmp/gh-aw/sandbox/firewall/logs/squid-logs", { recursive: true });
fs.writeFileSync("/tmp/gh-aw/sandbox/firewall/logs/squid-logs/access.log", '1761332530.474 172.30.0.20:35288 api.github.com:443 140.82.112.22:443 1.1 CONNECT 200 TCP_TUNNEL:HIER_DIRECT api.github.com:443 "-"\n');
fs.writeFileSync("/tmp/gh-aw/aw_info.json", JSON.stringify({ workflow_name: "firewall-workflow", firewall_enabled: true }));

await module.main(mockCore);

const summary = mockCore.summary.addRaw.mock.calls[0][0];
expect(summary).toContain("- **squid access.log present**: true");
expect(summary).not.toContain("egress traffic for this run cannot be audited");
});

it("reports squid access.log present when firewall enabled and file exists at legacy path", async () => {
fs.mkdirSync("/tmp/gh-aw/sandbox/firewall/logs", { recursive: true });
fs.writeFileSync("/tmp/gh-aw/sandbox/firewall/logs/access.log", '1761332530.474 172.30.0.20:35288 api.github.com:443 140.82.112.22:443 1.1 CONNECT 200 TCP_TUNNEL:HIER_DIRECT api.github.com:443 "-"\n');
fs.writeFileSync("/tmp/gh-aw/aw_info.json", JSON.stringify({ workflow_name: "firewall-workflow", firewall_enabled: true }));

await module.main(mockCore);

const summary = mockCore.summary.addRaw.mock.calls[0][0];
expect(summary).toContain("- **squid access.log present**: true");
expect(summary).not.toContain("egress traffic for this run cannot be audited");
});

it("warns when firewall enabled but squid access.log is missing", async () => {
fs.writeFileSync("/tmp/gh-aw/aw_info.json", JSON.stringify({ workflow_name: "firewall-workflow", firewall_enabled: true }));

await module.main(mockCore);

const summary = mockCore.summary.addRaw.mock.calls[0][0];
expect(summary).toContain("- **squid access.log present**: false");
expect(summary).toContain("Squid access.log not found; egress traffic for this run cannot be audited.");
});

it("omits squid access.log status when firewall is disabled", async () => {
fs.writeFileSync("/tmp/gh-aw/aw_info.json", JSON.stringify({ workflow_name: "no-firewall-workflow", firewall_enabled: false }));

await module.main(mockCore);

const summary = mockCore.summary.addRaw.mock.calls[0][0];
expect(summary).not.toContain("squid access.log");
expect(summary).not.toContain("egress traffic for this run cannot be audited");
});
});
15 changes: 15 additions & 0 deletions actions/setup/sh/print_firewall_logs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,18 @@ if command -v awf &> /dev/null; then
else
echo 'AWF binary not installed, skipping firewall log summary'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test suite covers this new warning path, so a shell-syntax regression (e.g. quoting/array issues) would silently ship undetected.

💡 Details

print_firewall_logs.sh has no accompanying test file in this diff, unlike generate_observability_summary.cjs which got four new Jest cases for the equivalent logic. The two implementations of the same "check current path, then legacy path" logic can now drift silently — e.g. if AWFProxyLogsDir changes in Go but only the .cjs array is updated (which is already stale per the ARC/DinD hardcoded-path issue noted elsewhere), this shell script would keep checking the old paths without any test catching the mismatch.

Consider adding a bats/shellspec test (or a minimal inline smoke test invoked from CI) that stubs AWF_LOGS_DIR, creates files at the current path, the legacy path, and neither, and asserts the WARNING line is/isn't emitted on stderr — mirroring the JS test matrix already added.

fi

# Warn if Squid access.log is missing (current layout: squid-logs/; legacy layout: directly under logs/).
# A missing access.log means egress traffic for this run cannot be audited.
ACCESS_LOG_FOUND=false
for candidate in \
"${AWF_LOGS_DIR}/squid-logs/access.log" \
"${AWF_LOGS_DIR}/access.log"; do
if [[ -f "${candidate}" ]]; then
Comment on lines +57 to +60
ACCESS_LOG_FOUND=true
break
fi
done

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The warning is written only to stderr (>&2) and is not appended to $GITHUB_STEP_SUMMARY. Unlike the AWF summary block above (which uses tee -a "${GITHUB_STEP_SUMMARY:-/dev/null}"), this warning will be invisible in the step-summary panel — easy to miss during triage.

Suggested fix:

if [[ "${ACCESS_LOG_FOUND}" == "false" ]]; then
  msg="WARNING: Squid access.log not found under ${AWF_LOGS_DIR}; egress traffic for this run cannot be audited."
  echo "${msg}" >&2
  echo "${msg}" >> "${GITHUB_STEP_SUMMARY:-/dev/null}"
fi

@copilot please address this.

if [[ "${ACCESS_LOG_FOUND}" == "false" ]]; then
echo "WARNING: Squid access.log not found under ${AWF_LOGS_DIR}; egress traffic for this run cannot be audited." >&2
fi
10 changes: 5 additions & 5 deletions pkg/workflow/testdata/TestWasmGolden_AllEngines/copilot.golden
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:
env:
GH_AW_SETUP_WORKFLOW_NAME: "engine-copilot-test"
GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/workflow.lock.yml@${{ github.ref }}
GH_AW_INFO_VERSION: "1.0.75"
GH_AW_INFO_VERSION: "COPILOT_VERSION"
GH_AW_INFO_AWF_VERSION: "vAWF_VERSION"
GH_AW_INFO_ENGINE_ID: "copilot"
- name: Generate agentic run info
Expand All @@ -71,8 +71,8 @@ jobs:
GH_AW_INFO_ENGINE_ID: "copilot"
GH_AW_INFO_ENGINE_NAME: "GitHub Copilot CLI"
GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'default' }}
GH_AW_INFO_VERSION: "1.0.75"
GH_AW_INFO_AGENT_VERSION: "1.0.75"
GH_AW_INFO_VERSION: "COPILOT_VERSION"
GH_AW_INFO_AGENT_VERSION: "COPILOT_VERSION"
GH_AW_INFO_WORKFLOW_NAME: "engine-copilot-test"
GH_AW_INFO_EXPERIMENTAL: "false"
GH_AW_INFO_SUPPORTS_TOOLS_ALLOWLIST: "true"
Expand Down Expand Up @@ -367,7 +367,7 @@ jobs:
env:
GH_AW_SETUP_WORKFLOW_NAME: "engine-copilot-test"
GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/workflow.lock.yml@${{ github.ref }}
GH_AW_INFO_VERSION: "1.0.75"
GH_AW_INFO_VERSION: "COPILOT_VERSION"
GH_AW_INFO_AWF_VERSION: "vAWF_VERSION"
GH_AW_INFO_ENGINE_ID: "copilot"
- name: Checkout repository
Expand Down Expand Up @@ -714,7 +714,7 @@ jobs:
env:
GH_AW_SETUP_WORKFLOW_NAME: "engine-copilot-test"
GH_AW_CURRENT_WORKFLOW_REF: ${{ github.repository }}/.github/workflows/workflow.lock.yml@${{ github.ref }}
GH_AW_INFO_VERSION: "1.0.75"
GH_AW_INFO_VERSION: "COPILOT_VERSION"
GH_AW_INFO_AWF_VERSION: "vAWF_VERSION"
GH_AW_INFO_ENGINE_ID: "copilot"
- name: Check team membership for workflow
Expand Down
9 changes: 9 additions & 0 deletions pkg/workflow/wasm_golden_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ var testDefaultAWFSchemaURLRE = regexp.MustCompile(`(releases/download/)` + rege
var testDefaultAWFImageTagRE = regexp.MustCompile(`("imageTag"\s*:\s*")(?:v)?` + regexp.QuoteMeta(strings.TrimPrefix(string(constants.DefaultFirewallVersion), "v")) + `"`)
var testDefaultMCPGImageRE = regexp.MustCompile(`(ghcr\.io/github/gh-aw-mcpg:)` + regexp.QuoteMeta(string(constants.DefaultMCPGatewayVersion)) + `\b`)
var testDefaultGitHubMCPServerImageRE = regexp.MustCompile(`(ghcr\.io/github/github-mcp-server:)` + regexp.QuoteMeta(string(constants.DefaultGitHubMCPServerVersion)) + `\b`)
var testDefaultCopilotInfoVersionRE = regexp.MustCompile(`GH_AW_INFO_VERSION: "` + regexp.QuoteMeta(string(constants.DefaultCopilotVersion)) + `"`)
var testDefaultCopilotAgentInfoVersionRE = regexp.MustCompile(`GH_AW_INFO_AGENT_VERSION: "` + regexp.QuoteMeta(string(constants.DefaultCopilotVersion)) + `"`)
var testDefaultCodexInfoVersionRE = regexp.MustCompile(`GH_AW_INFO_VERSION: "` + regexp.QuoteMeta(string(constants.DefaultCodexVersion)) + `"`)
var testDefaultCodexAgentInfoVersionRE = regexp.MustCompile(`GH_AW_INFO_AGENT_VERSION: "` + regexp.QuoteMeta(string(constants.DefaultCodexVersion)) + `"`)
var testDefaultCodexInstallVersionRE = regexp.MustCompile(`(@openai/codex@)` + regexp.QuoteMeta(string(constants.DefaultCodexVersion)) + `\b`)
Expand All @@ -48,6 +50,8 @@ func normalizeDefaultRuntimeVersions(content string) string {
normalized = testDefaultAWFImageRE.ReplaceAllString(normalized, `${1}AWF_VERSION`)
normalized = testDefaultAWFSchemaURLRE.ReplaceAllString(normalized, `${1}vAWF_VERSION$2`)
normalized = testDefaultAWFImageTagRE.ReplaceAllString(normalized, `${1}AWF_VERSION"`)
normalized = testDefaultCopilotInfoVersionRE.ReplaceAllString(normalized, `GH_AW_INFO_VERSION: "COPILOT_VERSION"`)
normalized = testDefaultCopilotAgentInfoVersionRE.ReplaceAllString(normalized, `GH_AW_INFO_AGENT_VERSION: "COPILOT_VERSION"`)
normalized = testDefaultCodexInfoVersionRE.ReplaceAllString(normalized, `GH_AW_INFO_VERSION: "CODEX_VERSION"`)
normalized = testDefaultCodexAgentInfoVersionRE.ReplaceAllString(normalized, `GH_AW_INFO_AGENT_VERSION: "CODEX_VERSION"`)
normalized = testDefaultCodexInstallVersionRE.ReplaceAllString(normalized, `${1}CODEX_VERSION`)
Expand Down Expand Up @@ -94,6 +98,8 @@ func TestNormalizeOutput_DefaultRuntimeVersions(t *testing.T) {
`run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:` + strings.TrimPrefix(string(constants.DefaultFirewallVersion), "v") + ` ghcr.io/github/gh-aw-firewall/api-proxy:` + strings.TrimPrefix(string(constants.DefaultFirewallVersion), "v") + ` ghcr.io/github/gh-aw-mcpg:` + string(constants.DefaultMCPGatewayVersion),
`{"schema":"https://github.com/github/gh-aw-firewall/releases/download/` + string(constants.DefaultFirewallVersion) + `/awf-config.schema.json","imageTag":"` + string(constants.DefaultFirewallVersion) + `"}`,
`GH_AW_MODEL_DETECTION_CLAUDE: ${{ vars.GH_AW_MODEL_DETECTION_CLAUDE || vars.GH_AW_DEFAULT_MODEL_CLAUDE || '` + constants.SonnetDefaultModel + `' }}`,
`GH_AW_INFO_VERSION: "` + string(constants.DefaultCopilotVersion) + `"`,
`GH_AW_INFO_AGENT_VERSION: "` + string(constants.DefaultCopilotVersion) + `"`,
`GH_AW_INFO_VERSION: "` + string(constants.DefaultCodexVersion) + `"`,
`GH_AW_INFO_AGENT_VERSION: "` + string(constants.DefaultCodexVersion) + `"`,
`GH_AW_INFO_VERSION: "` + string(constants.DefaultPiVersion) + `"`,
Expand All @@ -114,6 +120,8 @@ func TestNormalizeOutput_DefaultRuntimeVersions(t *testing.T) {
require.Contains(t, normalized, `releases/download/vAWF_VERSION/awf-config.schema.json`)
require.Contains(t, normalized, `"imageTag":"AWF_VERSION"`)
require.Contains(t, normalized, `GH_AW_MODEL_DETECTION_CLAUDE: ${{ vars.GH_AW_MODEL_DETECTION_CLAUDE || vars.GH_AW_DEFAULT_MODEL_CLAUDE || 'default' }}`)
require.Contains(t, normalized, `GH_AW_INFO_VERSION: "COPILOT_VERSION"`)
require.Contains(t, normalized, `GH_AW_INFO_AGENT_VERSION: "COPILOT_VERSION"`)
require.Contains(t, normalized, `GH_AW_INFO_VERSION: "CODEX_VERSION"`)
require.Contains(t, normalized, `GH_AW_INFO_AGENT_VERSION: "CODEX_VERSION"`)
require.Contains(t, normalized, `GH_AW_INFO_VERSION: "PI_VERSION"`)
Expand All @@ -126,6 +134,7 @@ func TestNormalizeOutput_DefaultRuntimeVersions(t *testing.T) {
require.NotContains(t, normalized, string(constants.DefaultFirewallVersion))
require.NotContains(t, normalized, string(constants.DefaultMCPGatewayVersion))
require.NotContains(t, normalized, constants.SonnetDefaultModel)
require.NotContains(t, normalized, string(constants.DefaultCopilotVersion))
require.NotContains(t, normalized, string(constants.DefaultCodexVersion))
require.NotContains(t, normalized, string(constants.DefaultPiVersion))
}
Expand Down