Add unauthenticated REST API fallback for remote workflow imports

[Copilot]

Objective

Add fallback code path for fetching remote workflow files without GitHub token

Plan

• Add unauthenticated REST API fallback in resolveRefToSHA function
• Add unauthenticated REST API fallback in downloadFileFromGitHub function
• Add JSON parsing for REST API responses
• Add tests for JSON parsing
• Document the fallback behavior
• Verify imports are cached in .github/aw/imports/ directory
• Restore action pins in actions-lock.json
• Add validation to prevent empty uses field in compiled workflows
• Move documentation to specs folder
• Address code review feedback

Implementation Summary

Changes Made

1. Modified resolveRefToSHA function - Added fallback to resolveRefToSHAUnauthenticated when gh CLI authentication fails

2. Modified downloadFileFromGitHub function - Added fallback to downloadFileFromGitHubUnauthenticated when gh CLI authentication fails

3. Added resolveRefToSHAUnauthenticated function:

   • Uses curl with GitHub's public REST API
   • Endpoint: GET /repos/{owner}/{repo}/commits/{ref}
   • Proper JSON parsing using encoding/json
   • Validates SHA format (40 hex characters)
   • Added -f flag to curl for better HTTP error handling

4. Added downloadFileFromGitHubUnauthenticated function:

   • Uses curl with GitHub's public REST API
   • Endpoint: GET /repos/{owner}/{repo}/contents/{path}?ref={ref}
   • Proper JSON parsing and base64 decoding
   • Error handling for Not Found, rate limits, etc.
   • Added -f flag to curl for better HTTP error handling
   • Uses Go's native base64 decoding instead of external base64 command for better portability

5. Added unit tests - TestJSONParsing validates JSON response parsing

6. Restored action pins - Fixed .github/aw/actions-lock.json which was accidentally emptied during build process

7. Added validation for empty uses field - Created getRequiredActionPin helper function in compiler_jobs.go that validates action pins exist before using them in compiled workflows. This prevents compilation errors where uses: would be empty for system actions like actions/checkout and actions/github-script.

8. Moved documentation to specs folder - Relocated UNAUTHENTICATED_FALLBACK.md to specs/ directory for better organization

9. Code review improvements:

   • Replaced external base64 -d command with Go's native encoding/base64 for better portability
   • Added -f flag to curl commands to fail on HTTP errors (404, 500, etc.) instead of trying to parse error HTML as JSON
   • Fixed documentation to use relative paths instead of hardcoded CI paths

How It Works

With GH_TOKEN (preferred):

• Uses gh CLI commands (existing behavior)
• Access to private repositories
• Higher rate limits (5000 requests/hour)

Without GH_TOKEN (fallback):

• Detects authentication failure from gh CLI
• Falls back to public REST API via curl
• Only works with public repositories
• Limited to 60 requests/hour per IP

Import Cache Integration

The fallback works seamlessly with the existing import cache:

• Downloaded files are cached by SHA in .github/aw/imports/{owner}/{repo}/{sha}/
• Reduces API calls on subsequent compilations
• Cache persists across workflow runs

Action Pin Validation

The compiler now validates that all required system actions (like actions/checkout and actions/github-script) have pins defined in actions-lock.json. This prevents compilation errors where the uses: field would be empty. Custom action repos (specified via action-repo in workflow frontmatter) can still be used without pins.

Testing

✅ Unit tests pass (TestJSONParsing)
✅ Action pins tests pass (TestActionPins*)
✅ All unit tests pass
✅ Build succeeds
✅ Linting passes

Documentation

See specs/UNAUTHENTICATED_FALLBACK.md for detailed documentation

Original prompt

Run the "update" command to receive the latest version of the daily-team-status agentic workflow. This version of the workflow has a remote import pointing to a file in the agentics repo.

Make sure that the "compile" command is able to compile this workflow.

Since the agent is running in an environment without GH_TOKEN, the "gh api" call will fail. Add a fallback code path that tries to fetch the source using the REST api without any token.

Make sure the import gets stored in the imports cache under .github/aw

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[pelikhan]

@copilot make sure to maintain existing pins, upsert resolved values

[Copilot]

Fixed in f47d79c. Restored all existing action pins from before my changes. The file was accidentally emptied during the build process but is now back to its original state with all 20+ action entries.

[pelikhan]

@copilot the value in "uses" can never be empty. This is a compilation error. Add a check.

[Copilot]

Fixed in 407e621. Added getRequiredActionPin helper that validates action pins exist before using them. This prevents empty uses: fields by returning an error during compilation if a required system action (like actions/checkout or actions/github-script) doesn't have a pin defined in actions-lock.json.

[Copilot]

Pull Request Overview

This PR adds unauthenticated REST API fallback functionality to enable fetching remote workflow files from public GitHub repositories when the gh CLI is not authenticated. The implementation detects authentication failures and transparently falls back to using curl with GitHub's public REST API.

Key Changes

• Added fallback code paths in remote_fetch.go that use curl and GitHub's public REST API when gh CLI authentication fails
• Added getRequiredActionPin helper function to validate that system actions have defined pins before using them in compiled workflows
• Added basic unit tests for JSON response parsing

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
specs/UNAUTHENTICATED_FALLBACK.md	New documentation describing the fallback implementation, behavior, and limitations
pkg/parser/remote_fetch.go	Added unauthenticated fallback functions using curl and GitHub REST API with JSON parsing and base64 decoding
pkg/parser/remote_fetch_test.go	Added unit tests for JSON response parsing (commit SHA, file content, and error responses)
pkg/workflow/compiler_jobs.go	Added getRequiredActionPin helper to validate action pins exist before use in pre-activation and activation jobs
pkg/workflow/action_pins.go	Updated comments to clarify that empty string return is expected for custom action repos

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The fallback detection logic only checks for specific strings in error output. This approach may be fragile and could miss authentication failures with different error messages.

Issue: The string matching uses || (OR) logic, which means any single match triggers the fallback, but the condition checks three different patterns:

• "GH_TOKEN" - could appear in various contexts
• "authentication" - generic word that might appear in non-auth errors
• "not logged into" - most specific pattern

Potential problem: If GitHub changes its error messages or if there's a different error that happens to contain one of these strings, the fallback could be triggered incorrectly.

Recommendation: Consider making the detection more specific by:

1. Checking the error type or exit code first
2. Using more specific error message patterns
3. Adding logging to track when fallback is used vs. when authentication truly failed

See below for a potential fix:

		if isAuthFailure(outputStr) {
			// Try fallback without authentication
			remoteLog.Printf("gh CLI authentication failed (exit code: %v, output: %q), trying unauthenticated REST API fallback", err, strings.TrimSpace(outputStr))

[Copilot]

The error handling logic checks for error messages in the response after successful JSON parsing, but this doesn't account for HTTP-level errors.

Issue: The code only handles cases where:

1. curl succeeds (exit code 0)
2. JSON parsing succeeds
3. The JSON contains a message field

However, it doesn't handle:

• HTTP errors where curl fails (non-zero exit code) but doesn't return JSON
• Network failures
• Malformed responses that aren't valid JSON
• Cases where the API returns an error without a message field

Example scenario: If curl can't reach api.github.com (network issue), the error at line 276 would just say "failed to resolve ref using unauthenticated API" without indicating it's a network problem.

Recommendation: Check the curl exit code and stderr separately before attempting JSON parsing. Add more specific error messages for different failure modes (network, HTTP errors, JSON parsing failures, API errors).

[Copilot]

[nitpick] Rate limit information is mentioned in documentation but the code doesn't parse rate limit headers from GitHub's API response.

Issue: The GitHub API returns rate limit information in response headers (X-RateLimit-Remaining, X-RateLimit-Reset), but curl is not configured to capture these headers. This means:

1. Users won't know how many API calls they have left
2. The code can't proactively warn users before hitting the limit
3. The 60 requests/hour limit mentioned in docs is not enforced or tracked

Impact: Users may hit rate limits unexpectedly and get cryptic error messages.

Recommendation:

1. Use curl -i to capture headers along with response body
2. Parse and log rate limit headers
3. Consider warning users when remaining requests are low
4. This addresses a "Future Improvement" mentioned in the documentation (line 87)

[Copilot]

The test command path in the documentation is hardcoded to a specific CI environment path.

Issue: Line 61 shows:

cd /home/runner/work/gh-aw/gh-aw

This path is specific to GitHub Actions runners and won't work for:

• Local development machines
• Other CI systems
• Users following the documentation

Recommendation: Use a relative path or make it clear this is just an example:

# From the repository root
go test ./pkg/parser -run TestJSONParsing -v

Suggested change

	cd /home/runner/work/gh-aw/gh-aw
	# From the repository root

[Copilot]

The new getRequiredActionPin function only validates system actions (actions/checkout and actions/github-script), but this pattern is inconsistent with many other places in the codebase where GetActionPin is called directly without validation.

Issue: Many other critical actions lack validation:

• actions/cache (cache.go, lines 242, 349)
• actions/upload-artifact (compiler_yaml.go, artifacts.go, etc.)
• actions/download-artifact (artifacts.go, line 38)
• actions/checkout (compiler_yaml.go, line 174)
• actions/github-script (compiler_yaml.go, multiple locations)

Why this matters: If action pins are missing, these would produce empty uses: fields, causing compilation errors.

Recommendation: Either:

1. Apply getRequiredActionPin consistently to all system actions throughout the codebase, OR
2. Remove this validation since GetActionPin is designed to return empty string for missing pins (which is the expected behavior for custom action repos per runtime_setup.go:357-367)

Given the codebase pattern where GetActionPin is designed to gracefully return empty string for custom actions, the current selective validation appears to be an over-engineered solution.

Suggested change

	// getRequiredActionPin gets an action pin and returns an error if it's not found
	// This ensures that required system actions always have pins defined
	func getRequiredActionPin(actionRepo string) (string, error) {
	pin := GetActionPin(actionRepo)
	if pin == "" {
	return "", fmt.Errorf("action pin not found for %s - this action must be added to .github/aw/actions-lock.json", actionRepo)
	}
	return pin, nil
	}

[Copilot]

The base64 decoding uses an external base64 command which may not be available in all environments where this code runs.

Issue: The code executes exec.Command("base64", "-d") which assumes:

1. The base64 command-line utility is installed
2. It supports the -d flag (GNU coreutils vs BSD variants)
3. The command is in the system PATH

Impact: The fallback code path could fail on systems that don't have the base64 utility, even though Go's standard library provides encoding/base64 which would work everywhere.

Recommendation: Use Go's built-in encoding/base64.StdEncoding.DecodeString() instead of shelling out to an external command. This would:

• Be more portable
• Remove a dependency on external tools
• Be faster (no process spawning)
• Be more secure (no shell command injection risks)

Example:

import "encoding/base64"

decoded, err := base64.StdEncoding.DecodeString(contentBase64)
if err != nil {
    return nil, fmt.Errorf("failed to decode base64 content: %w", err)
}
return decoded, nil

[Copilot]

The curl command uses -s (silent) flag but doesn't use -f (fail) flag, which can lead to confusing behavior.

Issue: When curl encounters HTTP errors (404, 500, etc.) without the -f flag:

• It still exits with code 0 (success)
• It returns the HTML error page in the response body
• The code tries to parse this HTML as JSON, leading to confusing error messages

Example scenario:

1. User requests a non-existent file
2. GitHub returns 404 with HTML error page
3. Code tries to parse HTML as JSON → "failed to parse JSON response"
4. User sees JSON parsing error instead of "file not found"

Recommendation: Add the -f flag to curl commands to make them fail on HTTP errors:

cmd := exec.Command("curl", "-s", "-f", "-H", "Accept: application/vnd.github.v3+json", url)

This way, HTTP errors will cause curl to exit with non-zero code, and you can handle them separately before attempting JSON parsing.

Suggested change

	cmd := exec.Command("curl", "-s", "-H", "Accept: application/vnd.github.v3+json", url)
	cmd := exec.Command("curl", "-s", "-f", "-H", "Accept: application/vnd.github.v3+json", url)

[pelikhan]

@copilot lint

[Copilot]

@copilot lint

Linting passes ✅ Also addressed several code review suggestions in 4c638dc:

• Replaced external base64 -d with Go's native encoding/base64 for portability
• Added -f flag to curl for better HTTP error handling
• Fixed documentation paths to use relative references