[ab-advisor] Experiment campaign for weekly-blog-post-writer: A/B test prefetch_strategy

[github-actions]

🧪 Experiment Campaign: weekly-blog-post-writer

Workflow file: .github/workflows/weekly-blog-post-writer.md
Selected dimension: prefetch_strategy
Triggered by: ab-testing-advisor on 2026-06-11

---

Background

The Weekly Blog Post Writer is a 405-line, 8-step workflow that gathers GitHub release data and merged PRs during its own agent turns (Steps 2 and 3) via two sequential tool calls before any writing begins. Prefetch strategy was randomly selected as the test dimension. By pre-fetching releases and merged PRs in steps: before the agent starts, we can eliminate these tool round-trips entirely — potentially reducing turn count and overall run cost. This test establishes a baseline understanding of prefetch savings on a moderately complex, multi-step data-gathering workflow.

Hypothesis

H0 (null): Pre-fetching GitHub releases and merged PRs in setup steps does NOT reduce agent turn count or total run duration compared to the agent fetching lazily via tools.

H1 (alternative): Eager pre-fetching reduces agent turn count by ≥20% and total run duration by ≥15% by eliminating 2 GitHub tool round-trips at the start of every run.

Experiment Configuration

Add the following experiments: block to the workflow frontmatter (rich object form, self-documenting):

experiments:
  prefetch_strategy:
    variants: [lazy, eager]
    description: "Tests whether pre-fetching GitHub releases and merged PRs in setup steps reduces agent turn count and run duration vs. letting the agent fetch lazily via tools"
    hypothesis: "H0: no change in agent_turn_count. H1: eager pre-fetching reduces agent turns by ≥20% and duration by ≥15% by eliminating tool round-trips for data gathering"
    metric: agent_turn_count
    secondary_metrics: [run_duration_ms, pr_creation_rate]
    guardrail_metrics:
      - name: pr_creation_success_rate
        direction: min
        threshold: 0.80
    min_samples: 10
    weight: [50, 50]
    start_date: "2026-06-11"
    issue: #aw_campaign_wbpw

Variant descriptions:

• lazy: Current baseline — the agent calls list_releases and list_pull_requests tools during its own turns (Steps 2 and 3)
• eager: Releases and merged PRs are pre-fetched in steps: before the agent starts; the agent reads from pre-fetched JSON files at /tmp/gh-aw/agent/releases.json and /tmp/gh-aw/agent/merged-prs.json instead of calling tools

Workflow Changes Required

View full diff details

1. Add conditional steps: pre-fetch block (after existing frontmatter, before ---)

+steps:
+  {{#if experiments.prefetch_strategy == "eager" }}
+  - name: Pre-fetch GitHub data
+    run: |
+      mkdir -p /tmp/gh-aw/agent
+      gh api repos/${{ github.repository }}/releases \
+        --jq '[.[] | select(.draft == false)] | .[0:10]' \
+        > /tmp/gh-aw/agent/releases.json
+      gh pr list --repo ${{ github.repository }} --state merged --limit 30 \
+        --json number,title,mergedAt,body,labels,url \
+        > /tmp/gh-aw/agent/merged-prs.json
+      echo "Pre-fetched $(jq length /tmp/gh-aw/agent/releases.json) releases"
+      echo "Pre-fetched $(jq length /tmp/gh-aw/agent/merged-prs.json) merged PRs"
+  {{/if}}

2. Wrap Step 2 with a conditional block

Before:

### Step 2: Review Recent Releases

Use the GitHub `list_releases` tool to fetch all releases in the repository. Look for any releases published in the past 7 days.

After:

{{#if experiments.prefetch_strategy == "eager" }}
### Step 2: Review Recent Releases (Pre-fetched)

Release data has already been fetched during setup. Read it from file:

```bash
cat /tmp/gh-aw/agent/releases.json

Look for any releases published in the past 7 days.
{{else}}

Step 2: Review Recent Releases

Use the GitHub list_releases tool to fetch all releases in the repository. Look for any releases published in the past 7 days.
{{/if}}

#### 3. Wrap Step 3 with a conditional block

**Before:**
```markdown
### Step 3: Review Recent Pull Requests

Use the GitHub `list_pull_requests` tool to fetch pull requests that were **merged** in the past 7 days.

After:

{{#if experiments.prefetch_strategy == "eager" }}
### Step 3: Review Recent Pull Requests (Pre-fetched)

Merged PR data has already been fetched during setup. Read it from file:

```bash
cat /tmp/gh-aw/agent/merged-prs.json

Look for PRs merged in the past 7 days.
{{else}}

Step 3: Review Recent Pull Requests

Use the GitHub list_pull_requests tool to fetch pull requests that were merged in the past 7 days.
{{/if}}

</details>

### Success Metrics

| Metric | Type | Target |
|--------|------|--------|
| `agent_turn_count` | Primary | Reduce by ≥20% in `eager` variant |
| `run_duration_ms` | Secondary | Reduce by ≥15% |
| `pr_creation_rate` | Secondary | Maintain ≥90% |
| `pr_creation_success_rate` | Guardrail | Must not drop below 80% |
| `empty_output_rate` | Guardrail | Must remain < 5% |

### Statistical Design

- **Variants**: `lazy` (baseline), `eager` (treatment)
- **Assignment**: Round-robin via `gh-aw` experiments runtime (cache-based)
- **Minimum runs per variant**: 10
- **Expected experiment duration**: ~20 weeks (weekly trigger, 50/50 split → ~1 run/week/variant)
- **Analysis approach**: Mann-Whitney U (non-parametric; robust to small samples typical of weekly workflows)

> ⚠️ **Note**: Weekly workflows accumulate data slowly. Consider triggering additional `workflow_dispatch` runs to accelerate data collection, or lowering `min_samples` to 6 if the effect size is clearly visible earlier.

### Implementation Steps

- [ ] Add `experiments:` section to frontmatter
- [ ] Add conditional `steps:` pre-fetch block using `{{#if experiments.prefetch_strategy == "eager" }}` (value-comparison form — never use the internal `__GH_AW_EXPERIMENTS__` env-var syntax)
- [ ] Add conditional blocks to Steps 2 and 3 in the prompt body using `{{#if experiments.prefetch_strategy == "eager" }}...{{else}}...{{/if}}`
- [ ] Run `gh aw compile weekly-blog-post-writer` to regenerate lock file
- [ ] Monitor experiment artifact uploaded per run to `/tmp/gh-aw/agent/experiments/state.json`
- [ ] After ≥10 runs per variant, analyze variant distribution via workflow run artifacts
- [ ] Document findings and promote winning variant

### References

- [A/B Testing in gh-aw](https://github.com/github/gh-aw/blob/main/.github/aw/github-agentic-workflows.md)
- Workflow file: `.github/workflows/weekly-blog-post-writer.md`


<!-- gh-aw-tracker-id: ab-testing-advisor -->




> Generated by [🧪 Daily A/B Testing Advisor](https://github.com/github/gh-aw/actions/runs/27344776762) · 384.4 AIC · ⌖ 33.5 AIC · ⊞ 22.4K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Fab-testing-advisor%22&type=issues)
> - [x] expires <!-- gh-aw-expires: 2026-06-25T12:07:17.548Z --> on Jun 25, 2026, 4:07 AM UTC-08:00

<!-- gh-aw-agentic-workflow: Daily A/B Testing Advisor, gh-aw-tracker-id: ab-testing-advisor, engine: copilot, version: 1.0.60, model: claude-sonnet-4.6, id: 27344776762, workflow_id: ab-testing-advisor, run: https://github.com/github/gh-aw/actions/runs/27344776762 -->

<!-- gh-aw-workflow-id: ab-testing-advisor -->
<!-- gh-aw-workflow-call-id: github/gh-aw/ab-testing-advisor -->
<!-- gh-aw-close-key: ab-testing-advisor -->