[ab-advisor] Experiment campaign for smoke-copilot-sub-agents: A/B test sub_agent_strategy

[github-actions]

🧪 Experiment Campaign: smoke-copilot-sub-agents

Workflow file: .github/workflows/smoke-copilot-sub-agents.md
Selected dimension: sub_agent_strategy
Triggered by: ab-testing-advisor on 2026-07-23

---

Background

This workflow is a smoke test for Copilot SDK inline sub-agents: it calls three fixed sub-agents, verifies exact identity strings, and creates a PASS/FAIL issue for each run. I chose sub_agent_strategy because the workflow is explicitly about sub-agent orchestration and recent runs show reliability instability, making it a good candidate to quantify whether the current decomposition is helping or hurting pass rate and runtime efficiency.

Hypothesis

H0: Changing the sub-agent strategy does not improve smoke-test pass rate compared to the current inline-sub-agent baseline.

H1: A simpler or more tightly constrained execution strategy improves pass rate by at least 15 percentage points and/or reduces operational failures without increasing empty/invalid outputs.

View Details

Recent 10 visible runs for smoke-copilot-sub-agents.lock.yml show 2 successes and 8 failures, so reliability is the dominant business outcome. The workflow prompt is minimal and highly constrained; the main thing to test is not writing style but orchestration structure and how much latitude the parent agent has while invoking sub-agents.

Characterization used for this campaign:

• purpose: verify that inline sub-agents resolve to the expected model identities and that the parent agent can orchestrate them correctly
• triggers: schedule, workflow_dispatch
• engine: copilot with copilot-sdk: true, bare: true, parent model gpt-5.3-codex
• prompt_density: minimal — short task list with exact-response constraints
• tools: safe output issue creation plus inline sub-agent calls
• outputs: one smoke-test issue containing per-agent expected/actual values and overall PASS/FAIL
• quality_signals: strict: true, explicit exact-match assertions, explicit ban on unnecessary tools, but weak observability about which orchestration mode was used
• infrastructure status: analysis_type, tags, and notify are already implemented and surfaced in experiment runtime metadata; remaining work is analytics/reporting integration rather than schema expansion

Experiment Configuration

Add the following experiments: block to the workflow frontmatter (use the rich object form so all metadata is self-documenting):

experiments:
  sub_agent_strategy:
    variants: [inline_strict, delegated_sequential, single_agent_control]
    description: "Measure whether inline sub-agent orchestration is the best reliability/cost tradeoff for a model-identity smoke test."
    hypothesis: "H0: no change in pass_rate. H1: delegated_sequential or single_agent_control improves pass_rate by >= 0.15 absolute versus inline_strict."
    metric: pass_rate
    secondary_metrics: [run_duration_seconds, output_validity_rate]
    guardrail_metrics:
      - name: empty_output_rate
        direction: min
        threshold: 0.01
      - name: false_pass_rate
        direction: min
        threshold: 0
    min_samples: 30
    weight: [34, 33, 33]
    start_date: "2026-07-23"
    issue: <this_issue_number>

Variant descriptions:

• inline_strict: Preserve the current inline sub-agent structure, but wrap the instruction text specifically around exact single-call behavior and exact string checking.
• delegated_sequential: Keep sub-agents, but force the parent to execute and validate them one at a time in a tighter sequence before composing the issue.
• single_agent_control: Remove sub-agent calls from the prompt path and have the parent agent produce a control run format that reports not_invoked for comparison; useful for measuring whether failures are caused by orchestration itself versus the reporting/output path.

Workflow Changes Required

List the exact changes needed in the workflow markdown body to implement the experiment using handlebars conditional blocks. Always compare against a specific variant value — the correct syntax is {{#if experiments.<name> == "<variant>" }}...{{else}}...{{/if}}. The compiler automatically expands experiments.<name> references at compile time; never write the internal env-var form (__GH_AW_EXPERIMENTS__<NAME>___<variant>) directly.

@@
 ## Tasks
 
-1. Call `haiku-whoami`, `mini-whoami`, and `nano-whoami` exactly once each.
-2. Ask each sub-agent exactly this question: `who am i?`
-3. Check the exact responses:
-   - `haiku-whoami` → `claude-haiku-4.5`
-   - `mini-whoami` → `gpt-5-mini`
-   - `nano-whoami` → `gpt-5-nano`
-4. Do not use any other agent or any unnecessary tool calls.
+{{#if experiments.sub_agent_strategy == "single_agent_control" }}
+1. Do not call any sub-agent.
+2. Produce the issue in the same format, but mark each agent as `not_invoked`.
+3. Set overall status to FAIL.
+4. Do not use unnecessary tool calls.
+{{else}}
+1. Call `haiku-whoami`, `mini-whoami`, and `nano-whoami` exactly once each.
+2. Ask each sub-agent exactly this question: `who am i?`
+3. Check the exact responses:
+   - `haiku-whoami` → `claude-haiku-4.5`
+   - `mini-whoami` → `gpt-5-mini`
+   - `nano-whoami` → `gpt-5-nano`
+4. {{#if experiments.sub_agent_strategy == "delegated_sequential" }}Execute and validate each sub-agent one at a time before moving to the next.{{else}}Validate all three results and keep the current inline orchestration behavior.{{/if}}
+5. Do not use any other agent or any unnecessary tool calls.
+{{/if}}
@@
 ## Output
 
 Always create an issue titled **"Smoke Test: Copilot Sub Agents - ${{ github.run_id }}"** with:
 - One line per sub-agent showing expected value, actual value, and ✅/❌
 - Overall status: PASS only if all three exact matches succeed, otherwise FAIL
+- Experiment variant: `{{ experiments.sub_agent_strategy }}`
 - Run URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

Success Metrics

Metric	Type	Target
pass_rate	Primary	+15 percentage points absolute vs baseline
run_duration_seconds	Secondary	No material regression; ideally <= current median
output_validity_rate	Secondary	>= 99% structured issue creation success
empty_output_rate	Guardrail	Must not exceed 1%

Statistical Design

• Null hypothesis: The variant does not improve pass_rate compared to baseline.
• Primary metric: pass_rate
• Secondary metrics: run_duration_seconds, output_validity_rate
• Guardrail metrics: empty_output_rate <= 1%, false_pass_rate = 0
• Minimum detectable effect: +15 percentage points absolute improvement in pass rate
• Required sample size: ~30 runs per variant for an 80% power first-pass directional decision, then confirm with more data if effect is marginal
• Variants: inline_strict, delegated_sequential, single_agent_control
• Assignment: Round-robin via gh-aw experiments runtime (cache-based)
• Minimum runs per variant: 30
• Expected experiment duration: ~90 scheduled runs total, or about 90 days at one scheduled run/day unless supplemented with manual dispatches
• Analysis approach: proportion test for pass_rate, descriptive median comparison for runtime

Implementation Steps

• Add experiments: section to frontmatter
• Add conditional blocks to workflow prompt body using {{#if experiments.sub_agent_strategy == "<variant>" }} (value-comparison form — never use the internal __GH_AW_EXPERIMENTS__ env-var syntax)
• Run gh aw compile smoke-copilot-sub-agents to regenerate lock file
• Monitor experiment artifact uploaded per run to /tmp/gh-aw/agent/experiments/state.json
• After sufficient runs, analyze variant distribution via workflow run artifacts
• Document findings and promote winning variant

References

• A/B Testing in gh-aw
• Workflow file: .github/workflows/smoke-copilot-sub-agents.md

Generated by 🧪 Daily A/B Testing Advisor · gpt54 18.3 AIC · ⌖ 9.93 AIC · ⊞ 7.2K · ◷

• expires on Aug 6, 2026, 3:13 AM UTC-08:00