Skip to content

[ab-advisor] Add sub_agent_strategy experiment to smoke-copilot-sub-agents #47557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 7 commits into from
Jul 23, 2026
Merged

[ab-advisor] Add sub_agent_strategy experiment to smoke-copilot-sub-agents #47557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e85d733
Initial plan
Copilot Jul 23, 2026
cad6cf7
Add sub_agent_strategy experiment to smoke-copilot-sub-agents
Copilot Jul 23, 2026
197d640
Fix template marker placement in smoke sub-agent experiment prompt
Copilot Jul 23, 2026
d09fa53
Warn on mid-line template separators in workflow prompts
Copilot Jul 23, 2026
2112212
docs(adr): draft ADR-47557 for A/B experiment infrastructure in smoke…
github-actions[bot] Jul 23, 2026
df8b024
fix: templateSeparatorPattern grammar and smoke workflow variant blocks
Copilot Jul 23, 2026
2c97cc1
fix: address grill-with-docs review threads on experiment spec
Copilot Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 123 additions & 1 deletion .github/workflows/smoke-copilot-sub-agents.lock.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

41 changes: 40 additions & 1 deletion .github/workflows/smoke-copilot-sub-agents.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,25 @@ engine:
id: copilot
copilot-sdk: true
bare: true
experiments:
sub_agent_strategy:
variants: [inline_strict, delegated_sequential, single_agent_control]
description: "Measure whether inline sub-agent orchestration is the best reliability/cost tradeoff for a model-identity smoke test."
hypothesis: "H0: no change in pass_rate between inline_strict and delegated_sequential. H1: delegated_sequential improves pass_rate by >= 0.15 absolute versus inline_strict. Note: single_agent_control is a synthetic negative baseline (always FAIL by design) excluded from H1 comparisons."
metric: pass_rate
secondary_metrics: [run_duration_seconds, output_validity_rate]
guardrail_metrics:
- name: empty_output_rate
direction: min
threshold: 0.01
- name: false_pass_rate
direction: min
threshold: 0.05
min_samples: 30
weight: [34, 33, 33]
start_date: "2026-07-23"

@github-actions github-actions Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] No end_date is defined — the experiment has no stopping rule, which means it will run indefinitely unless manually shut down.

💡 Suggestion

Add end_date or a max_samples field to the experiment spec. Without a stopping criterion, the experiment will either accumulate unbounded data or be forgotten and never analyzed. If the framework does not yet support end_date, this is worth a follow-up issue.

@copilot please address this.

Copilot AI Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the latest commit. Added end_date: "2026-10-23" (3 months from start_date) as a stopping criterion so the experiment does not run indefinitely.

end_date: "2026-10-23"
issue: 47551
safe-outputs:
create-issue:
expires: 2h
Expand All @@ -34,19 +53,39 @@ features:

## Tasks

{{#if experiments.sub_agent_strategy == 'single_agent_control' }}
1. Do not call any sub-agent.
2. Produce the issue in the same format, but mark each agent as `not_invoked`.
3. Set overall status to FAIL.

@github-actions github-actions Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] single_agent_control forces FAIL unconditionally — this contaminates pass_rate comparisons because its baseline is always 0, making any improvement over it trivially measurable and meaningless for H1.

💡 Suggestion

A true control arm should run the same task without modification and let the outcome be natural. If the goal is "no sub-agents", consider excluding this arm from the pass_rate primary metric and tracking it separately, or rename it to clarify the arm is a synthetic negative control (not a performance baseline).

@copilot please address this.

Copilot AI Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in the latest commit. The hypothesis now explicitly calls out that single_agent_control is a synthetic negative baseline (always FAIL by design, not a performance variant) and that H1 compares delegated_sequential vs inline_strict only. This makes the experimental intent clear without changing the arm's behavior.

4. Do not use unnecessary tool calls.
{{/if}}
{{#if experiments.sub_agent_strategy == 'delegated_sequential' }}
1. Call `haiku-whoami`, `mini-whoami`, and `nano-whoami` exactly once each.
2. Ask each sub-agent exactly this question: `who am i?`
3. Check the exact responses:
- `haiku-whoami` → `claude-haiku-4.5`
- `mini-whoami` → `gpt-5-mini`
- `nano-whoami` → `gpt-5-nano`
4. Do not use any other agent or any unnecessary tool calls.
4. Execute and validate each sub-agent one at a time before moving to the next.
5. Do not use any other agent or any unnecessary tool calls.
{{/if}}
{{#if experiments.sub_agent_strategy == 'inline_strict' }}
1. Call `haiku-whoami`, `mini-whoami`, and `nano-whoami` exactly once each.
2. Ask each sub-agent exactly this question: `who am i?`
3. Check the exact responses:
- `haiku-whoami` → `claude-haiku-4.5`
- `mini-whoami` → `gpt-5-mini`
- `nano-whoami` → `gpt-5-nano`
4. Validate all three results and keep the current inline orchestration behavior.
5. Do not use any other agent or any unnecessary tool calls.
{{/if}}

## Output

Always create an issue titled **"Smoke Test: Copilot Sub Agents - ${{ github.run_id }}"** with:
- One line per sub-agent showing expected value, actual value, and ✅/❌
- Overall status: PASS only if all three exact matches succeed, otherwise FAIL
- Experiment variant: `{{ experiments.sub_agent_strategy }}`
- Run URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

## agent: `haiku-whoami`
Expand Down
Loading
Loading