TCP-MT-9: Produce bootstrap 95% CIs for each EXP-3 arm's correctness rate and mean input-token cost; test whether the Arm A vs Arm C correctness difference (+6 pp) is statistically significant at n ≈ 36/arm; compute the minimum detectable effect size at 80% power.

[samscarrow]

Notion Work Item

Item Name: TCP-MT-9
Type: Measurement Track
Project: TCP Harness Prototype

Objective

Produce bootstrap 95% CIs for each EXP-3 arm's correctness rate and mean input-token cost; test whether the Arm A vs Arm C correctness difference (+6 pp) is statistically significant at n ≈ 36/arm; compute the minimum detectable effect size at 80% power.

Dataset

EXP-3 raw trial JSON files at repo root: layered-results.json, layered-results-v2.json, layered-results-imp5.json.

Kill/Stop Condition

If all four arms' 95% CIs overlap completely (i.e., no pairwise difference excludes zero), the EXP-3 sample size is insufficient to distinguish any arm — halt and report 'underpowered, need N = X per arm' with the power-analysis estimate.

Completion Contract

• Epistemic. Take measurements as specified. Report results with: (1) methodology, (2) raw data or summary statistics, (3) any measurement caveats or confounds encountered.
• No PR expected unless the objective requires instrumentation code. If instrumentation is needed, one PR for the instrumentation is acceptable — keep it separate from the measurement report itself.
• Handoff: Post measurements + methodology to this GitHub Issue → Close issue → Stop.

Metrics

• Per-arm bootstrap 95% CI (correctness %)
• Per-arm bootstrap 95% CI (mean input tokens)
• Pairwise Δ correctness with CI for A-vs-C and C-vs-D
• p-value or equivalent for A-vs-C difference
• Minimum detectable effect size at 80% power given n ≈ 36/arm

Expected Output Artifacts

1. bootstrap_ci_report.md — Human-readable summary
2. bootstrap_ci_results.json — Structured results
3. bootstrap_analysis.py (or .ts) — Reproducible analysis script

[samscarrow]

🎯 Dispatch Prompt — TCP-MT-9 → Claude Code

Generated by Prompt Architect | Target: Claude Code | Type: Measurement Track

---

Objective

Produce bootstrap 95% confidence intervals for each EXP-3 arm's correctness rate and mean input-token cost. Test whether the Arm A vs Arm C correctness difference (+6 pp) is statistically significant at n ≈ 36/arm. Compute the minimum detectable effect size (MDES) at 80% power.

This is a pure analysis track — no new LLM API calls are needed.

Completion Contract

• Epistemic. Take measurements as specified. Report results with: (1) methodology, (2) raw data or summary statistics, (3) any measurement caveats or confounds encountered.
• No PR expected unless the objective requires instrumentation code. If instrumentation is needed, one PR for the instrumentation is acceptable — keep it separate from the measurement report itself.
• Handoff: Post measurements + methodology to this GitHub Issue → Close issue → Stop.

Data Sources

Locate the EXP-3 trial-level data. Expected locations:

• layered-results.json (repo root)
• layered-results-v2.json (repo root)
• layered-results-imp5.json (repo root)

If not found at repo root, search recursively: find . -name 'layered-results*.json' -o -name '*exp*3*result*'. The files contain per-trial records from a 4-arm behavioral-description ablation experiment (144 total trials, ~36/arm). Each trial should have an arm assignment, a correctness indicator (binary), and an input token count.

EXP-3 point estimates for reference (use to sanity-check parsed data):

Arm	Description Style	Tool Count	Correctness	Mean Input Tokens
A	Realistic ungated	119	83%	26,021
B	Realistic filtered	46	75%	16,681
C	Minimal filtered	46	89%	9,890
D	Brief filtered	46	89%	10,194

Method (execute in order)

1. Load & parse the trial-level JSON files. Identify per-trial: arm assignment, correctness (binary 0/1), input token count. Validate that arm counts are ~36 each and point estimates match the reference table above (within rounding).

2. Bootstrap CIs (correctness) — For each arm, draw 10,000 bootstrap resamples (with replacement, size = arm's trial count). Compute mean correctness per resample. Report BCa 95% CI (bias-corrected and accelerated). If BCa is impractical in the chosen library, use percentile method and note it.

3. Bootstrap CIs (token cost) — Same procedure for mean input tokens per arm.

4. Pairwise difference tests — For key comparisons A vs C and C vs D:

   • Compute the bootstrap distribution of Δ (difference in means).
   • Report the 95% CI for Δ and whether it excludes zero.
   • Report a two-sided bootstrap p-value (proportion of resamples where Δ ≤ 0 or ≥ 0, doubled).

5. Power analysis / MDES — Given n ≈ 36/arm and observed variance:

   • Compute the minimum detectable effect size (in pp of correctness) at α = 0.05, power = 0.80 using a two-proportion z-test approximation.
   • Compute the required sample size per arm to detect a 6 pp difference at the same power level.

6. Write output artifacts:

   • bootstrap_ci_report.md — Human-readable summary with tables of CIs, pairwise tests, power analysis, and interpretation.
   • bootstrap_ci_results.json — Structured results: per-arm CIs, pairwise deltas with CIs and p-values, MDES, required N.
   • bootstrap_analysis.py — The reproducible analysis script.

Implementation Notes

• Use Python with numpy and scipy. Install via pip install numpy scipy if not already available.
• For BCa bootstrap: scipy.stats.bootstrap (scipy ≥ 1.7) supports BCa method natively.
• Seed the RNG (np.random.seed(42)) for reproducibility.
• The script must be self-contained and runnable: python bootstrap_analysis.py should produce both the .md and .json outputs.

Kill Condition

If all four arms' 95% CIs overlap completely — meaning no pairwise correctness difference excludes zero — the sample is underpowered for any claim. In that case, halt and report:

• "Underpowered at n ≈ 36/arm"
• The MDES and required N per arm for 6 pp detection
• Recommendation to re-run EXP-3 at the computed sample size

Success Criteria

• 95% CIs reported for all four arms on both correctness and token cost.
• A vs C pairwise Δ CI and p-value reported; significance conclusion stated.
• C vs D pairwise Δ CI reported; conclusion on practical equivalence stated.
• MDES at n = 36/arm computed and documented.
• Required sample size for 6 pp detection at 80% power computed.
• All numbers reproducible from the committed script + data files.

Terminal Action

After generating all artifacts:

1. git add bootstrap_analysis.py bootstrap_ci_report.md bootstrap_ci_results.json
2. git commit -m "feat(analysis): TCP-MT-9 bootstrap CI analysis for EXP-3 arms"
3. git push
4. Post the full content of bootstrap_ci_report.md as a comment on this issue.
5. Close this issue.
6. Stop.