TCP-MT-7: Run scale stress test for TCP per-task filtering at 500+ tool corpus

[samscarrow]

Notion Work Item

TCP-MT-7 (Measurement Track) — TCP Harness Prototype project

Objective

Run a scale stress test for TCP per-task filtering by expanding the current 90-tool corpus to 500+ tools with realistic synthetic supplements, then re-running the benchmark to test whether candidate-set size and correctness still hold at production-like scale.

Benchmark Design

Corpus Expansion

• Start from the current 90 real MCP-tool descriptors.
• Generate realistic synthetic tools to reach 500+ total tools.
• Synthetic tools must have plausible names, descriptions, and capability flags.
• The synthetic corpus must include near-collisions, adjacent capabilities, tempting-but-wrong candidates, mixed permission profiles, and realistic clusters so the filter is genuinely stressed.

Run Conditions

• Model: claude-sonnet-4-6 only
• Environment: offline only
• Repetitions: 3 reps
• Compare ungated vs per-task filtered behavior at 500+ scale

Primary Question

Does per-task filtering still produce roughly 1-3 tools per task rather than degrading to dozens, and does correctness still hold at this larger corpus size?

Secondary Questions

• Does preflight still pass cleanly at 500+ scale?
• Do token savings continue to scale roughly linearly with corpus growth?

Kill Condition

Stop if the synthetic corpus cannot be made realistic enough to stress the filter meaningfully, or if per-task filtering drops below 80% correctness at scale.

Required Evidence

• Corpus construction method and realism rationale
• Total corpus size and category distribution
• Candidate-set size per task
• Task correctness at 500+ scale
• Preflight outcome
• Input token deltas vs ungated
• Clear conclusion about whether per-task filtering remains production-credible at scale

Completion Contract

• Epistemic. Take measurements as specified. Report results with: (1) methodology, (2) raw data or summary statistics, (3) any measurement caveats or confounds encountered.
• No PR expected unless the objective requires instrumentation code.
• Handoff: Post measurements + methodology to this issue → Close issue → Stop.

---

Repo: git-scarrow/tool-capability-protocol | Branch: main | Environment: dev

[samscarrow]

Lab-to-Factory Handoff

Work Item: TCP-MT-7 — Scale Stress Test for Per-Task Filtering (500+ Tools)
Type: Measurement Track (Epistemic)
Provider: Claude Code (claude-sonnet-4-6)
Concurrency Group: tcp-harness

---

Mission

Run the final decisive TCP benchmark: a scale stress test that determines whether per-task cold-path filtering still holds when the tool corpus grows from 90 real MCP tools to 500+ total tools. This is the last measurement before archiving this line.

Environment

• Repo: git-scarrow/tool-capability-protocol
• Branch: main
• Environment: dev
• Model: claude-sonnet-4-6 only — no other models
• Mode: Offline environment only
• Repetitions: 3 reps per condition
• Base corpus: Current 90 real MCP-tool descriptors in the repo

Strategy

Use claude-sonnet-4-6 only. Offline environment only. 3 reps. Expand the corpus to 500+ tools by supplementing the 90 real MCP tools with synthetic tools that have realistic names, descriptions, and capability flags. Re-run the per-task filtering benchmark at this scale. Primary question: does per-task filtering still narrow to 1-3 tools per task rather than degrading to dozens, and does correctness still hold? Secondary: does preflight still pass, and do token savings scale linearly? Treat realism of the synthetic corpus as part of the benchmark validity, not just a data-generation step.

Detailed Spec

Phase 1: Corpus Expansion

1. Load the existing 90 real MCP-tool descriptors from the repo.
2. Generate realistic synthetic tools to bring the total to 500+ tools. Each synthetic tool must have:
   • A plausible, unique name following MCP naming conventions
   • A realistic description reflecting a believable capability
   • Appropriate capability flags consistent with the tool's described function
3. The synthetic corpus must include adversarial diversity:
   • Near-collisions: Tools with overlapping names or descriptions that could confuse the filter
   • Adjacent capabilities: Tools that do similar-but-distinct things
   • Tempting-but-wrong candidates: Tools that look relevant to certain tasks but are incorrect selections
   • Mixed permission profiles: Varied read/write/admin capability patterns
   • Realistic clusters: Groups of tools belonging to the same logical domain (e.g., 15 database tools, 20 CI/CD tools, etc.)
4. Document the corpus construction method and provide a rationale for why the synthetic corpus is realistic enough to stress the filter meaningfully.
5. Record total corpus size and category distribution.

Phase 2: Benchmark Execution

1. Run the per-task filtering benchmark against the 500+ tool corpus:
   • 3 repetitions per condition
   • Compare ungated (all tools presented) vs per-task filtered behavior at 500+ scale
2. For each task, record:
   • Candidate-set size after per-task filtering (target: 1-3 tools)
   • Whether the correct tool(s) were included in the candidate set
   • Task correctness (correct tool selected and used properly)
3. Run preflight checks at 500+ scale and record pass/fail.
4. Measure input token counts for ungated vs filtered runs and compute deltas.

Phase 3: Analysis

1. Compute and report:
   • Primary: Candidate-set size per task after filtering (mean, median, max). Target is 1-3.
   • Primary: Task correctness at 500+ scale. Must remain ≥ 80% or kill condition is met.
   • Secondary: Preflight pass/fail status at this scale
   • Secondary: Input token savings (filtered vs ungated) and whether savings scale roughly linearly with corpus growth compared to the 90-tool baseline
2. Provide a clear, unambiguous conclusion: does per-task filtering remain production-credible at 500+ tools, or does scale break the current approach?

Kill / Stop Condition

Stop if:

• The synthetic corpus cannot be made realistic enough to stress the filter in a meaningful way (document why and stop)
• Per-task filtering degrades below 80% correctness at 500+ tool scale

If the kill condition is triggered, report what was observed and stop. Do not attempt workarounds.

Success Condition

This work item passes if it shows that per-task filtering remains narrow (1-3 candidates), correct (≥ 80%), and operationally sane at 500+ tools — or cleanly falsifies that claim with evidence.

Required Deliverables

Post all of the following as findings on this issue:

1. Corpus construction method and realism rationale
2. Total corpus size and category distribution
3. Candidate-set size per task (mean, median, max across 3 reps)
4. Task correctness at 500+ scale (with comparison to 90-tool baseline if available)
5. Preflight outcome at 500+ scale
6. Input token deltas: filtered vs ungated at 500+, compared to 90-tool baseline
7. Clear conclusion statement

Metrics

• Primary: Candidate-set size per task after per-task filtering (target 1-3 tools), task correctness at 500+ tool scale
• Secondary: Preflight pass/fail status, input token savings vs ungated, and whether token savings scale roughly linearly with corpus growth

---

Lab Handoff Protocol

Completion Contract (Epistemic)

Posting findings to this GitHub Issue constitutes completion. No PRs are expected. Proceed directly to the Handoff Sequence once all required deliverables are posted.

Handoff Sequence

Step	Action
1. Report	Post findings, raw logs, and a summary as a comment on this issue.
2. Merge	Confirm all associated PRs are merged (skip — Epistemic).
3. Close	Close this GitHub Issue with a summary comment.
4. Stop	Stop immediately. Do not touch the Notion Work Item.