TCP-IMP-24: Improve expected_tool_name derivation for single-survivor emission

[samscarrow]

Notion Work Item

TCP-IMP-24 — Lab Work Item: https://www.notion.so/851e38f03b4c498786300c387467c072

Parent project: TCP Harness Prototype (https://www.notion.so/335e7cc701d5817b9f1fc07d2af25f0d)

Objective

Improve the expected-tool derivation pipeline so post-Stage-4 filtering can emit defensible single-survivor expected_tool_name annotations on a non-trivial fraction of real traffic, without returning to weak prompt-similarity oracle behavior.

Context

• TCP-MT-21 measured 146 post-IMP-22 rows and found 0 emitted expected-tool annotations: every row abstained.
• Diagnostics were present and consistent; the failure was not missing telemetry.
• Stage 4 never converged to a single survivor. The smallest nonzero survivor set was 47 tools, with common abstention buckets like ambiguous_196_survivors, ambiguous_164_survivors, and no_survivors.
• Agreement metrics and baseline comparisons remain blocked until the derivation layer can produce a clean emitted subset.

Scope

• Inspect current Stage 1–4 request-feature-to-tool-survival logic and identify why real traffic remains at dozens to hundreds of survivors after Stage 4.
• Add a conservative precision-improving derivation path that can reduce candidate sets to a single defensible expected tool when evidence is strong.
• Preserve explicit abstention when evidence is insufficient; do not reintroduce weak prompt-similarity emission.
• Keep the existing derivation diagnostics (schema/version fields, candidate-set phase metadata) compatible with TCP-MT-21 reruns.
• Add or update tests covering both successful single-survivor emission and abstention cases.

Deliverables

1. Implementation that increases discriminative power of expected-tool derivation while preserving abstention as the default for weak evidence.
2. Tests proving:
   • strong evidence can emit one expected tool
   • ambiguous candidate sets still abstain
   • no-survivor cases are reported cleanly
   • diagnostics remain present for measurement
3. Short implementation note explaining which evidence source or filtering rule was added and why it is defensible.
4. Recommendation for whether TCP-MT-21 should be repeated immediately after the implementation lands.

Dataset

Post-TCP-IMP-22/23 decisions.jsonl rows and targeted fixtures that exercise ambiguous_N_survivors, no_survivors, and strong-evidence single-survivor cases.

Stop / Kill Condition

Stop if the only available path to emission is weak prompt similarity, manual relabeling, or a threshold tweak that cannot be defended from request/tool evidence. Do not manufacture coverage at the expense of annotation validity.

Completion Contract (Implementation)

• Code-touching. One or more PRs expected.
• Work identity lives in the Notion Work Item; branches are execution artifacts.
• Preferred new branch name: work/tcp-imp-24 (or similar; must reference TCP-IMP-24).
• Before pushing, verify the branch contains only work for this Work Item. Rebase or cherry-pick onto a clean branch if scopes are mixed.
• PR title/body must reference TCP-IMP-24.
• Handoff: All PRs merged → Close issue → Stop.

[samscarrow]

Prompt Architect handoff — TCP-IMP-24 → Claude Code

Recommended downstream model type

Claude Code / code-capable implementation agent with repository access. This is a code-touching implementation task, not a measurement-only task. One or more PRs are expected.

Operating contract

• Work item: TCP-IMP-24
• Repo: git-scarrow/tool-capability-protocol
• Base branch: main
• Preferred branch: work/tcp-imp-24
• Keep the branch clean: before pushing, verify that git log main..HEAD --oneline contains only commits for TCP-IMP-24. If scope is mixed, rebase or cherry-pick onto a clean branch.
• PR title/body must reference TCP-IMP-24.
• Handoff completion: all PRs merged → close this issue → stop.

Objective

Improve expected_tool_name derivation so post-Stage-4 filtering can emit defensible single-survivor annotations on a non-trivial fraction of real traffic while preserving explicit abstention for weak evidence.

The goal is precision-first discriminative power, not coverage inflation.

Current known state

From the current repository surface:

• The relevant derivation point is in tcp/proxy/cc_proxy.py, especially _compute_expected_tool_name.
• Current emission is intentionally conservative: it emits expected_tool_name only when survivor_count == 1 and survivor_names_sorted has exactly one item.
• Ambiguous sets abstain with reasons such as ambiguous_N_survivors; zero survivors abstain as no_survivors; mismatch cases abstain defensively.
• top_survivor_by_similarity is present as diagnostic-only and must remain diagnostic-only. Do not promote weak prompt similarity back into the oracle path.
• Existing tests in tests/unit/test_cc_proxy_response_tap.py explicitly protect the current abstention behavior and diagnostic fields.
• TCP-MT-21 found 146 post-IMP-22 rows, 0 emitted expected-tool annotations, and survivor sets that remained too broad after Stage 4.

Required implementation strategy

1. Inspect the Stage 1–4 request-feature-to-tool-survival path in tcp/proxy/cc_proxy.py.

   • Identify why real traffic still has dozens to hundreds of survivors after Stage 4.
   • Preserve existing decision-log schema/version fields and candidate-set phase metadata.
   • Do not remove existing diagnostic fields; extend them if needed.

2. Add a conservative evidence path after the existing survivor calculation and before final expected_tool_name derivation.

   • The path must be defensible from request/tool evidence.
   • It may reduce an ambiguous survivor set to one expected tool only when the evidence is strong enough that downstream measurement can treat the row as labeled.
   • It must explicitly abstain otherwise.

3. Prefer evidence sources with deterministic semantics over similarity.
   Strong candidates to investigate:

   • exact tool-use intent exposed in request-derived features
   • tool/server capability resolutions already computed by the Capability Resolution Gate
   • explicit connector/server/tool mentions that map unambiguously to one surviving tool
   • unique action/capability-to-tool mappings among the current survivor set
   • exact name/alias matches after normalization, only if they are unambiguous among survivors

4. Preserve abstention as the default.

   • Ambiguous candidate sets still abstain.
   • No-survivor cases still report cleanly.
   • Count/list mismatches still abstain defensively.
   • Do not use alphabetical choice, arbitrary thresholds, manual relabeling, or prompt-similarity rank as emission evidence.

Concrete code guidance

Implement a small, testable derivation layer rather than burying logic inline.

A reasonable shape:

@dataclasses.dataclass(frozen=True)
class _ExpectedToolDerivation:
    expected_tool_name: str | None
    derivation_source: str | None
    candidate_set_size: int
    abstain_reason: str | None
    # optional: evidence fields, if needed and backward-compatible

Then add one or more helper functions with clear names, for example:

def _derive_expected_tool_from_strong_evidence(meta: dict[str, Any]) -> _ExpectedToolDerivation | None:
    ...

This helper should:

• operate only over the current survivor set or a clearly documented candidate subset derived from it
• return a derivation only when exactly one defensible candidate remains
• otherwise return None and allow _compute_expected_tool_name to keep abstaining

If additional metadata is needed in the decision log, add it in a backward-compatible way:

• keep existing fields intact
• use new optional keys rather than changing semantics of existing keys
• keep old readers safe with .get(...)

Required tests

Update or add unit tests, likely in tests/unit/test_cc_proxy_response_tap.py and any adjacent derivation/gating test module.

Tests must cover:

1. Strong-evidence single-survivor emission

   • Start with a candidate set that is ambiguous under survivor count alone.
   • Provide the new strong evidence.
   • Assert that expected_tool_name is emitted.
   • Assert expected_tool_derivation_source identifies the new evidence path, not single_survivor.

2. Ambiguous evidence still abstains

   • Multiple plausible surviving tools remain.
   • Assert expected_tool_name is None.
   • Assert the abstention reason remains explicit and stable enough for measurement.

3. No-survivor cases report cleanly

   • Preserve expected_tool_abstain_reason == "no_survivors" or equivalent existing behavior.
   • Do not mask no-survivor with the new evidence path.

4. Diagnostics remain present

   • Decision records still include:
     • expected_tool_name
     • expected_tool_derivation_source
     • expected_tool_candidate_set_size
     • expected_tool_abstain_reason
     • top_survivor_by_similarity when present
     • existing candidate/survivor telemetry
   • Ensure backward compatibility for readers that expect current fields.

5. Regression guard against weak prompt similarity

   • If only top_survivor_by_similarity points at a tool and no strong evidence exists, assert abstention.
   • Keep or strengthen the existing tests that say similarity is diagnostic-only.

Validation commands

Run the relevant fast tests first:

pytest tests/unit/test_cc_proxy_response_tap.py

Then run adjacent proxy/derivation tests if present:

pytest tests/unit/test_cc_proxy_runtime.py
pytest tests/unit/test_agent_loop.py

If the repo has a broader standard test target, run it before opening the PR.

Implementation note required in PR

The PR body must include a short implementation note answering:

1. What evidence source or filtering rule was added?
2. Why is it defensible from request/tool evidence?
3. What cases still abstain?
4. What diagnostics were preserved or added?
5. Should TCP-MT-21 be re-run immediately after merge?

Recommendation expectation

Unless the implementation changes only tests or documentation, the expected final recommendation is likely:

Re-run TCP-MT-21 immediately after merge on the same post-IMP-22/23 decisions.jsonl population plus targeted fixtures, segmented by emitted vs abstained vs legacy-excluded rows.

But do not make that recommendation blindly. Base it on the implemented evidence path and test results.

Stop condition

Stop and report instead of implementing if the only viable path to emission is:

• weak prompt similarity
• manual relabeling
• arbitrary threshold tuning
• broad heuristic ranking that cannot be defended from request/tool evidence

Do not trade annotation validity for coverage.