TCP-MT-22: Schema-3 survivor reducer promotion-gate measurement

[samscarrow]

Notion Work Item

TCP-MT-22 in Lab Work Items

Type

Measurement Track

Objective

Run the first auditable schema-3 survivor reducer promotion-gate measurement after TCP-IMP-25: redeploy/restart the enriched proxy, collect ≥200 eligible schema-3 rows, label them under the documented ground-truth methodology, compute macro precision/recall plus per-tool confusion, and emit a promote/hold/redesign recommendation.

Dataset

Source: ~/.tcp-shadow/proxy/decisions.jsonl rows emitted after TCP-IMP-25 redeploy/restart with DECISION_LOG_SCHEMA=3, prompt_text, and tool_capability_flags_by_name present.

Sample: ≥200 eligible labeled schema-3 rows only; exclude schema-2/pre-enrichment rows and rows below the documented confidence threshold.

Metrics

• Macro precision and recall of reducer_survivors vs correct_tool_set over ≥200 labeled schema-3 rows
• Per-tool TP/FP/FN/TN confusion matrix
• Inter-labeler agreement if multiple labelers are used
• Final promote/hold/redesign recommendation

Promotion Gate

• Promote only if precision ≥ 0.90 and recall ≥ 0.85 over ≥200 labeled schema-3 rows.
• Hold if the sample is valid but insufficient, too ambiguous, or operationally incomplete.
• Redesign if the sample is valid and below threshold.

Kill/Stop Condition

Stop and report hold if fewer than 200 eligible schema-3 rows can be collected, if prompt text contains unsafely retained PII/secrets, or if the labeling methodology cannot be applied reproducibly. Stop and report redesign if a valid ≥200-row sample fails precision ≥ 0.90 or recall ≥ 0.85. Do not promote shadow→live unless both thresholds pass.

Completion Contract (Measurement Track)

• Epistemic. Take measurements as specified. Report results with: (1) methodology, (2) raw data or summary statistics, (3) any measurement caveats or confounds encountered.
• No PR expected unless the objective requires instrumentation code. If instrumentation is needed, one PR for the instrumentation is acceptable — keep it separate from the measurement report itself.
• Handoff: Post measurements + methodology to this issue → Close issue → Stop.

Execution Context

• Dispatch Via: Claude Code
• Execution Lane: coder
• Environment: dev
• Branch: main
• Phase: Shadow → Live promotion gate

[samscarrow]

Prompt Architect handoff — TCP-MT-22 → Claude Code

Recommended downstream model type

Claude Code / code-capable measurement agent with repository and local runtime access. This is a Measurement Track handoff: epistemic measurement first; no PR is expected unless a small, separate instrumentation or reusable measurement script is required.

Operating contract

• Work item: TCP-MT-22
• Repo: git-scarrow/tool-capability-protocol
• GitHub issue: TCP-MT-22: Schema-3 survivor reducer promotion-gate measurement #82
• Base branch: main, unless the repository or predecessor work clearly documents a required TCP-IMP-25 branch/ref.
• Execution lane: coder
• Environment: dev
• Handoff completion: post measurements + methodology to this issue → close issue only if measurement is complete and not blocked → stop.
• Do not flip shadow→live gating as part of this work item.

Objective

Run the first auditable schema-3 survivor reducer promotion-gate measurement after TCP-IMP-25.

The measurement must:

1. redeploy/restart the proxy build that emits schema-3 decision rows,
2. collect at least 200 eligible schema-3 rows,
3. label them under the documented ground-truth methodology,
4. compute macro precision/recall plus per-tool confusion, and
5. emit exactly one final recommendation: promote, hold, or redesign.

Preflight verification — required before measurement

Before collecting or scoring any rows, verify all predecessor artifacts are actually present in the checkout/runtime you are using:

• DECISION_LOG_SCHEMA == 3 in the proxy code that is deployed/restarted.
• New decisions.jsonl rows include:
  • decision_log_schema / schema marker equal to 3
  • prompt_text
  • tool_capability_flags_by_name
  • reducer_survivors or the current schema-3 survivor-reducer output field(s)
• The documented methodology and seed/ground-truth artifact exist:
  • tcp/measurement/ground_truth_v1.json
  • tcp/measurement/GROUND_TRUTH_METHODOLOGY.md

If any of these are missing from main, do not invent replacements and do not silently fall back to schema-2 behavior. First inspect repository history/branches/PRs for the TCP-IMP-25 landed ref. If the schema-3 artifacts still cannot be located or deployed, stop and report hold with the exact missing precondition.

Dataset contract

Use only rows from:

~/.tcp-shadow/proxy/decisions.jsonl

Eligibility rules:

• Include only rows emitted after the TCP-IMP-25 schema-3 proxy redeploy/restart.
• Include only schema-3 rows.
• Require prompt_text and tool_capability_flags_by_name to be present.
• Require reducer survivor output fields needed to compare against labels.
• Exclude schema-2 / pre-enrichment / pre-restart rows from the denominator.
• Exclude rows below the confidence threshold specified in GROUND_TRUTH_METHODOLOGY.md.
• Do not mix historical rows into the ≥200-row gate.

Hard safety/quality hold

Run a PII/secrets hygiene check on retained prompt_text before labeling.

If retained prompt text contains unsafely retained PII/secrets:

• stop,
• report hold,
• include the hygiene failure class and affected row count,
• do not redact ad hoc and continue measuring.

Labeling contract

Apply tcp/measurement/GROUND_TRUTH_METHODOLOGY.md exactly.

Use tcp/measurement/ground_truth_v1.json as the documented ground-truth scaffold. Preserve all ambiguity, exclusion, oracle, LLM-assisted, and human-labeling rules from the methodology. Do not redesign the methodology during this work item.

For each included row, produce or reference a label with:

• row identifier / timestamp,
• prompt text reference or safe hash,
• reducer survivor set,
• correct tool set,
• inclusion/exclusion status,
• label source/method,
• ambiguity notes if applicable.

Metrics to compute

Compute over the included labeled schema-3 sample:

1. Row counts

   • total rows inspected,
   • schema-3 eligible rows,
   • rows excluded and why,
   • final labeled denominator,
   • whether denominator is ≥200.

2. Macro precision

   • For each included row:
     • precision_row = |reducer_survivors ∩ correct_tool_set| / |reducer_survivors|
   • Macro-average across included rows.
   • Define and document the behavior for empty survivor sets according to the methodology; if the methodology does not define it, report hold rather than inventing a rule.

3. Macro recall

   • For each included row:
     • recall_row = |reducer_survivors ∩ correct_tool_set| / |correct_tool_set|
   • Macro-average across included rows.
   • Define and document the behavior for empty correct-tool sets according to the methodology; if the methodology does not define it, report hold rather than inventing a rule.

4. Per-tool confusion

   • TP / FP / FN / TN for each tool over the labeled sample.

5. Inter-labeler agreement

   • Report if multiple labelers are used.
   • If only one labeler is used, explicitly state that agreement is not applicable and explain why that is acceptable under the methodology.

Promotion gate

The gate is binary and pre-declared:

• promote only if:
  • labeled denominator ≥ 200,
  • macro precision ≥ 0.90,
  • macro recall ≥ 0.85,
  • no hard safety/quality hold applies.
• hold if:
  • denominator < 200,
  • schema-3 artifacts or methodology are missing,
  • prompt-text hygiene fails,
  • methodology cannot be applied reproducibly,
  • sample is too ambiguous or operationally incomplete.
• redesign if:
  • a valid ≥200-row schema-3 sample is measurable,
  • the methodology is reproducible,
  • no hard hold applies,
  • and either precision < 0.90 or recall < 0.85.

Do not editorialize past the gate. The final recommendation must be one of exactly: promote, hold, redesign.

Suggested implementation shape

Prefer a small measurement script if no existing one already covers schema-3 reducer measurement, for example:

python scripts/mt22_schema3_reducer_gate.py \
  --log ~/.tcp-shadow/proxy/decisions.jsonl \
  --methodology tcp/measurement/GROUND_TRUTH_METHODOLOGY.md \
  --ground-truth tcp/measurement/ground_truth_v1.json \
  --min-rows 200 \
  --precision-threshold 0.90 \
  --recall-threshold 0.85

The script should print a paste-ready report and, if feasible, write machine-readable artifacts under an appropriate measurement/output path.

If adding a reusable script or instrumentation:

• keep it separate from the measurement report itself,
• keep the PR scoped,
• reference TCP-MT-22 in the PR,
• do not modify live gating behavior.

Validation commands

Use the repository’s documented commands after reading AGENTS.md. At minimum, run focused checks for any touched files. Likely candidates:

pytest tests/unit/test_cc_proxy_decisions_meta.py
pytest tests/unit/test_cc_proxy_response_tap.py

If you add a measurement script, add focused unit tests for:

• schema-3 eligibility filtering,
• schema-2 exclusion,
• missing field exclusion/hold,
• macro precision/recall calculations,
• per-tool confusion matrix,
• final promote/hold/redesign gate selection.

Required issue report format

Post the final report back to this issue in this structure:

## TCP-MT-22 schema-3 reducer promotion-gate report

### Methodology
- Repo ref / branch:
- Proxy redeploy/restart evidence:
- Log path:
- Time window:
- Schema version:
- Ground-truth artifact:
- Methodology document:
- Labeling method:
- PII/secrets hygiene check:

### Row accounting
- Total rows inspected:
- Schema-3 rows:
- Eligible rows:
- Excluded rows by reason:
- Final labeled denominator:
- Denominator ≥200: yes/no

### Metrics
- Macro precision:
- Macro recall:
- Per-tool confusion matrix:
- Inter-labeler agreement:

### Gate result
- Precision threshold met: yes/no
- Recall threshold met: yes/no
- Safety/quality hold: yes/no
- Final recommendation: promote / hold / redesign

### Caveats / confounds
- [sample, methodology, instrumentation, or runtime caveats]

### Artifacts
- [measurement script / output files / PR links if any]

Stop conditions

Stop and report hold if:

• fewer than 200 eligible schema-3 rows can be collected,
• schema-3 artifacts cannot be located/deployed,
• prompt_text hygiene fails,
• ground_truth_v1.json or GROUND_TRUTH_METHODOLOGY.md is missing,
• the methodology cannot be applied reproducibly.

Stop and report redesign if:

• the sample is valid,
• denominator is ≥200,
• the methodology is reproducible,
• and precision or recall misses the declared thresholds.

Anti-patterns

Do not:

• promote based on partial or schema-2 data,
• redesign the labeling methodology inside this measurement,
• silently change metric definitions,
• treat missing schema-3 fields as empty values,
• flip live gating,
• close the issue if the result is blocked/hold and follow-up is still required.