fix(early-stop): latch skill activation on any engagement, not first

The skill_triggered criterion decided activation on the FIRST skill the
agent engaged, so a row failed when the agent touched a wrong skill before
engaging the expected one -- even though reading a SKILL.md is comparison,
not commitment. Decide on ANY engagement instead:

- _check_impl and live_verdict now share _all_engaged_skill_names (the
  union across the whole trajectory). A positive criterion
  (skill_name == expected_skill) passes iff the expected skill is engaged
  anywhere in the run (recall); a distractor/negative criterion fails on
  any engagement of its skill (precision), so an extra off-target
  engagement lands on its own confusion cell rather than opening a
  precision hole. Negatives (expected_skill == "") still fail on any
  engagement of the target skill.

- Engagement is monotonic, so the live latch and the frozen-trajectory
  score agree by construction.

- live_decidable_polarities now narrows per instance: a positive can only
  live-pass, a distractor/negative only live-fail. Arming a positive with
  fail (or either with decided) is rejected at resolution instead of
  silently degrading to a full run.

Also confirms the early-stop watcher latches on a file-read engagement
(Read/Grep of skills/<name>/...), not only a Claude Skill tool call, so
early-stop fires for non-Claude agents.

Adds coverage for the any-engagement scoring, stacked recall/precision on
one trajectory, per-instance decidability, and the file-read latch.

Author: mohsen-uipath

src/coder_eval/criteria/skill_triggered.py

@@ -72,22 +72,31 @@ def _engaged_skill_names(cmd: CommandTelemetry) -> set[str]:
     return names
 
 
-def _first_engaged_skill_names(turn_records: list[TurnRecord]) -> set[str]:
-    """Skills engaged by the FIRST command that engages any skill (else empty).
-
-    Activation measures which skill the agent selects *first*, so scoring keys off
-    the first engaging command rather than "any command anywhere in the run". This
-    keeps the final check and the live verdict consistent and prevents a later,
-    incidental engagement (or a second skill invoked alongside the first) from
-    being counted as a competing activation and mis-scored as a false positive.
-    Commands are scanned in ``sequence_number`` order (turn records preserve it).
+def _all_engaged_skill_names(turn_records: list[TurnRecord]) -> set[str]:
+    """Union of every skill engaged anywhere in the trajectory (any-engagement).
+
+    Activation is scored on whether a skill was engaged *at all* during the run,
+    not on which skill was engaged first. This has two consequences that the
+    first-engagement policy could not express:
+
+    - **Recall (the positive criterion).** A row that engages the wrong skill
+      before eventually engaging the expected one is still credited for the
+      expected skill — reading a ``SKILL.md`` to compare candidates is
+      exploration, not commitment, so an earlier wrong touch must not fail the
+      row.
+    - **Precision (the distractor/negative criteria).** An unrelated skill
+      engaged *anywhere* is counted against its own criterion, so a positive row
+      that also fires an off-target skill (and a negative row that fires any
+      target skill) is penalized on that skill's confusion cell.
+
+    Order is irrelevant to a set union; the scan is left in ``sequence_number``
+    order purely for determinism.
     """
+    names: set[str] = set()
     for turn in turn_records:
         for cmd in turn.commands:
-            engaged = _engaged_skill_names(cmd)
-            if engaged:
-                return engaged
-    return set()
+            names.update(_engaged_skill_names(cmd))
+    return names
 
 
 @register_criterion
@@ -101,8 +110,11 @@ class SkillTriggeredChecker(BaseCriterion[SkillTriggeredCriterion]):
     criterion_type = "skill_triggered"
 
     # Observable mid-run: a Skill tool call (or a skill file read) is a positive
-    # event in the live stream, so both polarities are decidable the moment the
-    # agent first engages ANY skill (the first-engagement policy in live_verdict).
+    # event in the live stream. The TYPE can decide either polarity — a positive
+    # criterion live-passes when its expected skill is engaged, a
+    # distractor/negative one live-fails when its (wrong) skill is engaged — but
+    # any single INSTANCE decides only one of the two; see
+    # ``live_decidable_polarities``.
     live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"})
 
     def _check_impl(
@@ -123,10 +135,13 @@ def _check_impl(
                 error="turn_records not provided to checker",
             )
 
-        # First-engagement policy (mirrors ``live_verdict``): the run is scored on
-        # the FIRST skill the agent engages, so a second skill invoked alongside or
-        # after it is not counted as a competing activation.
-        triggered: bool = criterion.skill_name in _first_engaged_skill_names(turn_records)
+        # Any-engagement policy (mirrors ``live_verdict``): the row is scored on
+        # whether this skill was engaged AT ALL, regardless of order. A positive
+        # criterion (skill_name == expected_skill) passes iff the expected skill
+        # was engaged somewhere in the run — a wrong skill engaged first does not
+        # fail it (recall). A distractor/negative criterion fails on ANY
+        # engagement of its skill (precision).
+        triggered: bool = criterion.skill_name in _all_engaged_skill_names(turn_records)
         expected_yes: bool = criterion.expected_skill == criterion.skill_name
         score = 1.0 if triggered == expected_yes else 0.0
         observed = _YES if triggered else _NO
@@ -147,29 +162,53 @@ def live_verdict(
         criterion: SkillTriggeredCriterion,
         turn_records: list[TurnRecord],
     ) -> LiveVerdict:
-        """First-engagement policy: the FIRST observed skill engagement decides.
-
-        Activation measures which skill the agent selects *first*, so every
-        stacked ``skill_triggered`` criterion is decided simultaneously by the
-        first command that engages any skill:
-
-        - before any engagement -> ``"undecided"``;
-        - on the first command engaging some skill: ``observed = (skill_name in
-          engaged)``, ``expected = (expected_skill == skill_name)`` ->
-          ``"pass"`` iff they match, else ``"fail"``.
-
-        This covers the "wrong skill loads" case (a positive wrong signal) and
-        negative rows (``expected_skill == ""`` -> any engagement of the target
-        fails that criterion). ``_check_impl`` applies the SAME first-engagement
-        policy on the full trajectory, so the live verdict and the authoritative
-        score agree by construction — whether or not the run stopped early.
+        """Any-engagement latch: decide the instant THIS skill is engaged.
+
+        Mirrors ``_check_impl``'s any-engagement policy, latched monotonically
+        over the growing partial trajectory:
+
+        - this skill not engaged yet -> ``"undecided"`` (the final outcome still
+          depends on the rest of the run — the expected skill may load later, or
+          a distractor may yet fire);
+        - this skill engaged -> ``expected_skill == skill_name`` decides:
+          ``"pass"`` for a positive criterion (the expected skill loaded),
+          ``"fail"`` for a distractor/negative one (a wrong skill loaded).
+
+        Because engagement is monotonic (a skill, once engaged, stays engaged), a
+        latched verdict never flips, so it agrees with ``_check_impl`` on the
+        frozen trajectory by construction — whether or not the run stopped early.
+        A positive criterion can therefore only ever live-``pass`` and a
+        distractor/negative one only ever live-``fail``; their *absence* is never
+        decidable mid-run (see ``live_decidable_polarities``). This is the change
+        from first-engagement: a wrong skill engaged first no longer live-fails a
+        positive row — the run keeps going so the expected skill can still load.
         """
-        engaged = _first_engaged_skill_names(turn_records)
-        if not engaged:
+        if criterion.skill_name not in _all_engaged_skill_names(turn_records):
             return "undecided"
+        return "pass" if criterion.expected_skill == criterion.skill_name else "fail"
+
+    @classmethod
+    def live_decidable_polarities(cls, criterion: SkillTriggeredCriterion) -> frozenset[str]:
+        """Per-instance narrowing under the any-engagement latch.
+
+        Unlike the type-level capability (``live_stop_polarities`` = both), a
+        single instance decides exactly one polarity:
+
+        - a **positive** criterion (``skill_name == expected_skill``) can only
+          live-``pass`` (the expected skill engaging is a decidable hit; its
+          absence is not knowable mid-run);
+        - a **distractor/negative** criterion (``skill_name != expected_skill``,
+          including the ``expected_skill == ""`` negatives) can only
+          live-``fail`` (a wrong skill engaging is a decidable miss; its absence
+          is not).
+
+        ``validate_early_stop`` gates the requested ``stop_when`` on this set, so
+        arming a positive with ``fail`` / a distractor with ``pass`` — or either
+        with ``decided`` (which needs both) — is rejected at resolution rather
+        than silently degrading to a full run.
+        """
         expected_yes = criterion.expected_skill == criterion.skill_name
-        observed_yes = criterion.skill_name in engaged
-        return "pass" if observed_yes == expected_yes else "fail"
+        return frozenset({"pass"}) if expected_yes else frozenset({"fail"})
 
     def aggregate(
         self,

src/coder_eval/orchestration/early_stop.py

@@ -296,8 +296,9 @@ def _evaluate(self, in_flight: CommandTelemetry | None = None) -> None:
         record = self._collector.build_turn_record()
         if in_flight is not None:
             # The in-flight call has no ToolEnd yet, so the collector (which
-            # reduces commands from ToolEnd) has not captured it. Append it and
-            # re-sort by sequence so the verdict sees it in first-engagement order.
+            # reduces commands from ToolEnd) has not captured it. Append it so its
+            # engagement is visible to the verdict; re-sort by sequence to keep the
+            # partial trajectory in emission order.
             record.commands = sorted([*record.commands, in_flight], key=lambda c: c.sequence_number)
         records = [record]
         # An in-flight call has not been counted by a ToolEnd yet, so report it as