Evaluators: two evaluators on one case can't share a score field name — fail loud or auto-namespace

[renaudcepre]

Context

Each @evaluator returns a dataclass whose fields are tagged Verdict / Metric / Reason. If two evaluators attached to the same EvalCase expose a field with the same name (e.g. both declare ok: Annotated[bool, Verdict]), their scores collide. The current workaround is a bespoke dataclass per evaluator with globally-unique field names (count_ok/count_detail, alert_ok/alert_detail, shared_key_ok/shared_key_detail, …).

Impact

• You discover the rule only by hitting a collision, not from the API.
• It forces verbose, prefix-namespaced field names everywhere, which leaks the evaluator's identity into its result schema (every evaluator reinvents <name>_ok / <name>_detail).

Suggestion

Namespace scores by the evaluator's name automatically (e.g. count_matching.ok, alert_emitted.ok), so two evaluators can both use plain ok / detail. Failing that, raise an explicit error at registration/run time when two evaluators on the same case declare the same score name.

Note

If the constraint is intended, a loud error at collision time (instead of a silent overwrite) would already remove the papercut — the dangerous case is two evaluators silently clobbering each other's verdict.

[renaudcepre]

Two refinements worth folding into the in-flight namespacing fix (72c36a2), spotted while reviewing the evals layer:

1. Move the duplicate-name check before execution. The collision check in the wrapper runs after all evaluators (wrapper.py:89-96) — including judge calls, so a guaranteed crash is detected only after burning judge tokens. Duplicate ev.name values in the merged evaluator list are knowable before the loop; a pre-execution check on names catches virtually all collisions for free (keep the post-execution check as a backstop for exotic cases).

2. Namespace the ShortCircuit skipped placeholders too. When a dataclass-returning evaluator is skipped, the placeholder score uses the bare evaluator name (wrapper.py:270-273), while actual runs produce name.field scores. Score keys for the same case then differ between runs depending on whether the short-circuit fired — which breaks run-over-run history comparison. The placeholder should emit the same namespaced keys (or a single name.__skipped__ sentinel, as long as it's stable across runs).

Related silent-pass family: #117, #118.

[renaudcepre]

Both refinements landed in #115 (e92f49f):

1. Duplicate evaluator names are now checked on the merged (suite + per-case, ShortCircuit-flattened) list before any evaluator runs — a doomed case fails before burning judge tokens. The post-execution check stays as a backstop for exotic collisions the name check can't see.

2. ShortCircuit skipped placeholders now emit the same namespaced keys a real run would produce, via Evaluator.score_names() (derived from the return annotation, falling back to the bare name when the annotation is missing/unresolvable — a placeholder must never be stricter than the real run). Score keys are now identical whether or not the short-circuit fired.