Reachability claims drift silently: 15 false records in one campaign, each round fixing only what was in front of it

[localai-bot]

The pattern

The LTX-2.5 campaign (#435) produced fifteen instances of one defect: a comment or record asserting that a constant, a guard, or a capability is unreachable / unported / ungated, when measurement says otherwise. Not one was caught by reading; every one was caught by mutation.

A representative sample:

claim	measured
Ltx2ConvVideoDecoderConfig::norm_eps "upstream discards it"	norm3 reads it on 5 goldens; reds at 1.64e-2
kLtx2BweMelLogClamp "left EVERY golden green"	reds at 0.144965 — the same commit added the golden refuting it
kLtx2Res2sSigmaUpClamp "never binds, no value comparison can see it"	a 1% change reds 2 goldens on an arm the suite already runs
kLtx2UpsamplerNormEps "a mutation leaves every golden green"	at 100x (the class's own bar) reds 3 goldens
loader "refuses by name any other combination"	resolves ModelOpt/compressed-tensors linear scales silently and renders wrong
docs/USAGE.md "the Gemma-4 tower is still not ported"	made false by the PR that left the sentence in place

Why fixing instances has not worked

Four consecutive review rounds each repaired the instance in front of them and stopped. The pin list in test_ltx2_vae.cpp is now correct in every entry — and the identical defect class was found afterwards at five other sites nobody had enumerated (test_ltx2_pipeline.cpp, ltx2_upsampler.h, ltx2_connector.h, ltx2_pipeline.h, and one case in test_ltx2_vae.cpp outside the list).

The method that finally worked was not another review. It was a mechanical enumeration: grep every reachability claim in the tree, mutate each one, tabulate claimed-vs-measured. That single sweep found six false entries that four scoped rounds had missed.

Two sub-species worth naming

• Magnitude sensitivity, not unreachability. Several claims rested on a mutation at the minimum magnitude leaving goldens green. Escalating to the class's own stated 100x bar reds them hard. A green mutation is not a probe.
• An assertion true by construction. One "too-good" gate arm compared the oracle to a copy of itself, so it printed CHECK( 0 < 0.085 ) regardless of the code under test — and under the very mutation it was named for it was the only assertion pair that did not fire.

Proposal

A checker that enumerates reachability claims and refuses to let them drift. Concretely: recognise the project's own claim phrasings ("left every golden green", "invisible", "unreachable", "not ported", "is refused"), require each to name the constant or symbol it is about, and gate them — either by requiring a linked mutation-proof arm, or by a periodic sweep job that mutates each claimed-unreachable value and fails when a golden moves.

The weaker but still valuable version: a documented sweep script anyone can run before claiming a constant is unreachable, so the standard becomes a probe that fails to reach, never a mutation that happened not to move anything.

Related: #595 (docs/FEATURES.md is a lock), #567 (OpNameImpl is a second lock). Those are about surfaces every PR must write; this is about surfaces nothing verifies.