opsx: memory-relevance-gate — cross-encoder relevance gate for memory recall (design)

[Aaronontheweb]

Branch strategy: targets feature/memory-embeddings — this design depends wholly on the embedding stack (its spec deltas modify memory-embeddings, a capability that exists only as memory-core-redesign's proposed specs; its implementation reuses feature-branch infrastructure). Co-locating design with dependency means both arrive on dev in the same feature merge. Archive ordering: memory-core-redesign archives first, then this change.

What we found, in plain English

After the hybrid-recall work, the memory system is good at finding a relevant memory when one exists. The remaining problem is the opposite case. On about six out of ten real questions, nothing in memory is relevant, and the right move is to inject nothing. The system almost never makes that move. Its similarity floor keeps out the obviously unrelated stuff, but "similar" and "useful" are different questions: ask about GPU benchmark numbers and it will happily inject the GPU rack-layout doc. Same topic, zero help, wasted context.

We prototyped four fixes and measured all of them against real traffic, twice: once on the 93 labeled queries we had, then again on 450 fresh ones mined from the full session history, so nothing got to grade its own homework.

1. A statistics-only gate. Inject only when the best match stands out from the crowd. Looked great on the first 93 queries (70% correct silence). On the 450 fresh ones it fell apart: it started blocking real answers, and its wins turned out to be tuning luck. Dead.

2. A small trained model over cheap signals. Not enough data to trust it. Even after growing the dataset five-fold, real traffic only contains 39 examples of "a memory that survived the floor and was actually relevant." The model memorized the haystack instead of learning the needle, and fell apart on transfer. Shelved until we have far more labeled traffic.

3. Punishing repeat offenders — memories that keep getting injected and keep being useless. Sound idea, wrong corpus: 80% of what recall surfaces has never been injected before, so there is no track record to act on. Dead as a gate. The bookkeeping may return later as telemetry.

4. A cross-encoder. A 22MB model that reads the question and the memory together and scores "would this actually help answer it?" It was trained on millions of human relevance judgments, so it arrives already knowing what relevance looks like; our data only had to pick a threshold. This one held. On 450 queries it had never seen, it stays correctly silent on 87% of nothing-relevant questions (the floor alone manages 7%) while keeping 98% of the recalls we actually want. Cost: ~11ms per turn and ~100MB of memory, which leaves the 1GB pod budget intact.

One correction the bigger dataset forced on us: our first threshold choice (0.08) came from a sample with only 8 positive examples and looked like a free lunch — silence with no recall cost. With 39 positives the free lunch disappeared; 0.08 sits exactly on the edge of the recall budget. The design freezes 0.02 instead: we give up 2 points of silence accuracy to buy an 8-point safety margin on recall.

Two caveats ship with this, recorded rather than hidden. The cross-encoder learned relevance from web-search data, so it slightly undervalues procedural memories ("how we cut releases") — that costs about 1.7% of good recalls at the chosen threshold, and fixing it means domain calibration later. And the judges that labeled the expanded validation set agreed with each other less than the original round's judges did (κ 0.435 vs 0.754), so we aggregated labels conservatively: a memory only counts as relevant if the judges converged on it.

The numbers (reference)

design	mechanism	out-of-sample verdict
A — distribution-shape	z_top50 ≥ 2.80	Fails: 65.2% zero-inj, 86.5% recall retention (below the ≥90% constraint), F0.5 0.089 < 0.100 floor-only
B — cross-encoder (winner)	Xenova/ms-marco-MiniLM-L-6-v2 int8, pair scoring	86.8% zero-inj (95% CI 82.3–90.3), 98.3% retention, F0.5 0.130 vs 0.100, mean injected 0.251 vs 2.538 (S*=0.02)
C — learned feature gate	logistic/GBM over cosine/margin/z/length/age	Not viable: query-level OOF AUC 0.545 (chance); 39 positives still insufficient; recall collapse on transfer
D — offender priors	pollution/injection ratio per docId	Structurally dead: only 1.1% of top-3 candidates have 3+ injection history; 80.6% cold-start

Winner artifact: model_quantized.onnx, 23,143,499 bytes, SHA-256 e9d8ebf845c413e981c175bfe49a3bfa9b3dcce2a3ba54875ee5df5a58639fbe. Efficiency: ~11ms p50 / ~35ms p95 for 3 pairs (reference CPU), ~103MB incremental RSS; combined with int8 embeddings + daemon peak ≈ 763MB against the 1GB pod limit.

Architecture (reuses memory-core-redesign's infra wholesale)

• IRelevanceScorer seam in Netclaw.Actors (mirrors IMemoryEmbedder) + OnnxCrossEncoderScorer in Netclaw.Embeddings (pair encoding, dynamic seq length, sigmoid over the single-logit head).
• Provisioning extends the existing EmbeddingModelProvisioner allowlist: pinned URL/SHA-256/size plus the calibrated threshold, carried in the manifest like every other measured operating point. Nothing new for an operator to configure.
• Recall wiring in SQLiteMemoryRecallCoordinator: after the cosine floor, score the surviving candidates (≤3) under a ~60ms sub-budget; drop below threshold; zero survivors means inject nothing. Model unavailable or timed out means floor-only behavior with a rate-limited memory_recall_gate_degraded marker and doctor visibility.
• One mental switch: the gate follows Memory.Embeddings.Enabled. Memory.Recall.RelevanceGate { Enabled, Threshold } exists for explicit override only, both nullable.
• memory_retrieval_final gains gateScores + droppedByGate; a new eval case asserts the zero-injection behavior end to end.

Implementation target

Implementation lands on feature/memory-embeddings, not directly on dev.

Test plan

• openspec validate --changes memory-relevance-gate passes
• openspec status --change memory-relevance-gate shows all four artifacts done
• Reviewer sign-off on the frozen threshold (S*=0.02) and rejected alternatives before implementation begins

[Aaronontheweb]

TL;DR;, a search re-ranker powered by ONNX is the most competitive design here.