Reranker test plan: Postgres parity, live-provider smoke, quality regression harness (post-#1143)

[phernandez]

Context

PR #1143 (ironhalik) added the optional cross-encoder rerank stage: RerankProvider
protocol, FastEmbed (local ONNX) and LiteLLM (hosted) providers, factory with process-wide
singleton cache, opt-in config under reranker_*, fixed-prefix + demoted-tail pagination
stability, and fail-fast 503/502 error mapping. Unit coverage from the PR is strong
(pipeline math, provider contracts, factory, router mapping, redaction, config validation —
all in the default CI suites).

An audit of the merged feature found the remaining gaps concentrated in four areas, plus a
few small code fixes. This issue tracks closing them.

Phase 1 — CI-visible coverage gaps (one PR, no external deps)

• Postgres parity: the 9 repository-integration rerank tests in
  tests/repository/test_rerank_pipeline.py (:527-841) skip on Postgres
  ("sqlite-vec repository behavior is local SQLite-only"), so hybrid/vector rerank
  ordering on pgvector is asserted only by the single semantic-marked benchmark test.
  Port them to a Postgres variant (recording-reranker fixture; test-int/semantic/ +
  semantic marker if they need real vector tables, main suite otherwise).
• FTS negative guard: provider configured + FTS/title/permalink search → provider
  never invoked, ranking identical to baseline. (Only the inverse — no provider — is
  tested today: test_search_without_reranker_keeps_baseline.)
• Wiring seams: create_search_repository injects the singleton provider into both
  backends (search_repository.py:175-207); repository __init__ self-resolution
  (sqlite_search_repository.py:84-86, postgres_search_repository.py:100-102) respects
  semantic_search_enabled.
• Config-to-pipeline: reranker_max_document_chars set via config truncates the
  document text the provider receives (currently only the pure helper is tested).
• ChatGPT adapter opacity: chatgpt_tools.py:205's broad except Exception
  collapses rerank 503s into a generic "Internal search error". Distinguish retryable
  unavailability in the returned payload, and pin it with a test.

Small fixes (ride the Phase 1 PR as separate commits)

• Stale constructor default in fastembed_rerank_provider.py:71
  (Xenova/ms-marco-MiniLM-L-6-v2; the real default is jinaai/jina-reranker-v1-tiny-en
  from config_models.py:61) — dead today but a trap for direct construction.
• reranker_timeout config key: the LiteLLM provider hardcodes timeout=30.0
  (litellm_rerank_provider.py:34) and the factory never passes it. Add the config field
  (validated gt=0, default 30), thread it through the factory, document it in
  docs/semantic-search.md.

Phase 2 — real-model and live-provider verification

• Real-FastEmbed smoke (semantic-marked, runs in the test-semantic CI job with model
  caching): load actual jinaai/jina-reranker-v1-tiny-en, tiny fixed corpus, assert the
  relevant doc ranks first and scores are [0,1]-calibrated.
• Extend test-int/semantic/litellm_live_harness.py (embeddings-only today) with
  rerank: Cohere rerank-v3.5 + one alternate (Jina or Voyage), env-keyed, run manually
  with real keys — contract shape, ordering sanity, auth-failure classification
  (permanent vs transient). Mirrors the feat(core): add LiteLLM embedding provider #809 embedding live-smoke pattern.

Phase 3 — quality regression harness

• Golden-query eval on a realistic corpus (the Moby Dick demo project, 221 notes):
  ~20 queries with expected top notes; measure hit@k and MRR reranker-off vs reranker-on
  (FastEmbed). Build on the existing test-int/semantic/ metrics.py/corpus.py/
  report.py machinery. Assert no regression; report the delta. This is the durable guard
  the LoCoMo motivation (Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950) asked for, and the baseline for Add reranking step to search pipeline (local cross-encoder) #618/Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall #666/Entity-aware ranking boost: cross-conversation entity confusion dominates hard retrieval misses in LoCoMo benchmark #951 follow-ons.
• Latency benchmark (benchmark-marked): vector + hybrid P50/P95 with/without the local
  reranker — validates the "~20x headroom" claim and catches future model swaps that blow
  the budget.

Phase 4 — operational drills (manual, test-live style)

• Config UX: enable without semantic_search_enabled → clear error at
  bm config set time; model typo rejected at set-time; secrets absent from
  config list / diagnostics output.
• Live failure drill: kill network mid-query → API 503; multi-project MCP aggregate
  aborts rather than returning a partial page; recovery without restart.
• Pagination stability under a live provider: page 1 → concurrent writes → page 2,
  no duplicates or omissions.

Notes for implementers

• Reranking applies to vector and hybrid modes only; FTS/title/permalink keep their
  ranking (search_repository_base.py:1646). Scores replace fused scores for the fixed
  20-candidate prefix; the tail is demoted rank-only (rerank_provider.py:31-42) — the
  zero-floor tie-break is order-dependent, so downstream score-only re-sorts would break
  pagination stability.
• Failure semantics are deliberately fail-fast (no silent un-reranked fallback): a
  degraded page could duplicate/omit results relative to an already-served reranked page.
  Tests must not "fix" this.

Refs: #1143 (feature), #950 (motivation, closed), #618 #666 #951 #1155 (open follow-ons).

[claude]

Claude finished @phernandez's task in 53s —— View job

---

Triage

Type: Enhancement — test/QA coverage work for the reranker feature shipped in #1143, not a defect report.

Priority: Medium. No user-facing breakage; closes verification gaps (Postgres ordering parity, wiring seams, live-provider/quality regression coverage) for an opt-in feature that already has strong unit coverage.

Component: Search/semantic-search reranker (RerankProvider, search_repository.py, chatgpt_tools.py) and its test suites (tests/repository/, test-int/semantic/). Touches Postgres integration test paths used by cloud deployments, but isn't itself Cloud-specific behavior, so no cloud label applied.

Complexity: Mixed and clearly staged by the issue itself — Phase 1 (Postgres parity tests, FTS negative guard, wiring/config tests, two small code fixes) is medium and self-contained; Phase 2 (live-provider smoke) and Phase 3 (golden-query quality harness + latency benchmark) are larger and depend on external model/API access; Phase 4 is manual operational drills, not automatable CI work.

Duplicate check: No exact duplicate found among the issue's own cross-references (#618, #666, #950, #951, #1143, #1155) — this is a new tracking issue consolidating post-#1143 audit gaps rather than restating any of them.

Labels applied: enhancement.

[phernandez]

Phase 1 + small fixes are complete — delivered by #1232 (merged). All boxes ticked.

Scope decision for the remaining phases:

• Phase 4 is trimmed to a single short live smoke (enable fastembed rerank on a real
  instance, run searches, page through). The drills as originally listed re-verify behavior
  that now has automated coverage on both backends: config validation + secret redaction,
  503/502 mapping, multi-project aggregate abort, and cross-page pagination stability.
• Phase 2's live LiteLLM rerank run is deferred until hosted rerank matters (public
  recommendation or a cloud tier). It remains the one manual item with unique value — real
  provider response shapes through litellm.arerank can't be validated by stubs (same
  rationale as the feat(core): add LiteLLM embedding provider #809 embedding live smoke).
• Phase 2's real-FastEmbed smoke and Phase 3 (quality + latency harness) are automated
  work, not manual testing — Phase 3 is the gate for revisiting Entity-aware ranking boost: cross-conversation entity confusion dominates hard retrieval misses in LoCoMo benchmark #951 and closing Add reranking step to search pipeline (local cross-encoder) #618.

[phernandez]

Measured baseline (Phases 2-3, PR #1233)

Recorded here so the tracker carries the numbers, not just the PR description. All figures
from the real models (bge-small-en-v1.5 embeddings + jinaai/jina-reranker-v1-tiny-en
cross-encoder) on the curated golden ranking set; the harness re-prints these on every run
(test-int/semantic/test_reranker_quality.py, test_reranker_latency.py).

Ranking quality (golden set, curated ranking-sensitive queries)

Mode	hit@1	hit@5	MRR
Reranking off	0.833	1.000	0.917
Reranking on	1.000	1.000	1.000

No queries excluded; the tiny model degraded nothing. The suite asserts on ≥ off for
hit@1/MRR plus a hard hit@5 floor, so a silent rerank no-op fails CI.

Latency (local, warmed, 20 iterations)

Search	P50	P95
Vector, rerank off	9.5 ms	10.1 ms
Vector, rerank on	102.5 ms	238.4 ms
Hybrid, rerank off	13.2 ms	17.6 ms
Hybrid, rerank on	98.2 ms	463.8 ms

≈90 ms P50 overhead for the local tiny cross-encoder — inside the latency headroom that
motivated #950. The benchmark test enforces only a generous catastrophic-regression
ceiling; these figures are the reference point.

These numbers are the baseline for the #951 revisit (segment LoCoMo single-hop misses into
rerank-recovered vs absent-from-pool) and for closing #618 once the benchmark evidence is
accepted.

[phernandez]

LoCoMo rerank comparison complete — full results on #951 (#951 (comment)). Headline: overall R@5 +5.0pp / MRR +9.7pp with reranking on; single_hop (the entity-confusion category) gained most at +9.3pp R@5 / +11.9pp MRR. Latency caveat recorded there: p95 420ms → ~2.1s on the long-chunk LoCoMo corpus with the full ~100-candidate pool — mitigations are reranker_max_document_chars and a smaller pool. This closes the quality-evidence loop for #618 as well.

[phernandez]

Closing: everything actionable is delivered — Phase 1 + fixes (#1232), real-model smoke + quality harness + latency benchmark (#1233), and the LoCoMo before/after comparison recorded here and on #951 (overall R@5 +5.0pp / MRR +9.7pp; single_hop +9.3pp R@5). The hosted Cohere/Jina live check is dropped — not deferred — as there's no interest in hosted rerank providers; if that ever changes, the harness extension sketch lives in this issue's history. Only informal loose end: an optional ten-minute smoke with reranker_enabled on a personal instance. Follow-up filed for the one data-backed finding that deserves its own issue: reranker latency on long-document corpora.