feat(governance): RAG behaviour memory + enforced ranking safeguards

[BigSimmo]

Summary

The durable close-out of tonight's RAG cycle, per user direction: a permanent memory of what was learned, and enforced safeguards so retrieval/ranking behaviour cannot be changed casually by any future task or session.

RAG impact: no retrieval behaviour change — documentation, policy tooling, and source-pin tests only (confirmed by canary #56: reverted main green 36/36).

• docs/rag-behaviour/ (new, 4 files): README (standing rules + index), behaviour-map (verified mechanics: the four imputation sites, the comparator chains and their load-bearing relevance fallback, the gate ladder, second-stage engagement rules, live case↔path evidence), refuted-approaches (both live-refuted attempts with numbers, root causes, and the binding constraints for any third attempt), safeguards (the four-layer protection stack).
• AGENTS.md § RAG ranking protection: standing rules every agent session inherits — flag RAG impact before editing protected surfaces, canary pair for behaviour changes, never insert comparator keys above the relevance score.
• pr-policy blocking gate: PRs touching RAG-ranking protected surfaces (rag cluster, ranking/selection/release-order modules, eval harness, alias tables, golden fixture/snapshot, contract tests) now fail the PR-policy check without an explicit RAG impact: declaration — either no retrieval behaviour change — <reason> or a canary-pair reference. Self-test covers undeclared/vague (blocked) and no-change/canary (pass) cases. This very PR is gated by its own rule (see the declaration above).
• tests/rag-imputation-contract.test.ts: source-text pins on the imputation formulas (app + SQL) and the release comparator key ORDER (score → similarity → relevance → id). Red-proven: mutating one constant fails exactly the right test with a message routing the editor to the required protocol.

Also closes the loop live: canary #56 (run 29774459706) on reverted main = SUCCESS, restoring the 36/36 baseline and completing the #55-regression → revert → restore arc.

Verification

• node scripts/pr-policy.mjs --self-test + npm run check:pr-policy (workflow guard) — pass with the new gate cases
• Contract test 4/4 + red-proof executed (mutated formula → exact failure → restored)
• npm run test — 3025 passed / 1 failed: the known container-only pdf-extraction-budget artifact
• npm run lint + npm run typecheck + prettier — clean
• npm run docs:check-links — 1030 refs resolve (new docs folder included)
• npm run check:github-actions — pass (workflow edit is message/comment-only)

Risk and rollout

• Risk: low-medium (policy tooling). The new blocking gate could friction legitimate PRs — mitigated by a one-line declaration with two clearly documented forms, and scoped to a deliberate protected list (docs/tests-only repos paths don't trigger it). The contract test is deliberately brittle — that is its function; its failure message explains the protocol.
• Rollback: revert the PR (gate and pins disappear; docs remain harmless).
• Provider or production effects: None from this diff. Canary Simplify operational tooling and runbooks #56 was the user-approved confirmation dispatch (~$1–2; total cycle spend ≈$5–10 of the $20 cap).

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — no answer-path change
• No patient-identifiable document workflow was introduced or expanded
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — this PR only adds protection around those behaviours
• Deployment classification/TGA SaMD impact was checked: no clinical decision-support behaviour changed; safeguards reduce future regression risk

Notes

• Phase D is complete across feat(ci): eval-canary dispatch inputs — staged ranking-config override + branch evals (ADDENDUM 4 B-PR-1 / D-1) #1003 (branch-eval ref input), docs+tooling(eval): ADDENDUM 4 close-out — Phase C record, boundary-case policy, artifact trend tool (Phase D) #1006 (eval:trend, §3.1 policy), and this PR. Remaining documented plans (not implemented): word-boundary matcher (own PR), irrelevant@10 labeling audit, Phase E answer-side (separate approval).
• Ledger row appended (content commit 1d04436).

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

---

Generated by Claude Code

Summary by CodeRabbit

• New Features

  • Added automated checks for pull requests affecting retrieval and ranking behavior.
  • Requires a valid RAG impact: declaration, including either a no-change explanation or validation canary details.

• Documentation

  • Added guidance describing retrieval/ranking behavior, safeguards, evaluation procedures, and previously tested approaches.
  • Recorded recent validation results and follow-up plans.

• Tests

  • Added contract checks to protect score calculations, ranking order, and tie-breaking behavior.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 093f25f7-2a64-415e-a958-ae950bf146b9

📥 Commits

Reviewing files that changed from the base of the PR and between 40a4962 and 7b1bffa.

📒 Files selected for processing (9)

• .github/workflows/pr-policy.yml
• AGENTS.md
• docs/branch-review-ledger.md
• docs/rag-behaviour/README.md
• docs/rag-behaviour/behaviour-map.md
• docs/rag-behaviour/refuted-approaches.md
• docs/rag-behaviour/safeguards.md
• scripts/pr-policy.mjs
• tests/rag-imputation-contract.test.ts

---

📝 Walkthrough

Walkthrough

The PR adds a blocking RAG impact: policy gate for protected ranking surfaces, updates workflow messaging, introduces source-text contract tests for retrieval ordering and score formulas, and documents verified behavior, safeguards, refuted approaches, and review history.

Changes

RAG ranking protection

Layer / File(s)	Summary
RAG impact policy gate scripts/pr-policy.mjs, .github/workflows/pr-policy.yml	Classifies protected RAG files, validates explicit no-change or canary declarations, blocks invalid PRs, expands self-tests, and updates failure messaging.
Retrieval and ordering contract safeguards tests/rag-imputation-contract.test.ts, docs/rag-behaviour/safeguards.md	Pins score-imputation formulas, SQL caps, comparator ordering, coverage tie-breaking, and clamping semantics while documenting safeguard layers and rollback steps.
RAG behavior memory and constraints AGENTS.md, docs/rag-behaviour/*	Documents verified retrieval behavior, protected surfaces, canary requirements, comparator constraints, refuted experiments, and follow-up plans.
Review ledger record docs/branch-review-ledger.md	Records the safeguards, contract test, canary restoration, and Phase D completion.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Author
  participant PRPolicy
  participant RAGContractTests
  participant Workflow
  Author->>PRPolicy: Submit changed files and PR body
  PRPolicy->>PRPolicy: Classify RAG-ranking surfaces
  PRPolicy->>PRPolicy: Validate RAG impact declaration
  PRPolicy->>RAGContractTests: Run contract checks
  RAGContractTests-->>PRPolicy: Formula and ordering results
  PRPolicy->>Workflow: Report policy status
  Workflow-->>Author: Pass or blocking failure message

Loading

Possibly related PRs

• BigSimmo/Database#716: Extends the same PR-policy classification and evaluation flow.
• BigSimmo/Database#920: Covers related release-comparator ordering and release-score behavior.
• BigSimmo/Database#982: Overlaps with retrieval-selection tie-handling and content-coverage ordering.

Suggested labels: codex

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

---

_{Comment @coderabbitai help to get the list of available commands.}