docs: re-diagnose #018 (generation not retrieval), close #019, add #022 governance worklist#1073
Conversation
Free diagnosis from the run #61 + #57 eval-canary artifacts (no spend, no provider calls, read-only). - #19 closed as NOT REPRODUCIBLE: both runs place "Admission of Community Patients (AKG).pdf" at ranks 4-5 with missingDocumentSubstrings [], documentRecallAt5 1, contentRecallAt5 1, and zero failing cases across the 36-case retrieval suite. The claim is false at both endpoints of the stated #57-#61 range. - #18 re-scoped from "retrieval-depth" to answer generation. Retrieval is healthy (grounded, 4-5 citations, 100% doc/content recall, content_mrr_at_10 0.9344). Two of the three cases return the source_backed_review_fallback boilerplate, which cannot carry a schedule; the third emits degenerate extractive text and is misclassified as document_lookup. Suite-wide rather than monitoring-specific: 15/30 cases hit the fallback and all 7 targeting misses are route=extractive across dose/monitoring/pathway intents - substantially the same phenomenon as #21. No retrieval canary warranted. Docs-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ting gap Generated read-only at $0 from the run #61 + #57 eval-canary artifacts. - New docs/source-governance-refresh-worklist-2026-07-22.md (successor to the 2026-07-02 priority list): disambiguates the two governance rates (0.8409 is per-case over the 44-case answer suite; 0.5976 is per-slot over 676 retrieval top-result slots), gives the burn-down curve, and lists the top 25 documents by surfaced slots. - Key reframing: 59 of 124 surfaced documents are review-required, but 38 (64%) are the BMJ published-reference tier all at clinical_validation_status "unverified" -> ONE attestation-policy decision, not 38 clinical reviews. The remaining 21 are genuine local WA health-service reviews. Top-10 documents clear 44% of flagged slots; top-20 clear 66%. - #25 (new): the canary's own "## Source Governance" table prints all zeros (Top results | 0) even though topResults carry full governance metadata, so the operator-facing log does not surface this at all. Docs-only. No provider calls, no code change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe PR updates the outstanding-issues ledger and adds a regenerated source-governance worklist based on canary runs. It documents governance metrics, review-required classifications, a reporting gap, prioritized documents, burn-down calculations, execution order, and scope limits. ChangesSource governance documentation
Estimated code review effort: 1 (Trivial) | ~5 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d153f0bb5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/outstanding-issues.md`:
- Line 54: Update the `#025` detail in the outstanding-issues ledger so the
literal “Top results | 0” does not split the Markdown table row: escape the pipe
or rephrase the text, and add spaces around the affected inline code spans.
Preserve the issue’s meaning and keep the row aligned to seven cells.
In `@docs/source-governance-refresh-worklist-2026-07-22.md`:
- Around line 32-33: The BMJ policy decision entry must be constrained to the
metadata governance contract: replace the broad “ingestion default” option with
an explicit, auditable metadata update that preserves the third-party/unverified
distinction and never silently assigns current or approved status. Apply the
same clarification to the corresponding BMJ entry referenced around lines 88–90,
and ensure the debt remains open until the approved policy is implemented.
🪄 Autofix (Beta)
✅ Autofix completed
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0742619c-3965-44f2-bf2c-d99568eb6d04
📒 Files selected for processing (2)
docs/outstanding-issues.mddocs/source-governance-refresh-worklist-2026-07-22.md
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. ❌ Cannot run autofix: This PR has merge conflicts. Please resolve the conflicts with the base branch and try again. Alternatively, use |
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. ❌ Cannot run autofix: This PR has merge conflicts. Please resolve the conflicts with the base branch and try again. Alternatively, use |
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. ❌ Cannot run autofix: This PR has merge conflicts. Please resolve the conflicts with the base branch and try again. Alternatively, use |
|
@copilot resolve the merge conflicts on this branch. |
Head branch was pushed to by a user without write access
…ist, add #31, next-id=032
Merge conflicts resolved. The only conflicting file was
Commit: the latest push to this branch. |
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. Fixes Applied SuccessfullyFixed 1 file(s) based on 1 unresolved review comment. Files modified:
Commit: The changes have been pushed to the Time taken: |
Fixed 1 file(s) based on 1 unresolved review comment. Co-authored-by: CodeRabbit <noreply@coderabbit.ai>
CI triageCI failed on this PR. Automated classification of the 2 failed job(s):
Compared with main CI run #4453 (success). Classification is evidence routing, not permission to ignore a failure. Exact quarantined Playwright identities remain governed by the flake ledger. |
…updates Clarify that BMJ attestation must preserve third-party/unverified status and never silently default ingestion metadata to current or approved. Co-authored-by: Cursor <cursoragent@cursor.com>
…updates Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
Free ($0, read-only) diagnosis of the two P2 retrieval items plus the #22 governance worklist, all derived from the run #61 + #57 Eval Canary artifacts. No provider calls, no live queries, no code change.
#19 — closed, not reproducible
Both runs place
Admission of Community Patients (AKG).pdfat ranks 4–5, withmissingDocumentSubstrings: [],documentRecallAt5: 1,contentRecallAt5: 1, and zero failing cases across the 36-case retrieval suite. The claim ("misses on every canary #57–#61") is false at both endpoints of its own range. No retrieval canary was spent.#18 — re-scoped from retrieval to generation
Retrieval is healthy (grounded, 4–5 citations, 100% doc/content recall,
content_mrr_at_10 0.9344). The real cause:quality-lithium-monitoring-rangeandquality-adhd-medication-monitoringreturn thesource_backed_review_fallbackboilerplate — "a full written answer could not be completed just now" — which cannot contain a schedule, so targeting scores 0 by construction.quality-antipsychotic-metabolic-monitoringemits degenerate extractive text and is classifieddocument_lookupwhen the case expectsmedication_dose_risk.It is also not monitoring-specific: 15/30 cases hit the fallback and all 7 targeting misses are
route=extractiveacross dose/monitoring/pathway intents — substantially the same phenomenon as #21. A retrieval-side fix would have targeted the wrong layer.#22 — prioritized worklist generated
New
docs/source-governance-refresh-worklist-2026-07-22.md. Disambiguates the two rates (0.8409 = per-case over 44 answer cases; 0.5976 = per-slot over 676 retrieval slots) and reframes the work: of 124 surfaced documents, 59 are review-required, but 38 (64%) are the BMJ tier all atunverified→ one attestation-policy decision, not 38 reviews. Top-10 documents clear 44% of flagged slots; top-20 clear 66%.#25 — new
The canary's own
## Source Governancetable printsTop results | 0and all-zero rates despitetopResultscarrying full governance metadata, so operators cannot see any of this from the log.Verification
prettier --checkclean ·docs:check-linkspasses (1097 references)docs/outstanding-issues.md+ one new docGovernance
src/**touched.🤖 Generated with Claude Code
Summary by CodeRabbit
#022, and archived a non-reproducible item; added a new open issue#031about Source Governance table reports showing all zeros despite available governance metadata.