docs: add RAG failure mode checklist with healthcare and multilingual modes - #13717
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Automated Code Review
Reviewers: 4 | Confidence: 88%
✓ Correctness
This PR adds a new documentation-only markdown file (a RAG failure mode checklist). There are no code changes, so no runtime correctness issues exist. However, the document references several Semantic Kernel component names that do not exist in the codebase (
ChatHistoryPromptTemplate,TextSearchPluginas a class,IPromptFilter), which could mislead developers trying to find these types. The RAGAS GitHub URL also appears to point to a fork rather than the canonical repository.
✓ Security Reliability
This PR adds a purely documentation markdown file (rag-failure-mode-checklist.md) with no executable code, configuration changes, or dependency modifications. There are no direct security or reliability concerns introduced by this change. However, the References section links to unofficial forks and third-party repositories (e.g., vibrantlabsai/ragas instead of the official explodinggradients/ragas) rather than canonical upstream sources. If developers follow these links to install packages, they could inadvertently use unofficial forks, which is an indirect supply-chain concern. The code examples in the markdown are illustrative only and not executed by any build or test pipeline.
✓ Test Coverage
This PR adds a purely documentation markdown file (docs/rag-failure-mode-checklist.md) that describes RAG failure modes mapped to Semantic Kernel components. Since it contains no code changes—no new functions, classes, APIs, or behavioral modifications—there is no testable code surface and therefore no test coverage gap. The file is a reference checklist with advisory content and code snippets shown only as illustrative examples (not executable library code). No test changes are needed.
✗ Design Approach
This PR adds a new documentation file that presents itself as a general-purpose RAG failure mode checklist for Semantic Kernel but has several fundamental design problems. The document is structured as a personal research artifact—not official project documentation—with prominent personal author branding at top and bottom, the author's own personal repository in the references, and a dependency on an external third-party taxonomy ('WFGY 16-Problem Map') that SK should not be coupling its official docs to. Beyond provenance issues, the document contains incorrect SK component names (e.g.,
IPromptFilterdoesn't exist; the correct interface isIPromptRenderFilter), a non-compilable C# example with unimplemented method stubs, and a reference tovibrantlabsai/ragasrather than the canonicalexplodinggradients/ragasRAGAS framework. The healthcare-specific category (failures 11–13) is presented as a first-class concern of the SK framework rather than a domain-specific overlay. Taken together, the document does not follow the conventions of any other file indocs/and is better suited to an external blog post or personal repository.
Flagged Issues
- The document carries personal author attribution (name, affiliation, GitHub handle) at both top and bottom, and reference #4 points to the author's own repository (rajantripathi/soas-rag-evaluation). Official project docs must not serve as vehicles for personal promotion.
- The taxonomy is explicitly based on the 'WFGY 16-Problem Map' (onestardao/WFGY), coupling SK's official documentation to an obscure external project. Failure modes should be grounded in SK's own architecture and well-established RAG literature.
- References #2 and #3 point to
vibrantlabsai/ragas, an unofficial fork, instead of the canonicalexplodinggradients/ragas. Linking to unofficial forks from Microsoft documentation is misleading and poses a supply-chain risk if developers install packages from those sources. -
IPromptFilter(failure mode #11, lines 225 and 238) does not exist in Semantic Kernel; the correct interface isIPromptRenderFilter. Shipping docs with wrong API names actively harms developers. - The C#
RAGFailureDetectorexample callsEvaluateContextPrecision,EvaluateFaithfulness,EvaluateAnswerRelevancy, andDetectPHIbut provides no implementations. The snippet will not compile and gives no actionable starting point.
Suggestions
- Several SK component names are incorrect:
ChatHistoryPromptTemplate(lines 106/120/176) should beKernelPromptTemplate,HandlebarsPromptTemplate, orJinja2PromptTemplate;TextSearchPlugin(line 24) should beVectorStoreTextSearchand related extension methods. Use the actual class names or generic descriptions. - If a RAG failure mode guide is genuinely wanted for SK, it should be a first-party document: no personal attribution, component names verified against the actual API surface, working (or clearly pseudocode-labeled) examples, and references to canonical external projects only.
- Healthcare-specific failures (PHI leakage, medical terminology, clinical guidelines) are domain concerns, not SK framework concerns. They belong in a separate domain-specific guide or a clearly labelled section, not as peers to general retrieval/generation failures.
- Reference #1 (WFGY) and #4 (soas-rag-evaluation) are third-party repos without established community trust. Consider whether linking to them from official Microsoft documentation is appropriate, or add a disclaimer that these are external community resources.
- The RAGAS detection snippets reference metrics (
context_precision,faithfulness,answer_relevancy) without showing how they integrate with SK. Either provide a real integration example or remove the snippets and link to RAGAS docs directly.
Automated review by rajantripathi's agents
| # RAG Failure Mode Checklist for Semantic Kernel | ||
|
|
||
| **Author**: Dr. Rajan Prasad Tripathi | AUT AI Innovation Lab | ||
| **Reference**: [Semantic Kernel Issue #13581](https://github.com/microsoft/semantic-kernel/issues/13581) |
There was a problem hiding this comment.
Personal author attribution does not belong in official SK project documentation. No other file in docs/ carries individual author credits. If authorship must be acknowledged, it should appear only in the PR description or CHANGELOG.
| **Reference**: [Semantic Kernel Issue #13581](https://github.com/microsoft/semantic-kernel/issues/13581) | |
| **Reference**: [Semantic Kernel Issue #13581](https://github.com/microsoft/semantic-kernel/issues/13581) |
|
|
||
| **Author**: Dr. Rajan Prasad Tripathi | AUT AI Innovation Lab | ||
| **Reference**: [Semantic Kernel Issue #13581](https://github.com/microsoft/semantic-kernel/issues/13581) | ||
| **Based on**: WFGY 16-Problem Map |
There was a problem hiding this comment.
Basing the official SK taxonomy on the WFGY 16-Problem Map couples this document to an obscure external project. The taxonomy should stand on its own or cite widely-recognised RAG literature (e.g., the original RAG paper, RAGAS paper).
|
|
||
| **Semantic Kernel Components**: | ||
| - `TextSearchPlugin` | ||
| - `VectorStore` |
There was a problem hiding this comment.
TextSearchPlugin does not exist as a class in the Semantic Kernel codebase. The relevant search types are VectorStoreTextSearch and related extension methods in TextSearchServiceCollectionExtensions.
| **Semantic Kernel Components**: | ||
| - `ChatHistoryPromptTemplate` | ||
| - Token counting utilities | ||
|
|
There was a problem hiding this comment.
ChatHistoryPromptTemplate does not exist in the Semantic Kernel codebase. The actual prompt template implementations are KernelPromptTemplate, HandlebarsPromptTemplate, and Jinja2PromptTemplate. This also applies to lines 120 and 176.
| - PHI patterns differ across languages: | ||
| - English: SSN (XXX-XX-XXXX), MRN | ||
| - Chinese: 身份证号, 姓名 | ||
| - Arabic: الرقم الوطني, الاسم |
There was a problem hiding this comment.
IPromptFilter does not exist in Semantic Kernel (it was a proposed name in ADR-0033). The correct interface for intercepting prompt rendering is IPromptRenderFilter (see dotnet/src/SemanticKernel.Abstractions/Filters/Prompt/IPromptRenderFilter.cs). This also affects line 225.
| - Arabic: الرقم الوطني, الاسم | |
| - `IPromptRenderFilter` for PII detection |
|
|
||
| | # | Failure Mode | Primary Component | Detection Metric | Quick Fix | | ||
| |---|-------------|-------------------|------------------|-----------| | ||
| | 1 | Retrieval Collapse | TextSearchPlugin | context_precision < 0.3 | Increase top_k | |
There was a problem hiding this comment.
The EvaluateContextPrecision, EvaluateFaithfulness, EvaluateAnswerRelevancy, and DetectPHI methods are called but never implemented. This code will not compile and provides no value. Either provide real implementations or mark the block explicitly as pseudocode.
| | 1 | Retrieval Collapse | TextSearchPlugin | context_precision < 0.3 | Increase top_k | | |
| // NOTE: pseudocode — implement each Evaluate* method using your chosen | |
| // evaluation framework (e.g. RAGAS) and a PHI detection library. |
|
|
||
| --- | ||
|
|
||
| ## References |
There was a problem hiding this comment.
References #2 and #3 point to vibrantlabsai/ragas, an unofficial fork, not the canonical RAGAS project at explodinggradients/ragas. Linking to unofficial forks from Microsoft documentation could mislead developers into installing packages from untrusted sources (supply-chain risk). Update both references to the canonical repository.
| ## References | |
| 2. RAGAS Evaluation Framework: https://github.com/explodinggradients/ragas |
|
|
||
| ## References | ||
|
|
||
| 1. WFGY 16-Problem Map: https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md |
There was a problem hiding this comment.
Reference #4 is the author's personal repository. Personal repositories should not appear as normative references in official project documentation.
| 1. WFGY 16-Problem Map: https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md | ||
| 2. RAGAS Evaluation Framework: https://github.com/vibrantlabsai/ragas | ||
| 3. RAGAS Multilingual Fix (PR #2651): https://github.com/vibrantlabsai/ragas/pull/2651 | ||
| 4. SOAS RAG Evaluation Benchmark: https://github.com/rajantripathi/soas-rag-evaluation |
There was a problem hiding this comment.
Trailing personal attribution block (name, affiliation, GitHub handle, research focus) is self-promotional and inconsistent with every other file in docs/. It should be removed entirely.
There was a problem hiding this comment.
Pull request overview
Adds a new documentation page that provides a structured checklist of RAG failure modes and diagnostics guidance, mapped to Semantic Kernel concepts, including healthcare and multilingual considerations.
Changes:
- Introduces a 16 item RAG failure mode taxonomy with symptoms, detection ideas, and quick fixes.
- Adds a quick reference summary table for fast triage.
- Includes a C# oriented diagnostics example snippet and references to evaluation tooling.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| **Symptoms**: Irrelevant chunks returned; model hallucinates without evidence | ||
|
|
||
| **Semantic Kernel Components**: | ||
| - `TextSearchPlugin` |
There was a problem hiding this comment.
TextSearchPlugin is referenced as a Semantic Kernel component, but there does not appear to be a TextSearchPlugin type in this repo. In SK, text search is typically represented by ITextSearch / VectorStoreTextSearch<TRecord> and optionally exposed to the kernel via CreateWithGetTextSearchResults(...) (a KernelPlugin). Please rename this component reference to the actual SK abstraction(s) to avoid sending readers to a non existent API.
| - `TextSearchPlugin` | |
| - `ITextSearch` / `VectorStoreTextSearch<TRecord>` |
| **Symptoms**: Important information truncated; later chunks ignored | ||
|
|
||
| **Semantic Kernel Components**: | ||
| - `ChatHistoryPromptTemplate` |
There was a problem hiding this comment.
ChatHistoryPromptTemplate is called out as a Semantic Kernel component, but it does not exist in this repository. Consider updating these references to the current SK prompt and chat primitives (for example PromptTemplateConfig / IPromptTemplateFactory and ChatHistory), or reword this section to avoid naming a specific type that readers cannot find.
| - `ChatHistoryPromptTemplate` | |
| - `ChatHistory` | |
| - Prompt configuration (e.g., `PromptTemplateConfig`, `IPromptTemplateFactory`) |
|
|
||
| **Semantic Kernel Components**: | ||
| - `IChatCompletionService` | ||
| - `ChatHistoryPromptTemplate` |
There was a problem hiding this comment.
This section again references ChatHistoryPromptTemplate, which does not appear to be an actual SK type in this repo. Please update to the correct SK type names (or make the wording framework agnostic) so readers can map the guidance to real APIs.
| - `ChatHistoryPromptTemplate` | |
| - Chat-history–aware prompt template / system prompt configuration |
| | # | Failure Mode | Primary Component | Detection Metric | Quick Fix | | ||
| |---|-------------|-------------------|------------------|-----------| | ||
| | 1 | Retrieval Collapse | TextSearchPlugin | context_precision < 0.3 | Increase top_k | | ||
| | 2 | Semantic Mismatch | EmbeddingService | Similarity analysis | Switch embeddings | | ||
| | 3 | Chunk Drift | TextChunker | Mid-sentence splits | Add overlap | | ||
| | 4 | Language Mismatch | EmbeddingService | Cross-lingual test | Multilingual embeddings | |
There was a problem hiding this comment.
In the quick reference table, several "Primary Component" entries do not match actual Semantic Kernel types (for example TextSearchPlugin and EmbeddingService). Since the page frames these as SK component mappings, consider replacing them with the concrete SK abstractions used in this repo (ITextSearch / VectorStoreTextSearch, ITextEmbeddingGenerationService, etc.) or rename the column to something like "Layer" / "Subsystem".
| public class RAGFailureDetector | ||
| { | ||
| private readonly Kernel _kernel; | ||
|
|
||
| public async Task<RAGDiagnostics> DiagnoseAsync(string query, string response, List<string> contexts) | ||
| { | ||
| var diagnostics = new RAGDiagnostics(); | ||
|
|
||
| // Check 1: Retrieval Collapse | ||
| diagnostics.ContextPrecision = await EvaluateContextPrecision(query, contexts); | ||
|
|
||
| // Check 2: Hallucination | ||
| diagnostics.Faithfulness = await EvaluateFaithfulness(response, contexts); | ||
|
|
There was a problem hiding this comment.
The C# "Implementation Example" is presented as code, but it is not self contained: _kernel is never initialized or used, RAGDiagnostics is undefined, and methods like EvaluateContextPrecision / EvaluateFaithfulness / EvaluateAnswerRelevancy / DetectPHI are missing. Either mark this block explicitly as pseudocode or provide minimal definitions and wiring so the example compiles and can be adapted by readers.
| private string DeterminePrimaryFailure(RAGDiagnostics d) | ||
| { | ||
| if (d.ContextPrecision < 0.3) return "Retrieval Collapse"; | ||
| if (d.Faithfulness < 0.5) return "Hallucination"; | ||
| if (d.AnswerRelevancy < 0.6) return "Answer Relevance Drift"; | ||
| if (d.PHIDetected) return "PHI Leakage"; | ||
| return "None Detected"; | ||
| } |
There was a problem hiding this comment.
DeterminePrimaryFailure depends on hard coded thresholds, but the sample does not explain where these values come from or how they should be tuned per model or dataset. Consider documenting that these are example thresholds and should be calibrated (or make them configurable inputs) so users do not treat them as SK defaults.
|
@microsoft-github-policy-service agree |
|
Closing this PR because the linked proposal in #13581 was declined by the maintainers. Thank you for the review feedback and for considering the contribution. |
Description
This PR adds a comprehensive RAG Failure Mode Checklist for Semantic Kernel, addressing Issue #13581.
Contents
Categories Covered
Quick Reference Table
Each failure mode includes:
Related Work
Author: Dr. Rajan Prasad Tripathi | AUT AI Innovation Lab