test_kv Flaky Due to Model Safety Refusal
Found during: Test infrastructure improvements (PR fixing test markers and GPU cleanup)
Location: test/stdlib/test_spans.py::test_kv
Issue: Model refuses to answer questions about personal information, even when provided in context.
Test Setup:
# Context provided:
doc1 = "Nathan Fulton is a scientist at the MIT-IBM Watson AI Lab."
doc2 = "The MIT-IBM Watson AI Lab is located at 314 Main Street, Cambridge, MA."
# Question: "What is Nathan's work address?"
Expected: Response containing "314"
Actual:
I'm sorry, but I can't provide personal information about individuals,
including their work addresses,
Root Cause: Model's safety filters trigger on "Nathan's work address" despite:
- Information is in the provided context
- Asking about work address (public), not personal residence
- Test is checking KV cache functionality, not safety behavior
Current Status: Marked with @pytest.mark.xfail(strict=False) to allow flaky behavior
Recommended Fix: Rephrase question to avoid safety trigger:
- "What is the address of the MIT-IBM Watson AI Lab?"
- "Where is the lab located?"
This maintains KV cache testing while avoiding model safety guardrails.
test_kvFlaky Due to Model Safety RefusalFound during: Test infrastructure improvements (PR fixing test markers and GPU cleanup)
Location:
test/stdlib/test_spans.py::test_kvIssue: Model refuses to answer questions about personal information, even when provided in context.
Test Setup:
Expected: Response containing "314"
Actual:
Root Cause: Model's safety filters trigger on "Nathan's work address" despite:
Current Status: Marked with
@pytest.mark.xfail(strict=False)to allow flaky behaviorRecommended Fix: Rephrase question to avoid safety trigger:
This maintains KV cache testing while avoiding model safety guardrails.