Fix(#53): 정중한 어조 위장 사칭 문자가 Gemini 2차 검증을 못 받는 문제

[kite-pp]

📝 개요

"정중한 어조로 위장한 사칭 문자"가 나이브베이즈 1차 필터의 오판으로 Gemini 2차 검증을 아예 못 받고 통과하는 문제를 수정합니다. 변형 공격 테스트(#47) 중 발견했으며, 원인은 Gemini의 판단 오류가 아니라 하이브리드 로직의 조기 스킵이었습니다.

🔗 관련 이슈

• Closes [Fix] 정중한 어조 위장 사칭 문자가 Gemini 2차 검증을 못 받는 문제 #53

🎯 주요 변경 사항

• app/analysis/service.py: _analyze_text_hybrid가 rule_score를 받아, 규칙엔진이 이미 SUSPICIOUS 이상(rule_score ≥ RISK_MEDIUM_THRESHOLD)을 감지했으면 나이브베이즈의 SAFE 판정만으로 Gemini 2차 검증을 스킵하지 않도록 변경. analyze_pipeline에서 URL 추적 이전에 텍스트 기반 규칙 점수를 미리 계산해 전달.
• app/analysis/text/gemini_analyzer.py: SYSTEM_PROMPT에 "정중한 어조는 안전 신호가 아니다"는 명시적 지침(Tone Camouflage 지표 5번)과 few-shot 예시(Example 3) 추가.
• tests/analysis/test_service.py: 기존 하이브리드 스킵 테스트 4건에 rule_score 인자 반영, 신규 회귀 테스트 1건 추가.

📸 사진

생략

✅ PR 체크리스트

• 관련 이슈를 연결했습니다.
• 구현 범위와 변경 이유를 설명했습니다.
• 로컬 테스트(uvicorn 구동 또는 테스트 코드)를 통과했습니다. (222건 통과, kiwipiepy 미설치로 인한 무관한 기존 이슈 1건 제외)
• API 변경 사항이 있다면 Swagger / API 명세에 반영했습니다. (해당 없음 - 외부 API 스키마 변경 없음, 내부 로직만 변경)
• 민감 정보(API Key, 시크릿 키 등)가 코드·로그·테스트 데이터에 포함되지 않았습니다.
• 프론트엔드 또는 메인 백엔드(Spring)에 영향을 주는 응답 스키마 또는 Enum 변경이 있다면 팀에 공유했습니다. (해당 없음)
• 병합(Merge) 전 작업 브랜치를 삭제하지 않았습니다.

🧪 검증

실제 API(gemini-3.5-flash-lite, 무료 티어 대체 모델)로 기존 실패 사례 25건(tone_normalization 21 + urgency_softening 4)을 재평가한 결과 24건이 탐지로 전환됐습니다 (tone_normalization 16%→96%, urgency_softening 84%→100%). 프로덕션 기본 모델(gemini-flash-latest)로의 최종 재확인은 유료 티어 전환 후 별도 진행 필요합니다.

Summary by CodeRabbit

• Improvements
  • Improved smishing detection by recognizing formal, official-sounding messages that may disguise fraudulent intent.
  • Enhanced analysis to perform additional review when rule-based signals indicate elevated risk, even if an initial classifier considers the message safe.
  • Improved consistency of risk scoring across combined analysis methods.

[coderabbitai]

Warning

Review limit reached

@kite-pp, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 49 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f9ad0b9d-4c4a-4692-a6a9-b55bf343bf89

📥 Commits

Reviewing files that changed from the base of the PR and between 9afe296 and 53e669c.

📒 Files selected for processing (2)

• app/analysis/service.py
• tests/analysis/test_service.py

📝 Walkthrough

Walkthrough

The hybrid analysis pipeline now combines preliminary rule scores with Naive Bayes results before skipping Gemini. The Gemini prompt now treats formal wording as possible smishing camouflage and includes a formal bank impersonation example. Regression tests cover rule-score escalation.

Changes

Smishing escalation

Layer / File(s)	Summary
Rule-aware hybrid pipeline app/analysis/service.py, tests/analysis/test_service.py	The pipeline computes a preliminary rule score before text analysis. Gemini is skipped only when Naive Bayes reports SAFE and the rule score is below the medium-risk threshold. Tests cover score propagation, fallback behavior, and escalation.
Tone-camouflage detection guidance app/analysis/text/gemini_analyzer.py	The Gemini prompt identifies formal or polite wording as possible camouflage and adds a formal bank impersonation example with an unverified phone number.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• SafeFam/SafeFam_AI#20: Extends the hybrid Naive Bayes/Gemini pipeline with rule-score-aware escalation.
• SafeFam/SafeFam_AI#24: Introduces related hybrid escalation and Gemini-analysis behavior.
• SafeFam/SafeFam_AI#48: Evaluates related rule-score and pipeline escalation behavior.

Suggested reviewers: pearseona

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the fix for polite-tone impersonation messages that bypass Gemini verification.
Linked Issues check	✅ Passed	The changes implement the linked issue's service fix, prompt guidance, and regression coverage for Gemini escalation.
Out of Scope Changes check	✅ Passed	All changes remain within the linked issue scope: hybrid analysis, Gemini guidance, and regression tests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/tone-normalization-gemini-skip

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@app/analysis/service.py`:
- Around line 39-50: Update the rule preview exception handling used by
_analyze_text_hybrid so rule_score_preview is set to RISK_MEDIUM_THRESHOLD
rather than zero when rule_analyzer fails, and log the preview failure. Ensure
this fail-safe value prevents the SAFE Naive Bayes path from skipping Gemini
even when the final rule pass also fails. Add an integration test covering a
raising rule_analyzer, SAFE Naive Bayes output, and verification that Gemini is
invoked.

In `@tests/analysis/test_service.py`:
- Around line 157-192: The test
test_hybrid_text_track_escalates_when_rule_score_high_despite_naive_bayes_safe
currently uses a hardcoded rule score that does not verify the threshold
boundary. Import and pass RISK_MEDIUM_THRESHOLD as rule_score so the test
confirms Gemini is called when the score is exactly at least medium risk, while
preserving the existing assertions.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3d7ec44c-b551-42d6-9f48-361ff8c0ce91

📥 Commits

Reviewing files that changed from the base of the PR and between dc6b9fd and 9afe296.

📒 Files selected for processing (3)

• app/analysis/service.py
• app/analysis/text/gemini_analyzer.py
• tests/analysis/test_service.py

[kite-pp]

CodeRabbit 리뷰 반영 완료 (53e669c):

1. Major/보안: rule_score_preview 예외 시 폴백값을 0 → RISK_MEDIUM_THRESHOLD로 변경. 규칙 미리보기가 실패하는 상황까지 커버하는 통합 테스트(test_analyze_pipeline_escalates_to_gemini_when_rule_preview_raises) 추가.
2. Minor/테스트: 임의값(55) 대신 RISK_MEDIUM_THRESHOLD 상수로 경계값 테스트하도록 수정.

전체 테스트 223건 통과 (환경에 kiwipiepy 미설치로 무관한 test_stacking_analyzer.py만 제외).