feat: ミューテーションテスト週次実行と Slack 通知を導入（ADR-0017）#450
Conversation
テストスイートの検出力（弱い assertion / 実装なぞり）を可視化するため、
mutmut（backend）/ Stryker（web）を高価値な決定論的ビジネスロジック限定で
導入し、週次実行と用途別 Slack 通知を追加する。
- backend: requirements.txt に mutmut を追加し pyproject に [tool.mutmut]
（source_paths + only_mutate）を定義。mutmut の pytest 内部実行と干渉する
ため pytest addopts の --cov を Makefile / test.yml の呼び出し側へ移設
（カバレッジ出力は従来どおり）
- backend: test_error_code_sync.py のリポジトリルート解決を parents[2] 固定
から祖先探索へ変更（mutmut が tests/ を mutants/ 配下へコピーしても動作）
- web: @stryker-mutator/core + vitest-runner を devDeps に追加し
stryker.conf.json と test:mutation script を定義（vitest 4 互換確認済み）
- CI: mutation.yml（週次 月曜 3:00 JST + workflow_dispatch。BE/FE 並列・
warn-only・QUALITY 通知）、notify.yml（workflow_run("CI") で CI 失敗通知
+ main/stg/dev のデプロイ結果通知）、deps-notify.yml（pull_request_target
で Renovate/Dependabot の PR 起票通知）を新規作成
- Slack Webhook は SLACK_WEBHOOK_URL_{CI,DEPLOY,QUALITY,DEPS} の
プレースホルダー Secrets 参照（未登録の間は通知を静かに skip）
- Makefile に mutation-backend / mutation-web を追加、.gitignore に生成物を
追記、CLAUDE.md / docs/development.md / ADR-0017 を整備
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XcbPTeHXDTxNs4yyXwZJhW
github-actions
Bot
added
feature
|
Warning Review limit reached
Next review available in: 54 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds weekly mutation testing for backend and web, Slack notifications for CI/deploy/dependency events, backend coverage-flag relocation, a test path fix for copied runs, and ADR and development documentation for the new workflows. ChangesMutation Testing Setup
Estimated code review effort: 3 (Moderate) | ~30 minutes Slack Notification Workflows
Estimated code review effort: 3 (Moderate) | ~25 minutes ADR and Development Documentation
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant Scheduler
participant MutationWorkflow
participant MutationBackendJob
participant MutationWebJob
participant Slack
Scheduler->>MutationWorkflow: trigger weekly run or manual dispatch
MutationWorkflow->>MutationBackendJob: run mutmut and collect stats
MutationWorkflow->>MutationWebJob: run Stryker and collect stats
MutationBackendJob-->>MutationWorkflow: killed/survived/score outputs
MutationWebJob-->>MutationWorkflow: killed/survived/score outputs
MutationWorkflow->>Slack: post quality summary if webhook is set
sequenceDiagram
participant CIWorkflow
participant NotifyWorkflow
participant GitHubAPI
participant Slack
CIWorkflow->>NotifyWorkflow: workflow_run completed
NotifyWorkflow->>GitHubAPI: list run jobs
GitHubAPI-->>NotifyWorkflow: job statuses
NotifyWorkflow->>NotifyWorkflow: build CI or deploy summary
NotifyWorkflow->>Slack: send incoming webhook payload
Possibly related PRs
Suggested labels: ci, testing, documentation Suggested reviewers: yusuke0610 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (4)
.github/workflows/notify.yml (3)
25-25: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
timed_outconclusion is silently excluded from CI failure notifications.
conclusion == 'failure'won't catch a run that hittimeout-minutesand finished withconclusion: timed_out— that's a distinct enum value fromfailure. A hung/timed-out CI run currently produces no Slack notification even though it's arguably the failure case most worth surfacing.🐛 Proposed fix to also notify on timeout
- if: github.event.workflow_run.conclusion == 'failure' + if: contains(fromJSON('["failure", "timed_out"]'), github.event.workflow_run.conclusion)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/notify.yml at line 25, The CI notification gate in the workflow currently only checks workflow_run.conclusion for failure, so timed_out runs are missed. Update the condition in the notify workflow to include the timed_out conclusion as well, using the workflow_run event branch that handles Slack alerts, so both failure and timeout cases trigger the same notification path.
37-40: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
gh apifailures are silently swallowed, dropping notifications instead of failing loudly.Neither script uses
set -e/explicit error checks aroundgh api. If the API call errors (rate limit, transient network issue),names/linesend up empty: fornotify-cithe message just says "no job-level failures" (misleading but still sent); fornotify-deploythis makescount=0, which skips the entire deploy notification at Line 123/150 — a real deploy failure could go completely unreported.♻️ Proposed hardening for the deploy-collection step
run: | + set -euo pipefail # 再利用ワークフロー（deploy.yml）内のジョブは "deploy-dev / deploy-web" の # 形式で列挙される。"tests / deploy-gate" は startswith にマッチしない。 # skipped（deploy_gate 不通過・変更なし等）は通知対象にしない。 - lines=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}/jobs?per_page=100" --paginate \ - --jq '[.jobs[] | select(.name | startswith("deploy-")) - | select(.conclusion == "success" or .conclusion == "failure") - | "\(if .conclusion == "success" then "✅" else "❌" end) \(.name)"] | join("\n")') + if ! lines=$(gh api "repos/${REPO}/actions/runs/${RUN_ID}/jobs?per_page=100" --paginate \ + --jq '[.jobs[] | select(.name | startswith("deploy-")) + | select(.conclusion == "success" or .conclusion == "failure") + | "\(if .conclusion == "success" then "✅" else "❌" end) \(.name)"] | join("\n")'); then + echo "::warning::gh api call failed; falling back to a generic notice" + lines="⚠️ deploy ジョブ結果の取得に失敗しました" + fiAlso applies to: 109-113
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/notify.yml around lines 37 - 40, The job-collection steps in notify-ci and notify-deploy are swallowing gh api errors, which can turn API failures into empty outputs and hide real notification failures. Harden the collection logic by enabling immediate failure handling or adding an explicit exit check around gh api in the steps that build names and lines, so the workflow stops instead of emitting misleading empty results. Make sure the notify-ci branch that writes names to GITHUB_OUTPUT and the deploy collection step that computes count both fail loudly on API errors.
61-64: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winStatic heredoc delimiters combined with untrusted text (
TITLE/display_title) riskGITHUB_OUTPUTline injection.GitHub's own guidance is explicit here: make sure the delimiter you're using is randomly generated and unique for each run, per "Understanding the risk of script injections".
NOTIFY_EOF/DEPLOY_EOFare fixed, predictable strings; ifTITLE(workflow_run display_title, attacker-influenceable via PR title/commit subject) ever contains a line matching the delimiter exactly, subsequent lines could be parsed as additionalGITHUB_OUTPUTkey/value pairs. Actual blast radius here is small (onlysummary/bodyoutputs are consumed, safely viatoJSON()), but the fix is trivial.🔒 Proposed fix using a random delimiter
+ delimiter="NOTIFY_EOF_$(openssl rand -hex 8)" { echo "summary=❌ ${WF_NAME} failed (${BRANCH})" - echo "body<<NOTIFY_EOF" + echo "body<<${delimiter}" echo "$body" - echo "NOTIFY_EOF" + echo "${delimiter}" } >> "$GITHUB_OUTPUT"Also applies to: 116-119
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/notify.yml around lines 61 - 64, The heredoc blocks used to write to GITHUB_OUTPUT in the notification workflow rely on fixed delimiters, which can be abused if untrusted TITLE/display_title text contains the delimiter token. Update the output-writing logic in the notify job to generate a unique random delimiter per run for both the body and summary outputs, and use that delimiter consistently when emitting the multiline values so the GITHUB_OUTPUT parsing cannot be injected..github/workflows/deps-notify.yml (1)
22-22: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winSpoofable bot-actor check: prefer
github.event.pull_request.user.login.zizmor's bot-conditions rule applies here: this condition is spoofable: github.actor refers to the last actor to perform an "action" on the triggering context, and not necessarily the actor actually causing the trigger. An attacker can take advantage of this discrepancy to create a PR where the HEAD commit has github.actor == 'dependabot[bot]' but the rest of the branch history contains attacker-controlled code, bypassing the actor check. In general, checking a trigger's authenticity via github.actor is insufficient; instead, most users should use github.event.pull_request.user.login or similar, since that context refers to the actor that created the Pull Request rather than the last one to modify it. Since this workflow triggers only on
openedand doesn't check out code, exploitability is currently limited, but the fix is a one-line, defense-in-depth change onceSLACK_WEBHOOK_URL_DEPSis populated.🔒 Proposed fix using the PR-author context
- if: github.actor == 'renovate[bot]' || github.actor == 'dependabot[bot]' + if: github.event.pull_request.user.login == 'renovate[bot]' || github.event.pull_request.user.login == 'dependabot[bot]'🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/deps-notify.yml at line 22, The bot check in the deps notification workflow is using a spoofable actor field, so update the condition in the workflow job to key off the pull request author context instead of github.actor. Replace the actor comparison in deps-notify.yml with github.event.pull_request.user.login while keeping the existing renovate/dependabot allowlist, so the check is tied to the PR creator rather than the last actor on the event.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/workflows/deps-notify.yml:
- Line 22: The bot check in the deps notification workflow is using a spoofable
actor field, so update the condition in the workflow job to key off the pull
request author context instead of github.actor. Replace the actor comparison in
deps-notify.yml with github.event.pull_request.user.login while keeping the
existing renovate/dependabot allowlist, so the check is tied to the PR creator
rather than the last actor on the event.
In @.github/workflows/notify.yml:
- Line 25: The CI notification gate in the workflow currently only checks
workflow_run.conclusion for failure, so timed_out runs are missed. Update the
condition in the notify workflow to include the timed_out conclusion as well,
using the workflow_run event branch that handles Slack alerts, so both failure
and timeout cases trigger the same notification path.
- Around line 37-40: The job-collection steps in notify-ci and notify-deploy are
swallowing gh api errors, which can turn API failures into empty outputs and
hide real notification failures. Harden the collection logic by enabling
immediate failure handling or adding an explicit exit check around gh api in the
steps that build names and lines, so the workflow stops instead of emitting
misleading empty results. Make sure the notify-ci branch that writes names to
GITHUB_OUTPUT and the deploy collection step that computes count both fail
loudly on API errors.
- Around line 61-64: The heredoc blocks used to write to GITHUB_OUTPUT in the
notification workflow rely on fixed delimiters, which can be abused if untrusted
TITLE/display_title text contains the delimiter token. Update the output-writing
logic in the notify job to generate a unique random delimiter per run for both
the body and summary outputs, and use that delimiter consistently when emitting
the multiline values so the GITHUB_OUTPUT parsing cannot be injected.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ff237a62-0548-48a0-89f2-7b278f0a0e2d
⛔ Files ignored due to path filters (1)
web/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (14)
.claude/CLAUDE.md.github/workflows/deps-notify.yml.github/workflows/mutation.yml.github/workflows/notify.yml.github/workflows/test.yml.gitignoreMakefilebackend/pyproject.tomlbackend/requirements.txtbackend/tests/test_error_code_sync.pydocs/adr/0017-mutation-testing-and-slack-notifications.mddocs/development.mdweb/package.jsonweb/stryker.conf.json
CodeRabbit の指摘（defense-in-depth）に対応: - notify.yml: CI が timeout-minutes 到達で終了した場合の conclusion は timed_out であり failure 条件では拾えないため、両方を通知対象にする - notify.yml: 失敗ジョブ名 / デプロイ結果を収集する gh api の失敗を明示 チェックし、握りつぶさずフォールバック文言で通知する（特に deploy は count=0 で通知全体がスキップされ実際の失敗が埋もれるのを防止）。API 失敗 フォールバック時はデプロイ通知の絵文字を⚠️ にする - notify.yml / deps-notify.yml / mutation.yml: GITHUB_OUTPUT の heredoc デリミタを固定値から run 毎のランダム値（openssl rand）に変更し、 untrusted な display_title / PR タイトルによる出力インジェクションを防ぐ。 summary も heredoc 化して 1 行直書き（改行混入で壊れる）を解消 - deps-notify.yml: スプーフ可能な github.actor ではなく PR 作成者 （pull_request.user.login）でボット判定する Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XcbPTeHXDTxNs4yyXwZJhW
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/notify.yml (1)
123-129: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winStatic heredoc delimiter left unhardened for
linesoutput.Both other
GITHUB_OUTPUTmultiline writes in this file (lines 65-70 and 154-159) were hardened with a per-run random delimiter (GHOUT_$(openssl rand -hex 8)) to avoid collision with untrusted content, per the PR's stated security fix. This block still uses the staticDEPLOY_EOFdelimiter, so if anydeploy-*job name (or the fallback warning text) ever contained the literal stringDEPLOY_EOF, theGITHUB_OUTPUTparsing would break/desync.🔒 Apply the same random-delimiter pattern used elsewhere in this file
count=$(printf '%s' "$lines" | grep -c . || true) + delim="GHOUT_$(openssl rand -hex 8)" { echo "count=$count" - echo "lines<<DEPLOY_EOF" + echo "lines<<$delim" echo "$lines" - echo "DEPLOY_EOF" + echo "$delim" } >> "$GITHUB_OUTPUT"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/notify.yml around lines 123 - 129, The multiline GITHUB_OUTPUT write in the lines handling the `lines` output still uses a fixed heredoc marker, unlike the hardened patterns elsewhere in this workflow. Update the output block in `notify.yml` to use the same per-run random delimiter approach already used around the other `GITHUB_OUTPUT` writes, reusing the `lines`/deploy job output logic and ensuring the delimiter is generated uniquely before writing the heredoc.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/notify.yml:
- Around line 123-129: The multiline GITHUB_OUTPUT write in the lines handling
the `lines` output still uses a fixed heredoc marker, unlike the hardened
patterns elsewhere in this workflow. Update the output block in `notify.yml` to
use the same per-run random delimiter approach already used around the other
`GITHUB_OUTPUT` writes, reusing the `lines`/deploy job output logic and ensuring
the delimiter is generated uniquely before writing the heredoc.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7a124720-68c9-4246-8ffd-065cb1bb75f2
📒 Files selected for processing (3)
.github/workflows/deps-notify.yml.github/workflows/mutation.yml.github/workflows/notify.yml
🚧 Files skipped from review as they are similar to previous changes (2)
- .github/workflows/deps-notify.yml
- .github/workflows/mutation.yml
前コミットで compose の summary/body はランダムデリミタ化したが、 notify-deploy のデプロイ結果収集ステップの lines 出力に静的 DEPLOY_EOF が 残っていた。ジョブ名やフォールバック文言が万一デリミタと衝突しても GITHUB_OUTPUT が壊れないよう、他の出力と同じ openssl rand デリミタに統一する。 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XcbPTeHXDTxNs4yyXwZJhW
2 participants
変更概要
テストスイートの検出力（弱い assertion / 実装なぞりテスト）を可視化するため、mutmut（backend）/ Stryker（web）を高価値な決定論的ビジネスロジック限定で導入し、週次実行ワークフロー（
mutation.yml）と用途別 Slack 通知（CI 失敗 / デプロイ結果 / 品質 / 依存更新 PR の 4 チャンネル）を追加しました。Webhook はプレースホルダー Secrets（SLACK_WEBHOOK_URL_{CI,DEPLOY,QUALITY,DEPS}）参照で、未登録の間は通知が静かに skip されます（設計判断: ADR-0017）。主な変更点
mutmut==3.6.0をrequirements.txtに追加、pyproject.tomlに[tool.mutmut]（source_paths + only_mutate。blog スコアリング / billing / skills 推論 / repo_analyzer / shared / tasks/worker に限定）。mutmut の pytest 内部実行と干渉するため pytest addopts の--covを Makefile /test.ymlの呼び出し側へ移設（カバレッジ出力は従来どおり）tests/test_error_code_sync.pyのリポジトリルート解決をparents[2]固定から祖先探索へ変更（mutmut が tests/ をmutants/配下へコピーしても動作するように）@stryker-mutator/core+vitest-runnerを devDeps に追加（vitest 4 互換をスモーク実行で確認済み）、stryker.conf.json（utils / hooks / formMappers / payloadBuilders / slice に限定。generated.ts・定数・テストは除外）とtest:mutationscript を定義mutation.yml（週次 月曜 3:00 JST + workflow_dispatch のみ。PR/push では動かない warn-only。BE/FE 並列 → QUALITY 通知に killed / survived / score% を含め、score 80% 未満でnotify.yml（workflow_run("CI")で CI 失敗通知 + main/stg/dev のデプロイ結果通知。jobs API で失敗ジョブ名・環境別結果を補完）、deps-notify.yml（pull_request_target[opened] + actor 限定・checkout なしで Renovate/Dependabot の PR 起票を通知）mutation-backend/mutation-web、.gitignoreに生成物、CLAUDE.md/docs/development.md（ローカル実行・Secrets 登録手順）/ ADR-0017 を整備セルフレビューチェックリスト
必須確認
make ciが pass している（lint + test + build-web）※実行環境に nix devshell が無いため、CI と同一コマンドで個別に実施: ruff / pyright（0 errors）/ pytest 667 件 pass（--cov 出力維持を確認）/ lint-env-keys / eslint / lint-web-messages / vitest 365 件 + node:test pass / vite build / pip-audit clean / actionlint（新規・変更ワークフロー全 pass）条件付き確認（該当する場合のみ N/A と記入）
app/schemas//app/routers/の変更なし（codegen 不要。web/src/api/generated.tsに差分なし）web/src/への日本語メッセージ追加なし（設定ファイルと devDeps のみ）破壊的変更
ADR（設計判断を伴う変更の場合のみ）
docs/adr/0017-mutation-testing-and-slack-notifications.md）レビュー時の注意点
slackapi/slack-github-actionの SHA（91efab10… # v2.1.0）のみ未検証: 実装環境のネットワーク制限で外部リポジトリの SHA 解決ができなかったため。マージ後の初回gh workflow run mutation.ymlで action が解決できるか確認し、失敗する場合は SHA を差し替えてください（以後は Renovate のpinDigestsが追従管理）gh workflow run mutation.ymlで QUALITY 通知（Secrets 未登録なら skip）とレポート artifact、② dev への push でデプロイ通知のジョブ名形式（deploy-dev / deploy-web）を実地確認してください🤖 Generated with Claude Code
https://claude.ai/code/session_01XcbPTeHXDTxNs4yyXwZJhW
Generated by Claude Code
Summary by CodeRabbit