Feat: Report zizmor findings at any severity - #80
Conversation
The organisation scan pipeline is moving its zizmor floor from 'low' to 'informational' so every finding zizmor can report reaches the SARIF and code scanning. Match that here by cutting the zizmor category at INFORMATIONAL rather than LOW, so no published finding folds into the clean count. The previous cutoff assumed informational findings never reached the SARIF, which the 'low' floor guaranteed. That guarantee is going away: a scan of python-workflows produced 14 template-injection findings that the floor discarded, invisible to this report. zizmor emits both Low and Informational findings at SARIF level 'note', and the code-scanning alerts API exposes only that level, not the zizmor/severity property carried in the raw SARIF, so the two cannot be separated here. 'note' therefore still maps to LOW to avoid under-stating the estate, and the INFORMATIONAL cutoff makes the distinction moot for reporting: both tiers surface either way. The cutoff also closes a gap at SARIF level 'none', which normalises to INFORMATIONAL and previously passed silently. Add coverage pinning the zizmor cutoff and the unchanged MEDIUM global default, so a future change cannot silently reintroduce the gap. Co-authored-by: Claude <claude@anthropic.com> Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
There was a problem hiding this comment.
Pull request overview
This PR updates the Zizmor reporting category to treat findings at any severity as reportable failures by lowering its default fail_severity floor to Severity.INFORMATIONAL, aligning the report’s behavior with the organisation scan pipeline now publishing informational-level Zizmor results.
Changes:
- Lower Zizmor’s default
fail_severitycutoff fromLOWtoINFORMATIONALso informational findings are no longer folded into “clean”. - Update/refresh the inline documentation in
severity.pyandcategories.pyto reflect the new pipeline floor and the SARIFnoteambiguity. - Add tests that pin the Zizmor default cutoff and assert the global default remains
MEDIUM.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
tests/test_severity.py |
Adds regression tests locking down Zizmor’s cutoff at INFORMATIONAL and the global default at MEDIUM; updates SARIF note mapping rationale comments. |
src/github_security_report/severity.py |
Updates documentation explaining why SARIF note continues to map to LOW despite informational findings now being present. |
src/github_security_report/categories.py |
Lowers Zizmor category fail_severity to INFORMATIONAL and updates comments to match the new reporting posture. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
zxiiro
left a comment
There was a problem hiding this comment.
🤖 Auto-approved by agent: reviewed reporting-logic change for security and CI/CD impact, found low risk. Lowers the zizmor category's fail-severity floor to INFORMATIONAL to match the org's --min-severity informational scan pipeline (since zizmor's SARIF encoder can't distinguish Low from Informational at the 'note' level); well-tested, no permissions/secrets/network changes.
Feat: Report zizmor findings at any severity
Companion to .github#133, which moves the organisation scan pipeline's zizmor floor from
lowtoinformational. This makes the report count what that pipeline now publishes.Why
The zizmor category cut at
Severity.LOW, on the documented assumption that informational findings never reached the SARIF — which thelowfloor guaranteed. That guarantee is going away.The gap was not theoretical: a scan of
python-workflowsproduced 14template-injectionfindings that the floor discarded, while the report showed "Zizmor Static Analysis: ✅ 110 Clean" across the estate.Change
Plus the three stale comments (in
categories.pyandseverity.py) that justified the old cutoff.Why
notestill maps to LOWzizmor emits both Low and Informational findings at SARIF level
note. The raw SARIF distinguishes them via azizmor/severityproperty, but the code-scanning alerts API this tool consumes exposes onlyrule.severity— the SARIF level — so the two are genuinely indistinguishable here.Rather than guess,
notekeeps mapping toLOW(erring towards over- rather than under-stating), and the INFORMATIONAL cutoff makes the ambiguity moot for reporting: both tiers surface either way.The cutoff also closes a real gap at SARIF level
none, which normalises toINFORMATIONALand previously passed silently regardless of the floor.Scope
Only the zizmor category changes.
aislopkeeps itsLOWcutoff and the global default staysMEDIUM— the scan pipeline change is zizmor-only, so lowering aislop would alter its posture with no corresponding publisher change.Tests
Added
TestCategoryFailSeverity, pinning both the zizmor cutoff and the unchanged global default. Nothing previously asserted the zizmor default — only config overrides were covered — so this change would have passed the suite silently. It now can't regress unnoticed.uv run pytest— 405 passed (403 before, +2)pre-commit— ruff, ruff-format, mypy, basedpyright, reuse, codespell all passNote for operators
Anyone running the released CLI (v0.8.0) before this ships can get the same behaviour today via config:
{ "report": { "categories": { "zizmor": { "fail_severity": "informational" } } } }That override becomes redundant once this merges, but stays harmless.