Skip to content

Feat: Report zizmor findings at any severity - #80

Merged
askb merged 1 commit into
lfreleng-actions:mainfrom
modeseven-lfreleng-actions:feat/zizmor-informational-severity
Jul 28, 2026
Merged

Feat: Report zizmor findings at any severity#80
askb merged 1 commit into
lfreleng-actions:mainfrom
modeseven-lfreleng-actions:feat/zizmor-informational-severity

Conversation

@ModeSevenIndustrialSolutions

Copy link
Copy Markdown
Contributor

Feat: Report zizmor findings at any severity

Companion to .github#133, which moves the organisation scan pipeline's zizmor floor from low to informational. This makes the report count what that pipeline now publishes.

Why

The zizmor category cut at Severity.LOW, on the documented assumption that informational findings never reached the SARIF — which the low floor guaranteed. That guarantee is going away.

The gap was not theoretical: a scan of python-workflows produced 14 template-injection findings that the floor discarded, while the report showed "Zizmor Static Analysis: ✅ 110 Clean" across the estate.

Change

-        fail_severity=Severity.LOW,
+        fail_severity=Severity.INFORMATIONAL,

Plus the three stale comments (in categories.py and severity.py) that justified the old cutoff.

Why note still maps to LOW

zizmor emits both Low and Informational findings at SARIF level note. The raw SARIF distinguishes them via a zizmor/severity property, but the code-scanning alerts API this tool consumes exposes only rule.severity — the SARIF level — so the two are genuinely indistinguishable here.

Rather than guess, note keeps mapping to LOW (erring towards over- rather than under-stating), and the INFORMATIONAL cutoff makes the ambiguity moot for reporting: both tiers surface either way.

The cutoff also closes a real gap at SARIF level none, which normalises to INFORMATIONAL and previously passed silently regardless of the floor.

Scope

Only the zizmor category changes. aislop keeps its LOW cutoff and the global default stays MEDIUM — the scan pipeline change is zizmor-only, so lowering aislop would alter its posture with no corresponding publisher change.

Tests

Added TestCategoryFailSeverity, pinning both the zizmor cutoff and the unchanged global default. Nothing previously asserted the zizmor default — only config overrides were covered — so this change would have passed the suite silently. It now can't regress unnoticed.

  • uv run pytest405 passed (403 before, +2)
  • pre-commit — ruff, ruff-format, mypy, basedpyright, reuse, codespell all pass

Note for operators

Anyone running the released CLI (v0.8.0) before this ships can get the same behaviour today via config:

{ "report": { "categories": { "zizmor": { "fail_severity": "informational" } } } }

That override becomes redundant once this merges, but stays harmless.

The organisation scan pipeline is moving its zizmor floor from 'low'
to 'informational' so every finding zizmor can report reaches the
SARIF and code scanning. Match that here by cutting the zizmor
category at INFORMATIONAL rather than LOW, so no published finding
folds into the clean count.

The previous cutoff assumed informational findings never reached the
SARIF, which the 'low' floor guaranteed. That guarantee is going
away: a scan of python-workflows produced 14 template-injection
findings that the floor discarded, invisible to this report.

zizmor emits both Low and Informational findings at SARIF level
'note', and the code-scanning alerts API exposes only that level, not
the zizmor/severity property carried in the raw SARIF, so the two
cannot be separated here. 'note' therefore still maps to LOW to avoid
under-stating the estate, and the INFORMATIONAL cutoff makes the
distinction moot for reporting: both tiers surface either way. The
cutoff also closes a gap at SARIF level 'none', which normalises to
INFORMATIONAL and previously passed silently.

Add coverage pinning the zizmor cutoff and the unchanged MEDIUM
global default, so a future change cannot silently reintroduce the
gap.

Co-authored-by: Claude <claude@anthropic.com>
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Zizmor reporting category to treat findings at any severity as reportable failures by lowering its default fail_severity floor to Severity.INFORMATIONAL, aligning the report’s behavior with the organisation scan pipeline now publishing informational-level Zizmor results.

Changes:

  • Lower Zizmor’s default fail_severity cutoff from LOW to INFORMATIONAL so informational findings are no longer folded into “clean”.
  • Update/refresh the inline documentation in severity.py and categories.py to reflect the new pipeline floor and the SARIF note ambiguity.
  • Add tests that pin the Zizmor default cutoff and assert the global default remains MEDIUM.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
tests/test_severity.py Adds regression tests locking down Zizmor’s cutoff at INFORMATIONAL and the global default at MEDIUM; updates SARIF note mapping rationale comments.
src/github_security_report/severity.py Updates documentation explaining why SARIF note continues to map to LOW despite informational findings now being present.
src/github_security_report/categories.py Lowers Zizmor category fail_severity to INFORMATIONAL and updates comments to match the new reporting posture.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@zxiiro zxiiro left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Auto-approved by agent: reviewed reporting-logic change for security and CI/CD impact, found low risk. Lowers the zizmor category's fail-severity floor to INFORMATIONAL to match the org's --min-severity informational scan pipeline (since zizmor's SARIF encoder can't distinguish Low from Informational at the 'note' level); well-tested, no permissions/secrets/network changes.

@askb
askb merged commit 59683eb into lfreleng-actions:main Jul 28, 2026
25 checks passed
@ModeSevenIndustrialSolutions
ModeSevenIndustrialSolutions deleted the feat/zizmor-informational-severity branch July 28, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants