Add --exclude glob flag to skill scan#17
Conversation
Adds a repeatable `--exclude PATTERN` option to `skillspector scan`. Patterns use fnmatch semantics against the path relative to the scan root and are applied during file discovery in build_context, so excluded files are absent from both the components list and any analyzer findings. The motivating case is binary assets (e.g. a marketing-template PDF in `assets/`) whose raw bytes happen to match static regex patterns like `shell=True` or `-rf /`, producing HIGH-severity false positives that block adoption in CI. With this flag, those files can be filtered without moving them out of the skill bundle. Excluded files are logged at DEBUG, surfaced via `--verbose`. Excluding everything is valid; the scan succeeds with no findings.
8d77ad4 to
43dcb4a
Compare
|
This may need more discussion. Imagine if a someone uses |
rng1995
left a comment
There was a problem hiding this comment.
Review: COMMENT — solid, well-tested change, but it needs a rebase before it can land.
Thanks for this. The feature is well-scoped and the approach is sound: filtering once in _walk_skill_files, before components, file_cache, and component_metadata are built, means every downstream analyzer inherits the exclusion for free. I confirmed that the static pattern runner and the YARA node both iterate state["components"] and read from state["file_cache"] (and no analyzer re-walks the skill directory), so excluded files genuinely produce no findings — the "absent from the Components table and any analyzer findings" claim checks out. Docs and tests are thorough (directory glob, no-match, exclude-everything, plus an end-to-end CLI test against a binary fixture).
Blocking (rebase): this branch currently conflicts with main. It is not mergeable as-is — the conflict is in src/skillspector/nodes/build_context.py, against the recently merged change that switched path collection to rel.as_posix() (#86 / #87). When resolving, please make the final append use the posix form rather than str(rel):
# after the exclude check
paths.append(rel_str) # rel_str = rel.as_posix(), already computed aboveKeeping paths.append(str(rel)) would re-introduce OS-specific backslashes in component paths on Windows and undo that portability fix (these strings are used as dict keys and as SARIF/URI locations). Since you already compute rel_str = rel.as_posix() for matching, reusing it keeps everything consistent.
Minor / optional (non-blocking):
fnmatch.fnmatchapplies OS-dependent case normalization (case-sensitive on Linux, case-insensitive on Windows), so--exclude '*.pdf'would not matchREPORT.PDFon Linux but would on Windows. Given the repo's cross-platform care, considerfnmatch.fnmatchcasefor deterministic behavior (and document the casing), or keepfnmatchintentionally.- "Exclude everything" exits 0 with no findings. That is a reasonable, tested choice, but for a security scanner an over-broad pattern could mask a real result; a single INFO/DEBUG note when an exclusion removes all files might prevent a confusing "clean" outcome. Optional.
Nice, focused PR overall — once it is rebased with the posix-path resolution above, this looks good to me.
rng1995
left a comment
There was a problem hiding this comment.
[Automated SkillSpector Review]
Summary: Adds a repeatable --exclude PATTERN flag to skillspector scan, filtering files during discovery in _walk_skill_files (fnmatch against scan-root-relative paths) so excluded files are absent from components, file_cache, component_metadata, and all analyzer findings. Includes node-level and end-to-end CLI tests plus README docs. The core approach (filter once at discovery) is sound and the tests are generally thorough, but there are blocking issues.
Blockers
- Branch conflicts with
main(not mergeable). The conflict is insrc/skillspector/nodes/build_context.py: main now appendsrel.as_posix()(#86/#87, commitsbc6c542/54d9074) and_resolve_skill_dirnow raisesValueErrorinstead of returningNone(a8d284f), so the hunk anchored onreturn _minimal_update()no longer applies. A rebase is required. - Windows path regression baked into the resolution. The new code computes
rel_str = rel.as_posix()for matching but then doespaths.append(str(rel)), which re-introduces OS-specific backslashes in component paths on Windows — undoing the #87 fix (these strings are dict keys and SARIF URI locations). Usepaths.append(rel_str). - Exclusions are invisible in every report format — silent detection gap. Excluded files leave no trace in JSON/SARIF/markdown output (only a DEBUG log). This repo's existing false-positive mechanism (
src/skillspector/suppression.pybaseline) deliberately records every suppression in the report (suppressed,suppressed_count, requiredreason);--excludebypasses that convention. A CI job running--exclude '*'(or--exclude 'SKILL.md') exits 0 with a SAFE verdict and zero audit trail — and the PR's tests enshrine this as valid. Maintainer @keshprad raised exactly this on the thread. Please: (a) record applied exclude patterns and excluded-file count in report metadata across formats, and (b) emit a visible warning (or error) when exclusions remove all files orSKILL.md. - Post-rebase:
--excludewill silently not apply to multi-skill scans. On current main,scanroutes multi-skill directories through_scan_multi_skill(), which calls_scan_state(str(skill.path), format, no_llm, yara_rules_dir=yara_dir)without exclude patterns (src/skillspector/cli.py:375). When rebasing, either threadexclude_patternsthrough (deciding pattern semantics relative to each skill root vs. the scan root) or explicitly reject/warn when--excludeis combined with a multi-skill input.
Non-blocking
fnmatch.fnmatchappliesos.path.normcase, so matching is case-insensitive on Windows but case-sensitive on Linux/macOS (--exclude '*.pdf'missesREPORT.PDFon Linux only). Preferfnmatch.fnmatchcasefor deterministic cross-platform behavior and document the casing rule.- Excluding
SKILL.mddoes not exclude the manifest:_parse_manifest(skill_dir)readsSKILL.mddirectly regardless ofcomponents, so manifest-derived findings can still fire against an "excluded" file. Either document this or make the behavior consistent. tests/unit/test_cli.py::test_cli_scan_exclude_drops_pdf_from_components_and_findingsnever asserts that the PDF does produce a TM1 finding without--exclude, so it passes vacuously if the fixture stops triggering the pattern. Add a control scan without the flag asserting the finding is present.- README: consider cross-referencing the baseline suppression mechanism as the audit-friendly alternative for finding-level false positives, so users pick the right tool.
| if matched is not None: | ||
| logger.debug("Excluded by --exclude %r: %s", matched, rel_str) | ||
| continue | ||
| paths.append(str(rel)) |
There was a problem hiding this comment.
This should be paths.append(rel_str). str(rel) produces backslash-separated paths on Windows, regressing the as_posix() portability fix merged in #87 (these strings are used as file_cache keys and SARIF URI locations). You already compute rel_str = rel.as_posix() above — reuse it. Note this function is also the site of the merge conflict with main, so this needs to be resolved as part of the rebase.
| logger.debug("Skipping path (not under skill_dir): %s", item) | ||
| continue | ||
| rel_str = rel.as_posix() | ||
| matched = next((p for p in patterns if fnmatch.fnmatch(rel_str, p)), None) |
There was a problem hiding this comment.
fnmatch.fnmatch applies os.path.normcase, making matching case-insensitive on Windows but case-sensitive on POSIX — --exclude '*.pdf' will not match REPORT.PDF on Linux but will on Windows. Consider fnmatch.fnmatchcase for deterministic cross-platform behavior, and document the casing rule in the README either way.
| rel_str = rel.as_posix() | ||
| matched = next((p for p in patterns if fnmatch.fnmatch(rel_str, p)), None) | ||
| if matched is not None: | ||
| logger.debug("Excluded by --exclude %r: %s", matched, rel_str) |
There was a problem hiding this comment.
Exclusions are only visible at DEBUG level and leave no trace in JSON/SARIF/markdown output. The repo's suppression baseline records every suppressed finding in the report (suppressed, suppressed_count); --exclude should similarly surface applied patterns and excluded-file count in report metadata, and warn when exclusions remove all files or SKILL.md — otherwise --exclude '*' in CI yields a silent SAFE verdict with no audit trail.
| return skill_dir | ||
|
|
||
|
|
||
| def test_cli_scan_exclude_drops_pdf_from_components_and_findings(tmp_path: Path) -> None: |
There was a problem hiding this comment.
This test asserts the PDF produces no findings with --exclude, but never establishes that it produces a TM1 finding without the flag — if the fixture bytes stop triggering the pattern, the test passes vacuously. Add a control invocation without --exclude asserting the TM1 finding is present against assets/template-style.pdf.
Summary
--exclude PATTERNoption toskillspector scanbuild_context, so excluded files are absent from both the Components table and any analyzer findingsfnmatchsemantics against the path relative to the scan rootProblem
SkillSpector's static analyzers read every file with
utf-8 + errors='replace'and run regex patterns against the result. When a skill bundle ships binary assets (PDFs, images), raw bytes can incidentally match patterns likeshell=True,-rf /, or--no-verify, producing HIGH-severity false positives. There is currently no way to suppress these short of moving the file out of the skill folder, which is a non-starter for adoption in CI: every push surfaces a HIGH finding that has to be dismissed manually in the GitHub Security tab.Solution
List[str]via Typer), threaded into graph state asexclude_patterns_walk_skill_files, beforecomponents,file_cache, andcomponent_metadataare built — so the rest of the pipeline is unchanged--verbose)Tests
tests/nodes/test_build_context.py: exclusion filters components/file_cache/metadata, directory glob patterns, no-match leaves the list intact, exclude-everything yields empty statetests/unit/test_cli.py: end-to-end CLI invocation with a fixture skill containing a PDF whose bytes match TM1 (Tool Parameter Abuse) — asserts the PDF is absent fromcomponentsand produces noissues; also covers the repeatable form and the exclude-everything casemake test-unit)Test plan
make test-unitis greenmake lintandmake format-checkare clean--exclude '*.pdf'and reappears without it