Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .skillspector-baseline.example.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# SkillSpector baseline (example)
#
# A baseline suppresses known/accepted findings so re-scans surface only NEW
# issues. Pass it with: skillspector scan <path> --baseline <this-file>
# Generate a fingerprint baseline automatically: skillspector baseline <path>
#
# See docs/SUPPRESSION.md for the full reference. All identifiers below are
# placeholders — replace them with your own rule ids, paths, and reasons.

version: 1

# Glob rules — human-authored, drift-tolerant (survive line/wording changes).
# A finding is suppressed when EVERY field a rule sets glob-matches it.
# Unspecified fields match anything. `reason` is required for auditability.
rules:
# Suppress an entire rule across all skills (global pattern suppression).
- id: "SQP-1"
reason: "Trigger-phrase breadth is a skill-description nit, not a vulnerability"

# Suppress a rule family with a glob, scoped by message substring.
- id: "SQP-*"
message: "*telemetry*"
reason: "First-party internal telemetry; reviewed and accepted"

# Skill/file-scoped suppression of a specific false positive.
- id: "SSD-2"
path: "example-skill/SKILL.md"
message: "*example false-positive phrase*"
reason: "False positive: phrase is a benign trigger, not an instruction"

# Fingerprints — exact, machine-generated suppressions (one per accepted
# finding). Regenerate with `skillspector baseline` when a skill changes.
fingerprints:
- hash: "sha256:0123456789abcdef"
rule_id: "SDI-2"
file: "example-skill/SKILL.md"
reason: "Accepted: reads its own environment ($EXAMPLE_TOKEN) for context"
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ SkillSpector helps you answer: **"Is this skill safe to install?"**
- **Live vulnerability lookups**: SC4 queries [OSV.dev](https://osv.dev) for real-time CVE data with automatic offline fallback
- **Multiple output formats**: Terminal, JSON, Markdown, and SARIF reports
- **Risk scoring**: 0-100 score with severity labels and clear recommendations
- **Baseline / false-positive suppression**: Accept known findings via a glob-rule or fingerprint baseline so re-scans surface only *new* issues ([docs](docs/SUPPRESSION.md))

## Quick Start

Expand Down Expand Up @@ -146,6 +147,26 @@ skillspector scan ./my-skill/ --format markdown --output report.md
skillspector scan ./my-skill/ --format sarif --output report.sarif
```

### Suppressing False Positives (baseline)

Suppress known/accepted findings so the risk score reflects only un-triaged
issues and re-scans surface only *new* findings. See the
[suppression guide](docs/SUPPRESSION.md) for the full reference.

```bash
# Accept all current findings into a baseline (run once), then commit it.
skillspector baseline ./my-skill/ -o .skillspector-baseline.yaml

# Scan against the baseline — only NEW findings are reported and scored.
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml

# Review what was suppressed (still excluded from the score).
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml --show-suppressed
```

A baseline can also use drift-tolerant glob rules (by rule id, file path, or
message) — see [`.skillspector-baseline.example.yaml`](.skillspector-baseline.example.yaml).

### LLM Analysis

For the best results, configure an OpenAI-compatible LLM endpoint for
Expand Down Expand Up @@ -436,8 +457,14 @@ Options:
-f, --format [terminal|json|markdown|sarif] Output format [default: terminal]
-o, --output PATH Output file path
--no-llm Skip LLM analysis (static only)
--yara-rules-dir PATH Extra YARA rules directory
-b, --baseline PATH Suppress findings listed in a baseline
--show-suppressed List baseline-suppressed findings
-V, --verbose Show detailed progress
--help Show this message and exit

# Generate a baseline of all current findings (see docs/SUPPRESSION.md)
skillspector baseline <path> [-o FILE] [--no-llm] [--reason TEXT]
```

## Integrating SkillSpector
Expand Down
6 changes: 5 additions & 1 deletion docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,9 @@ All targets assume the virtual environment is **already created and activated**.
| `output_format` | Requested report format: `terminal`, `json`, `markdown`, or `sarif` |
| `report_body` | Formatted report string (set by report node from `output_format`) |
| `use_llm` | When False, meta_analyzer skips LLM and uses fallback (e.g. for `--no-llm`) |
| `baseline` | Loaded `suppression.Baseline` (set by CLI/API from `--baseline`); report node drops matching findings before scoring |
| `show_suppressed` | When True, baseline-suppressed findings are listed in the report (still excluded from the risk score) |
| `suppressed_findings` | List of `SuppressedFinding` (finding + reason) produced by the report node |
| `findings` | All raw findings from analyzers (reducer: `operator.add`) |
| `filtered_findings` | Findings after meta_analyzer |
| `model_config` | Optional model IDs per node (e.g. default, meta_analyzer) |
Expand Down Expand Up @@ -126,7 +129,7 @@ There are no conditional edges: after `resolve_input` → `build_context`, all a
| **build_context** | Reads `skill_path`, populates `components`, `file_cache`, `ast_cache`, `manifest`, `component_metadata`, `has_executable_scripts` | [build_context.py](../src/skillspector/nodes/build_context.py) |
| **Analyzers** | 20 nodes; each returns `AnalyzerNodeResponse` (list of `Finding`). State reducer appends to `findings`. | [nodes/analyzers/__init__.py](../src/skillspector/nodes/analyzers/__init__.py) (`ANALYZER_NODE_IDS`, `ANALYZER_NODES`) |
| **meta_analyzer** | Per-file LLM filter/enrich of `findings` → `filtered_findings` via `LLMMetaAnalyzer`; one LLM call per file (or per chunk for oversized files); token budgets from `constants.py`; falls back when `use_llm` is False | [meta_analyzer.py](../src/skillspector/nodes/meta_analyzer.py), [llm_analyzer_base.py](../src/skillspector/nodes/llm_analyzer_base.py) |
| **report** | Builds SARIF 2.1.0, computes `risk_score`, `risk_severity`, `risk_recommendation`; writes `report_body` from `output_format` (terminal/json/markdown/sarif) | [report.py](../src/skillspector/nodes/report.py) |
| **report** | Applies baseline suppression (`state["baseline"]`), then builds SARIF 2.1.0, computes `risk_score`, `risk_severity`, `risk_recommendation` from the non-suppressed findings; writes `report_body` from `output_format` (terminal/json/markdown/sarif) | [report.py](../src/skillspector/nodes/report.py) |

---

Expand All @@ -142,6 +145,7 @@ There are no conditional edges: after `resolve_input` → `build_context`, all a
| `llm_utils.py` | `chat_completion()` for OpenAI-compatible / NVIDIA Inference API |
| `cli.py` | Typer app: `scan` (with input resolution, `--format`, `--no-llm`), `--version` |
| `input_handler.py` | Resolves Git URL, file URL, .zip, single file, or directory to a local directory path |
| `suppression.py` | Baseline / false-positive suppression: `Baseline`, `SuppressionRule`, `load_baseline`, `partition_findings`, `finding_fingerprint`, `build_baseline_dict` (see [SUPPRESSION.md](SUPPRESSION.md)) |
| `__init__.py` | Package version (from pyproject.toml via `importlib.metadata`) |
| `sarif_models.py` | SARIF 2.1.0 Pydantic models and `validate_sarif_report()` |
| **nodes/** | |
Expand Down
119 changes: 119 additions & 0 deletions docs/SUPPRESSION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Baseline / False-Positive Suppression

SkillSpector's analyzers — especially the LLM semantic ones — can produce
findings that are correct in general but not actionable for *your* skills
(framework/architectural patterns, first-party tooling conventions, accepted
lab practices). A **baseline** lets you suppress those known findings so that:

- the risk score reflects only **un-triaged** issues,
- re-scans surface only **new** findings (incremental CI/CD), and
- every suppression carries an auditable **reason**.

Suppressed findings never count toward the risk score and are excluded from the
SARIF results. They are shown in the terminal/Markdown report only when you pass
`--show-suppressed`, and are always listed (machine-readable) in the JSON report
under `suppressed` / `suppressed_count`.

> Addresses [issue #88](https://github.com/NVIDIA/SkillSpector/issues/88).

## Quick start

```bash
# 1. Accept all current findings into a baseline (run once).
skillspector baseline ./my-skill/ -o .skillspector-baseline.yaml

# 2. Commit the baseline, then scan against it. Only NEW findings are reported.
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml

# Review what was suppressed.
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml --show-suppressed
```

## CLI

| Command / option | Description |
|------------------|-------------|
| `skillspector baseline <path> [-o FILE] [--no-llm] [--reason TEXT]` | Scan and write a baseline that fingerprint-suppresses every current finding. Default output: `.skillspector-baseline.yaml`. |
| `skillspector scan <path> --baseline FILE` (`-b`) | Suppress findings matching the baseline before scoring/reporting. |
| `skillspector scan <path> --baseline FILE --show-suppressed` | Also list the suppressed findings (they still don't affect the score). |

A missing or malformed baseline file exits with code 2.

## Baseline file format

YAML or JSON (the `.json` extension selects JSON output when generating). Two
complementary mechanisms:

```yaml
version: 1

rules: # human-authored, glob-based, drift-tolerant
- id: "SQP-1" # glob over the finding's rule id
reason: "Trigger-phrase breadth is a description nit, not a vuln"
- id: "SSD-2"
path: "example-skill/SKILL.md" # glob over the finding's file
message: "*example false-positive phrase*" # glob over the finding's message
reason: "False positive: benign trigger phrase, not an instruction"

fingerprints: # machine-generated, exact
- hash: "sha256:1a2b3c4d5e6f7081"
rule_id: "SDI-2" # informational (for humans reading the file)
file: "example-skill/SKILL.md"
reason: "Accepted — reads its own environment for context"
```

### `rules` — glob suppression

A finding is suppressed when **every** field a rule specifies matches it;
unspecified fields match anything. Use this for:

- **Global pattern suppression** — `id: "SQP-1"` (or `id: "SQP-*"`) drops a rule
or rule family across all skills.
- **Skill/file-scoped suppression** — add `path:` (and optionally `message:`) to
scope the suppression to a specific skill, file, or message.

Field reference:

| Field | Matches against | Notes |
|-------|-----------------|-------|
| `id` (or `rule_id`) | `Finding.rule_id` | glob |
| `path` (or `file`) | `Finding.file` | glob; `*` crosses `/`, `**` is an alias for `*` |
| `message` | `Finding.message` | glob, case-insensitive; wrap a keyword in `*` for substring |
| `reason` | — | required; recorded in reports and audits |

Glob matching uses Python's [`fnmatch`](https://docs.python.org/3/library/fnmatch.html),
so `*` matches across path separators (`*SKILL.md` matches `a/b/SKILL.md`).
Rules are **drift-tolerant**: they keep working after line numbers shift or
content is reworded.

### `fingerprints` — exact suppression

Each entry is the stable hash of one finding
(`sha256(rule_id|file|start_line|end_line|message)`, truncated). Generated by
`skillspector baseline`. Because the hash includes the line span and message,
editing a skill so a finding moves or is reworded changes its fingerprint —
**regenerate the baseline** after material changes, or prefer `rules` for
suppressions you want to survive edits.

An entry may be a bare string (`"sha256:..."`) or a mapping with `hash`,
optional `reason`, and informational `rule_id` / `file`.

## How it fits the pipeline

Suppression is applied in the **report node** (`skillspector/nodes/report.py`),
the single place where findings are scored and formatted, so the CLI and any
future REST API behave identically. The CLI loads the baseline file into a
`skillspector.suppression.Baseline` and passes it via graph state
(`state["baseline"]`, `state["show_suppressed"]`); the report node partitions
findings into kept vs. suppressed via
`skillspector.suppression.partition_findings`.

## Recommended workflow

1. Triage the first scan. For genuine false positives, prefer a `rules` entry
with a clear `reason` (drift-tolerant). For "accept everything as-is right
now", run `skillspector baseline` to fingerprint them.
2. Commit the baseline file to the repo.
3. In CI, run `skillspector scan <path> --baseline <file>`; the build fails
(exit 1) only when a **new** finding pushes the risk score above threshold.
4. Periodically review with `--show-suppressed` and prune stale entries.
110 changes: 109 additions & 1 deletion src/skillspector/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
from skillspector.graph import graph
from skillspector.logging_config import get_logger, set_level
from skillspector.multi_skill import MultiSkillDetectionResult, detect_skills
from skillspector.suppression import build_baseline_dict, dump_baseline, load_baseline

logger = get_logger(__name__)

Expand Down Expand Up @@ -92,6 +93,8 @@ def _scan_state(
format: FormatChoice,
no_llm: bool,
yara_rules_dir: str | None = None,
baseline: Path | None = None,
show_suppressed: bool = False,
) -> dict[str, object]:
"""Build initial graph state from scan CLI args."""
state: dict[str, object] = {
Expand All @@ -101,6 +104,10 @@ def _scan_state(
}
if yara_rules_dir is not None:
state["yara_rules_dir"] = yara_rules_dir
if baseline is not None:
# Loading may raise FileNotFoundError/ValueError, mapped to exit code 2 by scan().
state["baseline"] = load_baseline(baseline)
state["show_suppressed"] = show_suppressed
return state


Expand Down Expand Up @@ -180,6 +187,23 @@ def scan(
help="Scan directories containing multiple skills (immediate subdirectories with SKILL.md) independently.",
),
] = False,
baseline: Annotated[
Path | None,
typer.Option(
"--baseline",
"-b",
help="Baseline file (YAML/JSON) of suppressed findings. Matching findings "
"are dropped before scoring. Generate one with 'skillspector baseline'.",
),
] = None,
show_suppressed: Annotated[
bool,
typer.Option(
"--show-suppressed",
help="List findings suppressed by the baseline in the report (they still "
"do not count toward the risk score).",
),
] = False,
verbose: Annotated[
bool,
typer.Option(
Expand Down Expand Up @@ -240,7 +264,14 @@ def scan(
result = None
try:
yara_dir = str(yara_rules_dir.resolve()) if yara_rules_dir else None
state = _scan_state(input_path, format, no_llm, yara_rules_dir=yara_dir)
state = _scan_state(
input_path,
format,
no_llm,
yara_rules_dir=yara_dir,
baseline=baseline,
show_suppressed=show_suppressed,
)
if verbose:
console.print("[dim]Running scan...[/dim]")
logger.debug(
Expand Down Expand Up @@ -374,5 +405,82 @@ def _scan_multi_skill(
raise typer.Exit(code=1)


@app.command()
def baseline(
input_path: Annotated[
str,
typer.Argument(
help="Path or URL to scan. Supports: Git URL, file URL, zip file, .md file, or directory.",
),
],
output: Annotated[
Path,
typer.Option(
"--output",
"-o",
help="Where to write the baseline file (YAML; .json extension writes JSON).",
),
] = Path(".skillspector-baseline.yaml"),
no_llm: Annotated[
bool,
typer.Option(
"--no-llm",
help="Skip LLM analysis when generating the baseline (static analysis only).",
),
] = False,
reason: Annotated[
str,
typer.Option(
"--reason",
help="Reason recorded for every suppressed finding in the baseline.",
),
] = "Accepted finding (auto-generated baseline)",
verbose: Annotated[
bool,
typer.Option("--verbose", "-V", help="Show detailed progress."),
] = False,
) -> None:
"""
Generate a baseline file that suppresses every finding in the current scan.

Run this once to accept all existing findings, then commit the file and pass
it to future scans with --baseline so only NEW findings are reported.

Examples:

skillspector baseline ./my-skill/
skillspector baseline ./my-skill/ -o team-baseline.yaml --no-llm
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml
"""
result = None
try:
if verbose:
set_level("DEBUG")
console.print("[dim]Scanning to build baseline...[/dim]")
# output_format is irrelevant here; we consume findings, not report_body.
state = _scan_state(input_path, FormatChoice.json, no_llm)
result = graph.invoke(state)
findings = result.get("filtered_findings") or result.get("findings") or []
data = build_baseline_dict(findings, reason=reason)
dump_baseline(data, output)
console.print(
f"[green]Wrote baseline with {len(findings)} suppressed finding(s) to:[/green] {output}"
)
except typer.Exit:
raise
except (FileNotFoundError, ValueError) as e:
console.print(f"[red]Error:[/red] {e}")
raise typer.Exit(code=2) from e
except Exception as e:
if verbose:
console.print_exception()
else:
console.print(f"[red]Error:[/red] {e}")
raise typer.Exit(code=2) from e
finally:
if result is not None:
_cleanup_result(result)


if __name__ == "__main__":
app()
Loading