Skip to content

[Feature] Multi-skill directories scanned as one monolithic entity, inflating score and losing granularity #136

Description

@mimran-khan

Summary

Imagine grading students, but instead of giving each student their own exam score, you add up all their mistakes into one number and give the entire class a single grade. A class of 20 average students would fail, while a class of 1 brilliant student would pass — not because one group is worse, but because more people means more total mistakes.

That's how SkillSpector handles directories containing multiple skills. In practice, skills are often organized into collections — a jira-tools/ folder might contain epic-planner/, comment-poster/, backlog-manager/, each with their own SKILL.md. When you point SkillSpector at the parent directory, it doesn't recognize these as independent skills. Instead, it treats the entire tree as one monolithic skill, sums all findings from all sub-skills into a single score, and reports the skill name as "unknown" (because there's no root-level SKILL.md).

This means you can't scan a skill registry or hub directory in one pass and get meaningful per-skill results. You're forced to manually identify and scan each sub-skill individually, which defeats the purpose of an automated scanner.

Reproduction

Create a skill collection directory:

skill-collection/
├── weather-lookup/
│   ├── SKILL.md
│   ├── tool.py
│   └── tests/
│       └── test_tool.py
├── email-sender/
│   ├── SKILL.md
│   ├── send.py
│   └── templates/
│       └── welcome.html
└── file-manager/
    ├── SKILL.md
    ├── manager.py
    └── docs/
        └── usage.md

Each subdirectory is an independent skill with its own SKILL.md. There is no root-level SKILL.md.

skillspector scan ./skill-collection/
# Skill name: "unknown"
# Components: all files from all 3 sub-skills combined
# Score: sum of ALL findings from ALL sub-skills

Compare with scanning a single sub-skill:

skillspector scan ./skill-collection/weather-lookup/
# Skill name: "weather-lookup"
# Components: only this skill's files
# Score: only this skill's findings

Root Cause

The resolve_inputbuild_context pipeline walks the entire directory tree without checking for multiple SKILL.md files:

  1. resolve_input resolves the path to a directory
  2. build_context recursively collects all files from that directory
  3. The manifest is parsed from the first/only SKILL.md found — if none exists at root, name is "unknown"
  4. All files across all subdirectories are treated as components of one skill
  5. All findings accumulate into one findings list via operator.add

There is no logic to detect that the scanned path contains multiple independent skills.

Impact

  • Unusable for skill registries/hubs that organize skills in directories
  • No per-skill granularity in reports — users can't identify which specific sub-skill has issues
  • Guaranteed score inflation due to accumulated findings from unrelated sub-skills (compounds the scoring saturation issue)
  • Reports "unknown" as skill name — confusing output with no attribution

Suggested Fix

  1. Auto-detect multi-skill directories: If the scanned path has no root SKILL.md but immediate subdirectories contain SKILL.md files, scan each subdirectory independently and produce per-skill reports.
  2. Aggregate report for directories: Optionally produce a summary report listing each sub-skill's individual score, plus an aggregate view.
  3. CLI flag: --recursive to explicitly enable multi-skill scanning with per-skill output.
  4. Warn on ambiguous input: If the scanner detects multiple SKILL.md files in subdirectories, warn the user: "Found N skills in this directory. Use --recursive to scan each independently."

Affected Version

SkillSpector v2.2.3

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions