correct license-header criteria and add eval suite#205
Conversation
|
yep just fixing that now -issue is mpostly that the example can't follow the linting rules we have |
AH.. We should make some rules for exclusions then in AGENTS.md |
paulk-asert
left a comment
There was a problem hiding this comment.
I hadn't thought about whether an empty jar should be allowed or not in a src distribution. We obviously don't want any opaque binaries. I can see a case for also banning empty jars. And I can see a case for a more lenient interpretation that perhaps allowed a jar containing fully readable javadoc html. But I am happy with what is captured now as a starting point - all these files will evolve once the rubber hits the road.
|
Empty jars do exist in existing projects. I've also come across jars that contain no code, but that's very rare. |
Yeah. We also have almost empty - just metadata .whl files (which are also .zip files BTW. |
Follow-up to #195: fix license-header criteria and add eval suite
PR #195 added the
## License headerssection tocriteria.md. This PR corrects several issues found during review and adds a full behavioral eval suite covering the skill's Step 3, 4, and 6 sub-checks.Criteria fixes
Correctness fixes
`nit` / no finding— changed tono finding.Calibration fixes
minorfinding even if the files in this PR carry correct headers, because the pattern silently degrades the tool for all future PRs.Exemption gaps
Added exemptions that were missing from the original list:
.md,.rst,.txt) — ASF projects are conventionally lenient.READMEfiles in any format.LICENSE,NOTICE,DISCLAIMER, and similar legal declaration files — these are the licence artefact and must not carry an Apache header.Category A attribution rule
The previous wording stated that a
licenses/directory entry and aLICENSEupdate were both expected. This is incorrect: alicenses/directory is a common good practice but is not required by ASF policy. The sole determining factor is whetherLICENSEorLICENSE.txtwas updated with an attribution notice.Eval suite
Adds
tools/skill-evals/evals/pr-management-code-review/— 36 cases across 6 suites covering the skill's deterministic sub-checks:step-3-security-disclosure-scanstep-4-third-party-licenselicenses/-only → majorstep-4-compiled-artifacts.jar/.pyc/.so/.whldetection; major vs blocking escalationstep-4-image-ipstep-4-license-headersstep-6-dispositionAPPROVE/REQUEST_CHANGES/COMMENTauto-pick logicCurrently all test pass