Skip to content

Test fidelity check should fail CI when upstream tests aren't ported #53

Description

@patrick-chinchill

Process gap — silent test coverage erosion.

The problem

scripts/verify_test_fidelity.py reports every TS it("...") that has no matching Python test_*():

```
MISSING: [Options Load] should call onOptionsLoad handler for a matching action ID
MISSING: [Options Load] should prefer specific handlers before catch-all handlers
...
TOTAL: 548/588 matched (93%), 40 missing, 3 absorbers
```

But it exits 0 regardless of how many are missing. CI prints "Run with --fix to generate stubs for missing tests" and moves on. New upstream tests land unported, undetected, for releases at a time.

Why it's bad

This is how we ended up with:

Every one of those shows up in fidelity output. Every one got ignored because the script exits 0.

Fix

Option A — strict: exit non-zero when `missing > 0`. Simple, forces discipline.

Option B — graduated: exit non-zero when `missing > K` (where K is the current count, i.e. treat today's gap as the baseline and ratchet down). Allows incremental adoption without a big-bang PR.

Option C — per-file threshold: allow configured gaps per file in a `fidelity-baseline.json` (like pyrefly's). New misses must be explicitly acknowledged.

I'd recommend Option A for post-4.26 PRs + a separate catch-up PR that either ports the 40 missing tests or explicitly marks them as "intentionally skipped" with a reason.

Acceptance

  • `verify_test_fidelity.py --strict` (or equivalent) exits non-zero on missing tests
  • CI workflow invokes the strict mode
  • Existing 40 gaps handled (either ported or baselined with rationale)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions