Dashboard: Stop counting skipped/xfailed tests as passing coverage (#211) - #327
Merged
Conversation
tadani3
changed the title
Performance Comparison
|
|
The author of this PR, tadani3, is not an activated member of this organization on Codecov. |
Closed
added 4 commits
June 2, 2026 18:05
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Member
|
@tadani3 thanks! Could you share two screenshots of before and after the change? |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes the confidence dashboard’s coverage accounting so skipped and xfailed tests no longer inflate “passing” coverage, by correcting the producer-side flags in scripts/generate_dashboard.py and aligning the UI’s L2/L3 semantics with the new pass-gating.
Changes:
- Fix producer logic to gate L4/L5 golden coverage on YAML
skip_reason, and gate L2/L3 “passing” on explicit status (l2_status == "pass",l3_status == "pass"). - Add L2 status scanning (parse+graph vs graph-only xfail) and expose L2 status/configured fields to the rendered JSON for correct dot/badge rendering.
- Add a comprehensive regression test suite for the dashboard generators/scanners/aggregations/rendering pipeline.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
scripts/generate_dashboard.py |
Corrects producer flags and introduces L2 status scanning + pass-gating (l2_passes/l3_passes) across confidence, summary, and JSON emission. |
scripts/templates/dashboard.html.j2 |
Updates dot rendering and adds L2 xfail UI affordances (badges, detail rows, summary annotations). |
tests/generate_dashboard_test.py |
Adds end-to-end regression tests covering scanner, aggregation, rendering, and meta-invariants against the live repo. |
added 8 commits
June 3, 2026 21:18
… improve awaiting data count
…nd add xfail status extraction
…attributes for clarity
Member
|
Please ping me again when this is ready to review, thanks! |
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> Signed-off-by: Tommaso Adani <83273681+tadani3@users.noreply.github.com>
justinchuby
approved these changes
Jun 5, 2026
Contributor
Author
|
Before (DinoV3 Original Issue) After (DinoV3 Original Issue) |
Contributor
Author
|
Before (Dashboard Flat Page) After (Dashboard Flat Page) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a class of user-visible dashboard defects whose common root cause is the
data-collection layer of
scripts/generate_dashboard.pymarking tests as"passed" when they did not actually run, then layers on L2 expected-failure
tracking, a unified per-level status model, and a round of UI/legibility
polish.
Per the repo guideline "root-cause before remedy — never mask symptoms by
adding compensating logic", correctness fixes are applied at the producers
(scanners +
ModelInfo/summary properties), not at the downstream consumers.1. Coverage-correctness fixes (skipped/xfailed tests no longer count as passing)
Two independent producers were setting "this test passed" flags for tests that
never ran:
_scan_l4_golden_files/_scan_l5_generation_goldensetl4_golden_files/l5_generation_goldenwhenever a JSON file existed undertestdata/golden/,even when the YAML case had a
skip_reason.ModelInfo.confidence_leveland one_compute_summarybranch read the rawl3_synthetic_parityflag without checkingl3_status == "pass".Bugs addressed (Issue #211):
l4_golden_files,confidence_levell4_golden_files,l5_generation_goldenl4_golden_files,l5_generation_golden,confidence_level(L3)l4_golden_files,l5_generation_goldenl4_golden_filesl4_golden_filesL4:Ncounts skipped modelsl4_golden_filesl4_golden_files,confidence_levelconfidence_levelL3 analogs (skipped/xfailed L3 inflating
confidence_leveland escaping the"Not Tested" bucket) are fixed via the new
ModelInfo.l3_passesproperty and thel3_status == "pass"guards.Additionally,
min_token_match_ratiois no longer recorded for skipped YAMLcases, so the dashboard never shows an "L5 token match threshold" for a test that
doesn't run.
2. L2 expected-failure tracking + unified status model
_XFAIL_REASONS) with dots, badges, and CI-skipannotations.
authoritative via the
model_typefield (disambiguates registry entries thatshare a
test_model_id, e.g.dinov2/dinov3_vit).ModelInforeworked so L2–L5 share a{status, configured}contract(
status ∈ {pass, xfail, skip, None}). Status badge classes unified acrosslevels.
3. UI / legibility polish
all use the
--l1..--l5palette; xfail given a distinct purple outside theL0–L5 spectrum. Matrix cells switched to solid fills so same-hue neighbors
(L1/L2 amber, L3/L4 green) are distinguishable.
positive states rely on the Confidence column + dot strip (removes
redundant/duplicate badges).
confidence labels, tooltips, and the matrix legend.
nowrapso the L1–L5 family badges stay on one row.by_level[0]semantics ("no passing coverage at anylevel"), L2 xfail bucketing, and the YAML disambiguation rule.
Template note: the Jinja template's skipped-aware branches already existed;
once the upstream flags are correct they render the right output.
Files changed
scripts/generate_dashboard.pyscripts/templates/dashboard.html.j2