Skip to content

chore: drop 19 MB of committed per-case scores.jsonl from incident-triage demo - #233

Merged
Minsoo Thigpen (minthigpen) merged 2 commits into
mainfrom
changliu2/drop-committed-scores-jsonl
Jun 12, 2026
Merged

chore: drop 19 MB of committed per-case scores.jsonl from incident-triage demo#233
Minsoo Thigpen (minthigpen) merged 2 commits into
mainfrom
changliu2/drop-committed-scores-jsonl

Conversation

@changliu2

Copy link
Copy Markdown
Collaborator

What

The four incident-triage variant runs committed their full per-test-case judge outputs (\scores.jsonl, ~5 MB each = 19.1 MB total), bloating every clone and CI checkout. This untracks them and re-ignores the pattern.

Kept (the README + viewer rely on these)

  • \metrics.json\ — the aggregate numbers cited in the README results table
  • \manifest.json, \config.yaml\ — per-variant run metadata
  • \ rade_off.png\ — the chart the README embeds (unchanged)

\scripts/render_trade_off.py\ already falls back to the documented case-study numbers when \scores.jsonl\ is absent (see its module docstring), so the chart still re-renders.

Tradeoff

Viewer per-case drill-down on these four pre-computed demo runs now shows summary metrics only (no per-row judge data). Re-run the suite locally to regenerate \scores.jsonl\ (gitignored going forward).

Scope

  • 4 files deleted, 19.1 MB
  • .gitignore: re-ignore \�xamples/incident_triage_agent/artifacts/**/scores.jsonl\ (the explicit artifact negation above it still allows the small curated files)
  • Local working copies preserved (\git rm --cached\ only)

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

…iage demo

The four incident-triage variant runs committed their full per-test-case judge outputs (scores.jsonl, ~5 MB each = 19.1 MB total), which bloats every clone and CI checkout. Untrack them and re-ignore the pattern. The curated summaries the README and viewer rely on stay committed: metrics.json (the cited aggregate numbers), manifest.json, config.yaml, and trade_off.png. render_trade_off.py already falls back to the documented case-study numbers when scores.jsonl is absent, so the chart still re-renders. Only per-case drill-down in the viewer is affected for these four pre-computed demo runs; re-run locally to regenerate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I verified the ignore rule behaves the way the PR intends: the generated incident-triage scores.jsonl files are ignored, while the curated metrics.json, manifest.json, config.yaml, and trade_off.png artifacts remain trackable. I also checked there are no tracked incident-triage scores.jsonl files left and ran python scripts/render_trade_off.py --suite incident-triage-agent-v1 --out /tmp/incident-triage-tradeoff-pr233.png to confirm the fallback path still renders without the committed score rows.

One non-blocking follow-up worth considering later: the incident-triage README still mentions scores.jsonl in a couple of artifact path summaries. That is still true for freshly regenerated local runs, but less true for the checked-in demo snapshot after this cleanup.

…tted (Jake's #233 follow-up)

Addresses the non-blocking note on #233: the per-variant artifact summaries (B.1 naive-prompt, B.2 guarded-with-shield-gepa) listed scores.jsonl among the committed files, but after the cleanup it is gitignored and regenerated locally. Updated both summaries to list the committed curated files (config.yaml, manifest.json, metrics.json) and note scores.jsonl is regenerated on a fresh run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@changliu2

Copy link
Copy Markdown
Collaborator Author

Addressed your non-blocking follow-up in bc4aede (thanks for the thorough verify on the render fallback).

The two per-variant artifact summaries (Appendix B.1 naive-prompt and B.2 guarded-with-shield-gepa) listed scores.jsonl among the committed files. After this cleanup it's gitignored and regenerated locally, so both summaries now list the committed curated files — {config.yaml, manifest.json, metrics.json} — and note scores.jsonl is regenerated on a fresh run. The render-fallback mentions (the scripts/render_trade_off.py section) and the general "each run writes … scores.jsonl" line are left as-is since they're accurate for fresh local runs.

Docs-only change on top of your approval — re-requesting a quick re-approval per the new commit.

@minthigpen

Copy link
Copy Markdown
Contributor

Approved docs-only change on top of Jake's first approval.

@minthigpen
Minsoo Thigpen (minthigpen) merged commit 5643d11 into main Jun 12, 2026
16 checks passed
@minthigpen
Minsoo Thigpen (minthigpen) deleted the changliu2/drop-committed-scores-jsonl branch June 12, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants