chore: drop 19 MB of committed per-case scores.jsonl from incident-triage demo - #233
Conversation
…iage demo The four incident-triage variant runs committed their full per-test-case judge outputs (scores.jsonl, ~5 MB each = 19.1 MB total), which bloats every clone and CI checkout. Untrack them and re-ignore the pattern. The curated summaries the README and viewer rely on stay committed: metrics.json (the cited aggregate numbers), manifest.json, config.yaml, and trade_off.png. render_trade_off.py already falls back to the documented case-study numbers when scores.jsonl is absent, so the chart still re-renders. Only per-case drill-down in the viewer is affected for these four pre-computed demo runs; re-run locally to regenerate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jake Present (jakepresent)
left a comment
There was a problem hiding this comment.
Approved. I verified the ignore rule behaves the way the PR intends: the generated incident-triage scores.jsonl files are ignored, while the curated metrics.json, manifest.json, config.yaml, and trade_off.png artifacts remain trackable. I also checked there are no tracked incident-triage scores.jsonl files left and ran python scripts/render_trade_off.py --suite incident-triage-agent-v1 --out /tmp/incident-triage-tradeoff-pr233.png to confirm the fallback path still renders without the committed score rows.
One non-blocking follow-up worth considering later: the incident-triage README still mentions scores.jsonl in a couple of artifact path summaries. That is still true for freshly regenerated local runs, but less true for the checked-in demo snapshot after this cleanup.
…tted (Jake's #233 follow-up) Addresses the non-blocking note on #233: the per-variant artifact summaries (B.1 naive-prompt, B.2 guarded-with-shield-gepa) listed scores.jsonl among the committed files, but after the cleanup it is gitignored and regenerated locally. Updated both summaries to list the committed curated files (config.yaml, manifest.json, metrics.json) and note scores.jsonl is regenerated on a fresh run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Addressed your non-blocking follow-up in The two per-variant artifact summaries (Appendix B.1 Docs-only change on top of your approval — re-requesting a quick re-approval per the new commit. |
|
Approved docs-only change on top of Jake's first approval. |
What
The four incident-triage variant runs committed their full per-test-case judge outputs (\scores.jsonl, ~5 MB each = 19.1 MB total), bloating every clone and CI checkout. This untracks them and re-ignores the pattern.
Kept (the README + viewer rely on these)
\scripts/render_trade_off.py\ already falls back to the documented case-study numbers when \scores.jsonl\ is absent (see its module docstring), so the chart still re-renders.
Tradeoff
Viewer per-case drill-down on these four pre-computed demo runs now shows summary metrics only (no per-row judge data). Re-run the suite locally to regenerate \scores.jsonl\ (gitignored going forward).
Scope
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com