tend ci-fix: transient classifier ignores repeat occurrences

[prql-bot]

Problem

The bundled ci-fix skill's transient-vs-structural classifier does not track repeat occurrences. A failure that recurs across multiple commits and multiple ci-fix/triage sessions within hours can still be classified "transient" each time, because each session evaluates the failure in isolation against the skill's three criteria in §3a (same code path succeeded recently, filesystem/network-level shape, matching upstream incident).

The macos-15 cargo→rustup-init failure in PRQL is a clean example: the bot misclassified it as transient at least four times in 8.5 hours before finally diagnosing it as structural and shipping the fix.

Evidence (PRQL, 2026-05-14)

The same macos-15 Swatinem/rust-cache@v2 cache-bin corruption surfaced four ci-fix/triage runs and four review runs in the same window, all classified transient, all rerunning the failed job, all reruns happening to pass:

Time	Workflow	Run	Disposition
11:44	ci-fix	25858204087	Transient. Opened + closed #5885. $4.06.
11:44	triage (on #5885)	25858203080	Transient. Rerun passed. $2.45.
15:08	ci-fix	25867840990	Transient. Session text says "This matches issue #5885 exactly — same-day transient ... Per the ci-fix skill's step 3a, this is transient." $2.05.
15:18	triage (on #5886)	25868370551	Acknowledged already-closed.
17:18–17:37	review × 4	25874429943, 25874436469, 25874493395, 25875379565	Each independently reran the same flake on a different dependabot PR. Combined cost ~$10.
20:10	ci-fix	25882958133	Correctly diagnosed structural — filed PR #5897 (cache-bin: false workaround). Merged 23:01Z.

The 15:08 session is the smoking gun: the bot saw a literal match against an issue it had filed the same morning and still applied §3a's transient path.

Prior windows have shown the same shape — yesterday's review-runs tracking comment (#5852 comment id 4404889246) notes: "identical shape to multiple prior windows' macOS bootstrap flakes." So the misclassification is structural across weeks, not a one-day quirk.

Root cause

§3a of ci-fix/SKILL.md reads:

Use this path when:

• The same code path succeeded on a recent prior run with no relevant changes between runs
• The failure shape is filesystem/network-level, not anything the project's code does
• An upstream status incident matches the timing and components

If you can't tell whether it's transient, treat it as durable and create a fix PR.

All three criteria evaluate the current failure in isolation. There is no signal for "the bot has classified this same shape as transient N times in the recent past." The implicit assumption that a rerun-pass settles the question is wrong when the underlying bug is intermittent (cache contents depend on prior runs' state, runner image rotation, etc.) — the rerun passes some of the time, and each pass reinforces the wrong classification.

Proposed fix

Add a repeat-occurrence escalation to §3a. Before applying the transient path, query bot-filed ci-fix: issues from the past 7 days for the same failure shape; if 2+ matches exist, treat as durable and create a fix PR.

Concrete addition (drop in under §3a's existing bullet list):

### Repeat-occurrence escalation

Before applying the transient path, check whether the same failure shape has
been classified transient recently:

\`\`\`bash
BOT_LOGIN=$(gh api user --jq '.login')
gh issue list --state all --author "$BOT_LOGIN" --search "ci-fix: in:title" \
  --json number,title,createdAt,closedAt \
  --jq "[.[] | select(.createdAt >= (now - 7*86400 | todateiso8601))] | length"
\`\`\`

Match by failure-shape keyword (e.g. \`rustup-init\`, \`composer connect timeout\`,
\`docker pull rate limit\`) not by job name — the same root cause can surface on
multiple jobs.

If 2+ prior matches exist within the past 7 days, escalate to durable: a fault
that re-fires every 1–3 days is not transient even when individual reruns pass.
Search for an upstream-documented workaround (\`gh issue search\` against the
action's repo, the action's README, GitHub Community threads) and apply it as
the fix. If no upstream workaround is documented, file a fix PR proposing a
minimal mitigation (e.g. pin runner image, skip the affected leg, disable the
relevant cache layer) and link the upstream tracking issue.

Optional secondary tightening: in the bullet "same code path succeeded on a recent prior run," replace "recent prior run" with "the most recent N=5 runs" so that an isolated success amid a streak of failures doesn't trigger the transient path.

Request

I'd like to file this as an issue on max-sixty/tend — would a maintainer approve cross-posting? The text above is reusable verbatim. If approved, I'll open the upstream issue and reference this one.

I can also open a smaller follow-up here adding repo-local guidance to running-tend mentioning the specific macos-15 cache-bin symptom and pointing at the upstream fix path, so future sessions in this repo short-circuit faster even before the bundled skill is updated.

[max-sixty]

yes, raise in tend

[prql-bot]

Thanks — filed upstream as max-sixty/tend#505.