Fix it: make the shared push_experiments_state job merge state.json field-by-field instead of aborting on any git rebase conflict — concurrent writers to the same experiments branch are expected, not exceptional.
Problem: push_experiments_state (used by Test Quality Sentinel and 20+ other daily/experiment workflows — grep confirms agent-performance-analyzer.lock.yml, many daily-* workflows, etc. all share it) fails when two runs push to the same experiments/<workflow> branch close together. git rebase --onto hits CONFLICT (content): Merge conflict in state.json after 4 retry attempts and gives up, dropping the state update for that run entirely.
Affected workflow / run: Test Quality Sentinel — run 30630027716 (2026-07-31T12:16:46Z), job push_experiments_state (91155184042). Agent job itself succeeded (agent_job_conclusion: success) — only the state-push step failed.
Comparator: A later run of the same workflow, 30630104145 (2026-07-31T12:18:04Z, about 1.5 minutes after the failed one), completed with conclusion: success — confirming this is a transient concurrency race, not a permanent break. audit-diff between the two runs shows zero firewall anomalies and near-identical GitHub API call counts (13 vs 13, core consumed 2584 vs 2550) — the failure is purely a git-level race, not a tooling/auth regression.
Probable root cause: The push script rebases onto the GraphQL parent and aborts on any state.json conflict rather than merging fields, so two concurrent writers to the same experiments branch collide and one loses its update.
Proposed remediation:
- Add a field-level JSON merge (or explicit last-writer-wins per key) for
state.json before falling back to git rebase --abort.
- Alternatively, serialize writes per
experiments/<workflow> branch with a lock/backoff at the branch level instead of relying on commit-level rebase retries.
- Fix once in the shared action/script since this job is reused by dozens of workflows — do not patch per-workflow.
Success criteria: Two runs of the same experiment-tracking workflow that start within the same push-retry window both persist their state.json changes (merged, not lost), with zero push_experiments_state job failures over a 48h monitoring window.
Parent report: #49245. Analyzed via agenticworkflows audit-diff (base 30630027716 vs compare 30630104145) during the 2026-07-31 13:21 UTC failure-investigation pass.
Related to #49245
Generated by 🔍 [aw] Failure Investigator (6h) · agent · 76.1 AIC · ⌖ 13.7 AIC · ⊞ 6.8K · ◷
Fix it: make the shared
push_experiments_statejob mergestate.jsonfield-by-field instead of aborting on any git rebase conflict — concurrent writers to the same experiments branch are expected, not exceptional.Problem:
push_experiments_state(used by Test Quality Sentinel and 20+ other daily/experiment workflows — grep confirmsagent-performance-analyzer.lock.yml, manydaily-*workflows, etc. all share it) fails when two runs push to the sameexperiments/<workflow>branch close together.git rebase --ontohitsCONFLICT (content): Merge conflict in state.jsonafter 4 retry attempts and gives up, dropping the state update for that run entirely.Affected workflow / run: Test Quality Sentinel — run 30630027716 (2026-07-31T12:16:46Z), job
push_experiments_state(91155184042). Agent job itself succeeded (agent_job_conclusion: success) — only the state-push step failed.Comparator: A later run of the same workflow, 30630104145 (2026-07-31T12:18:04Z, about 1.5 minutes after the failed one), completed with
conclusion: success— confirming this is a transient concurrency race, not a permanent break.audit-diffbetween the two runs shows zero firewall anomalies and near-identical GitHub API call counts (13 vs 13, core consumed 2584 vs 2550) — the failure is purely a git-level race, not a tooling/auth regression.Probable root cause: The push script rebases onto the GraphQL parent and aborts on any
state.jsonconflict rather than merging fields, so two concurrent writers to the same experiments branch collide and one loses its update.Proposed remediation:
state.jsonbefore falling back togit rebase --abort.experiments/<workflow>branch with a lock/backoff at the branch level instead of relying on commit-level rebase retries.Success criteria: Two runs of the same experiment-tracking workflow that start within the same push-retry window both persist their
state.jsonchanges (merged, not lost), with zeropush_experiments_statejob failures over a 48h monitoring window.Parent report: #49245. Analyzed via
agenticworkflows audit-diff(base 30630027716 vs compare 30630104145) during the 2026-07-31 13:21 UTC failure-investigation pass.Related to #49245