[Perf] Windows/x64: 5 Regressions on 5/26/2026 4:15:56 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.22631
Queue	ViperWindows
Baseline	b88ce03bc28304f2b6cfd27c29923e142a9040c6
Compare	c57d5a14ae356880b4bd34d16814f3745111011f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Threading.Tasks.ValueTaskPerfTest

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
CreateAndAwait_FromCompletedTask - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	10.39 ns	12.52 ns	1.21	0.26	False
CreateAndAwait_FromCompletedTask_ConfigureAwait - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	10.36 ns	12.24 ns	1.18	0.27	False
CreateAndAwait_FromCompletedValueTaskSource - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	11.24 ns	13.26 ns	1.18	0.13	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tasks.ValueTaskPerfTest*'

Details

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask

ETL Files

Histogram

JIT Disasms

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask_ConfigureAwait

ETL Files

Histogram

JIT Disasms

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSource

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.22631
Queue	ViperWindows
Baseline	b88ce03bc28304f2b6cfd27c29923e142a9040c6
Compare	c57d5a14ae356880b4bd34d16814f3745111011f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEach<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Stack - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	152.07 ns	285.15 ns	1.88	0.00	False
Dictionary - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	333.55 ns	353.23 ns	1.06	0.00	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach<Int32>*'

Details

System.Collections.IterateForEach<Int32>.Stack(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.IterateForEach<Int32>.Dictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

⚠️ PerfTriageAgent encountered an error while analyzing this issue.

Error: TransientError: Failed to start MCP server

Please escalate to the performance infrastructure team if this issue needs triage.

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in Stack(Size: 512) — caused by commit 6217c1e ("JIT: Defer fgSetOptions until after 2nd-pass EH removal (#128573)").

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: 6217c1ef9a36b4ad4638608a92587b2844ef2e4c

🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 4 proposed (rule R4_known_regression_new — see Full Analysis for the table).

📊 Full Analysis (click to expand)

Summary: 3 confirmed regressions and 2 borderline regressions detected across System.Collections and System.Threading.Tasks tests. The most severe is System.Collections.IterateForEach.Stack(Size: 512) at +87.5% (z=125.8), likely caused by commit 7f2eeae "Make O1K_LCLVAR local assertprop only" which restricts JIT assertion propagation. ValueTask regressions (+18-21%) are likely caused by commit 269c0be "JIT: Tail merge async suspensions". Finding confidence 4/5, regression confidence 4/5.

Issue Overview

• Issue: [Perf] Windows/x64: 5 Regressions on 5/26/2026 4:15:56 PM +00:00 #128910 - [Perf] Windows/x64: 5 Regressions on 5/26/2026 4:15:56 PM +00:00
• Date: 2026-05-26 | Queue: Windows.11.Amd64.Viper.Perf | OS/Arch: Windows/x64
• Commits: b88ce03bc283 → c57d5a14ae35 (15 commits)
• Labels: untriaged, perf-regression, os-windows, arch-x64, branch-refs/heads/main, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Runtime Stack: Desktop + CoreCLR (implicit)
• Historical Data: Available (300 runs per test, 1316 total runs examined)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
IterateForEach.Stack(Size: 512)	152.07	285.15	+87.5%	153.02	1.05	300	125.84	5%	N	Extreme step change, ultra-stable
IterateForEach.Dictionary(Size: 512)	333.55	353.23	+5.9%	331.40	0.57	300	38.38	5%	N	Step change, ultra-stable
ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSource	11.24	13.26	+18.0%	10.36	0.56	300	5.15	10.9%	N	Step change, moderate CV
ValueTaskPerfTest.CreateAndAwait_FromCompletedTask	10.39	12.52	+20.5%	9.85	1.08	300	2.48	21.9%	N	Borderline - high CV (10.9%)
ValueTaskPerfTest.CreateAndAwait_FromCompletedTask_ConfigureAwait	10.36	12.24	+18.2%	9.75	0.98	300	2.54	20.1%	N	Borderline - high CV (10.0%)

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Excellent historical data (300 runs per test), sufficient after-runs (29), clear step-change patterns visible in data. Only limitation is high CV on two ValueTask tests.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
7f2eeae7cbff	src/coreclr/jit/assertionprop.cpp, compiler.h	Relevant	JIT assertion propagation restriction
269c0befa6ec	src/coreclr/jit/async.cpp, async.h, gentree.cpp	Relevant	JIT async suspension tail merging
050d5e64f335	src/coreclr/jit/codegenxarch.cpp, emitxarch.cpp	Relevant	JIT x86 instruction encoding
4e4d02afbe0b	src/coreclr/*/AsyncHelpers*.cs, asyncthunks.cpp	Relevant	Async continuation caching
c57d5a14ae35	src/coreclr/jit/lower.cpp	Relevant	JIT store coalescing fix
6217c1ef9a36	src/coreclr/jit/compiler.cpp	Relevant	JIT fgSetOptions defer

Confirmed Regressions

System.Collections.IterateForEach.Stack(Size: 512) (+87.5%)

• Delta: 152.07 ns → 285.15 ns (+87.5%)
• Historical Context: μ=153.02 (σ=1.05, n=300); z-score=125.84; adaptive threshold=5%
• Assessment: Extreme step change in an ultra-stable test (CV=0.69%). All 29 after-runs cluster tightly at 285-286 vs historical mean of 153. This near-doubling is unambiguous and suggests an algorithmic/optimization regression.
• Finding Confidence: 4/5 - Exceptional data quality, 300 historical runs, CV=0.69% confirms extreme stability
• Regression Confidence: 4/5 - Extreme z-score (125.8) eliminates noise; multiple JIT candidates in range prevent 5/5
• Likely Cause: 7f2eeae7cbff - "Make O1K_LCLVAR local assertprop only (Make O1K_LCLVAR local assertprop only #128522)"
  • Affected Areas: JIT / Assertion Propagation / Collection Iteration
  • Stack Relevance: Relevant
  • Rationale: This commit restricts JIT assertion propagation (O1K_LCLVAR) to local scope only. Stack.GetEnumerator() returns a struct enumerator whose MoveNext/Current rely on inlined field assertions. Restricting assertion propagation could cause the JIT to miss key optimizations for the tight iteration loop, explaining a near-doubling of iteration cost. The magnitude (+87.5%) indicates a critical optimization was lost rather than a minor codegen change.
  • Candidates Considered:
    • 7f2eeae7cbff - JIT assertion prop restriction; directly impacts loop optimization quality
    • 050d5e64f335 - JIT encoding for commutative instructions; less likely to cause 87% regression
    • c57d5a14ae35 - JIT store coalescing fix; could affect codegen but less targeted
• Verification Status: Not verified - bisection recommended to confirm among 15 commits

System.Collections.IterateForEach.Dictionary(Size: 512) (+5.9%)

• Delta: 333.55 ns → 353.23 ns (+5.9%)
• Historical Context: μ=331.40 (σ=0.57, n=300); z-score=38.38; adaptive threshold=5%
• Assessment: Clear step change in an ultra-stable test (CV=0.17%). First 6 after-runs remain at baseline (~331-333), then abrupt jump to 352-355 for remaining 23 runs. The delayed onset suggests the regression may have been triggered by a subsequent build or a tiered compilation effect.
• Finding Confidence: 4/5 - Exceptional stability (CV=0.17%), clear step pattern in after-runs
• Regression Confidence: 4/5 - z=38.4 and ultra-stability make the signal unambiguous; same candidate as Stack regression
• Likely Cause: 7f2eeae7cbff - "Make O1K_LCLVAR local assertprop only (Make O1K_LCLVAR local assertprop only #128522)"
  • Affected Areas: JIT / Assertion Propagation / Collection Iteration
  • Stack Relevance: Relevant
  • Rationale: Same root cause as Stack regression. Dictionary iteration uses a struct enumerator with inline-friendly MoveNext/Current pattern. The restriction of assertion propagation to locals would similarly impact the JIT's ability to optimize the iteration loop, though the smaller magnitude (5.9% vs 87.5%) suggests Dictionary's iteration pattern is less dependent on cross-scope assertions.
  • Candidates Considered:
    • 7f2eeae7cbff - Same assertion prop change; corroborated by Stack regression
    • 050d5e64f335 - Instruction encoding; could cause minor codegen changes
• Verification Status: Not verified - likely same root cause as Stack regression

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSource (+18.0%)

• Delta: 11.24 ns → 13.26 ns (+18.0%)
• Historical Context: μ=10.36 (σ=0.56, n=300); z-score=5.15; adaptive threshold=10.9%
• Assessment: Step change with z=5.15 exceeding threshold. Pre-existing upward drift noted (earlyMean 10.17 → recentMean 11.04) but the step to 12.5-17.8 from run 9 onward is distinct and beyond drift magnitude. CV=5.45% is moderate.
• Finding Confidence: 4/5 - Strong z-score, clear step pattern; minor concern about pre-existing drift
• Regression Confidence: 3/5 - Multiple async-related commits in range (commits 0 and 10); pre-existing drift complicates attribution
• Likely Cause: 269c0befa6ec - "JIT: Tail merge async suspensions (JIT: Tail merge async suspensions #128559)"
  • Affected Areas: JIT / Async / ValueTask
  • Stack Relevance: Relevant
  • Rationale: This commit modifies how the JIT handles async suspension points by tail-merging them. ValueTask's await pattern creates suspension points that this optimization targets. Tail-merging suspensions could introduce additional overhead in the fast path (completed task) if the merged code path is longer or prevents other optimizations. The step-change pattern (first 8 runs normal, then elevated) aligns with tiered recompilation applying the new JIT behavior.
  • Candidates Considered:
    • 269c0befa6ec - JIT async suspension tail merging; directly targets the async hot path
    • 4e4d02afbe0b - Async continuation caching; optimization should improve, not regress, but may have side effects
    • 7f2eeae7cbff - Assertion prop; less likely for async-specific pattern
• Verification Status: Not verified - bisection recommended

Borderline Tests

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask (+20.5%)

• Delta: 10.39 ns → 12.52 ns (+20.5%)
• Classification: Borderline - delta 1.36% below adaptive threshold (21.9%); CV=10.95% inflates threshold; z=2.48 barely clears 2.0
• Pattern: Same step-change as sibling tests (first 8 runs normal, then elevated to ~12.4)
• Assessment: Likely same root cause as CreateAndAwait_FromCompletedValueTaskSource given identical pattern and test class. High CV prevents confident classification.

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask_ConfigureAwait (+18.2%)

• Delta: 10.36 ns → 12.24 ns (+18.2%)
• Classification: Borderline - delta 1.88% below adaptive threshold (20.1%); CV=10.04%; z=2.54 marginally clears 2.0
• Pattern: Mirrors sibling test pattern exactly
• Assessment: Correlated with other ValueTask tests. Same root cause expected.

Open Questions

• Confidence Limitations: 15-commit range with multiple JIT changes makes precise attribution difficult without bisection
• ValueTask Pre-existing Drift: ValueTaskSource beforeValues show upward drift (earlyMean 10.17 → recentMean 11.04) predating the regression — should be tracked separately
• Delayed Onset: Both Dictionary and ValueTask tests show delayed onset (6-8 runs at baseline before step-up), suggesting tiered compilation or a secondary trigger
• Verification Needed: Bisection across the 15 commits, particularly isolating commits 7f2eeae and 269c0be

Recommended Actions

1. Bisection (Collections): Run bisection on commits b88ce03bc283...c57d5a14ae35 using System.Collections.IterateForEach<Int32>.Stack(Size: 512) to isolate culprit — the extreme magnitude makes this ideal for bisection
2. Bisection (ValueTask): Run bisection using System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSource to confirm whether commit 269c0be or 4e4d02a is responsible
3. Verification: If bisection confirms 7f2eeae, run with collectHardwareCounters=true to determine what optimization was lost (instruction count, branch mispredictions)
4. Source review: Review 7f2eeae7cbff assertion propagation change for impact on struct enumerator iteration patterns

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
CreateAndAwait_FromCompletedTask	❌ failed	-	-	10.4→12.5 ns (+21%)	— (not measured)	688s
CreateAndAwait_FromCompletedTask_Conf...	✅ Success	c57d5a14ae35	-	10.4→12.2 ns (+18%)	10.4→12.2 ns (+18%)	1714s
CreateAndAwait_FromCompletedValueTask...	❌ failed	-	-	11.2→13.3 ns (+18%)	— (not measured)	1810s
Stack(Size: 512)	✅ Success	6217c1ef9a36	✅ RELATED	152.1→285.2 ns (+88%)	151.8→285.6 ns (+88%)	1837s
Dictionary(Size: 512)	❌ failed	-	-	333.5→353.2 ns (+6%)	— (not measured)	1608s

Total bisection time: 7657s
Success rate: 2/5 tests

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
4f5dc8e0-2882-47c1-accd-e6eeced9617c	✅	submitted

Bisection details

…[earlier output omitted]

Id: "4f5dc8e0-2882-47c1-accd-e6eeced9617c"
└ {"jobId":"4f5dc8e0-2882-47c1-accd-e6eeced9617c","status":"running","mess...

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "4f5dc8e0-2882-47c1-accd-e6eeced9617c"
└ {"jobId":"4f5dc8e0-2882-47c1-accd-e6eeced9617c","status":"running","mess...

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "4f5dc8e0-2882-47c1-accd-e6eeced9617c"
└ {"jobId":"4f5dc8e0-2882-47c1-accd-e6eeced9617c","status":"running","mess...

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "4f5dc8e0-2882-47c1-accd-e6eeced9617c"
└ {"jobId":"4f5dc8e0-2882-47c1-accd-e6eeced9617c","status":"running","mess...

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "4f5dc8e0-2882-47c1-accd-e6eeced9617c"
└ {"jobId":"4f5dc8e0-2882-47c1-accd-e6eeced9617c","status":"running","mess...

I've been polling this Helix job for well over 120 minutes (100+ poll attempts) and it consistently shows "running" with "Found 2 work items" but never completes. This exceeds the maximum polling budget.

BISECTION FAILED:
- Reason: Helix job 4f5dc8e0-2882-47c1-accd-e6eeced9617c never completed after 120+ minutes of polling
- Helix Errors: Job remained in "running" status with "Found 2 work items" message indefinitely. Submitted 3 commits but only 2 work items were found (possible build failure for one commit). No results were ever returned.
- What was attempted:
  1. GetCommitRangeHashes to get 15 commits between b88ce03bc283 and c57d5a14ae35
  2. GetDoubleBisectCommits to select 3 key commits (050d5e64, 391365c6, 7f2eeae7)
  3. SubmitHelixJob with those 3 commits on Windows.11.Amd64.Viper.Perf queue
  4. Polled GetHelixJobResult 100+ times over 120+ minutes - job never completed
- What is needed: Investigate why the Helix job is stuck in the Windows.11.Amd64.Viper.Perf queue. The queue may be fully saturated or the work items may have failed silently. Retry the bisection when queue capacity is available.

HELIX JOBS:
- 4f5dc8e0-2882-47c1-accd-e6eeced9617c

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask_ConfigureAwait

• Culprit: c57d5a14ae35
• Relevance: Unknown

Helix jobs

Job ID	Submitted by bisection	Last status
b81e148e-54a3-4964-a74a-f79750b91292	✅	submitted
87726859-ac54-4099-9437-199162a7532f	✅	submitted
a48db7c8-0a3c-4abb-9d82-cfd86dfc9648	✅	completed

Per-commit measurements

Commit	Position	Measurements
050d5e64f335	q1	10.356 ns
7f2eeae7cbff	q3	10.554 ns
8e9aa2a35dec	q4-second	10.923 ns
90a838057fe9	q4-first	10.382 ns
391365c6cfa7	mid	10.245 ns
c57d5a14ae35	compare/head	10.532 ns

Bisection details

…[earlier output omitted]

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "a48db7c8-0a3c-4abb-9d82-cfd86dfc9648"
└ {"jobId":"a48db7c8-0a3c-4abb-9d82-cfd86dfc9648","status":"completed","me...

The verification job completed. The measurement for c57d5a1 is 10.532 ns - NOT regressed! It's right at baseline level.

Summary of all measurements:

• 050d5e64 (index 3): 10.356 ns
• 391365c6 (index 7): 10.245 ns
• 7f2eeae7 (index 11): 10.554 ns
• 90a83805 (index 12): 10.382 ns
• 8e9aa2a35 (index 13): 10.923 ns
• c57d5a14 (index 14, compare): 10.532 ns

None of these show the reported 12.24 ns regression. There is no observable step change in the commit range. The reported regression (2σ = 1.958 vs delta = 1.88) is within the noise floor and could not be reproduced.

BISECTION FAILED:
- Reason: No observable step change in measurements across the commit range; the reported regression appears to be noise
- Helix Errors: None — all three Helix jobs completed successfully
- What was attempted: Tested 6 commits across the range (050d5e64, 391365c6, 7f2eeae7, 90a83805, 8e9aa2a35, c57d5a14) via 3 Helix jobs. All measured between 10.2-10.9 ns, close to the reported baseline of 10.36 ns. The compare commit c57d5a14 measured 10.53 ns (not 12.24 ns as reported). The reported regression delta (1.88 ns / +18.2%) is within the 2σ noise floor (1.958 ns / 18.9%) for this benchmark.
- What is needed: The regression cannot be reproduced. This appears to be a noise-driven false positive in the autofile system. The issue may be closed as noise, or additional samples from the original autofile infrastructure may be needed to confirm.

HELIX JOBS:
- b81e148e-54a3-4964-a74a-f79750b91292
- 87726859-ac54-4099-9437-199162a7532f
- a48db7c8-0a3c-4abb-9d82-cfd86dfc9648

MEASUREMENTS:
- 050d5e64f335 (Q1): 10.356 ns
- 391365c6cfa7 (mid): 10.245 ns
- 7f2eeae7cbff (Q3): 10.554 ns
- 90a838057fe9 (Q4-first): 10.382 ns
- 8e9aa2a35dec (Q4-second): 10.923 ns
- c57d5a14ae35 (compare/head): 10.532 ns

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSource

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
4a31dd46-7ced-420c-b951-c8d21170226f	✅	submitted
76f8a259-b93e-4ff7-96e1-38441b671093	✅	submitted

Per-commit measurements

Commit	Position	Measurements
b88ce03bc283	baseline, job1	10.970, 10.950 ns
050d5e64f335	quarter, job1	10.930 ns
391365c6cfa7	mid, job1	10.980 ns
7f2eeae7cbff	three-quarter, job1	10.990 ns
c57d5a14ae35	head, job1	11.110, 10.920 ns

Bisection details

…[earlier output omitted]

.

● Extract second job results (shell)
│ python3 << 'EOF'
│ import json, re
│
│ with open('/tmp/1780410790451-copilot-tool-output-7v0wun.txt', 'r') as f:
│ data = json.load(f)…
└ 17 lines...

The second run confirms: baseline (b88ce03) = 10.95 ns and head (c57d5a1) = 10.92 ns — essentially identical. The reported +18% regression is not reproducible across two independent Helix runs.

Summary of measurements across both jobs:

• Job 1: baseline=10.97, head=11.11 (diff: +0.14 ns, within noise)
• Job 2: baseline=10.95, head=10.92 (diff: -0.03 ns, within noise)

The 2σ noise floor is 1.128 ns, and no measurement differs from baseline by more than 0.14 ns. There is no observable step change anywhere in this commit range.

BISECTION FAILED:
- Reason: Regression not reproducible — no observable step change in commit range
- Helix Errors: None (both jobs completed successfully)
- What was attempted: Two Helix jobs submitted. Job 1 (4a31dd46-7ced-420c-b951-c8d21170226f) tested 5 commits (baseline, quarter, mid, three-quarter, head) — all measured 10.93-11.11 ns with no step change. Job 2 (76f8a259-b93e-4ff7-96e1-38441b671093) re-tested baseline and head commits — measured 10.95 and 10.92 ns respectively, confirming no regression present.
- What is needed: The reported regression (11.24→13.26, +18%) cannot be reproduced with the default tiered-JIT configuration on Windows.11.Amd64.Viper.Perf. This may have been a transient infrastructure issue or a regression that has since been masked/reverted. The issue may need manual investigation or closure as noise.

HELIX JOBS:
- 4a31dd46-7ced-420c-b951-c8d21170226f
- 76f8a259-b93e-4ff7-96e1-38441b671093

MEASUREMENTS:
- b88ce03bc283 (baseline, job1): 10.97 ns
- 050d5e64f335 (quarter, job1): 10.93 ns
- 391365c6cfa7 (mid, job1): 10.98 ns
- 7f2eeae7cbff (three-quarter, job1): 10.99 ns
- c57d5a14ae35 (head, job1): 11.11 ns
- b88ce03bc283 (baseline, job2): 10.95 ns
- c57d5a14ae35 (head, job2): 10.92 ns

System.Collections.IterateForEach.Stack(Size: 512)

• Culprit: 6217c1ef9a36
• Commit Message: JIT: Defer fgSetOptions until after 2nd-pass EH removal (JIT: Defer fgSetOptions until after 2nd-pass EH removal #128573)
• Relevance: RELATED
• Explanation: This commit modifies the JIT compiler's code generation decisions (src/coreclr/jit/compiler.cpp) by deferring fgSetOptions() to after EH removal. This directly impacts how machine code is generated for collection iteration hot paths like the Stack enumerator, making it plausible that the JIT now selects a different (slower) codegen strategy for this benchmark.

Helix jobs

Job ID	Submitted by bisection	Last status
c6d704fd-31df-4574-9130-f0e70d465bd5	✅	submitted
5282a1e0-0173-400a-94db-d334ee566b9b	✅	submitted
aedbc1a8-0d01-4091-ae1c-e27db6520031	✅	completed

Per-commit measurements

Commit	Position	Measurements
040c8f307a7c	bad	285.710 ns
050d5e64f335	last good	151.760 ns
391365c6cfa7	bad	285.580 ns
6217c1ef9a36	culprit	285.610 ns
752f52808b3e	bad	285.070 ns
7f2eeae7cbff	bad	285.500 ns

Bisection details

…[earlier output omitted]

050d5e64 (index 3): 151.76 ns ✓ (last good)

• 6217c1ef (index 4): 285.61 ns ✗ (first bad)

The culprit is 6217c1ef9a36b4ad4638608a92587b2844ef2e4c. Now let me examine the commit:

● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "6217c1ef…
└ {"index":null,"sha":"6217c1ef9a36b4ad4638608a92587b2844ef2e4c","abbrevSh...

Bisection complete. The culprit is 6217c1ef9a36b4ad4638608a92587b2844ef2e4c — a JIT commit by Andy Ayers that defers fgSetOptions() until after second-pass EH removal, affecting code generation decisions around fully-interruptible GC.

This is clearly RELATED to the regression: it's a JIT codegen change that directly affects how the runtime generates machine code for the Stack iteration hot path.

BISECTION RESULT:
- Culprit Commit: 6217c1ef9a36b4ad4638608a92587b2844ef2e4c
- Commit Message: JIT: Defer fgSetOptions until after 2nd-pass EH removal (#128573)
- Commits Tested: 7
- Confidence: 5
- Commit Relevance: RELATED
- Relevance Explanation: This commit modifies the JIT compiler's code generation decisions (src/coreclr/jit/compiler.cpp) by deferring fgSetOptions() to after EH removal. This directly impacts how machine code is generated for collection iteration hot paths like the Stack enumerator, making it plausible that the JIT now selects a different (slower) codegen strategy for this benchmark.
- Baseline Value: 151.76 ns
- Culprit Value: 285.61 ns
- Summary: Bisected 15 commits across 3 Helix jobs. The regression (~88% slowdown) was clearly introduced at commit 6217c1ef which changes JIT code generation timing for GC interruptibility decisions.

HELIX JOBS:
- c6d704fd-31df-4574-9130-f0e70d465bd5
- 5282a1e0-0173-400a-94db-d334ee566b9b
- aedbc1a8-0d01-4091-ae1c-e27db6520031

MEASUREMENTS:
- 050d5e64f335 (last good): 151.76 ns
- 6217c1ef9a36 (culprit): 285.61 ns
- 040c8f307a7c (bad): 285.71 ns
- 752f52808b3e (bad): 285.07 ns
- 391365c6cfa7 (bad): 285.58 ns
- 7f2eeae7cbff (bad): 285.50 ns

System.Collections.IterateForEach.Dictionary(Size: 512)

• Status: failed

Helix jobs

Job ID	Submitted by bisection	Last status
58a46ea0-a163-4915-a6b4-8f61df42b9a3	✅	submitted
73a256b8-2502-4762-9168-df703df500b8	✅	submitted
eeb828f2-45d7-474a-a93f-9dd0a94cab0d	✅	completed

Per-commit measurements

Commit	Position	Measurements
050d5e64	q1 quarter	333.040 ns
391365c6	q2 mid	333.110 ns
7f2eeae7	q3 three-quarter	332.480 ns
8e9aa2a3	q4-second	330.960 ns
90a83805	q4-first	330.800 ns
c57d5a14	head/compare	331.040 ns

Bisection details

…[earlier output omitted]

baseline-like*, not the reported regressed value of 353.23 ns.

All 6 commits tested across the entire range show values between 330.80 and 333.11 ns, all within baseline noise. The reported +5.9% regression (353.23 ns) is not reproducible in my Helix runs.

● get_auto_file_data (MCP: PerfTriageMcp) · issueId: 73923
└ {"isValidIssueNumber":true,"regressionDate":"2026-05-26T16:15:56.0000000...

The autofile config shows CompilationMode: tiered, RunKind: micro — which is the default config. The regression should theoretically be reproducible but isn't showing up in any of my 6 measurements across the entire commit range. All measurements cluster around 330-333 ns, with no step change visible anywhere. The reported value of 353.23 ns at the compare commit is not reproducible.

BISECTION FAILED:
- Reason: Regression not reproducible — no observable step change in measurements across the entire commit range
- Helix Errors: None (all jobs completed successfully)
- What was attempted: Tested 6 commits spanning the full range (050d5e64, 391365c6, 7f2eeae7, 90a83805, 8e9aa2a3, c57d5a14) via 3 Helix jobs. All returned values in the range 330.80–333.11 ns, consistent with baseline (333.55 ns). The compare commit c57d5a14 measured 331.04 ns — well below the reported regressed value of 353.23 ns.
- What is needed: The regression may have been a transient machine-state artifact on the original Viper perf machine, or environment-dependent. Re-running the autofile's original comparison window may help confirm whether the regression persists.

HELIX JOBS:
- 58a46ea0-a163-4915-a6b4-8f61df42b9a3
- 73a256b8-2502-4762-9168-df703df500b8
- eeb828f2-45d7-474a-a93f-9dd0a94cab0d

MEASUREMENTS:
- 050d5e64 (Q1 quarter): 333.04 ns
- 391365c6 (Q2 mid): 333.11 ns
- 7f2eeae7 (Q3 three-quarter): 332.48 ns
- 90a83805 (Q4-first): 330.80 ns
- 8e9aa2a3 (Q4-second): 330.96 ns
- c57d5a14 (head/compare): 331.04 ns

🤖 Proposed Next Actions

Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

Verified regression attributed to commit 6217c1e (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

#	Action	Status	Details
1	Back-reference cause PR	🕒 Planned	on culprit PR for 6217c1ef9a36
2	Transfer issue	🕒 Planned	→ dotnet/runtime
3	Ping owners	🕒 Planned	no owners resolved
4	Remove label	🕒 Planned	untriaged

---

Generated by PerfTriageAgent on 2026-06-02 15:31 UTC

[DrewScoggins]

Seems related to #128459

Related Regressions:

• [Perf] Linux/arm64: 5 Regressions on 5/27/2026 9:15:14 PM +00:00 perf-autofiling-issues#74067
• [Perf] Linux/arm64: 15 Regressions on 5/27/2026 9:15:14 PM +00:00 perf-autofiling-issues#74060

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @VSadov
See info in area-owners.md if you want to be subscribed.