[Perf] Burgers.Test3 Regression on x64

[alexcovington]

Description

The Burgers.Test3 scenario from dotnet/performance is regressing in 6.0. I can reproduce by running the following after cloning dotnet/performance:

$ py scripts\benchmarks_ci.py -c Release -f netcoreapp5.0 net6.0 --filter Burgers.Test3

Configuration

Configuration below in Data.

Regression?

Regression from .NET 5.0 -> .NET 6.0.

Data

AMD Ryzen 7 5800

.NET 5.0 Results

// * Summary *

BenchmarkDotNet=v0.12.1.1466-nightly, OS=Windows 10.0.19041.804 (2004/May2020Update/20H1)
AMD Ryzen 7 5800, 1 CPU, 16 logical and 8 physical cores
.NET SDK=6.0.100-preview.3.21153.9
  [Host]     : .NET 5.0.3 (5.0.321.7212), X64 RyuJIT
  Job-TDIWMD : .NET 5.0.3 (5.0.321.7212), X64 RyuJIT

 PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  IterationTime=250.0000 ms
 MaxIterationCount=20  MinIterationCount=15  WarmupCount=1
 |    Method |     Mean |    Error |   StdDev |   Median |      Min |      Max | Gen 0 | Gen 1 | Gen 2 | Allocated |
 |---------- |---------:|---------:|---------:|---------:|---------:|---------:|------:|------:|------:|----------:|
 | Burgers_3 | 87.11 ms | 0.200 ms | 0.167 ms | 87.15 ms | 86.82 ms | 87.44 ms |     - |     - |     - |    156 KB |

.NET 6.0 Results

// * Summary *

BenchmarkDotNet=v0.12.1.1466-nightly, OS=Windows 10.0.19041.804 (2004/May2020Update/20H1)
AMD Ryzen 7 5800, 1 CPU, 16 logical and 8 physical cores
.NET SDK=6.0.100-preview.3.21153.9
[Host]     : .NET 6.0.0 (6.0.21.15201), X64 RyuJIT
Job-HJLOTG : .NET 6.0.0 (6.0.21.15201), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  IterationTime=250.0000 ms
MaxIterationCount=20  MinIterationCount=15  WarmupCount=1

|    Method |    Mean |    Error |   StdDev |  Median |     Min |     Max | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---------- |--------:|---------:|---------:|--------:|--------:|--------:|------:|------:|------:|----------:|
| Burgers_3 | 1.443 s | 0.0050 s | 0.0047 s | 1.442 s | 1.438 s | 1.455 s |     - |     - |     - |    156 KB |

Intel i9 10900k

.NET 5.0

// * Summary *
 
BenchmarkDotNet=v0.12.1.1466-nightly, OS=Windows 10.0.19042
Intel Core i9-10900K CPU 3.70GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK=6.0.100-preview.3.21153.9
   [Host]     : .NET 5.0.3 (5.0.321.7212), X64 RyuJIT
   Job-QBUPZJ : .NET 5.0.3 (5.0.321.7212), X64 RyuJIT
 
PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  IterationTime=250.0000 ms
MaxIterationCount=20  MinIterationCount=15  WarmupCount=1
 
|    Method |     Mean |   Error |  StdDev |   Median |      Min |      Max | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---------- |---------:|--------:|--------:|---------:|---------:|---------:|------:|------:|------:|----------:|
| Burgers_3 | 117.2 ms | 1.50 ms | 1.25 ms | 117.2 ms | 116.1 ms | 120.2 ms |     - |     - |     - |    159 KB |

.NET 6.0

// * Summary *
 
BenchmarkDotNet=v0.12.1.1466-nightly, OS=Windows 10.0.19042
Intel Core i9-10900K CPU 3.70GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK=6.0.100-preview.3.21153.9
  [Host]     : .NET 6.0.0 (6.0.21.15201), X64 RyuJIT
  Job-JFEUYZ : .NET 6.0.0 (6.0.21.15201), X64 RyuJIT
 
PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  IterationTime=250.0000 ms
MaxIterationCount=20  MinIterationCount=15  WarmupCount=1
 
|    Method |    Mean |    Error |   StdDev |  Median |     Min |     Max | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---------- |--------:|---------:|---------:|--------:|--------:|--------:|------:|------:|------:|----------:|
| Burgers_3 | 1.450 s | 0.0006 s | 0.0005 s | 1.450 s | 1.449 s | 1.451 s |     - |     - |     - |    156 KB |

Analysis

This looks to be a SIMD issue of some kind, but I'm not exactly sure where the issue is. The following analysis was done on the Ryzen 7 machine but holds true for both machines I tested on.

I collected some ETW traces and found that the benchmark was spending most of its time in the GetCalculated3 method, which looks like this:

private static double[] GetCalculated3(int nt, int nx, double dx, double dt, double nu, double[] initial)
    {
        var nx2 = nx + (Vector<double>.Count - (nx % Vector<double>.Count));

        double[] u = new double[nx2];
        double[] un = new double[nx2];
        Array.Copy(initial, un, initial.Length);

        double factor = Math.Pow(nu * dt / dx, 2.0);

        for (int tStep = 0; tStep < nt; tStep++)
        {
            for (int i = 1; i < nx2 - Vector<double>.Count + 1; i += Vector<double>.Count)
            {
                var vectorIn0 = new Vector<double>(un, i);
                var vectorInPrev = new Vector<double>(un, i - 1);
                var vectorInNext = new Vector<double>(un, i + 1);

                var vectorOut = vectorIn0 - vectorIn0 * (dt / dx) * (vectorIn0 - vectorInPrev) + factor *
                    (vectorInNext - 2.0 * vectorIn0 + vectorInPrev);

                vectorOut.CopyTo(u, i);
            }

            u[0] = un[0] - un[0] * dt / dx * (un[0] - un[nx - 1]) + factor *
                        (un[1] - 2 * un[0] + un[nx - 1]);

            u[nx - 1] = un[nx - 1] - un[nx - 1] * dt / dx * (un[nx - 1] - un[nx - 2]) + factor *
                        (un[0] - 2 * un[nx - 1] + un[nx - 2]);

            double[] swap = u;
            u = un;
            un = swap;
        }

        return un;
    }

Lots of vectorization, so I had BDN generate the disassembly for both .NET 5.0 and 6.0 to see if the issue was with codegen.

BDN Disassembly

.NET 5.0.3 (5.0.321.7212), X64 RyuJIT

; Burgers.Test3()
       sub       rsp,38
       vzeroupper
       vmovsd    xmm2,qword ptr [7FFDE628E770]
       vmovsd    qword ptr [rsp+20],xmm2
       mov       rdx,[rcx+10]
       mov       [rsp+28],rdx
       vmovsd    xmm2,qword ptr [rcx+18]
       vmovsd    xmm3,qword ptr [rcx+20]
       mov       ecx,4E20
       mov       edx,2711
       call      Burgers.GetCalculated3(Int32, Int32, Double, Double, Double, Double[])
       nop
       add       rsp,38
       ret
; Total bytes of code 61

; Burgers.GetCalculated3(Int32, Int32, Double, Double, Double, Double[])
       push      r15
       push      r14
       push      r12
       push      rdi
       push      rsi
       push      rbp
       push      rbx
       sub       rsp,50
       vzeroupper
       vmovaps   [rsp+40],xmm6
       vmovaps   [rsp+30],xmm7
       vmovaps   [rsp+20],xmm8
       mov       edi,ecx
       mov       esi,edx
       vmovaps   xmm6,xmm2
       vmovaps   xmm7,xmm3
       mov       rbx,[rsp+0B8]
       mov       edx,esi
       sar       edx,1F
       and       edx,3
       add       edx,esi
       and       edx,0FFFFFFFC
       mov       ecx,esi
       sub       ecx,edx
       mov       edx,ecx
       neg       edx
       lea       ebp,[rdx+rsi+4]
       movsxd    rdx,ebp
       mov       rcx,offset MT_System.Double[]
       call      CORINFO_HELP_NEWARR_1_VC
       mov       r14,rax
       movsxd    rdx,ebp
       mov       rcx,offset MT_System.Double[]
       call      CORINFO_HELP_NEWARR_1_VC
       mov       r15,rax
       mov       r8d,[rbx+8]
       mov       rcx,rbx
       mov       rdx,r15
       call      System.Array.Copy(System.Array, System.Array, Int32)
       vmulsd    xmm0,xmm7,qword ptr [rsp+0B0]
       vdivsd    xmm0,xmm0,xmm6
       vmovsd    xmm1,qword ptr [7FFDE6281AD8]
       call      System.Math.Pow(Double, Double)
       xor       edx,edx
       test      edi,edi
       jle       near ptr M01_L03
       add       ebp,0FFFFFFFD
       lea       eax,[rsi+0FFFF]
       movsxd    rcx,eax
       add       esi,0FFFFFFFE
       movsxd    r8,esi
M01_L00:
       mov       r9d,1
       cmp       ebp,1
       jle       near ptr M01_L02
       mov       r10d,[r15+8]
       vdivsd    xmm1,xmm7,xmm6
M01_L01:
       cmp       r9d,r10d
       jae       near ptr M01_L05
       lea       ebx,[r9+3]
       cmp       ebx,r10d
       jae       near ptr M01_L05
       vmovupd   ymm2,[r15+r9*8+10]
       lea       r11d,[r9+0FFFF]
       cmp       r11d,r10d
       jae       near ptr M01_L05
       lea       r12d,[r9+2]
       cmp       r12d,r10d
       jae       near ptr M01_L05
       vmovupd   ymm3,[r15+r11*8+10]
       lea       r11d,[r9+1]
       cmp       r11d,r10d
       jae       near ptr M01_L05
       lea       r12d,[r9+4]
       cmp       r12d,r10d
       jae       near ptr M01_L05
       vmovupd   ymm4,[r15+r11*8+10]
       vbroadcastsd ymm5,xmm1
       vmulpd    ymm5,ymm5,ymm2
       vsubpd    ymm8,ymm2,ymm3
       vmulpd    ymm5,ymm5,ymm8
       vsubpd    ymm5,ymm2,ymm5
       vbroadcastsd ymm8,qword ptr [7FFDE6281AE8]
       vmulpd    ymm2,ymm8,ymm2
       vsubpd    ymm2,ymm4,ymm2
       vaddpd    ymm2,ymm2,ymm3
       vbroadcastsd ymm3,xmm0
       vmulpd    ymm2,ymm3,ymm2
       vaddpd    ymm2,ymm5,ymm2
       mov       r11d,[r14+8]
       cmp       r9d,r11d
       jae       near ptr M01_L06
       cmp       ebx,r11d
       jae       near ptr M01_L07
       vmovupd   [r14+r9*8+10],ymm2
       mov       r9d,r12d
       cmp       r9d,ebp
       jl        near ptr M01_L01
M01_L02:
       mov       r10d,[r15+8]
       cmp       r10d,0
       jbe       near ptr M01_L05
       vmovsd    xmm1,qword ptr [r15+10]
       vmulsd    xmm2,xmm1,xmm7
       vdivsd    xmm2,xmm2,xmm6
       cmp       eax,r10d
       jae       near ptr M01_L05
       vmovsd    xmm3,qword ptr [r15+rcx*8+10]
       vsubsd    xmm4,xmm1,xmm3
       vmulsd    xmm2,xmm2,xmm4
       vsubsd    xmm2,xmm1,xmm2
       cmp       r10d,1
       jbe       near ptr M01_L05
       vmovsd    xmm4,qword ptr [r15+18]
       vmulsd    xmm1,xmm1,qword ptr [7FFDE6281AF8]
       vsubsd    xmm1,xmm4,xmm1
       vaddsd    xmm1,xmm1,xmm3
       vmulsd    xmm1,xmm1,xmm0
       vaddsd    xmm1,xmm2,xmm1
       mov       r11d,[r14+8]
       cmp       r11d,0
       jbe       near ptr M01_L05
       vmovsd    qword ptr [r14+10],xmm1
       vmovsd    xmm1,qword ptr [r15+rcx*8+10]
       vmulsd    xmm2,xmm1,xmm7
       vdivsd    xmm2,xmm2,xmm6
       cmp       esi,r10d
       jae       near ptr M01_L05
       vmovsd    xmm3,qword ptr [r15+r8*8+10]
       vsubsd    xmm4,xmm1,xmm3
       vmulsd    xmm2,xmm2,xmm4
       vsubsd    xmm2,xmm1,xmm2
       vaddsd    xmm1,xmm1,xmm1
       vmovsd    xmm4,qword ptr [r15+10]
       vsubsd    xmm1,xmm4,xmm1
       vaddsd    xmm1,xmm1,xmm3
       vmulsd    xmm1,xmm1,xmm0
       vaddsd    xmm1,xmm2,xmm1
       cmp       eax,r11d
       jae       short M01_L05
       vmovsd    qword ptr [r14+rcx*8+10],xmm1
       inc       edx
       cmp       edx,edi
       jl        short M01_L04
       mov       r15,r14
M01_L03:
       mov       rax,r15
       vmovaps   xmm6,[rsp+40]
       vmovaps   xmm7,[rsp+30]
       vmovaps   xmm8,[rsp+20]
       vzeroupper
       add       rsp,50
       pop       rbx
       pop       rbp
       pop       rsi
       pop       rdi
       pop       r12
       pop       r14
       pop       r15
       ret
M01_L04:
       xchg      r14,r15
       jmp       near ptr M01_L00
M01_L05:
       call      CORINFO_HELP_RNGCHKFAIL
M01_L06:
       call      CORINFO_HELP_THROW_ARGUMENTOUTOFRANGEEXCEPTION
M01_L07:
       call      CORINFO_HELP_THROW_ARGUMENTEXCEPTION
       int       3
; Total bytes of code 672

.NET 6.0.0 (6.0.21.15201), X64 RyuJIT

; Burgers.Test3()
       push      rbp
       sub       rsp,30
       vzeroupper
       lea       rbp,[rsp+30]
       mov       [rbp+10],rcx
       vmovsd    xmm2,qword ptr [7FFDE8B4C1F8]
       vmovsd    qword ptr [rsp+20],xmm2
       mov       rcx,[rbp+10]
       mov       rcx,[rcx+10]
       mov       [rsp+28],rcx
       mov       rcx,[rbp+10]
       vmovsd    xmm2,qword ptr [rcx+18]
       mov       rcx,[rbp+10]
       vmovsd    xmm3,qword ptr [rcx+20]
       mov       ecx,4E20
       mov       edx,2711
       call      Burgers.GetCalculated3(Int32, Int32, Double, Double, Double, Double[])
       nop
       lea       rsp,[rbp]
       pop       rbp
       ret
; Total bytes of code 84

; Burgers.GetCalculated3(Int32, Int32, Double, Double, Double, Double[])
       push      r15
       push      r14
       push      rdi
       push      rsi
       push      rbp
       push      rbx
       sub       rsp,148
       vzeroupper
       vmovaps   [rsp+130],xmm6
       vmovaps   [rsp+120],xmm7
       vmovaps   [rsp+110],xmm8
       vmovaps   [rsp+100],xmm9
       vmovaps   [rsp+0F0],xmm10
       vxorps    xmm4,xmm4,xmm4
       vmovdqa   xmmword ptr [rsp+40],xmm4
       mov       rax,0FFFFFFFFFF70
M01_L00:
       vmovdqa   xmmword ptr [rsp+rax+0E0],xmm4
       vmovdqa   xmmword ptr [rsp+rax+0F0],xmm4
       vmovdqa   xmmword ptr [rsp+rax+100],xmm4
       add       rax,30
       jne       short M01_L00
       mov       edi,ecx
       mov       esi,edx
       vmovaps   xmm6,xmm2
       vmovaps   xmm7,xmm3
       mov       rbx,[rsp+1A8]
       mov       edx,esi
       sar       edx,1F
       and       edx,3
       add       edx,esi
       and       edx,0FFFFFFFC
       mov       ecx,esi
       sub       ecx,edx
       mov       edx,ecx
       neg       edx
       lea       ebp,[rdx+rsi+4]
       movsxd    rdx,ebp
       mov       rcx,offset MT_System.Double[]
       call      CORINFO_HELP_NEWARR_1_VC
       mov       r14,rax
       movsxd    rdx,ebp
       mov       rcx,offset MT_System.Double[]
       call      CORINFO_HELP_NEWARR_1_VC
       mov       r15,rax
       mov       r8d,[rbx+8]
       mov       rcx,rbx
       mov       rdx,r15
       call      System.Array.Copy(System.Array, System.Array, Int32)
       vmulsd    xmm0,xmm7,qword ptr [rsp+1A0]
       vdivsd    xmm0,xmm0,xmm6
       vmovsd    xmm8,qword ptr [7FFDE8B4C8B8]
       vmovaps   xmm1,xmm8
       call      System.Math.Pow(Double, Double)
       xor       edx,edx
       test      edi,edi
       jle       near ptr M01_L07
M01_L01:
       mov       eax,1
       lea       ecx,[rbp+0FFFD]
       cmp       ecx,1
       jle       near ptr M01_L06
M01_L02:
       mov       r8d,[r15+8]
       cmp       eax,r8d
       jae       near ptr M01_L09
       lea       r9d,[rax+3]
       cmp       r9d,r8d
       jae       near ptr M01_L09
       vmovupd   ymm1,[r15+rax*8+10]
       lea       r10d,[rax+0FFFF]
       cmp       r10d,r8d
       jae       near ptr M01_L09
       lea       r11d,[rax+2]
       cmp       r11d,r8d
       jae       near ptr M01_L09
       vmovupd   ymm2,[r15+r10*8+10]
       lea       r10d,[rax+1]
       cmp       r10d,r8d
       jae       near ptr M01_L09
       lea       r11d,[rax+4]
       cmp       r11d,r8d
       jae       near ptr M01_L09
       vmovupd   ymm3,[r15+r10*8+10]
       vmovaps   ymm4,ymm1
       vmovupd   [rsp+0A0],ymm4
       vxorps    ymm5,ymm5,ymm5
       vmovupd   [rsp+0C0],ymm5
       xor       r10d,r10d
       vdivsd    xmm9,xmm7,xmm6
M01_L03:
       lea       r8,[rsp+0A0]
       vmovsd    xmm10,qword ptr [r8+r10*8]
       vmulsd    xmm10,xmm9,xmm10
       lea       r8,[rsp+0C0]
       vmovsd    qword ptr [r8+r10*8],xmm10
       inc       r10
       cmp       r10,4
       jl        short M01_L03
       vmovupd   ymm9,[rsp+0C0]
       vsubpd    ymm4,ymm4,ymm2
       vmulpd    ymm4,ymm9,ymm4
       vsubpd    ymm4,ymm1,ymm4
       vmovupd   [rsp+60],ymm1
       vmovupd   [rsp+80],ymm5
       xor       r8d,r8d
M01_L04:
       lea       r10,[rsp+60]
       vmovsd    xmm1,qword ptr [r10+r8*8]
       vaddsd    xmm1,xmm1,xmm1
       lea       r10,[rsp+80]
       vmovsd    qword ptr [r10+r8*8],xmm1
       inc       r8
       cmp       r8,4
       jl        short M01_L04
       vmovupd   ymm1,[rsp+80]
       vsubpd    ymm1,ymm3,ymm1
       vaddpd    ymm1,ymm1,ymm2
       vmovupd   [rsp+20],ymm1
       vmovupd   [rsp+40],ymm5
       xor       r8d,r8d
M01_L05:
       lea       r10,[rsp+20]
       vmovsd    xmm1,qword ptr [r10+r8*8]
       vmulsd    xmm1,xmm1,xmm0
       lea       r10,[rsp+40]
       vmovsd    qword ptr [r10+r8*8],xmm1
       inc       r8
       cmp       r8,4
       jl        short M01_L05
       vmovupd   ymm1,[rsp+40]
       vaddpd    ymm1,ymm4,ymm1
       mov       r8d,[r14+8]
       cmp       eax,r8d
       jae       near ptr M01_L10
       cmp       r9d,r8d
       jae       near ptr M01_L11
       vmovupd   [r14+rax*8+10],ymm1
       mov       eax,r11d
       cmp       eax,ecx
       jl        near ptr M01_L02
M01_L06:
       mov       r8d,[r15+8]
       test      r8d,r8d
       je        near ptr M01_L09
       vmovsd    xmm2,qword ptr [r15+10]
       vmulsd    xmm4,xmm2,xmm7
       vdivsd    xmm1,xmm4,xmm6
       lea       r10d,[rsi+0FFFF]
       cmp       r10d,r8d
       jae       near ptr M01_L09
       movsxd    rax,r10d
       vmovsd    xmm3,qword ptr [r15+rax*8+10]
       vsubsd    xmm5,xmm2,xmm3
       vmulsd    xmm1,xmm1,xmm5
       vsubsd    xmm1,xmm2,xmm1
       cmp       r8d,1
       jbe       near ptr M01_L09
       vmovsd    xmm4,qword ptr [r15+18]
       vmulsd    xmm2,xmm2,xmm8
       vsubsd    xmm2,xmm4,xmm2
       vaddsd    xmm2,xmm2,xmm3
       vmulsd    xmm2,xmm2,xmm0
       vaddsd    xmm1,xmm1,xmm2
       mov       r11d,[r14+8]
       test      r11d,r11d
       je        near ptr M01_L09
       vmovsd    qword ptr [r14+10],xmm1
       vmovsd    xmm1,qword ptr [r15+rax*8+10]
       vmulsd    xmm2,xmm1,xmm7
       vdivsd    xmm2,xmm2,xmm6
       lea       r9d,[rsi+0FFFE]
       cmp       r9d,r8d
       jae       near ptr M01_L09
       movsxd    rcx,r9d
       vmovsd    xmm3,qword ptr [r15+rcx*8+10]
       vsubsd    xmm4,xmm1,xmm3
       vmulsd    xmm2,xmm2,xmm4
       vsubsd    xmm2,xmm1,xmm2
       vmovsd    xmm4,qword ptr [r15+10]
       vmulsd    xmm1,xmm1,xmm8
       vsubsd    xmm1,xmm4,xmm1
       vaddsd    xmm1,xmm1,xmm3
       vmulsd    xmm1,xmm1,xmm0
       vaddsd    xmm1,xmm2,xmm1
       cmp       r10d,r11d
       jae       short M01_L09
       vmovsd    qword ptr [r14+rax*8+10],xmm1
       inc       edx
       cmp       edx,edi
       jl        short M01_L08
       mov       r15,r14
M01_L07:
       mov       rax,r15
       vmovaps   xmm6,[rsp+130]
       vmovaps   xmm7,[rsp+120]
       vmovaps   xmm8,[rsp+110]
       vmovaps   xmm9,[rsp+100]
       vmovaps   xmm10,[rsp+0F0]
       vzeroupper
       add       rsp,148
       pop       rbx
       pop       rbp
       pop       rsi
       pop       rdi
       pop       r14
       pop       r15
       ret
M01_L08:
       xchg      r14,r15
       jmp       near ptr M01_L01
M01_L09:
       call      CORINFO_HELP_RNGCHKFAIL
M01_L10:
       call      CORINFO_HELP_THROW_ARGUMENTOUTOFRANGEEXCEPTION
M01_L11:
       call      CORINFO_HELP_THROW_ARGUMENTEXCEPTION
       int       3
; Total bytes of code 951

There are a few things that jump out to me as possibly causing the regression:

• Mixing SSE+AVX in the hot loop. The .NET 6.0 dump shows use of both SSE and AVX features in the hot loop and alternates between them. The .NET 5.0 dump seems to be scheduling these a little better and keeping the AVX instructions together and then moving to the SSE instructions. I know that on some CPU architectures switching between these modes can cause a significant perf drop, so I'm wondering if that's what I'm seeing here because that transition happens inside the hot loop.
• Some kind of alignment issue, although when running the repro and collecting a profile with AMDuProf, I didn't see very many reported misaligned accesses for the duration of the benchmark. I have been having some issues getting AMDuProf to properly collect information on managed stack frames/function calls though, so this may need more investigation.

Please let me know if there is any other information I can provide or if I can clarify any of the above.

[alexcovington]

After looking at this more, it seems more like the regression is caused by mixing SSE and AVX, so inserting some zveroupper in between those transitions might help. Starting to look into this now.

[alexcovington]

Oh just found #10484, might be related.

[AndyAyersMS]

Perf history shows:

Implicated commit range is e0df256...db6c320

Nothing obvious jumps out there at first glance. Unfortunately we didn't get a lot of runs during that time frame.

Maybe #41898 or #42050?

@DrewScoggins weren't we going to open an issue for this? I recall it slipped past our screening somehow.

[DrewScoggins]

Yeah, this was missed because of a bug in the reporting that got fixed as a result of this, but I don't think a bug ever got cut to actually fix this issue.

[JulieLeeMSFT]

@alexcovington assigning this to you for now. Please let us know if JIT team needs to take it over.
CC @kunalspathak

[alexcovington]

Unfortunately, inserting the extra zveroupper instructions did not resolve the issue. I created this branch to insert the instruction when switching between usage of the ymm and xmm registers. Here is the assembly:

Assembly with added `zveroupper` instructions

; Assembly listing for method BurgersRepro.Program:GetCalculated3(int,int,double,double,double,System.Double[]):System.Double[]
; Emitting BLENDED_CODE for X64 CPU with AVX - Windows
; optimized code
; rsp based frame
; fully interruptible
; Final local variable assignments
;
;  V00 arg0         [V00,T17] (  4,  7   )     int  ->  rdi        
;  V01 arg1         [V01,T11] (  8, 14   )     int  ->  rsi        
;  V02 arg2         [V02,T31] (  6, 15   )  double  ->  [rsp+0x1E0]  
;  V03 arg3         [V03,T32] (  6, 15   )  double  ->  mm7        
;  V04 arg4         [V04,T39] (  1,  1   )  double  ->  [rsp+0x1F0]  
;  V05 arg5         [V05,T19] (  2,  2   )     ref  ->  rbx         class-hnd
;  V06 loc0         [V06,T18] (  4,  7   )     int  ->  rbp        
;  V07 loc1         [V07,T07] (  8, 53   )     ref  ->  registers   class-hnd
;  V08 loc2         [V08,T02] ( 16, 91   )     ref  ->  registers   class-hnd
;  V09 loc3         [V09,T23] (  4, 25   )  double  ->  mm9        
;  V10 loc4         [V10,T15] (  4, 13   )     int  ->  rbx        
;  V11 loc5         [V11,T00] ( 14,212   )     int  ->  r12        
;  V12 loc6         [V12,T20] (  5, 80   )  simd32  ->  mm11         ld-addr-op
;  V13 loc7         [V13,T22] (  3, 48   )  simd32  ->  mm12         ld-addr-op
;  V14 loc8         [V14,T24] (  2, 32   )  simd32  ->  mm13         ld-addr-op
;  V15 loc9         [V15,T25] (  2, 32   )  simd32  ->  mm0         ld-addr-op
;  V16 OutArgs      [V16    ] (  1,  1   )  lclBlk (32) [rsp+0x00]   "OutgoingArgSpace"
;* V17 tmp1         [V17    ] (  0,  0   )  simd32  ->  zero-ref    "non-inline candidate call"
;  V18 tmp2         [V18    ] (  2, 64   )  simd32  ->  [rsp+0xB0]   do-not-enreg[XS] addr-exposed "struct address for call/obj"
;  V19 tmp3         [V19,T21] (  2, 64   )  simd32  ->  [rsp+0x90]   "non-inline candidate call"
;* V20 tmp4         [V20    ] (  0,  0   )  simd32  ->  zero-ref    "non-inline candidate call"
;  V21 tmp5         [V21    ] (  2, 64   )  simd32  ->  [rsp+0x70]   do-not-enreg[XS] addr-exposed "struct address for call/obj"
;  V22 tmp6         [V22    ] (  2, 64   )  simd32  ->  [rsp+0x50]   do-not-enreg[XS] addr-exposed "struct address for call/obj"
;  V23 tmp7         [V23,T33] (  2, 16   )  double  ->  mm2         "Strict ordering of exceptions for Array store"
;  V24 tmp8         [V24,T34] (  2, 16   )  double  ->  mm2         "Strict ordering of exceptions for Array store"
;  V25 tmp9         [V25,T14] (  2, 16   )     ref  ->  r14         class-hnd "impSpillLclRefs"
;  V26 tmp10        [V26    ] (  6,192   )  simd32  ->  [rsp+0x30]   do-not-enreg[XS] addr-exposed "by-value struct argument"
;  V27 tmp11        [V27,T03] (  2, 64   )    long  ->  rcx         "argument with side effect"
;  V28 tmp12        [V28,T04] (  2, 64   )    long  ->  rcx         "argument with side effect"
;  V29 tmp13        [V29,T05] (  2, 64   )    long  ->  rcx         "argument with side effect"
;* V30 tmp14        [V30,T26] (  0,  0   )  double  ->  zero-ref    "argument with side effect"
;  V31 cse0         [V31,T30] (  2, 20   )  double  ->  mm10         "CSE - moderate"
;  V32 cse1         [V32,T28] (  5, 20   )  double  ->  mm2         "CSE - moderate"
;  V33 cse2         [V33,T35] (  3, 12   )  double  ->  mm1         "CSE - moderate"
;  V34 cse3         [V34,T36] (  3, 12   )  double  ->  mm1         "CSE - moderate"
;  V35 cse4         [V35,T29] (  5, 20   )  double  ->  mm2         "CSE - moderate"
;  V36 cse5         [V36,T12] (  4, 16   )    long  ->  rdx         "CSE - moderate"
;  V37 cse6         [V37,T01] ( 12,120   )     int  ->  [rsp+0x2C]   "CSE - aggressive"
;  V38 cse7         [V38,T13] (  4, 16   )     int  ->  rcx         "CSE - moderate"
;  V39 cse8         [V39,T27] (  5, 26   )  double  ->  mm8         "CSE - moderate"
;  V40 cse9         [V40,T06] (  6, 60   )     int  ->  rax         "CSE - moderate"
;  V41 cse10        [V41,T10] (  3, 24   )     int  ->  r13         "CSE - moderate"
;  V42 cse11        [V42,T08] (  3, 48   )     int  ->  rcx         "CSE - moderate"
;  V43 cse12        [V43,T09] (  3, 48   )     int  ->  rcx         "CSE - moderate"
;  V44 cse13        [V44,T16] (  3, 12   )     int  ->   r8         "CSE - moderate"
;  V45 rat0         [V45,T37] (  2,  4   )  double  ->  mm0         "argument with side effect"
;  V46 rat1         [V46,T38] (  2,  4   )  double  ->  mm1         "argument with side effect"
;
; Lcl frame size = 392

G_M49761_IG01:              ;; offset=0000H
       4157                 push     r15
       4156                 push     r14
       4155                 push     r13
       4154                 push     r12
       57                   push     rdi
       56                   push     rsi
       55                   push     rbp
       53                   push     rbx
       4881EC88010000       sub      rsp, 392
       C5F877               vzeroupper 
       C5F829B42470010000   vmovaps  qword ptr [rsp+170H], xmm6
       C5F829BC2460010000   vmovaps  qword ptr [rsp+160H], xmm7
       C57829842450010000   vmovaps  qword ptr [rsp+150H], xmm8
       C578298C2440010000   vmovaps  qword ptr [rsp+140H], xmm9
       C57829942430010000   vmovaps  qword ptr [rsp+130H], xmm10
       C578299C2420010000   vmovaps  qword ptr [rsp+120H], xmm11
       C57829A42410010000   vmovaps  qword ptr [rsp+110H], xmm12
       C57829AC2400010000   vmovaps  qword ptr [rsp+100H], xmm13
       C57829B424F0000000   vmovaps  qword ptr [rsp+F0H], xmm14
       C57829BC24E0000000   vmovaps  qword ptr [rsp+E0H], xmm15
       8BF9                 mov      edi, ecx
       8BF2                 mov      esi, edx
       C5F828F2             vmovaps  xmm6, xmm2
       C5F828FB             vmovaps  xmm7, xmm3
       488B9C24F8010000     mov      rbx, gword ptr [rsp+1F8H]
						;; bbWeight=1    PerfScore 41.25
G_M49761_IG02:              ;; offset=0084H
       8BD6                 mov      edx, esi
       C1FA1F               sar      edx, 31
       83E203               and      edx, 3
       03D6                 add      edx, esi
       83E2FC               and      edx, -4
       8BCE                 mov      ecx, esi
       2BCA                 sub      ecx, edx
       8BD1                 mov      edx, ecx
       F7DA                 neg      edx
       8D6C3204             lea      ebp, [rdx+rsi+4]
       4863D5               movsxd   rdx, ebp
       48B95019E3F3FB7F0000 mov      rcx, 0x7FFBF3E31950
       E8B1C0885E           call     CORINFO_HELP_NEWARR_1_VC
       4C8BF0               mov      r14, rax
       4863D5               movsxd   rdx, ebp
       48B95019E3F3FB7F0000 mov      rcx, 0x7FFBF3E31950
       E89CC0885E           call     CORINFO_HELP_NEWARR_1_VC
       4C8BF8               mov      r15, rax
       448B4308             mov      r8d, dword ptr [rbx+8]
       488BCB               mov      rcx, rbx
       498BD7               mov      rdx, r15
       E82A3AAFFF           call     System.Array:Copy(System.Array,System.Array,int)
       C5C3598424F0010000   vmulsd   xmm0, xmm7, qword ptr [rsp+1F0H]
       C5FB5EC6             vdivsd   xmm0, xmm0, xmm6
       C57B100545030000     vmovsd   xmm8, qword ptr [reloc @RWD00]
       C4C17828C8           vmovaps  xmm1, xmm8
       E80B20AE5E           call     System.Math:Pow(double,double):double
       C57828C8             vmovaps  xmm9, xmm0
       33DB                 xor      ebx, ebx
       85FF                 test     edi, edi
       0F8E8B020000         jle      G_M49761_IG09
						;; bbWeight=1    PerfScore 34.00
G_M49761_IG03:              ;; offset=0103H
       41BC01000000         mov      r12d, 1
       448D6DFD             lea      r13d, [rbp-3]
       4183FD01             cmp      r13d, 1
       0F8E84010000         jle      G_M49761_IG07
       418B4F08             mov      ecx, dword ptr [r15+8]
       448BC1               mov      r8d, ecx
       C5FB11B424E0010000   vmovsd   qword ptr [rsp+1E0H], xmm6
       C5435ED6             vdivsd   xmm10, xmm7, xmm6
						;; bbWeight=4    PerfScore 67.00
G_M49761_IG04:              ;; offset=012BH
       453BE0               cmp      r12d, r8d
       0F83DA020000         jae      G_M49761_IG12
       418D4C2403           lea      ecx, [r12+3]
       413BC8               cmp      ecx, r8d
       0F83CC020000         jae      G_M49761_IG12
       C4017D105CE710       vmovupd  ymm11, ymmword ptr[r15+8*r12+16]
       418D4C24FF           lea      ecx, [r12-1]
       413BC8               cmp      ecx, r8d
       0F83B7020000         jae      G_M49761_IG12
       418D542402           lea      edx, [r12+2]
       413BD0               cmp      edx, r8d
       0F83A9020000         jae      G_M49761_IG12
       C4417D1064CF10       vmovupd  ymm12, ymmword ptr[r15+8*rcx+16]
       418D4C2401           lea      ecx, [r12+1]
       413BC8               cmp      ecx, r8d
       0F8394020000         jae      G_M49761_IG12
       418D542404           lea      edx, [r12+4]
       448944242C           mov      dword ptr [rsp+2CH], r8d
       413BD0               cmp      edx, r8d
       0F8381020000         jae      G_M49761_IG12
       C4417D106CCF10       vmovupd  ymm13, ymmword ptr[r15+8*rcx+16]
       488D8C24B0000000     lea      rcx, [rsp+B0H]
       C57D115C2430         vmovupd  ymmword ptr[rsp+30H], ymm11
       488D542430           lea      rdx, bword ptr [rsp+30H]
       C4C17828D2           vmovaps  xmm2, xmm10
       C4437D19DE01         vextractf128 ymm14, ymm11, 1
       C4437D19E701         vextractf128 ymm15, ymm12, 1
       C4637D19EE01         vextractf128 ymm6, ymm13, 1
       E80DF2FFFF           call     System.Numerics.Vector`1[Double][System.Double]:op_Multiply(System.Numerics.Vector`1[Double],double):System.Numerics.Vector`1[Double]
       C4432518DE01         vinsertf128 ymm11, ymm11, ymm14, 1
       C4431D18E701         vinsertf128 ymm12, ymm12, ymm15, 1
       C4C1255CCC           vsubpd   ymm1, ymm11, ymm12
       C5F5598C24B0000000   vmulpd   ymm1, ymm1, ymmword ptr[rsp+B0H]
       C5A55CC9             vsubpd   ymm1, ymm11, ymm1
       C5FD118C2490000000   vmovupd  ymmword ptr[rsp+90H], ymm1
       488D4C2470           lea      rcx, [rsp+70H]
       C57D115C2430         vmovupd  ymmword ptr[rsp+30H], ymm11
       4C8D442430           lea      r8, bword ptr [rsp+30H]
       C5F877               vzeroupper 
       C4C17828C8           vmovaps  xmm1, xmm8
       C4437D19E701         vextractf128 ymm15, ymm12, 1
       E8CBF1FFFF           call     System.Numerics.Vector`1[Double][System.Double]:op_Multiply(double,System.Numerics.Vector`1[Double]):System.Numerics.Vector`1[Double]
       488D4C2450           lea      rcx, [rsp+50H]
       C4631518EE01         vinsertf128 ymm13, ymm13, ymm6, 1
       C5955C4C2470         vsubpd   ymm1, ymm13, ymmword ptr[rsp+70H]
       C4431D18E701         vinsertf128 ymm12, ymm12, ymm15, 1
       C4C17558CC           vaddpd   ymm1, ymm1, ymm12
       C5FD114C2430         vmovupd  ymmword ptr[rsp+30H], ymm1
       C5F877               vzeroupper 
       C4C17828C9           vmovaps  xmm1, xmm9
       4C8D442430           lea      r8, bword ptr [rsp+30H]
       E897F1FFFF           call     System.Numerics.Vector`1[Double][System.Double]:op_Multiply(double,System.Numerics.Vector`1[Double]):System.Numerics.Vector`1[Double]
       C57D109C2490000000   vmovupd  ymm11, ymmword ptr[rsp+90H]
       C5A558442450         vaddpd   ymm0, ymm11, ymmword ptr[rsp+50H]
       418B4608             mov      eax, dword ptr [r14+8]
						;; bbWeight=16    PerfScore 1484.00
G_M49761_IG05:              ;; offset=0254H
       443BE0               cmp      r12d, eax
       0F83AD010000         jae      G_M49761_IG13
       418D542403           lea      edx, [r12+3]
       3BD0                 cmp      edx, eax
       0F83A5010000         jae      G_M49761_IG14
       C4817D1144E610       vmovupd  ymmword ptr[r14+8*r12+16], ymm0
       4183C404             add      r12d, 4
       453BE5               cmp      r12d, r13d
       448B44242C           mov      r8d, dword ptr [rsp+2CH]
       0F8CA8FEFFFF         jl       G_M49761_IG04
						;; bbWeight=16    PerfScore 136.00
G_M49761_IG06:              ;; offset=0283H
       C5F877               vzeroupper 
       C5FB10B424E0010000   vmovsd   xmm6, qword ptr [rsp+1E0H]
						;; bbWeight=8    PerfScore 24.00
G_M49761_IG07:              ;; offset=028FH
       458B6F08             mov      r13d, dword ptr [r15+8]
       4585ED               test     r13d, r13d
       0F8468010000         je       G_M49761_IG12
       C4C17B105710         vmovsd   xmm2, qword ptr [r15+16]
       C5EB59C7             vmulsd   xmm0, xmm2, xmm7
       C5FB5EC6             vdivsd   xmm0, xmm0, xmm6
       8D4EFF               lea      ecx, [rsi-1]
       413BCD               cmp      ecx, r13d
       0F834E010000         jae      G_M49761_IG12
       4863D1               movsxd   rdx, ecx
       C4C17B104CD710       vmovsd   xmm1, qword ptr [r15+8*rdx+16]
       C5EB5CD9             vsubsd   xmm3, xmm2, xmm1
       C5FB59C3             vmulsd   xmm0, xmm0, xmm3
       C5EB5CC0             vsubsd   xmm0, xmm2, xmm0
       4183FD01             cmp      r13d, 1
       0F862E010000         jbe      G_M49761_IG12
       C4C17B105F18         vmovsd   xmm3, qword ptr [r15+24]
       C4C16B59D0           vmulsd   xmm2, xmm2, xmm8
       C5E35CD2             vsubsd   xmm2, xmm3, xmm2
       C5EB58D1             vaddsd   xmm2, xmm2, xmm1
       C4C16B59D1           vmulsd   xmm2, xmm2, xmm9
       C5FB58D2             vaddsd   xmm2, xmm0, xmm2
       418B4608             mov      eax, dword ptr [r14+8]
       85C0                 test     eax, eax
       0F8406010000         je       G_M49761_IG12
       C4C17B115610         vmovsd   qword ptr [r14+16], xmm2
       C4C17B1054D710       vmovsd   xmm2, qword ptr [r15+8*rdx+16]
       C5EB59C7             vmulsd   xmm0, xmm2, xmm7
       C5FB5EC6             vdivsd   xmm0, xmm0, xmm6
       448D46FE             lea      r8d, [rsi-2]
       453BC5               cmp      r8d, r13d
       0F83E4000000         jae      G_M49761_IG12
       4D63C0               movsxd   r8, r8d
       C4817B104CC710       vmovsd   xmm1, qword ptr [r15+8*r8+16]
       C5EB5CD9             vsubsd   xmm3, xmm2, xmm1
       C5FB59C3             vmulsd   xmm0, xmm0, xmm3
       C5EB5CC0             vsubsd   xmm0, xmm2, xmm0
       C4C17B105F10         vmovsd   xmm3, qword ptr [r15+16]
       C4C16B59D0           vmulsd   xmm2, xmm2, xmm8
       C5E35CD2             vsubsd   xmm2, xmm3, xmm2
       C5EB58D1             vaddsd   xmm2, xmm2, xmm1
       C4C16B59D1           vmulsd   xmm2, xmm2, xmm9
       C5FB58D2             vaddsd   xmm2, xmm0, xmm2
       3BC8                 cmp      ecx, eax
       0F83AA000000         jae      G_M49761_IG12
       C4C17B1154D610       vmovsd   qword ptr [r14+8*rdx+16], xmm2
       FFC3                 inc      ebx
       3BDF                 cmp      ebx, edi
       0F8C91000000         jl       G_M49761_IG11
						;; bbWeight=4    PerfScore 422.00
G_M49761_IG08:              ;; offset=036BH
       4D8BFE               mov      r15, r14
						;; bbWeight=2    PerfScore 0.50
G_M49761_IG09:              ;; offset=036EH
       498BC7               mov      rax, r15
						;; bbWeight=1    PerfScore 0.25
G_M49761_IG10:              ;; offset=0371H
       C5F828B42470010000   vmovaps  xmm6, qword ptr [rsp+170H]
       C5F828BC2460010000   vmovaps  xmm7, qword ptr [rsp+160H]
       C57828842450010000   vmovaps  xmm8, qword ptr [rsp+150H]
       C578288C2440010000   vmovaps  xmm9, qword ptr [rsp+140H]
       C57828942430010000   vmovaps  xmm10, qword ptr [rsp+130H]
       C578289C2420010000   vmovaps  xmm11, qword ptr [rsp+120H]
       C57828A42410010000   vmovaps  xmm12, qword ptr [rsp+110H]
       C57828AC2400010000   vmovaps  xmm13, qword ptr [rsp+100H]
       C57828B424F0000000   vmovaps  xmm14, qword ptr [rsp+F0H]
       C57828BC24E0000000   vmovaps  xmm15, qword ptr [rsp+E0H]
       C5F877               vzeroupper 
       4881C488010000       add      rsp, 392
       5B                   pop      rbx
       5D                   pop      rbp
       5E                   pop      rsi
       5F                   pop      rdi
       415C                 pop      r12
       415D                 pop      r13
       415E                 pop      r14
       415F                 pop      r15
       C3                   ret      
						;; bbWeight=1    PerfScore 46.25
G_M49761_IG11:              ;; offset=03E2H
       4D87F7               xchg     r14, r15
       E919FDFFFF           jmp      G_M49761_IG03
						;; bbWeight=2    PerfScore 6.00
G_M49761_IG12:              ;; offset=03EAH
       E811B73F5E           call     CORINFO_HELP_RNGCHKFAIL
						;; bbWeight=0    PerfScore 0.00
G_M49761_IG13:              ;; offset=03EFH
       E80CBD3F5E           call     CORINFO_HELP_THROW_ARGUMENTOUTOFRANGEEXCEPTION
						;; bbWeight=0    PerfScore 0.00
G_M49761_IG14:              ;; offset=03F4H
       E807BA3F5E           call     CORINFO_HELP_THROW_ARGUMENTEXCEPTION
       CC                   int3     
						;; bbWeight=0    PerfScore 0.00
RWD00  	dq	4000000000000000h	;            2


; Total bytes of code 1018, prolog size 132, PerfScore 2368.45, instruction count 209, allocated bytes for code 1072 (MethodHash=9b453d9e) for method BurgersRepro.Program:GetCalculated3(int,int,double,double,double,System.Double[]):System.Double[]
; ============================================================

Looking at the original assembly, I'm noticing that there might be some loops that aren't being unrolled or are being unrolled differently between 5.0 and 6.0? In the 6.0 assembly, I see the following section:

M01_L03:
       lea       r8,[rsp+0A0]
       vmovsd    xmm10,qword ptr [r8+r10*8]
       vmulsd    xmm10,xmm9,xmm10
       lea       r8,[rsp+0C0]
       vmovsd    qword ptr [r8+r10*8],xmm10
       inc       r10
       cmp       r10,4
       jl        short M01_L03
       vmovupd   ymm9,[rsp+0C0]
       vsubpd    ymm4,ymm4,ymm2
       vmulpd    ymm4,ymm9,ymm4
       vsubpd    ymm4,ymm1,ymm4
       vmovupd   [rsp+60],ymm1
       vmovupd   [rsp+80],ymm5
       xor       r8d,r8d
M01_L04:
       lea       r10,[rsp+60]
       vmovsd    xmm1,qword ptr [r10+r8*8]
       vaddsd    xmm1,xmm1,xmm1
       lea       r10,[rsp+80]
       vmovsd    qword ptr [r10+r8*8],xmm1
       inc       r8
       cmp       r8,4
       jl        short M01_L04
       vmovupd   ymm1,[rsp+80]
       vsubpd    ymm1,ymm3,ymm1
       vaddpd    ymm1,ymm1,ymm2
       vmovupd   [rsp+20],ymm1
       vmovupd   [rsp+40],ymm5
       xor       r8d,r8d
M01_L05:
       lea       r10,[rsp+20]
       vmovsd    xmm1,qword ptr [r10+r8*8]
       vmulsd    xmm1,xmm1,xmm0
       lea       r10,[rsp+40]
       vmovsd    qword ptr [r10+r8*8],xmm1
       inc       r8
       cmp       r8,4
       jl        short M01_L05

But I don't see that same pattern in the 5.0 assmebly, I just see the one loop:

M01_L01:
       cmp       r9d,r10d
       jae       near ptr M01_L05
       lea       ebx,[r9+3]
       cmp       ebx,r10d
       jae       near ptr M01_L05
       vmovupd   ymm2,[r15+r9*8+10]
       lea       r11d,[r9+0FFFF]
       cmp       r11d,r10d
       jae       near ptr M01_L05
       lea       r12d,[r9+2]
       cmp       r12d,r10d
       jae       near ptr M01_L05
       vmovupd   ymm3,[r15+r11*8+10]
       lea       r11d,[r9+1]
       cmp       r11d,r10d
       jae       near ptr M01_L05
       lea       r12d,[r9+4]
       cmp       r12d,r10d
       jae       near ptr M01_L05
       vmovupd   ymm4,[r15+r11*8+10]
       vbroadcastsd ymm5,xmm1
       vmulpd    ymm5,ymm5,ymm2
       vsubpd    ymm8,ymm2,ymm3
       vmulpd    ymm5,ymm5,ymm8
       vsubpd    ymm5,ymm2,ymm5
       vbroadcastsd ymm8,qword ptr [7FFBE2A71AE8]
       vmulpd    ymm2,ymm8,ymm2
       vsubpd    ymm2,ymm4,ymm2
       vaddpd    ymm2,ymm2,ymm3
       vbroadcastsd ymm3,xmm0
       vmulpd    ymm2,ymm3,ymm2
       vaddpd    ymm2,ymm5,ymm2
       mov       r11d,[r14+8]
       cmp       r9d,r11d
       jae       near ptr M01_L06
       cmp       ebx,r11d
       jae       near ptr M01_L07
       vmovupd   [r14+r9*8+10],ymm2
       mov       r9d,r12d
       cmp       r9d,ebp
       jl        near ptr M01_L01

@kunalspathak Wondering if you know the best way to configure any loop unrolling options in the JIT when generating a dump to help see if this is the issue?

[kunalspathak]

I verified that the regressions comes from #42050.

Asmdiffs: https://www.diffchecker.com/BUv4WFCr

@tannergooding - I am assigning this to you.