[Perf] Changes at 12/26/2021 10:58:11 AM

[performanceautofiler]

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	71192eacd5194e8afce9686e701aa5f072972b8d
Compare	7ab969c84ef05ba948c0075392716ce335b47744
Diff	Diff

Regressions in PerfLabTests.GetMember

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GetMethod10 - Duration of single invocation	867.00 μs	1.15 ms	1.33	0.11	True
GetMethod5 - Duration of single invocation	540.78 μs	589.19 μs	1.09	0.12	True
GetMethod12 - Duration of single invocation	1.04 ms	1.41 ms	1.35	0.08	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.GetMember*'

Details

Payloads

Baseline
Compare

Histogram

PerfLabTests.GetMember.GetMethod10

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1516062156593405 > 908.7084953399124.
IsChangePoint: Marked as a change because one of 12/26/2021 4:30:45 AM, 1/5/2022 6:46:20 PM falls between 12/6/2021 2:46:54 PM and 1/5/2022 6:46:20 PM.
IsRegressionStdDev: Marked as regression because -32.249640928760144 (T) = (0 -1165738.8169220851) / Math.Sqrt((2417218554.266879 / (83)) + (1465316332.3535087 / (31))) is less than -1.9813718148759274 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (83) + (31) - 2, .025) and -0.31890306372968186 = (883869.9742083632 - 1165738.8169220851) / 883869.9742083632 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### PerfLabTests.GetMember.GetMethod5

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 589.1945524691358 > 457.07850260416666.
IsChangePoint: Marked as a change because one of 12/26/2021 4:30:45 AM, 1/5/2022 6:46:20 PM falls between 12/6/2021 2:46:54 PM and 1/5/2022 6:46:20 PM.
IsRegressionStdDev: Marked as regression because -26.17071255316827 (T) = (0 -595822.2093277765) / Math.Sqrt((973207935.473748 / (83)) + (692759957.5964096 / (31))) is less than -1.9813718148759274 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (83) + (31) - 2, .025) and -0.3447906618513562 = (443059.4487528011 - 595822.2093277765) / 443059.4487528011 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### PerfLabTests.GetMember.GetMethod12

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4062647023809525 > 1.0935164550781251.
IsChangePoint: Marked as a change because one of 12/26/2021 4:30:45 AM, 1/5/2022 6:46:20 PM falls between 12/6/2021 2:46:54 PM and 1/5/2022 6:46:20 PM.
IsRegressionStdDev: Marked as regression because -33.97323270602115 (T) = (0 -1388667.1066422376) / Math.Sqrt((6334608012.472459 / (83)) + (377140135.0807543 / (31))) is less than -1.9813718148759274 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (83) + (31) - 2, .025) and -0.29892392718979804 = (1069090.4044293014 - 1388667.1066422376) / 1069090.4044293014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

category:performance
theme:benchmarks

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[kunalspathak]

Possibly related to #63129
@jkotas

[jkotas]

#63129 was 100% platform agnostic change and it only affected exception throwing paths.

What is the trend for x64? Do you see the same drop on x64?

[kunalspathak]

What is the trend for x64? Do you see the same drop on x64?

We haven't triaged x64 issues yet. @DrewScoggins , can you share your findings once you triage them?

[DrewScoggins]

We are seeing the same behavior on Windows x64 as well

GetMethod5
GetMethod12
GetMethod10

Tagging subscribers to this area: @dotnet/area-system-reflection
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	71192eacd5194e8afce9686e701aa5f072972b8d
Compare	7ab969c84ef05ba948c0075392716ce335b47744
Diff	Diff

Regressions in PerfLabTests.GetMember

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GetMethod10 - Duration of single invocation	867.00 μs	1.15 ms	1.33	0.11	True
GetMethod5 - Duration of single invocation	540.78 μs	589.19 μs	1.09	0.12	True
GetMethod12 - Duration of single invocation	1.04 ms	1.41 ms	1.35	0.08	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.GetMember*'

Details

Payloads

Baseline
Compare

Histogram

PerfLabTests.GetMember.GetMethod10

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1516062156593405 > 908.7084953399124.
IsChangePoint: Marked as a change because one of 12/26/2021 4:30:45 AM, 1/5/2022 6:46:20 PM falls between 12/6/2021 2:46:54 PM and 1/5/2022 6:46:20 PM.
IsRegressionStdDev: Marked as regression because -32.249640928760144 (T) = (0 -1165738.8169220851) / Math.Sqrt((2417218554.266879 / (83)) + (1465316332.3535087 / (31))) is less than -1.9813718148759274 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (83) + (31) - 2, .025) and -0.31890306372968186 = (883869.9742083632 - 1165738.8169220851) / 883869.9742083632 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### PerfLabTests.GetMember.GetMethod5

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 589.1945524691358 > 457.07850260416666.
IsChangePoint: Marked as a change because one of 12/26/2021 4:30:45 AM, 1/5/2022 6:46:20 PM falls between 12/6/2021 2:46:54 PM and 1/5/2022 6:46:20 PM.
IsRegressionStdDev: Marked as regression because -26.17071255316827 (T) = (0 -595822.2093277765) / Math.Sqrt((973207935.473748 / (83)) + (692759957.5964096 / (31))) is less than -1.9813718148759274 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (83) + (31) - 2, .025) and -0.3447906618513562 = (443059.4487528011 - 595822.2093277765) / 443059.4487528011 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### PerfLabTests.GetMember.GetMethod12

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4062647023809525 > 1.0935164550781251.
IsChangePoint: Marked as a change because one of 12/26/2021 4:30:45 AM, 1/5/2022 6:46:20 PM falls between 12/6/2021 2:46:54 PM and 1/5/2022 6:46:20 PM.
IsRegressionStdDev: Marked as regression because -33.97323270602115 (T) = (0 -1388667.1066422376) / Math.Sqrt((6334608012.472459 / (83)) + (377140135.0807543 / (31))) is less than -1.9813718148759274 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (83) + (31) - 2, .025) and -0.29892392718979804 = (1069090.4044293014 - 1388667.1066422376) / 1069090.4044293014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	arch-arm64, area-System.Reflection, os-windows, tenet-performance, tenet-performance-benchmarks
Milestone:	-

[jkotas]

I am not able to reproduce the regression.

The benchmark is basically just a several relatively large methods (a lot of code expansion due to inlining) with loops. The trend of the bechmark suggests that it is multi-modal, likely due to sensitivity to code alignment. My guess is that one of the PRs in the range moved things around and affected the code alignment.