[PERF] Regression on AMD

[sebastienros]

Between 3.1 and master (5.0) ... 33% degradation

These numbers are for Plaintext, but the same regression appears on all scenarios.

-n Plaintext --webHost KestrelSockets -j /var/jenkins_home/workspace/baseline_citrine_amd/build/../src/Benchmarks/benchmarks.plaintext.json --description Baseline31 --aspnetCoreVersion 3.1 --runtimeVersion 3.1 --sdk 3.1.102

RequestsPerSecond:           3,980,990
Max CPU (%):                 89
WorkingSet (MB):             96
Avg. Latency (ms):           3.25
Startup (ms):                275
First Request (ms):          30.53
Latency (ms):                0.1
Total Requests:              80,017,797
Duration: (ms)               20,100
Socket Errors:               0
Bad Responses:               0
Build Time (ms):             8,502
Published Size (KB):         115,446
SDK:                         3.1.102
Runtime:                     3.1.2
ASP.NET Core:                3.1.2

-n Plaintext --webHost KestrelSockets -j /var/jenkins_home/workspace/baseline_citrine_amd/build/../src/Benchmarks/benchmarks.plaintext.json --description Baseline 

RequestsPerSecond:           2,988,782
Max CPU (%):                 77
WorkingSet (MB):             100
Avg. Latency (ms):           3.42
Startup (ms):                253
First Request (ms):          28.98
Latency (ms):                0.16
Total Requests:              60,073,627
Duration: (ms)               20,100
Socket Errors:               0
Bad Responses:               0
Build Time (ms):             3,501
Published Size (KB):         119,145
SDK:                         5.0.100-preview.2.20120.3
Runtime:                     5.0.0-preview.2.20125.16
ASP.NET Core:                5.0.0-preview.2.20126.7

[benaadams]

Running runtime with the AMD fix? #31904

[gfoidl]

Do these benchmarks hit dotnet/aspnetcore#19009?
The two PRs should mitigate the regression on AMD.

[sebastienros]

This is using the latest builds, so not older than a day. You can see the exact versions that contain the day of the month in it.

We can get the whole published folder, and also traces if needed.

[sebastienros]

I have isolated the regression between:

• 5.0.0-alpha.1.20103.10 https://github.com/dotnet/runtime/tree/a7f8cdc0de945561ae3fcaa72440c0ec56d69060 (Feb 3 12:26 PM PST)
• 5.0.0-alpha.1.20111.2 https://github.com/dotnet/runtime/tree/f496497de42b80b6d031d5de445ec74b2b01bb17 (Feb 11 1:59 AM PST)

By running the Plaintext NonPipelined scenario (60% regression) like so:

dotnet run -- --server "http://asp-citrine-amd:5001" --client "http://asp-citrine-load:5002" -n plaintextnonpipelined -j ..\Benchmarks\benchmarks.plaintext.json --warmup 5 --duration 10 --self-contained  --aspnetcoreversion 5.0.0-preview.1.20107.8 --sdk 5.0.100-preview.1.20110.1 --runtimeversion 5.0.0-alpha.1.20111.2

The aspnet core version was pinned for both runs showing that the reason seems to be a runtime change. Unfortunately there was no successful runtime build+publication during this window so I can't get more precise that this.

At that point I suggest that someone tries to identify which commits might be related, and try to rebuild the impacted assets of these commits, so we can test them on the benchmark to identity the root cause.

[stephentoub]

Does the regression appear only on Linux or also on Windows?

If it weren't for the fact that I know you and @adamsitnik said you did lots of testing on #2346, that would be my first guess.

[benaadams]

Why I thought it might need the AMD fix #31904 is recently Kestrel changed from using its own ASCII encoding to use Encoding.ASCII.GetBytes dotnet/aspnetcore#18404 so would be dependent on and effected by any runtime regressions. However that was merged on 11 Feb so should be in the before and not the after.

[benaadams]

on #2346, that would be my first guess.

Is in the commit range

[adamsitnik]

It's interesting that the CPU consumption was never 100% and the RPS for this 48 core machine is smaller compared to the machine which has only 12 cores.

I suspect that there is some underlying scalability issue and my change in #2346 has for some reason made it even worse.

As soon as we get PerfCollect to work on this machine I would be more than happy to analyze the trace files and identify the problem.

[sebastienros]

The AMD machine has been available for a week only, and is on Linux.

I have seen other unusual behaviors on this machine that I need to investigate, related to the number of cores, so this might be related to Adam's suspicion. I will gather some data.

[adamsitnik]

While I was working on the TechEmpower recently, I found out that the max number of IO Threads is 16 (for some reason, imho it should be Math.Max(Environment.ProcessorCount / 2, 1)).

Could you try to run some benchmarks on this machine with a bigger number? If I remember correctly, it's configurable via --kestrelthreadcount argument

https://github.com/dotnet/aspnetcore/blob/34ee3324fb4a8c8de4ebdee6a6c8ecdbe5891236/src/Servers/Kestrel/Transport.Sockets/src/SocketTransportOptions.cs#L17