-
Notifications
You must be signed in to change notification settings - Fork 4
[Perf] Changes at 2/26/2022 2:38:27 AM #3784
Copy link
Copy link
Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | alpine 3.12 |
| Baseline | 0c33372da6e56b1150d6111ee279ab6f52231a98 |
| Compare | c8da2fd5d8b841fc34845218158399cf35eb8bcf |
| Diff | Diff |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 4.85 ms | 4.49 ms | 0.92 | 0.01 | False | |||||
| Count - Duration of single invocation | 5.13 ms | 4.67 ms | 0.91 | 0.02 | False | |||||
| Count - Duration of single invocation | 3.75 ms | 3.31 ms | 0.88 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\s[a-zA-Z]{0,12}ing\s", Options: NonBacktracking)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.48737708030303 < 4.6141636706958264.
IsChangePoint: Marked as a change because one of 2/8/2022 12:37:33 AM, 2/11/2022 9:28:46 PM, 2/25/2022 11:02:46 PM, 3/1/2022 5:49:50 AM falls between 2/20/2022 5:50:14 AM and 3/1/2022 5:49:50 AM.
IsImprovementStdDev: Marked as improvement because 64.34634832129687 (T) = (0 -4500460.316874301) / Math.Sqrt((10627152922.546461 / (299)) + (196129875.5654471 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.0997509558949912 = (4999128.126093682 - 4500460.316874301) / 4999128.126093682 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\s[a-zA-Z]{0,12}ing\s", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.669069198660715 < 4.949552988700397.
IsChangePoint: Marked as a change because one of 1/15/2022 9:05:03 AM, 2/15/2022 12:48:49 AM, 2/26/2022 10:27:55 AM, 3/1/2022 5:49:50 AM falls between 2/20/2022 5:50:14 AM and 3/1/2022 5:49:50 AM.
IsImprovementStdDev: Marked as improvement because 6.302454281717755 (T) = (0 -4728492.344083105) / Math.Sqrt((28863042239.856823 / (299)) + (9380445112.794176 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.05588967449838557 = (5008410.793061514 - 4728492.344083105) / 5008410.793061514 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+\s+Holmes\s+\w+", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.306345409777778 < 3.546261253896588.
IsChangePoint: Marked as a change because one of 2/15/2022 12:48:49 AM, 2/26/2022 10:27:55 AM, 3/1/2022 5:49:50 AM falls between 2/20/2022 5:50:14 AM and 3/1/2022 5:49:50 AM.
IsImprovementStdDev: Marked as improvement because 20.6297238685053 (T) = (0 -3283154.7027353174) / Math.Sqrt((16881288828.491781 / (299)) + (235785646.5320235 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.06011603460739471 = (3493148.966919431 - 3283154.7027353174) / 3493148.966919431 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | alpine 3.12 |
| Baseline | 0c33372da6e56b1150d6111ee279ab6f52231a98 |
| Compare | c8da2fd5d8b841fc34845218158399cf35eb8bcf |
| Diff | Diff |
Improvements in System.Linq.Tests.Perf_Enumerable
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| FirstWithPredicate_LastElementMatches - Duration of single invocation | 518.21 ns | 459.86 ns | 0.89 | 0.11 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'Details
Payloads
Histogram
System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IEnumerable)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 459.85831484385005 < 491.0330063684158.
IsChangePoint: Marked as a change because one of 2/8/2022 5:20:12 PM, 2/17/2022 3:48:53 PM, 2/19/2022 1:08:37 AM, 2/25/2022 11:02:46 PM, 3/1/2022 5:49:50 AM falls between 2/20/2022 5:50:14 AM and 3/1/2022 5:49:50 AM.
IsImprovementStdDev: Marked as improvement because 3.7533798392740776 (T) = (0 -466.3102948640107) / Math.Sqrt((375.5884232080384 / (282)) + (350.5934243325014 / (8))) is greater than 1.9682351736146575 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (282) + (8) - 2, .975) and 0.05131311636195156 = (491.5323516182623 - 466.3102948640107) / 491.5323516182623 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | alpine 3.12 |
| Baseline | 0c33372da6e56b1150d6111ee279ab6f52231a98 |
| Compare | c8da2fd5d8b841fc34845218158399cf35eb8bcf |
| Diff | Diff |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 117.10 ms | 101.09 ms | 0.86 | 0.03 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: NonBacktracking)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 101.0861945 < 108.23783633633242.
IsChangePoint: Marked as a change because one of 2/8/2022 12:37:33 AM, 2/11/2022 9:28:46 PM, 2/25/2022 11:02:46 PM, 3/1/2022 5:49:50 AM falls between 2/20/2022 5:50:14 AM and 3/1/2022 5:49:50 AM.
IsImprovementStdDev: Marked as improvement because 81.58236125039869 (T) = (0 -101233818.40238094) / Math.Sqrt((6385515615224.782 / (299)) + (29264686117.397625 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.11304852716683375 = (114136817.51833881 - 101233818.40238094) / 114136817.51833881 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | alpine 3.12 |
| Baseline | 0c33372da6e56b1150d6111ee279ab6f52231a98 |
| Compare | c8da2fd5d8b841fc34845218158399cf35eb8bcf |
| Diff | Diff |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IsMatch - Duration of single invocation | 126.00 ns | 118.00 ns | 0.94 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 9, Options: NonBacktracking)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 118.002386307482 < 120.53650479435872.
IsChangePoint: Marked as a change because one of 2/19/2022 11:51:48 PM, 2/25/2022 11:02:46 PM, 3/1/2022 5:49:50 AM falls between 2/20/2022 5:50:14 AM and 3/1/2022 5:49:50 AM.
IsImprovementStdDev: Marked as improvement because 32.25764577201101 (T) = (0 -118.86229416933939) / Math.Sqrt((3.74158353594347 / (299)) + (0.49709969444614915 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.06903067099119957 = (127.67584330183212 - 118.86229416933939) / 127.67584330183212 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Reactions are currently unavailable
Metadata
Metadata
Assignees
Type
Fields
Give feedbackNo fields configured for issues without a type.


