Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Perf] Windows/x64: 13 Regressions on 4/17/2023 11:29:15 PM #16917

Closed
performanceautofiler bot opened this issue Apr 25, 2023 · 1 comment
Closed

[Perf] Windows/x64: 13 Regressions on 4/17/2023 11:29:15 PM #16917

performanceautofiler bot opened this issue Apr 25, 2023 · 1 comment
Assignees
@kunalspathak
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 25, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 14.24 ns 15.75 ns 1.11 0.27 False 204.00979345918162 219.8935752502178 1.0778579377083395) Trace Trace
Parse - Duration of single invocation 14.80 ns 16.83 ns 1.14 0.25 False Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.Parse(value: "32767")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.751831320143213 > 14.978052890908078.
IsChangePoint: Marked as a change because one of 2/14/2023 11:21:41 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -11.606809317088151 (T) = (0 -16.03423588104406) / Math.Sqrt((0.12664414931152215 / (16)) + (0.36516795961662746 / (32))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (32) - 2, .025) and -0.11189419305702006 = (14.42064899804886 - 16.03423588104406) / 14.42064899804886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Int16.Parse(value: "-32768")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.830216513264055 > 15.53724792024754.
IsChangePoint: Marked as a change because one of 2/14/2023 10:45:34 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -9.840558849842646 (T) = (0 -16.615741230810286) / Math.Sqrt((0.13024933625889365 / (16)) + (0.6745015015612567 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.11126226503685574 = (14.95213304148253 - 16.615741230810286) / 14.95213304148253 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Span - Duration of single invocation 15.87 μs 16.93 μs 1.07 0.10 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.Span(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.932781397032212 > 16.683166888297873.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 3/6/2023 11:14:52 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -5.989289773657847 (T) = (0 -16804.783423752997) / Math.Sqrt((348561.70925745624 / (16)) + (282600.2106729922 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.06619894290712855 = (15761.395690313286 - 16804.783423752997) / 15761.395690313286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare 90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_SByte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 12.59 ns 14.13 ns 1.12 0.34 False Trace Trace
Parse - Duration of single invocation 12.91 ns 14.66 ns 1.14 0.29 False 181.14563474898938 197.08599698417459 1.087997495811994) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.Parse(value: "127")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.125992006793247 > 13.13879364677171.
IsChangePoint: Marked as a change because one of 2/11/2023 7:07:55 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -8.415273937149585 (T) = (0 -14.315951816285791) / Math.Sqrt((0.11640928630621405 / (16)) + (0.9279620208269205 / (31))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (31) - 2, .025) and -0.12789235318471096 = (12.692657925965491 - 14.315951816285791) / 12.692657925965491 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_SByte.Parse(value: "-128")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.658769729450594 > 13.54416071918037.
IsChangePoint: Marked as a change because one of 2/14/2023 10:45:34 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -3.8192623284078584 (T) = (0 -14.372523513282488) / Math.Sqrt((1.204133288180699 / (16)) + (0.02890431388143468 / (32))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (32) - 2, .025) and -0.07913925432032819 = (13.318506815262412 - 14.372523513282488) / 13.318506815262412 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse3 - Duration of single invocation 35.11 ns 40.16 ns 1.14 0.01 True 462.16797425839036 482.8721567706881 1.0447979601908166) Trace Trace
Parse2 - Duration of single invocation 32.69 ns 35.75 ns 1.09 0.01 True Trace Trace
TryParse2 - Duration of single invocation 29.88 ns 31.65 ns 1.06 0.01 True 364.8384807482681 377.639329647514 1.0350863452588441) Trace Trace
TryParse3 - Duration of single invocation 33.36 ns 36.00 ns 1.08 0.00 True 440.74124955297214 456.3303458265436 1.0353701776028064) Trace Trace
TryParse4 - Duration of single invocation 53.68 ns 58.62 ns 1.09 0.00 True 668.4554801601473 694.6383600183013 1.0391692201429499) Trace Trace

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.Parse3

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.16239255984396 > 36.63214849062534.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -28.95887521621109 (T) = (0 -39.700695537142565) / Math.Sqrt((0.20806615789870309 / (16)) + (0.37431010585617047 / (32))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (32) - 2, .025) and -0.12948658663859075 = (35.149328913496745 - 39.700695537142565) / 35.149328913496745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Version.Parse2

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.74567315032882 > 34.081954046224475.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -27.091471124027155 (T) = (0 -36.14794780669832) / Math.Sqrt((0.06313603042932722 / (14)) + (0.4749522994469745 / (33))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (33) - 2, .025) and -0.11487673725819167 = (32.42326850912389 - 36.14794780669832) / 32.42326850912389 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Version.TryParse2

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.652192988197786 > 31.303923134806077.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -23.34920674498986 (T) = (0 -31.859615987564073) / Math.Sqrt((0.08355550403450145 / (16)) + (0.08853216256766265 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.06970207267896955 = (29.783634902917054 - 31.859615987564073) / 29.783634902917054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Version.TryParse3

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.00109161822378 > 35.53914430458859.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -23.292671631450187 (T) = (0 -35.774148034066634) / Math.Sqrt((0.0488523955741817 / (16)) + (0.15408316065648017 / (32))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (32) - 2, .025) and -0.061295532743701184 = (33.70799831935781 - 35.774148034066634) / 33.70799831935781 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Version.TryParse4

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 58.61993016121663 > 55.93540562628913.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -35.67831036192813 (T) = (0 -58.47952651838256) / Math.Sqrt((0.1654132031536049 / (15)) + (0.2968013437079938 / (32))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (32) - 2, .025) and -0.09520783354760168 = (53.39582563882459 - 58.47952651838256) / 53.39582563882459 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorDefaultSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedSet - Duration of single invocation 15.83 ns 17.38 ns 1.10 0.28 False Trace Trace
Queue - Duration of single invocation 16.77 ns 18.05 ns 1.08 0.25 False Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorDefaultSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorDefaultSize<String>.SortedSet

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.382008559427064 > 16.41625176468302.
IsChangePoint: Marked as a change because one of 2/9/2023 4:10:34 PM, 2/10/2023 11:47:22 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -8.760654349032466 (T) = (0 -17.459606466371195) / Math.Sqrt((0.08681036968921212 / (16)) + (1.0117143125387675 / (32))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (32) - 2, .025) and -0.10689410189284915 = (15.773511157494035 - 17.459606466371195) / 15.773511157494035 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Collections.CtorDefaultSize<String>.Queue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.053603535841948 > 17.50234080557936.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -9.651644675042178 (T) = (0 -18.359146894027567) / Math.Sqrt((0.20518103375654215 / (16)) + (0.35026819710846036 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.08752856281347339 = (16.88153076782814 - 18.359146894027567) / 16.88153076782814 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_Same_Upper - Duration of single invocation 1.68 μs 2.02 μs 1.20 0.03 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringEquality*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0183136377859996 > 1.763121480401568.
IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 3/17/2023 10:06:19 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -12.946235857352336 (T) = (0 -2008.7237428800436) / Math.Sqrt((10806.883184041426 / (16)) + (18.007742160027576 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.201298110535889 = (1672.12761367282 - 2008.7237428800436) / 1672.12761367282 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@kunalspathak kunalspathak

Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kunalspathak