Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Perf] Windows/arm64: 7 Regressions on 4/18/2023 2:18:01 AM #16763

Closed
performanceautofiler bot opened this issue Apr 20, 2023 · 1 comment
Closed

[Perf] Windows/arm64: 7 Regressions on 4/18/2023 2:18:01 AM #16763

performanceautofiler bot opened this issue Apr 20, 2023 · 1 comment
Assignees
@EgorBo
Labels
arch-arm64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 20, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare 634cc1e04e91b2b264a57798a2c733ae261758f9
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_SByte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 12.60 ns 13.78 ns 1.09 0.07 False
TryParse - Duration of single invocation 11.62 ns 13.18 ns 1.13 0.13 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.TryParse(value: "-128")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.784010946589708 > 13.072777265683868.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/20/2023 12:14:19 AM falls between 4/11/2023 5:52:15 AM and 4/20/2023 12:14:19 AM.
IsRegressionStdDev: Marked as regression because -14.072503360994446 (T) = (0 -13.831573577618375) / Math.Sqrt((0.12224441537409077 / (19)) + (0.0053956326260793394 / (5))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (5) - 2, .025) and -0.09671687031240189 = (12.61179977442895 - 13.831573577618375) / 12.61179977442895 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_SByte.TryParse(value: "127")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.178725167736376 > 12.327657887007549.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/20/2023 12:14:19 AM falls between 4/11/2023 5:52:15 AM and 4/20/2023 12:14:19 AM.
IsRegressionStdDev: Marked as regression because -11.560680494955447 (T) = (0 -13.027801961724492) / Math.Sqrt((0.04825601504674728 / (19)) + (0.04689586536009053 / (6))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (6) - 2, .025) and -0.09926742630395309 = (11.851349043906117 - 13.027801961724492) / 11.851349043906117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare 90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 14.88 ns 16.23 ns 1.09 0.13 False
Parse - Duration of single invocation 14.37 ns 16.08 ns 1.12 0.13 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt16.TryParse(value: "65535")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.22813793182175 > 15.654946838795647.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/20/2023 12:14:19 AM falls between 4/11/2023 5:52:15 AM and 4/20/2023 12:14:19 AM.
IsRegressionStdDev: Marked as regression because -12.30160796979925 (T) = (0 -16.282375474381162) / Math.Sqrt((0.17570017067013816 / (19)) + (0.010960299334759529 / (6))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (6) - 2, .025) and -0.08637282249885606 = (14.987833952738915 - 16.282375474381162) / 14.987833952738915 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_UInt16.Parse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.080071113803793 > 15.76063170069148.
IsChangePoint: Marked as a change because one of 4/17/2023 5:51:51 AM, 4/20/2023 12:14:19 AM falls between 4/11/2023 5:52:15 AM and 4/20/2023 12:14:19 AM.
IsRegressionStdDev: Marked as regression because -12.910085926642264 (T) = (0 -16.12191109500379) / Math.Sqrt((0.12347736024800872 / (17)) + (0.002623617407474447 / (8))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (8) - 2, .025) and -0.07500346083369458 = (14.997078318706812 - 16.12191109500379) / 14.997078318706812 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare 90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 3.86 ms 4.60 ms 1.19 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<MyEventsListerViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.6004375 > 4.012463588578089.
IsChangePoint: Marked as a change because one of 4/8/2023 9:16:19 PM, 4/9/2023 1:58:52 PM, 4/17/2023 8:43:19 PM, 4/20/2023 12:14:19 AM falls between 4/11/2023 5:52:15 AM and 4/20/2023 12:14:19 AM.
IsRegressionStdDev: Marked as regression because -3.1158777352535765 (T) = (0 -4447872.652777777) / Math.Sqrt((110488054186.93439 / (19)) + (66169414811.44008 / (6))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (6) - 2, .025) and -0.10000901366550996 = (4043487.460122107 - 4447872.652777777) / 4043487.460122107 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare 90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 5.06 μs 6.00 μs 1.18 0.05 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<Int32, Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.997328524424197 > 5.3560194773457095.
IsChangePoint: Marked as a change because one of 1/5/2023 2:25:12 AM, 1/15/2023 4:47:52 PM, 3/25/2023 12:18:49 AM, 4/9/2023 11:04:55 PM, 4/17/2023 8:43:19 PM, 4/20/2023 12:14:19 AM falls between 4/11/2023 5:52:15 AM and 4/20/2023 12:14:19 AM.
IsRegressionStdDev: Marked as regression because -6.001295749885055 (T) = (0 -5974.069421220879) / Math.Sqrt((204945.30730221627 / (19)) + (2609.125932355143 / (6))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (6) - 2, .025) and -0.11908661313734537 = (5338.344102314521 - 5974.069421220879) / 5338.344102314521 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare 90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrueComparer<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableSortedSet - Duration of single invocation 317.49 μs 341.52 μs 1.08 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrueComparer<String>.ImmutableSortedSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 341.51778846153843 > 330.92679677884615.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 1/26/2023 1:49:07 PM, 2/9/2023 1:41:04 PM, 3/5/2023 11:54:18 PM, 4/17/2023 8:43:19 PM, 4/20/2023 12:14:19 AM falls between 4/11/2023 5:52:15 AM and 4/20/2023 12:14:19 AM.
IsRegressionStdDev: Marked as regression because -11.111460999057925 (T) = (0 -343758.0745588695) / Math.Sqrt((18186709.074191596 / (19)) + (24974281.792317376 / (6))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (6) - 2, .025) and -0.07890772794152787 = (318616.750678701 - 343758.0745588695) / 318616.750678701 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@EgorBo EgorBo

Labels
arch-arm64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@EgorBo