Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Perf] Windows/arm64: 5 Regressions on 6/8/2023 3:52:42 PM #18864

Open
performanceautofiler bot opened this issue Jun 15, 2023 · 1 comment
Open

[Perf] Windows/arm64: 5 Regressions on 6/8/2023 3:52:42 PM #18864

performanceautofiler bot opened this issue Jun 15, 2023 · 1 comment
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro os-windows perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jun 15, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS Windows 10.0.25094
Queue AmpereWindows
Baseline dd715a0747ebeb85a234c8abec7e01bb6652c8ce
Compare 9bcf52342b5a6573ecd2c81fbfd541df0ebc42f4
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.DictionarySequentialKeys

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryGetValue_17_Int_Int - Duration of single invocation 5.88 ns 7.40 ns 1.26 0.72 False
TryGetValue_3k_Int_Int - Duration of single invocation 5.61 ns 7.64 ns 1.36 0.68 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.DictionarySequentialKeys*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_17_Int_Int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.395976936880073 > 5.7943076452128945.
IsChangePoint: Marked as a change because one of 3/31/2023 7:02:21 PM, 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -7.695170562714825 (T) = (0 -8.018789895969261) / Math.Sqrt((0.18774556681341403 / (12)) + (1.3012597196780862 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.484735265675485 = (5.40082133249599 - 8.018789895969261) / 5.40082133249599 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_3k_Int_Int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.641963550983901 > 5.879509724413551.
IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -13.113548564576806 (T) = (0 -8.743435290828566) / Math.Sqrt((0.17092856038841 / (12)) + (0.6791435685309158 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.6305911085842962 = (5.362126191415179 - 8.743435290828566) / 5.362126191415179 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.25094
Queue AmpereWindows
Baseline dd715a0747ebeb85a234c8abec7e01bb6652c8ce
Compare 9bcf52342b5a6573ecd2c81fbfd541df0ebc42f4
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParseDateTimeOffset - Duration of single invocation 5.97 ns 7.64 ns 1.28 0.83 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8ParserTests.TryParseDateTimeOffset(value: 12/30/2017 3:45:22 AM -08:00)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.640911329696776 > 6.302415903491371.
IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -7.262938294741985 (T) = (0 -9.710078430474258) / Math.Sqrt((0.9973057405368995 / (12)) + (2.751057774974143 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.6836997855265233 = (5.7671079570980295 - 9.710078430474258) / 5.7671079570980295 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.25094
Queue AmpereWindows
Baseline dd715a0747ebeb85a234c8abec7e01bb6652c8ce
Compare 9bcf52342b5a6573ecd2c81fbfd541df0ebc42f4
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Replace_String - Duration of single invocation 73.13 ns 103.03 ns 1.41 0.58 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 103.02587738588925 > 79.04393196166279.
IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -6.95275929703907 (T) = (0 -93.03306473253484) / Math.Sqrt((15.875856829350282 / (12)) + (64.28241464298715 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.23016574351153538 = (75.62644726796724 - 93.03306473253484) / 75.62644726796724 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.25094
Queue AmpereWindows
Baseline dd715a0747ebeb85a234c8abec7e01bb6652c8ce
Compare 9bcf52342b5a6573ecd2c81fbfd541df0ebc42f4
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Get

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetUInt16 - Duration of single invocation 1.28 μs 1.42 μs 1.11 0.67 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Get.GetUInt16

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4243274439721787 > 1.1817496321292034.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/26/2023 9:55:14 AM, 6/8/2023 5:31:08 AM, 6/14/2023 3:35:53 PM falls between 6/5/2023 10:43:22 AM and 6/14/2023 3:35:53 PM.
IsRegressionStdDev: Marked as regression because -7.208603559815226 (T) = (0 -1474.2133425890993) / Math.Sqrt((6162.145617042773 / (12)) + (15447.056533549852 / (13))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (13) - 2, .025) and -0.25268554569773194 = (1176.8423030442 - 1474.2133425890993) / 1176.8423030442 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@AndyAyersMS
Copy link
Member

Likely dotnet/runtime#87115

@EgorBo FYI

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro os-windows perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AndyAyersMS