Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Perf] Linux/arm64: 9 Regressions on 5/3/2023 4:46:09 PM #17669

Closed
performanceautofiler bot opened this issue May 11, 2023 · 2 comments
Closed

[Perf] Linux/arm64: 9 Regressions on 5/3/2023 4:46:09 PM #17669

performanceautofiler bot opened this issue May 11, 2023 · 2 comments
Assignees
@AndyAyersMS
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented May 11, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline c5303ae33513e1a2eddceb2986639efd7505115e
Compare f510c5ab2d8a9b94b84f63ff9d2d8318243e390f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOf - Duration of single invocation 0.00 ns 5.61 ns ∞ 0.35 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.LastIndexOf

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.607638176836272 > 3.638212119947284.
IsChangePoint: Marked as a change because one of 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -8.342327248424626 (T) = (0 -5.662088339810326) / Math.Sqrt((0.9629267264086404 / (12)) + (0.32225623083190635 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.8310437359132798 = (3.092273673619388 - 5.662088339810326) / 3.092273673619388 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline c5303ae33513e1a2eddceb2986639efd7505115e
Compare f510c5ab2d8a9b94b84f63ff9d2d8318243e390f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfChar - Duration of single invocation 8.57 ns 9.96 ns 1.16 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.IndexOfChar

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.962848700818892 > 9.085712045960994.
IsChangePoint: Marked as a change because one of 2/17/2023 6:56:12 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -34.389456880632274 (T) = (0 -9.688005491798132) / Math.Sqrt((0.002513082466834866 / (12)) + (0.01467211266888041 / (21))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (21) - 2, .025) and -0.11978158036066272 = (8.651692135066009 - 9.688005491798132) / 8.651692135066009 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline c5303ae33513e1a2eddceb2986639efd7505115e
Compare f510c5ab2d8a9b94b84f63ff9d2d8318243e390f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfValue - Duration of single invocation 13.95 ns 15.63 ns 1.12 0.03 False
IndexOfValue - Duration of single invocation 2.25 ns 3.74 ns 1.66 0.48 False
IndexOfAnyFourValues - Duration of single invocation 8.31 ns 9.67 ns 1.16 0.10 False
BinarySearch - Duration of single invocation 12.20 ns 13.95 ns 1.14 0.05 False
BinarySearch - Duration of single invocation 6.18 ns 7.83 ns 1.27 0.07 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.IndexOfValue(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.626589995119188 > 14.844753391152661.
IsChangePoint: Marked as a change because one of 2/7/2023 2:48:42 AM, 3/2/2023 11:02:42 PM, 4/5/2023 12:23:38 PM, 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -57.17064366716211 (T) = (0 -15.787050063318283) / Math.Sqrt((0.006248804590791376 / (12)) + (0.007495819091510038 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.11892979456522196 = (14.109062194963352 - 15.787050063318283) / 14.109062194963352 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Byte>.IndexOfValue(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.73517810265894 > 2.492323805767218.
IsChangePoint: Marked as a change because one of 2/9/2023 1:41:04 PM, 4/12/2023 8:39:35 PM, 5/3/2023 1:18:01 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -6.637974987607555 (T) = (0 -3.188651461034731) / Math.Sqrt((0.11193091543841652 / (11)) + (0.06045034050710598 / (23))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (23) - 2, .025) and -0.3081440625433966 = (2.4375384579853567 - 3.188651461034731) / 2.4375384579853567 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.672091755337897 > 8.675655571897074.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -37.47399176339142 (T) = (0 -9.560802687016526) / Math.Sqrt((0.008371299992431592 / (12)) + (0.013157403486968058 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.16426058041386535 = (8.21190964278624 - 9.560802687016526) / 8.21190964278624 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Byte>.BinarySearch(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.945162040919739 > 12.815905925913881.
IsChangePoint: Marked as a change because one of 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -15.618899884094315 (T) = (0 -14.06800025311069) / Math.Sqrt((0.03949751474632863 / (12)) + (0.2212336555032102 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.14714166215007007 = (12.263524826343811 - 14.06800025311069) / 12.263524826343811 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Byte>.BinarySearch(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.832914457976153 > 6.723584356819076.
IsChangePoint: Marked as a change because one of 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -27.107570105201628 (T) = (0 -7.83194690108983) / Math.Sqrt((0.011205000649243848 / (12)) + (0.04093754651069887 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.22394319799493728 = (6.398946384047984 - 7.83194690108983) / 6.398946384047984 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline c5303ae33513e1a2eddceb2986639efd7505115e
Compare f510c5ab2d8a9b94b84f63ff9d2d8318243e390f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Base64Tests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Base64DecodeDestinationTooSmall - Duration of single invocation 332.15 ns 353.93 ns 1.07 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Base64Tests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Base64Tests.Base64DecodeDestinationTooSmall(NumberOfBytes: 1000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 353.93367808338246 > 348.7689482304825.
IsChangePoint: Marked as a change because one of 1/21/2023 9:40:41 PM, 1/27/2023 7:12:54 PM, 2/7/2023 2:48:42 AM, 4/6/2023 11:23:56 PM, 4/17/2023 5:51:51 AM, 4/28/2023 1:33:49 PM, 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -56.36319479183408 (T) = (0 -351.289541485181) / Math.Sqrt((0.00772448339261608 / (12)) + (2.5254696073645126 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.057656602817105655 = (332.1395059128917 - 351.289541485181) / 332.1395059128917 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline c5303ae33513e1a2eddceb2986639efd7505115e
Compare f510c5ab2d8a9b94b84f63ff9d2d8318243e390f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NextSingle_unseeded - Duration of single invocation 4.55 ns 5.72 ns 1.26 0.33 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Random*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Random.NextSingle_unseeded

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.722543108387424 > 4.7377454273414665.
IsChangePoint: Marked as a change because one of 5/3/2023 7:05:50 AM, 5/10/2023 10:36:48 PM falls between 5/2/2023 4:25:00 AM and 5/10/2023 10:36:48 PM.
IsRegressionStdDev: Marked as regression because -5.228892499272883 (T) = (0 -6.261226691380106) / Math.Sqrt((0.01254655476208119 / (12)) + (2.679250942265444 / (22))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (22) - 2, .025) and -0.41379924086406084 = (4.428653312583108 - 6.261226691380106) / 4.428653312583108 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added ampere arch-arm64 branch-refs/heads/main kind-micro os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr untriaged labels May 11, 2023
@EgorBo
Copy link
Member

EgorBo commented May 16, 2023

Either dotnet/runtime#85559 or dotnet/runtime#85654

@EgorBo
Copy link
Member

EgorBo commented May 16, 2023

dotnet/runtime#85988

@EgorBo EgorBo closed this as completed May 16, 2023
@EgorBo EgorBo removed the untriaged label May 16, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@AndyAyersMS AndyAyersMS

Labels
ampere arch-arm64 branch-refs/heads/main kind-micro os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EgorBo @AndyAyersMS