improve cold_path() #133852

x17jiri · 2024-12-04T11:58:04Z

#120370 added a new instrinsic cold_path() and used it to fix likely and unlikely

However, in order to limit scope, the information about cold code paths is only used in 2-target switch instructions. This is sufficient for likely and unlikely, but limits usefulness of cold_path for idiomatic rust. For example, code like this:

if let Some(x) = y { ... }

may generate 3-target switch:

switch y.discriminator:
0 => true branch
1 = > false branch
_ => unreachable

and therefore marking a branch as cold will have no effect.

This PR improves cold_path() to work with arbitrary switch instructions.

Note that for 2-target switches, we can use llvm.expect, but for multiple targets we need to manually emit branch weights. I checked Clang and it also emits weights in this situation. The Clang's weight calculation is more complex that this PR, which I believe is mainly because switch in C/C++ can have multiple cases going to the same target.

rustbot · 2024-12-04T11:58:14Z

r? @estebank

rustbot has assigned @estebank.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

x17jiri · 2024-12-20T14:47:05Z

Just found that this also works:

	let new_buf = Global.allocate(layout).map_err(|_| {
		cold_path();
		Error::new_alloc_failed("Cannot allocate memory.")
	})?;

As long as the closure is inlined, branch weights will be emited.

estebank · 2024-12-31T19:30:24Z

I'm not the best person to review this, sorry

r? compiler

bors · 2025-01-02T03:45:48Z

☔ The latest upstream changes (presumably #130060) made this pull request unmergeable. Please resolve the merge conflicts.

petrochenkov · 2025-01-02T14:12:39Z

r? compiler

lcnr · 2025-01-06T08:32:04Z

maybe r? @nikic 😅

saethlin · 2025-02-10T04:48:49Z

Anything like this should be re-rolled to the codegen group (which I am in), not the compiler overall. The compiler group is very big and will often just toss around review on things a lot. And @x17jiri if you feel lost in the review process don't hesitate to reach out on the Zulip https://rust-lang.zulipchat.com/

r? saethlin

I can get to this in a few days max. If someone else wants to approve it before then, feel free.

saethlin · 2025-02-11T22:17:11Z

compiler/rustc_codegen_llvm/src/builder.rs

+        let cold_weight = unsafe { llvm::LLVMValueAsMetadata(self.cx.const_u32(1)) };
+        let hot_weight = unsafe { llvm::LLVMValueAsMetadata(self.cx.const_u32(2000)) };


Are 1 and 2000 derived from anything in particular? If these are the magic values clang uses or something like that, a comment would be great.

These values are used by llvm.expect for branches with 2 targets. I added a comment.

compiler/rustc_codegen_llvm/src/builder.rs

saethlin · 2025-02-11T22:27:52Z

Just a few nits, and you have a merge conflict.

Then I'd like to see this go through a perf run before we merge; this isn't just making cold_path better, it also makes #[cold] more powerful, which ought to be an improvement to codegen but you better know. So we'll know at least before this lands by doing a perf run.

saethlin · 2025-02-11T22:28:25Z

tests/codegen/intrinsics/cold_path3.rs

These are good tests. Thank you.

saethlin · 2025-02-13T17:38:52Z

@rustbot label: +perf-regression-triaged
Handful of low-significance opt regressions. Expected because this PR adds more LLVM attributes. The perf suite is a poor representation of the effectiveness of this PR because the compiler is built with PGO, which is likely to obscure improvements from branch weight information.

@bors r+

bors · 2025-02-13T17:38:55Z

📌 Commit 7968501 has been approved by saethlin

It is now in the queue for this repository.

improve cold_path() rust-lang#120370 added a new instrinsic `cold_path()` and used it to fix `likely` and `unlikely` However, in order to limit scope, the information about cold code paths is only used in 2-target switch instructions. This is sufficient for `likely` and `unlikely`, but limits usefulness of `cold_path` for idiomatic rust. For example, code like this: ``` if let Some(x) = y { ... } ``` may generate 3-target switch: ``` switch y.discriminator: 0 => true branch 1 = > false branch _ => unreachable ``` and therefore marking a branch as cold will have no effect. This PR improves `cold_path()` to work with arbitrary switch instructions. Note that for 2-target switches, we can use `llvm.expect`, but for multiple targets we need to manually emit branch weights. I checked Clang and it also emits weights in this situation. The Clang's weight calculation is more complex that this PR, which I believe is mainly because `switch` in `C/C++` can have multiple cases going to the same target.

bors · 2025-02-13T23:59:22Z

⌛ Testing commit 7968501 with merge 7c77e1a...

bors · 2025-02-14T00:36:58Z

💔 Test failed - checks-actions

x17jiri · 2025-02-14T11:18:40Z

It seems the test is failing because the metadata have different numbers on the apple-darwin arch. I think I can fix this using regex. Is there a way for me to run the test on another arch? (Linux specifically)

saethlin · 2025-02-14T17:45:30Z

Nope. But if you push a fix that uses a regex and works on Linux, I'll adjust the PR description to run try-job on apple and a few other platforms: https://rustc-dev-guide.rust-lang.org/tests/ci.html#try-builds

x17jiri · 2025-02-17T07:07:37Z

@saethlin It should be fixed

saethlin · 2025-02-17T13:40:53Z

@bors try

improve cold_path() rust-lang#120370 added a new instrinsic `cold_path()` and used it to fix `likely` and `unlikely` However, in order to limit scope, the information about cold code paths is only used in 2-target switch instructions. This is sufficient for `likely` and `unlikely`, but limits usefulness of `cold_path` for idiomatic rust. For example, code like this: ``` if let Some(x) = y { ... } ``` may generate 3-target switch: ``` switch y.discriminator: 0 => true branch 1 = > false branch _ => unreachable ``` and therefore marking a branch as cold will have no effect. This PR improves `cold_path()` to work with arbitrary switch instructions. Note that for 2-target switches, we can use `llvm.expect`, but for multiple targets we need to manually emit branch weights. I checked Clang and it also emits weights in this situation. The Clang's weight calculation is more complex that this PR, which I believe is mainly because `switch` in `C/C++` can have multiple cases going to the same target. try-job: aarch64-apple try-job: test-various

bors · 2025-02-17T13:42:13Z

⌛ Trying commit 7bb5f4d with merge e1f68a1...

bors · 2025-02-17T15:09:39Z

☀️ Try build successful - checks-actions
Build commit: e1f68a1 (e1f68a1662ea16c66f5e4b710c83b29008886fce)

saethlin · 2025-02-17T19:32:49Z

@bors r+

bors · 2025-02-17T19:32:51Z

📌 Commit 7bb5f4d has been approved by saethlin

It is now in the queue for this repository.

bors · 2025-02-18T07:49:14Z

⌛ Testing commit 7bb5f4d with merge 3b022d8...

bors · 2025-02-18T10:59:22Z

☀️ Test successful - checks-actions
Approved by: saethlin
Pushing 3b022d8 to master...

rust-timer · 2025-02-18T12:47:20Z

Finished benchmarking commit (3b022d8): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

If the regression was expected or you think it can be justified,
please write a comment with sufficient written justification, and add
@rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
If you think that you know of a way to resolve the regression, try to create
a new PR with a fix for the regression.
If you do not understand the regression or you think that it is just noise,
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	0.4%	[0.1%, 0.9%]	7
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.7%	[-0.7%, -0.7%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[-0.7%, 0.9%]	8

Max RSS (memory usage)

Results (primary -1.9%, secondary 0.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.9%	[3.9%, 3.9%]	1
Improvements ✅ (primary)	-1.9%	[-2.7%, -0.9%]	7
Improvements ✅ (secondary)	-3.6%	[-3.6%, -3.6%]	1
All ❌✅ (primary)	-1.9%	[-2.7%, -0.9%]	7

Cycles

Results (primary 1.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.0%	[1.0%, 1.0%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.0%	[1.0%, 1.0%]	1

Binary size

Results (primary 0.0%, secondary 0.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.0%, 0.1%]	8
Regressions ❌ (secondary)	0.1%	[0.1%, 0.1%]	38
Improvements ✅ (primary)	-0.1%	[-0.1%, -0.1%]	3
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[-0.1%, 0.1%]	11

Bootstrap: 774.633s -> 775.584s (0.12%)
Artifact size: 362.36 MiB -> 360.27 MiB (-0.58%)

rustbot assigned estebank Dec 4, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 4, 2024

x17jiri mentioned this pull request Dec 21, 2024

Export cold_path() in std::hint rust-lang/libs-team#510

Closed

ojeda mentioned this pull request Dec 22, 2024

Rust wanted features Rust-for-Linux/linux#354

Open

46 tasks

rustbot assigned petrochenkov and unassigned estebank Dec 31, 2024

rustbot assigned lcnr and unassigned petrochenkov Jan 2, 2025

rustbot assigned nikic and unassigned lcnr Jan 6, 2025

x17jiri mentioned this pull request Jan 15, 2025

Reexport likely/unlikely in std::hint #133695

Merged

safinaskar mentioned this pull request Feb 10, 2025

#[cold] on match arms #120193

Open

rustbot assigned saethlin and unassigned nikic Feb 10, 2025

safinaskar mentioned this pull request Feb 10, 2025

parallel: rustc_data_structures: sync: implement RwLock as enum { Sync(parking_lot::RwLock), NoSync(RefCell) } #136772

Closed

saethlin reviewed Feb 11, 2025

View reviewed changes

compiler/rustc_codegen_llvm/src/builder.rs Outdated Show resolved Hide resolved

saethlin reviewed Feb 11, 2025

View reviewed changes

compiler/rustc_codegen_llvm/src/builder.rs Outdated Show resolved Hide resolved

saethlin reviewed Feb 11, 2025

View reviewed changes

tests/codegen/intrinsics/cold_path3.rs Outdated

Copy link

Member

saethlin Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are good tests. Thank you.

x17jiri reacted with thumbs up emoji

saethlin added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 13, 2025

rustbot added the perf-regression-triaged The performance regression has been triaged. label Feb 13, 2025

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Feb 13, 2025

This comment has been minimized.

# to view

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Feb 14, 2025

improve cold_path()

7bb5f4d

x17jiri force-pushed the cold_path branch from 7968501 to 7bb5f4d Compare February 17, 2025 05:40

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 17, 2025

bors added the merged-by-bors This PR was explicitly merged by bors. label Feb 18, 2025

bors merged commit 3b022d8 into rust-lang:master Feb 18, 2025
7 checks passed

rustbot added this to the 1.87.0 milestone Feb 18, 2025

		let cold_weight = unsafe { llvm::LLVMValueAsMetadata(self.cx.const_u32(1)) };
		let hot_weight = unsafe { llvm::LLVMValueAsMetadata(self.cx.const_u32(2000)) };

improve cold_path() #133852

improve cold_path() #133852

Uh oh!

Conversation

x17jiri commented Dec 4, 2024 • edited by saethlin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Dec 4, 2024

Uh oh!

x17jiri commented Dec 20, 2024

Uh oh!

estebank commented Dec 31, 2024

Uh oh!

bors commented Jan 2, 2025

Uh oh!

petrochenkov commented Jan 2, 2025

Uh oh!

lcnr commented Jan 6, 2025

Uh oh!

saethlin commented Feb 10, 2025

Uh oh!

saethlin Feb 11, 2025

Choose a reason for hiding this comment

Uh oh!

x17jiri Feb 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

saethlin commented Feb 11, 2025

Uh oh!

saethlin Feb 11, 2025

Choose a reason for hiding this comment

Uh oh!

saethlin commented Feb 13, 2025

Uh oh!

bors commented Feb 13, 2025

Uh oh!

bors commented Feb 13, 2025

Uh oh!

This comment has been minimized.

bors commented Feb 14, 2025

Uh oh!

x17jiri commented Feb 14, 2025

Uh oh!

saethlin commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

x17jiri commented Feb 17, 2025

Uh oh!

saethlin commented Feb 17, 2025

Uh oh!

bors commented Feb 17, 2025

Uh oh!

bors commented Feb 17, 2025

Uh oh!

saethlin commented Feb 17, 2025

Uh oh!

bors commented Feb 17, 2025

Uh oh!

bors commented Feb 18, 2025

Uh oh!

bors commented Feb 18, 2025

Uh oh!

Uh oh!

rust-timer commented Feb 18, 2025

Overall result: ❌✅ regressions and improvements - please read the text below

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

Uh oh!

x17jiri commented Dec 4, 2024 •

edited by saethlin

Loading

saethlin commented Feb 14, 2025 •

edited

Loading