Vec: avoid creating slices to the elements #61114

RalfJung · 2019-05-24T07:22:32Z

Instead of self.deref_mut().as_mut_ptr() to get a raw pointer to the buffer, use self.buf.ptr_mut(). This (a) avoids creating a unique reference to all existing elements without any need, and (b) creates a pointer that can actually be used for the entire buffer, and not just for the part of it covered by self.deref_mut().

I also got worried about RawVec::ptr returning a *mut T from an &self, so I added both a mutable and an immutable version.

Cc @gankro in particular for the assume changes -- I don't know why that is not in Unique, but I moved it up from Vec::deref to RawVec::ptr to avoid having to repeat it everywhere.

Fixes #60847

rust-highfive · 2019-05-24T07:22:43Z

r? @sfackler

(rust_highfive has picked a reviewer for you, use r? to override)

RalfJung · 2019-05-24T08:19:00Z

There's just too many things using RawVec, I am going to give up on that "better mutability tracking". Not sure if y'all would want that anyway.

sfackler · 2019-05-24T14:33:52Z

src/liballoc/raw_vec.rs

@@ -194,7 +195,9 @@ impl<T, A: Alloc> RawVec<T, A> {
    /// Unique::empty() if `cap = 0` or T is zero-sized. In the former case, you must
    /// be careful.
    pub fn ptr(&self) -> *mut T {


Should this just return a NonNull<T>?

Maybe? There might be a reason for why this looks the way it does. Possibly because NonNull is not very ergonomic.

yeah, NonNull is only useful as a storage type, it's useless for doing actual work.

hanna-kruppe · 2019-05-24T15:20:58Z

(b) creates a pointer that can actually be used for the entire buffer, and not just for the part of it covered by self.deref_mut().

This seems like it would also be a problem for anyone outside liballoc who uses as_mut_ptr to e.g.

initialize parts of a vector before calling set_len
disassemble a vector and later reconstruct it with Vec::from_raw_parts
keep a cursor into the Vec as its length changes (similar to the test you added, but moving the pointer around, into portions of the buffer that are added later)

Is that right? And the workaround used here (accessing the internal RawVec) isn't available outside liballoc, right?

So perhaps we should add Vec::as_mut_ptr that returns a pointer covering the vector's entire buffer, not just the 0..len portion? (Plus Vec::as_ptr for symmetry, I guess.)

RalfJung · 2019-05-24T15:25:51Z

Yes that is correct. Vec::as_mut_ptr (via DerefMut) may only be used to access the elements 0..len with len taken at the time of the invocation.

That's what Stacked Borrows says, anyway. ;) This might also be an indication that the model is too strict? The desired interaction with unsized types is not really clear. One custom DST are a thing, I doubt we want Stacked Borrows to run the arbitrary user code to compute the size...

hanna-kruppe · 2019-05-24T15:50:44Z

Assuming that this restriction on the use of <[T]>::{as_ptr, as_mut_ptr} prevails in some form, what do you think about (rephrasing the last part of my previous comment for clarity) adding inherent methods as_ptr, as_mut_ptr to Vec that shadow the slice methods and differ only in that they return a pointer that may be used to access elements 0..capacity?

RalfJung · 2019-05-24T16:00:30Z

That sounds very reasonable!

I didn't do that here because I was not sure how that shadowing would work out.

Gankra · 2019-05-24T22:45:47Z

re: assume: I believe historically we found that llvm gets really overexcited when it finds assumes and wastes waaay too much time trying to profitably use them. So we moved to only inserting them in strategic hotspots, instead of everywhere we could.

Gankra · 2019-05-24T22:50:03Z

Also I agree with @rkruppe. If as_ptr is a semantic footgun for the implementation, it's a footgun for our users too. We should add the shadowing implementation (which works fine, Vec shadows slice methods in a few places already).

RalfJung · 2019-05-25T08:13:56Z

I will have to make them insta-stable though to not regress people doing vec.as_mut_ptr(), right?

RalfJung · 2019-05-25T08:40:22Z

All right, I rewrote this PR to add shadowing methods instead. I also moved the assume there.

RalfJung · 2019-05-25T08:42:08Z

src/liballoc/vec.rs

@@ -1754,7 +1819,6 @@ impl<T> IntoIterator for Vec<T> {
    fn into_iter(mut self) -> IntoIter<T> {
        unsafe {
            let begin = self.as_mut_ptr();
-            assume(!begin.is_null());


as_mut_ptr now contains an assume so I think I can remove this? Unfortunately whoever added this did not leave a comment, because it seemed redundant before already.

sfackler · 2019-05-25T14:36:12Z

r? @gankro

Gankra · 2019-05-25T15:05:01Z

@bors r+

bors · 2019-05-25T15:05:02Z

@gankro: 🔑 Insufficient privileges: Not in reviewers

RalfJung · 2019-05-25T15:28:27Z

@bors r=Gankro

bors · 2019-05-25T15:28:28Z

📌 Commit 428ab7e has been approved by Gankro

Vec: avoid creating slices to the elements Instead of `self.deref_mut().as_mut_ptr()` to get a raw pointer to the buffer, use `self.buf.ptr_mut()`. This (a) avoids creating a unique reference to all existing elements without any need, and (b) creates a pointer that can actually be used for the *entire* buffer, and not just for the part of it covered by `self.deref_mut()`. I also got worried about `RawVec::ptr` returning a `*mut T` from an `&self`, so I added both a mutable and an immutable version. Cc @gankro in particular for the `assume` changes -- I don't know why that is not in `Unique`, but I moved it up from `Vec::deref` to `RawVec::ptr` to avoid having to repeat it everywhere. Fixes rust-lang#60847

@ghost

Rollup of 9 pull requests Successful merges: - #61087 (Tweak `self` arg not as first argument of a method diagnostic) - #61114 (Vec: avoid creating slices to the elements) - #61144 (Suggest borrowing for loop head on move error) - #61149 (Fix spelling in release notes) - #61161 (MaybeUninit doctest: remove unnecessary type ascription) - #61173 (Auto-derive Encode and Decode implementations of DefPathTable) - #61184 (Add additional trace statements to the const propagator) - #61189 (Turn turbo 🐟 🍨 into an error) - #61193 (Add comment to explain why we change the layout for Projection) Failed merges: r? @ghost

Before, they went through `&str` and `&mut str`, which created intermediary references, shrinking provenance to only the initialized parts. `Vec<T>` already has such inherent methods added in rust-lang#61114.

rust-highfive assigned sfackler May 24, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 24, 2019

This comment has been minimized.

# to view

RalfJung force-pushed the vec branch from 64a0e07 to 98afa8f Compare May 24, 2019 07:54

This comment has been minimized.

# to view

RalfJung force-pushed the vec branch 2 times, most recently from 684f4a5 to 487a108 Compare May 24, 2019 08:24

RalfJung mentioned this pull request May 24, 2019

running tests under miri fails thomcc/rust-typed-arena#26

Closed

sfackler reviewed May 24, 2019

View reviewed changes

add test checking that Vec push/pop does not invalidate pointers

9d82826

shadow as_ptr as as_mut_ptr in Vec to avoid going through Deref

428ab7e

RalfJung force-pushed the vec branch from 487a108 to 428ab7e Compare May 25, 2019 08:39

RalfJung commented May 25, 2019

View reviewed changes

rust-highfive assigned Gankra and unassigned sfackler May 25, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 25, 2019

Centril mentioned this pull request May 25, 2019

Rollup of 9 pull requests #61198

Closed

Centril mentioned this pull request May 26, 2019

Rollup of 9 pull requests #61201

Merged

bors merged commit 428ab7e into rust-lang:master May 26, 2019

RalfJung deleted the vec branch May 29, 2019 08:01

RalfJung mentioned this pull request Apr 23, 2020

Fix leak on panic in insert_many. servo/rust-smallvec#213

Merged

cuviper mentioned this pull request Jun 25, 2020

Switch to hashbrown's RawTable internally indexmap-rs/indexmap#131

Merged

Noratrieb mentioned this pull request May 28, 2022

Add as_ptr and as_mut_ptr inherent method to String #97483

Closed

guevara mentioned this pull request Jun 19, 2022

Fast and Simple Rust Interner guevara/read-it-later#8489

Open

guevara mentioned this pull request Sep 1, 2022

Fast and Simple Rust Interner guevara/read-it-later#8769

Open

pnkfelix mentioned this pull request Mar 21, 2023

Remove the assume(!is_null) from Vec::as_ptr #106967

Merged

thomcc mentioned this pull request Jul 19, 2023

Add note that Vec::as_mut_ptr() does not materialize a reference to the internal buffer #113859

Merged

guevara mentioned this pull request Jan 23, 2024

Fast and Simple Rust Interner guevara/read-it-later#10646

Open

Vec: avoid creating slices to the elements #61114

Vec: avoid creating slices to the elements #61114

Uh oh!

Conversation

RalfJung commented May 24, 2019

Uh oh!

rust-highfive commented May 24, 2019

Uh oh!

This comment has been minimized.

This comment has been minimized.

RalfJung commented May 24, 2019

Uh oh!

sfackler May 24, 2019

Choose a reason for hiding this comment

Uh oh!

RalfJung May 24, 2019

Choose a reason for hiding this comment

Uh oh!

Gankra May 24, 2019

Choose a reason for hiding this comment

Uh oh!

hanna-kruppe commented May 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RalfJung commented May 24, 2019

Uh oh!

hanna-kruppe commented May 24, 2019

Uh oh!

RalfJung commented May 24, 2019

Uh oh!

Gankra commented May 24, 2019

Uh oh!

Gankra commented May 24, 2019

Uh oh!

RalfJung commented May 25, 2019

Uh oh!

RalfJung commented May 25, 2019

Uh oh!

RalfJung May 25, 2019

Choose a reason for hiding this comment

Uh oh!

sfackler commented May 25, 2019

Uh oh!

Gankra commented May 25, 2019

Uh oh!

bors commented May 25, 2019

Uh oh!

RalfJung commented May 25, 2019

Uh oh!

bors commented May 25, 2019

Uh oh!

Uh oh!

hanna-kruppe commented May 24, 2019 •

edited

Loading