New lint: transmute(Vec) is undefined behavior

[Shnatsel]

Documentation on std::mem::transmute() specifically calls out that transmuting a Vec of any type is UB.

Code with UB from documentation:

let store = [0, 1, 2, 3];
let mut v_orig = store.iter().collect::<Vec<&i32>>();

// Using transmute: this is Undefined Behavior, and a bad idea.
let v_transmuted = unsafe {
    std::mem::transmute::<Vec<&i32>, Vec<Option<&i32>>>(v_orig)
};

Correct code from documentation (assuming interior types can be transmuted):

let store = [0, 1, 2, 3];
let mut v_orig = store.iter().collect::<Vec<&i32>>();
// Using from_raw_parts
let v_from_raw = unsafe {
    Vec::from_raw_parts(v_orig.as_mut_ptr() as *mut Option<&i32>,
                        v_orig.len(),
                        v_orig.capacity())
};
std::mem::forget(v_orig);

What exactly is wrong with it is not currently documented (I've opened rust-lang/rust#64073 to clarify this) but my understanding is that the problem stems from the following:

Most fundamentally, Vec is and always will be a (pointer, capacity, length) triplet. No more, no less. The order of these fields is completely unspecified, and you should use the appropriate methods to modify these.

I.e. while it's safe to transmute the data on the heap assuming the types are compatible, it's never safe to transmute the (pointer, capacity, length) triplet on the stack.

I'm not 100% convinced that the "correct" code proposed in documentation correctly upholds aliasing invariants. It's possible that std::mem::forget() must be called before the new Vec is constructed, or that it's only safe to do with ManuallyDrop.

cc @RalfJung

[danielhenrymantilla]

It's possible that std::mem::forget() must be called before the new Vec is constructed, or that it's only safe to do with ManuallyDrop.

Yes, the potential issue is that

    Vec::from_raw_parts(v_orig.as_mut_ptr() as *mut Option<&i32>,
                        v_orig.len(),
                        v_orig.capacity())

asserts / requires that v_org.as_mut_ptr() not be aliased (due to Vec::from_raw_parts creating (unique) ownership out of that pointer)

However, the original pointer it comes from, the one from v_orig, is "used" right afterwards in mem::forget. Unless forget becomes a special lang construct, the current stacked borrows model fails with this pattern.

The solution is to "forget" in advance, thanks to ManuallyDrop:

let store = [0, 1, 2, 3];
let v_orig = store.iter().collect::<Vec<&i32>>();
// Using from_raw_parts
let v_from_raw = unsafe {
    // prepare for an auto-forget of the initial vec:
    let v_orig: &mut Vec<_> = &mut *ManuallyDrop::new(v_orig);
    Vec::from_raw_parts(v_orig.as_mut_ptr() as *mut Option<&i32>,
                        v_orig.len(),
                        v_orig.capacity())
    // v_orig is never used again, so no aliasing issue
};

[llogiq]

That's a lot of code. Perhaps it might be a good idea to encapsulate it in a Vec::transmute::<Item>()-method?

[Lokathor]

There is currently no trait for this in core or std, so crates have to make up their own.

[Shnatsel]

@llogiq sounds like a good idea to me. Care to get the ball rolling with an issue against stdlib and/or an RFC?

[Lokathor]

This actually could potentially be a lot more broad and useful, though I don't know how much of this info that clippy really has access to when doing an analysis:

• Transmuting any repr(Rust) type with more than one field is UB.

Related:

• Transmuting any Box<H> to Box<K> is UB when you drop the new Box unless the new type's size and alignment are both exactly the same. This includes anything that transitively contains a Box

[llogiq]

@Shnatsel Sure thing. I'll write up that RFC.

[llogiq]

rust-lang/rfcs#2756

[CAD97]

As a follow-up to @Lokathor: currently, a lint would be always correct to trigger when both type arguments to transmute are repr(Rust), unless they are fully equivalent. Any defined transmutes must involve at least one repr(C) or repr(transparent) type (counting built-in types with defined layouts as repr(C)).

A lint specifically for Vec and/or other smart pointers is a great opportunity for teaching, but until we get more defined layouts, it's UB for any two distinct repr(Rust) types even when they're identical newtypes.

[RalfJung]

a lint would be always correct to trigger when both type arguments to transmute are repr(Rust)

I think that's basically rust-lang/rust#50842.

I'm not 100% convinced that the "correct" code proposed in documentation correctly upholds aliasing invariants. It's possible that std::mem::forget() must be called before the new Vec is constructed, or that it's only safe to do with ManuallyDrop.

@llogiq has since corrected that in the docs.