Document #[repr(u8)] enum to enable LLVM optimizations

[geofft]

See rust-lang/rust#27303 and rust-lang/libc#48 for discussion of the
optimization, or just try compiling unsafe {free(malloc(16))} with various
Rust definitions of void * in the FFI binding.

[Gankra]

This is a bit unclear to me as written. I would maybe zoom out from LLVM a bit and focus on the ABI implications of C-like enums: it will be treated as an integer, and not a struct-containing-an-integer.

CC @mystor

[geofft]

Oh, hm, is a #[repr(u8)] enum Foo {Bar(())} represented as a struct, and thus passed on the stack in ABIs where structs are unconditionally on the stack? Then yeah, that's worth documenting too! Let me test that and then rework this paragraph, thanks.

[mystor]

It might also be good to note that controlling the #[repr] of the enum doesn't change the set of legal values, and that unlike C/C++ enums, it is undefined behaviour to have a #[repr(C)] enum Foo { A = 0, B = 1 } with the value 10.

[geofft]

OK, I've rewritten it to take the above comments into account.

This has drifted a fair bit from my original subject; the specific optimization LLVM does with malloc and free (and particular, that LLVM considers void * to be a * u8) is more than I can fit in a passing comment. However, the comments on std::os::raw::c_void and libc::c_void, plus the paragraph here about the enum needing to be C-like, are probably clear enough.

[strega-nil]

If a C-like enum is missing a repr(u*) or repr(C), it is treated like a one-member struct for ABI compatibility purposes.

This seems like a silly guarantee to make, since they should not be treated as a one-member struct for ABI purposes.

However, a non-C-like enum will always be treated like a struct (even if the only data is () or PhantomData). Adding repr simply forces the discriminant to be represented in memory by the specified integer size. This does have the effect of inhibiting certain optimizations like the null-pointer optimization.

Again, why are we guaranteeing this? The null pointer optimization inhibition, sure, but otherwise this seems like it should mostly be deleted.

[steveklabnik]

I have no idea what's right here...

[mystor]

I generally agree with @ubsan That any properties of repr(rust) enums, with the potential exception of Option<&T> and Option<Box<T>>, should not be documented. Any internal implementation details of them should remain internal.

[Gankra]

Agreed, the only property of repr(rust) that's guaranteed is that OptionLike<&T> has the same abi as &T and *const T.

[steveklabnik]

Great. @geofft care to make these changes?

[geofft]

Oh yeah, good points about not documenting the exact properties of repr(rust). Thanks!

I've updated the pull request. This doesn't seem like the right area to discuss OptionLike, so I just removed all discussion of repr(rust) representation, but I can add a bit about that if that's helpful.

[steveklabnik]

Let's . Thanks!