Possible undefined behavior in out parameter handling

[jsha]

A lot of our functions take out parameters, e.g.:

rustls_result rustls_connection_read(struct rustls_connection *conn,
                                     uint8_t *buf,
                                     size_t count,
                                     size_t *out_n);

The implementation is usually like:

            let out_n: &mut size_t = try_mut_from_ptr!(out_n);
            ...
            *out_n = n_read;

I think this creates undefined behavior if the caller passes a pointer to an uninitialized size_t, since we're then creating a reference to uninitialized memory. It would be better to keep out_n as a raw pointer, do the null check at the top of the function, and then dereference it in an unsafe block at the end of the function.

Another possibility would be to define an OutParam wrapper type, something like:

#[repr(transparent)]
struct OutParam<T> {
  inner: NonNull<MaybeUninit<T>>
}

impl<T> OutParam<T> {
  fn from_ptr(p: *mut T) -> Option<Self> {...}
  fn write(T) { ... }
}

That would allow us to use the type system to ensure we've checked for null at the top. And would also allow us, on the Rust side, to clearly mark out parameters (though they are already marked rather clearly by *mut and occurring at the end of a function signature).

[jsha]

I started work on this in #256. I concluded that defining an OutParam type added complexity without corresponding benefit. In particular, fn write would need to be unsafe, so we'd need an unsafe block at the end of the function anyhow.

[jsha]

Related links:

https://lucumr.pocoo.org/2022/1/30/unsafe-rust/
rust-lang/miri#1240
https://github.com/CAD97/pointer-utils/pull/45/files

In particular those last two made me aware of ptr::slice_from_raw_parts (vs slice::from_raw_parts):

Forms a raw slice from a pointer and a length.

This function is safe, but actually using the return value is unsafe. See the documentation of slice::from_raw_parts for slice safety requirements.