Current __gc logic for userdata is unsound

[jonas-schievink]

The manual states:

Moreover, if the finalizer marks a finalizing object for finalization again, its finalizer will be called again in the next cycle where the object is unreachable.

This means that a destructor assigned to the __gc metamethod might be run multiple times on the same userdata. Inside fn destructor, rlua currently just does this:

        let obj = &mut *(ffi::lua_touserdata(state, 1) as *mut T);
        mem::replace(obj, mem::uninitialized());
        0

If one figures out a way to resurrect the userdata object from within drop, this leads to a double free. However, I am not sure if this is possible, since all userdata must outlive 'static.

If this is by design, it would be helpful to document that. In general, rluas internals are barely documented which makes it hard to understand and debug stuff like #18.

EDIT: See comment below for use-after-free demonstration

[jonas-schievink]

There's no need to use drop. In fact, that's impossible because the userdata can't be accessed by-value inside drop. I can just use an external object holding the only reference to the userdata to resurrect it:

extern crate rlua;

use rlua::*;

struct Userdata {
    id: u8,
}

impl Drop for Userdata {
    fn drop(&mut self) {
        println!("dropping {}", self.id);
    }
}

impl LuaUserDataType for Userdata {
    fn add_methods(methods: &mut LuaUserDataMethods<Self>) {
        methods.add_method("access", |lua, this, args| {
            println!("accessing userdata {}", this.id);
            lua.pack(())
        });
    }
}

fn main() {
    let lua = Lua::new();
    {
        let globals = lua.globals();
        globals.set("userdata", Userdata { id: 123 }).unwrap();
    }

    lua.eval::<()>(r#"
        local tbl = setmetatable({
            userdata = userdata
        }, { __gc = function(self)
            -- resurrect userdata
            hatch = self.userdata
        end })

        print("userdata = ", userdata)
        print("hatch = ", hatch)
        print "collecting..."
        tbl = nil
        userdata = nil  -- make table and userdata collectable
        collectgarbage("collect")
        print("userdata = ", userdata)
        print("hatch = ", hatch)
        hatch:access()
    "#, None).unwrap();
}

This demonstrates not a double-drop, but a use-after-free in hatch:access():

userdata = 	userdata: 0x7f68e0c306a8
hatch = 	nil
collecting...
dropping 123
userdata = 	nil
hatch = 	userdata: 0x7f68e0c306a8
accessing userdata 32

[kyren]

So, I apparently had a bad understanding of how __gc metamethods worked from lua circa 5.1, because I was unaware that somewhere between 5.1 and 5.3, tables got the ability to have __gc methods that could be assigned by scripts.

Apparently, ANY userdata may be resurrected, and may have its __gc methods called twice, correct? I now need to.. I guess store all userdata in a form where it can be nulled, and panic on access if it is nulled, and allow it to be nulled more than once safely. Something like, I suppose storing Option, and simply setting it to None on gc instead of dropping it, then I can check to make sure it is not None before returning a pointer.

I can do that, but what it means is it may be.. very hard or impossible to prevent a lua script from forcing a panic. I suppose practically that is not a HUGE deal, since preventing a lua script from causing a DOS attack, even with execution limits, is probably already extremely extremely difficult.

[kyren]

I believe this may be fixed by 36134e6. I don't necessarily like the fact that some userdata now may have multiple levels of Option, and may be able to be collapsed somewhat.

[kyren]

Actually no, there's something I didn't think of. Since Lua 5.3 allows __gc methods on lua tables, that means that any function that may trigger a gc metamethod may longjmp, which makes my entire longjmp story unsound. I was relying on only methods marked as 'e' in the reference manual causing longjmp, but this means that any method marked 'm' may ALSO cause longjmp, which is more or less everything. It's noted in the reference manual that 'm' methods may trigger a __gc metamethod, but I was relying on only userdata having __gc methods, and not being defined in scripts.

This may mean that it is technically impossible to write sound lua bindings.

Okay, so probably just extremely hard not impossible. I think the sanest way to do it would be to, as your first step, write an entire C API that guarantees never to longjmp past the caller.

[kyren]

If this is by design, it would be helpful to document that. In general, rluas internals are barely documented which makes it hard to understand and debug stuff like #18.

I completely agree that it's not well documented. It's not well documented because it has until this point been in dramatic flux. When we finally (FINALLY) stop finding fundamental flaws in my view of the universe, I may rewrite it to be consistent and actually document it.

This is not the first time that I have discovered that a lua bindings system I work on is fundamentally unsound, and it probably won't be the last. Does ANYBODY get this right, if it's even possible to GET right? I'm legitimately considering writing my own 5.3 compatible interpreter at this point, it would be faster than wrapping everything in the universe to make sure it doesn't longjmp.

[jonas-schievink]

I did briefly consider a more low-level wrapper around Lua's C API, which would handle this and abstract away some Lua version differences, too. But I don't yet know how such a library would have to look to be useful, reasonably fast but still safe.

I'm legitimately considering writing my own 5.3 compatible interpreter at this point

I was interested in this for other reasons (and found a way to implement a pretty slow GC in safe Rust), but it is unlikely that such an implementation wouldn't have some bugs, too. Maybe no safety issues, but bugs in the language itself, which isn't really pleasant to debug, either.

This library is at least the third attempt at safe Lua bindings for Rust, at some point it has to work, even if just because of the infinite monkey theorem ;)

[kyren]

Sometimes I think if I do end up writing something that works, it is probably only due to the infinite monkey theorem.

[jonas-schievink]

A reasonably easy way to "fix" the remaining issue would be to replace setmetatable with a wrapper that puts __gc behind a pcall. Should the pcall fail, we can either just abort the program (not good in the presence of user-written code) or call a Rust handler that can log the error, set a flag, etc.

[kyren]

That's not a bad idea!

[kyren]

The documentation for setmetatable says:

Sets the metatable for the given table. (To change the metatable of other types from Lua code, you must use the debug library (§6.10).) If metatable is nil, removes the metatable of the given table. If the original metatable has a __metatable field, raises an error.

But, if you go look at the documentation for the debug library, it says, for debug.setmetatable:

Sets the metatable for the given value to the given table (which can be nil). Returns value.

So, I don't know if debug.setmetatable can be used to set the metatable on something other than a table, but if it can, that is also a soundness hole.

[kyren]

I could wrap both setmetatable and debug.setmetatable, I could only wrap setmetatable, or I could wrap setmetatable and not load the debug library. I feel like other things in the debug library could be used to cause unsoundness as well.. somehow. I mean, you can do things like set hooks with things that cause 'error' or replace arbitrary upvalues. I know it's drastic but I'm leaning towards removing the debug library somehow.