Connection level memory limits

[zenhack]

As discussed in #194, we should have the ability to specify connection-level memory limits, to protect against DoS. In particular, we want:

1. "soft" limit on the size of unreleased incoming Call messages. Reading from the transport should block if we are above this limit.
2. A "hard" limit on all memory allocated by the transport; if we exceed this we should drop the connection entirely.

We should not reach these limits under normal circumstances; cooperative flow control should prevent them. Notes:

• In principle, (1) could cause deadlocks; the scenario where this occurs is if you have mutually recursive calls bouncing back and forth across a connection until you hit a limit -- so this is somewhat like a stack overflow, except you get a deadlock instead of a panic :/.
• (1) is insufficient to protect from DoS attacks, since it does not account for (potentially large) return messages for which no finish message has come in.

I suggested adding these to rpc.Options, but I think the cleanest way to implement these is as Transports that impose the limits -- I don't think any of the rest of the rpc subsystem needs to change. We could add these to rpc.Options anyway as a convenience, if we wanted.

We should also figure out how to make the API "safe by default" in this regard.

[zenhack]

I'm going to stick this on the 3.0 milestone; making this safe by default might involve adding default limits, which could otherwise be a breaking change, so best to do that before we stabilize.

[lthibault]

Instead of deadlocking when the limit is reached, could we close the connection and log a warning? I'd prefer to have this fail noisily.

[zenhack]

That sounds like just having limit 2? We could also maybe have a timeout for the first one, (or both) after which it drops the connection.

[lthibault]

We could also maybe have a timeout for the first one, (or both) after which it drops the connection.

Yeah, this is the kind of thing I had in mind. Having a hard-vs-soft limit makes sense, but I'd also like the soft limit to become a hard one when it deadlocks. A timeout mechanism seems like a sensible way to do this. 👍