MQTT with OAuth 2: investigate if credential expiration can be checked before or after a keepalive frame is sent

[michaelklishin]

See #11854 for background.

Currently when a JWT token is used for authN/authZ and the token expires, the plugin will refuse all "new" operations:

• New connections that use this token
• New publishes on connections that use this token
• New subscriptions will be refused

and so on. However, existing consumers that never publish or send anything other than acks and/or heartbeats, will continue being connected.

Solutions Considered and Rejected

Two potential solutions were considered and rejected during the 3.13 MQTT plugin redesign:

• Check for token expiration every time we deliver a batch of messages. This has substantial negative effect on throughput in the "happy case"
• Add a timer to check for token expiration. This adds to per-connection memory consumption and worsens CPU context switching in environments with 100s of thousands or million connections, even if they never consume and only publish (that is, the problem outlined above does not apply)

A Third Alternative

MQTT connections already use a timer for keepalives. It has nothing to do with credentials or authentication but it could be abused
to evaluate a function periodically, and close the connection with an expired token with a DISCONNECT frame.

When non-JWT authorization mechanisms are used,
the keepalive callback would not be used (or would do nothing).

At least this is a specific option that can be investigated.

[michaelklishin]

This was rejected in favor of a solution similar to #10299:

• Credential expiration is entirely orthogonal to keepalives/heartbeats
• Keepalives and stats timers can be disabled
• Take expiry timestamp into account in stream connections #10299 avoids waking up potentially millions of Erlang processes for millions of connections every, say, minute

[ansd]

I prefer taking the same approach as done for Streams in #10299 because:

1. We can easily re-use that same mechanism across multiple protocols
2. Checking for expired JWT tokens is orthogonal (i.e. a different concern to) to KeepAlive and to the stats timer: There are MQTT environments where both stats timers and KeepAlives are disabled. In fact our docs recommends to switch off management metrics and therefore the stats timer. In environments, where both are switched off, we still want to expire JWT tokens.
3. Scheduling an Erlang timer is cheap, even if there are millions of connections. (What's less cheap is running millions of processes every minute or so, but that's not the case here since it's a one-off timer for now.)