Improve consumer metric cleanup when a channel goes down

[SimonUnge]

Proposed Changes

This update improves metric cleanup of consumer data when a channel goes down by:

• No longer emitting consumer_deleted for each consumer of a channel
• bulk/pattern deletion of metric data in ets tables.

See #9320 for background.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality) (not really)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask on the mailing list.
We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc.

[michaelklishin]

With this PR, when I kill an app process that has 10K consumers on one channel, I observe a short spike of CPU usage from ≈ 0.3% to ≈ 5% which then instantly goes back to ≈ 0.3%.

[michaelklishin]

Ah, the original test was with over 100K consumers on a channel. I re-tested with 200K consumers and CPU usage drops to 14% within a few seconds, and to virtually zero in the next 40s-60s, so this definitely provides an improvement over the behavior described in #9320.

[michaelklishin]

This improves sample cleanup for a few channels with an extremely large number of consumers but regresses when there's a large number of channels with one consumer, which is more common.

Passing consumer info as context when a channel is closed would help avoid the scan introduced in this PR entirely.

[michaelklishin]

Or, alternatively, the two approaches can be combined:

• Do a scan when the number of consumers is > 1000 (or another constant)
• Include consumer details when there are just a few

[SimonUnge]

To clarify, its ets:match_delete that is the culprit then, and should be avoided if there are few consumers in the channel. Do we see a need to also revert the removal of emit_consumer_delete in rabbit_amqqueue_process, i.e add back the event notification?