Rework SLO metrics

[joe-elliott]

What this PR does:
This PR reworks the way our SLO metrics are recorded with these goals in mind;

• Push the calculations out to be as close the http.Server as possible
• Clearly metric both total queries and SLO queries in one spot
• Cleanup and remove unused metrics

This PR accomplishes the above with the following sharp corners:

• In order to pass the throughput up to the code that calculates the SLOs a *float was added to the context. This works, but is brittle and difficult to detect if it breaks.
• Specialized code needed to be added to the streaming endpoint

TODO

• correct changelog
• add tests

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[zalegrala]

This looks reasonable to me. Just a question about tracking the other operations. Have you had a chance to test this out?

[mdisibio]

LGTM, accuracy of the metrics for SLO calculation is important and I think this is a good change. I think this is Tempo's first case of context smuggling so it gets a pass. 😄

[zalegrala]

Nice work Joe. Thanks for giving this area some ❤️ .