metrics: Refactor metrics label filter logic

[lambdanis]

Tetragon has a functionality to disable some of the high-cardinality metrics
labels. So far it worked like this:

1. Configure option.Config.MetricsLabelFilter as a set (map[string]interface{})
   of enabled labels.
2. metrics.granularLabelFilter wraps enabled labels and configurable labels in
   one global variable.
3. Define a metric using a "granular metric" wrapper, which uses
   granularLabelFilter.
4. When the metric is registered, the inner Prometheus metric is registered
   with the enabled labels only.
5. Then we update the metric like a regular Prometheus metric, using
   WithLabelValues.

Refactor this logic so that now it works like this:

1. Configure MetricsLabelFilter similarly like before, but as map[string]bool
2. metrics.ProcessLabels struct implements metrics.FilteredLabels interface and
   contains label values of configurable labels. Disabled labels are set to ""
   when the struct is instantiated.
3. Define a metric using a "granular metric" wrapper, which now uses Go generics
   to specify configurable labels (FilteredMetrics type).
4. The inner Prometheus metric is now registered with all labels.
5. Then we update the metric using a slightly different implementation of
   WithLabelValues, which takes a generic FilteredLabels struct and any
   additional label values as strings.

This approach provides better types, so that it's not as easy for developers to
make a mistake and pass incorrect labels when updating a metric. Additionally,
it makes it easy to define a different set of configurable labels (another
FilteredMetrics type) and to implement dynamic metrics labels configuration (as
it doesn't require re-registering metrics when the filter changes).

[netlify]

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	3b11b2f
🔍 Latest deploy log	https://app.netlify.com/sites/tetragon/deploys/661d721b7ba3b80007444c61
😎 Deploy Preview	https://deploy-preview-2321--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[lambdanis]

I'm still doing some testing so I converted it to draft, but please review 🙇‍♀️

[kkourt]

LGTM, thanks.

[kkourt]

1. Configure MetricsLabelFilter similarly like before, but as map[string]bool
2. metrics.ProcessLabels struct implements metrics.FilteredLabels interface and
   contains label values of configurable labels. Disabled labels are set to ""
   when the struct is instantiated.
3. Define a metric using a "granular metric" wrapper, which now uses Go generics
   to specify configurable labels (FilteredMetrics type).
4. The inner Prometheus metric is now registered with all labels.
5. Then we update the metric using a slightly different implementation of
   WithLabelValues, which takes a generic FileredLabels struct and any
   additional label values as strings.

AFAIU, the disabled labels are set to "". The thing I'm not sure I understand is where does the filtering happens? Is it .WithLabelValues()?

[lambdanis]

When calling WithLabelValues() you need to pass a ProcessLabels struct. If it's created with NewProcessLabels then disabled labels are set to "", what from Prometheus perspective is equivalent to not having these labels at all. So filtering relies on always creating ProcessLabels with NewProcessLabels. I'll add a comment about it, as it's not obvious indeed.

[lambdanis]

I only added one commit with a docstring and fixed a typo in a previous commit. Will merge after CI passes.