Add Customization Options for KRR Notifications (Exclude Metrics and Set Thresholds)

[ErezWeiss]

Is your feature request related to a problem? Please describe.
We are looking to automate weekly KRR reports, but the current notification system includes details that are not actionable for us at the moment. Specifically, we don’t want to receive notifications related to memory usage and tiny CPU changes. These notifications make it harder to focus on the issues that matter the most.

Describe the solution you'd like
We would like an option to customize KRR notifications to exclude specific metrics (e.g., memory) or set thresholds to avoid alerts for minor changes (e.g., CPU fluctuations below a certain percentage). This would help streamline our weekly reporting and make the data more actionable.

Describe alternatives you’ve considered
Filtering the notifications post-generation using external scripts.
Manually ignoring irrelevant notifications, which is not sustainable long-term.

Are you interested in contributing a PR for this?
No.

Additional context
We use Robusta's KRR for weekly insights into our system's performance. Customizing notifications based on our current focus areas would greatly improve the usability of the reports.

[aantn]

Thanks. Are you requirements the same as #381?

[ErezWeiss]

@aantn Thanks!
Yes, plus the option to disable the memory recommendations.

[ErezWeiss]

@aantn by the way, do you have a recommendation for how to utilize Java workloads? As krr can't look inside the JVM

[aantn]

@ErezWeiss got it, thanks. Any interest in contributing a PR for this? We'd be happy to review and merge it!

Re. Java, what are the outcomes you'd like? To exclude Java workloads from recommendations based on some label? To calculate recommendations based on JVM specific metrics?

[ErezWeiss]

@aantn
I'll try to look into it.

Re. java - as the memory recommendations are irrelevant, we want to disable it now.
For a future feature, considering the JVM metrics would be awesome.

[aantn]

@ErezWeiss is there a way to identify all java pods/containers in your environment?

[aantn]

We could add an option to skip certain workloads by name, or possibly by labels on the workloads.

[ErezWeiss]

@aantn
We use the label: microservice-type=java

[ErezWeiss]

as for JVM, we use the metrics of jvm_memory_used_bytes and jvm_memory_max_bytes

[aantn]

Got it, can you filter out the java workloads with something like --selector='microservice-type!=java' for now?

[ErezWeiss]

Doesn't work, I still get all of the workloads

[aantn]

Got it, thanks.