Show scylla_sstables_bloom_filter_memory_size on one of the dashboards

[michoecho]

We saw production nodes OOMing due to bloated bloom filters multiple times lately. (2 times in the last week).

And every time people seem to forget that we have a metric for this, and they waste time e.g. logging into the cluster and duing the bloom filter files.

Since the metric proved useful many times, maybe we should put it into one of the dashboards.

Perhaps it should be normalized by total memory. (E.g. sum by (...) (scylla_sstables_bloom_filter_memory_size) / sum by (...) (scylla_memory_total_memory)).

[amnonh]

@michoecho can you also suggest an alert based on that?

[amnonh]

@michoecho can you look at the memory metrics you put, it's the same one

[michoecho]

@michoecho can you also suggest an alert based on that?

@avikivity @denesb @mykaul What do you think? Should we have an alert if bloom filters exceed some fraction of memory? And if yes, what should be the threshold? 0.1, or is that too aggressive?

[avikivity]

@michoecho can you also suggest an alert based on that?

@avikivity @denesb @mykaul What do you think? Should we have an alert if bloom filters exceed some fraction of memory?

yes

And if yes, what should be the threshold? 0.1, or is that too aggressive?

I think it's a reasonable starting point.

[mykaul]

I've asked @d-helios to scan our cloud to see where we are at right now, I think based on the results we can determine the alert threshold

[amnonh]

@michoecho ping, please look at the metric you put, one is missing

[michoecho]

@michoecho ping, please look at the metric you put, one is missing

missing

If you mean the fact that I typed

sum by (...) (scylla_sstables_bloom_filter_memory_size) / sum by (...) (scylla_sstables_bloom_filter_memory_size)

then the answer is that I meant

sum by (...) (scylla_sstables_bloom_filter_memory_size) / sum by (...) (scylla_memory_total_memory)

Otherwise I don't understand what you are asking for.

[amnonh]

@michoecho yes, that is exactly what I was asking about

[amnonh]

I've checked the cloud, I think 0.1 is a good threshold