Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Disk space alerts not working due to wrong metrics used #1282

Closed
ybalt opened this issue Feb 15, 2021 · 0 comments · Fixed by #1283
Closed

Disk space alerts not working due to wrong metrics used #1282

ybalt opened this issue Feb 15, 2021 · 0 comments · Fixed by #1283
Labels
bug Something isn't working right

Comments

@ybalt
Copy link

ybalt commented Feb 15, 2021

Download
https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-3.5.tar.gz
unpack and run

Panel Name: -
Dashboard Name: -
Scylla-Monitoring Version: 3.5
Scylla-Version: 4.1

All alerts that calculate free disk space are not working, for example

alert: DiskFull
expr: node_filesystem_avail{mountpoint="/var/lib/scylla"}
  / node_filesystem_size{mountpoint="/var/lib/scylla"} * 100 < 25
for: 30s
labels:
  severity: "2"
annotations:
  description: '{{ $labels.instance }} has less than 25% free disk space.'
  summary: Instance {{ $labels.instance }} low disk space

the reason is a wrong metric name, there is no metric with name 'node_filesystem_avail' present on scylla prometheus, but
'node_filesystem_avail_bytes' is present. The same with 'node_filesystem_size', it should be replaced with 'node_filesystem_size_bytes' to correct work.

This bug is very dangerous, as may result no free space on disk and cluster down without any notifications

@ybalt ybalt added the bug Something isn't working right label Feb 15, 2021
@amnonh amnonh added this to the monitoring 3.7 milestone Feb 15, 2021
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working right
Projects
None yet
2 participants