First of all I don’t have a problem with CheckMK, but in the way that ‘on the job’ it can create unwanted notification. How would to solve this? Do you have tips or idea’s?
Background
Current we are using CheckMK Enterprise 1.6.0p13 with the following rule (as example)
But when people are patching, create database dump or something else, a filesystem can fill up from 0% to 50% in short time. In this example /var/cache will be fill and cleaned after work is done. Currently we get the following notification:
Service: Filesystem /var/cache, Status: WARNING, Status information: WARN - 55.13% used (1.1 of 1.99 GB), (warn/crit at 84.14%/92.07%), trend: +753.89 MB / 24 hours - only 29 hours until disk fullWARN
Idea’s that I had:
Ask CheckMK for a minimum parameter. So that notification only be create when filesystem is higher then 60%.
Exclude filesystem from the main rule and create rule without Trend calculation
This you can do with defining the unwanted filesystems in you rule and negate it (all other not this).
You have then only to create a second rule without trend limits that is applied to all ignored filesystems from the first rule.
You get the warning not because of used space but because you have set levels on the time left until full (72h / 24h).
This is a test that is only suitable for filesystems with a low and constant growth. If you have spikes (like database dumps etc) the growth trend will compute a very large value in MB/day which comes down to a small time left, hence the WARN.
Based on the feedback of other admin, they say that this covert ~70%. Because take /data as a filesystem, where (for example) the database can grow.
When users import data it grow with a large precent of data. This peak of data trigger the Trend calculation. On the other side the admin like the Trend calculation, for warning them before hand.
We already extend almost every filesystem, so compensate for this. Dumps are almost a weekly thing.
Was hoping to say to other admin, they there is other options that ‘disable the trend or have trend and accept the warnings’.
Trend calculation and Trend alerting are two different things. I let calculate the trend on most filesystems but alerts are generated only on a few ones. That is the same situation as @r.sander mentioned it in his post. Trend alerting is not suitable on all filesystems.
A oke, now I get it. I will create new for for the Trend computation for those filesystem.
As a side note. Would it even possible to like ‘filter’ out these notifications, if certain host/filesystem gives notification about trend computation?
I think this is possible.
At the notification rule level you can only set it based on “Match the output of the check plugin” I guess. If you disable notifications for Filesystem in the parameters, then you would get no notification at all. Even for when the filesystem is at 95% full.
