I’m administering a modest number of Linux servers all running various crons with mk-job. Some of these jobs run very frequently.
Those same servers all have an automated reboot cycle with in which all the high-frequency jobs are terminated in a way that registers as an error with mk-job.
Under normal circumstances, 1 check attempt is enough to count as a hard state and I’d like to be notified. This is because that hard state notification is just what I need to be prompted to look deeper into a job even if the job recovers during subsequent runs.
An API call registers the downtime during the 10 minutes before the reboot but I’ve not yet worked out a method for increasing the check attempts. As a CheckMK novice (former Nagios/Icinga2 user) I’m keen to approach this idiomatically instead of brute-forcing it so I’m interested in recommendations. I’m aware that “increasing the check attempts” might be the wrong solution to my needs.
Rebooting is fast so I’m looking for a grace period that lasts during the 2-5 minutes of time taken to reboot and the very brief 5 minutes afterwards when the most frequently run jobs should re-run and go green again.
I’ve explored creating a plugin to set a host label when the host is in its pre-flight for reboot but it looks like I’d need to look into tweaking service discovery. Again, I think I’m XY-problem-ing things.
Finally, it’s worth mentioning that I’m using CheckMK Community right now.
if you have an api call already creating a downtime, why not keep that downtime for the mk-job services for another 5-10 minutes depending on how long they need to recover after reboot?
retries would hide a problem the same way (unless I’m missing a detail in your requirements :))
That’s definitely one of the options I considered and good to hear it’s one that seemed sensible to someone else. However, and this is where I confess my ignorance, will the original CRIT notifications still send once the downtime ends even if the error has cleared? Or, if the jobs return to an OK/green state will this mean no noise?
This is a crucial one for me because I’m keen to share this with a teammate who I don’t want to have to explain, “look every Monday morning your inbox will have notifications you don’t need to read and bother me about if they look like X, Y, Z.”
It’s fine for me btw. To paraphrase a line out of The Matrix: “When I look at my inbox, I don’t even see the code…”
Despite this solving my issue I wanted to follow up with a bit of a eureka moment that I had while on holiday and that I’ve now tested successfully.
The problem for me is that the automated reboot is taking place often during periods when frequently run cron jobs are active. Due to the application architecture used by my team, the reboot process triggers an error state in the mini automated programs and this is being caught as an error beacuse mk-job wraps each job.
The automated reboot is run with a mini shell program of our own design that:
Triggers an API call to CheckMK’s downtime endpoint to set up a fixed downtime period
Triggers the system reboot
The fix for my issues was to insert a step in-between that terminates any crons wrapped with mk-job i.e.:
pkill -1 -f mk-job
For my environment, I know that it is safe to kill these processes because we have a whole 10 minute pre-flight process to deal with winding down anything that isn’t safe to run up until the reboot occurs.
With this in place, I actually don’t need to create such a long downtime period at all. Killing it this way appears to allow mk-job to clean itself up without causing errors to be logged.