Scheduled notifications possible? API? Scheduled test of notification rules?

[dmayer (Dirk Mayer)]

Hello,

in the past, for our old Nagios we used a cronjob to trigger daily custom notifications via the Nagios external command file. We used this to test if the correct persons will receive an SMS notification for oncall duty for their respective team and if SMS notification is working at all.

That same approach does not work as intended in checkmk. I tried all combinations of the options “force” or “broadcast”, but the command will either send no notification at all or send notifications to everybody, who currently has not temporarily disabled notifications.

Is it possible to implement a scheduled notification, that behaves identical to a configured notification rule and only alerts the responsible contacts? Did I miss something in the checkmk API, which can trigger notifications?

Or am I missing something regarding the notification settings? All current users have a role, where they can edit their own folder, but can see everything else. When triggering custom alerts, people receive notifications, even if they are not a contact of that host. Do checkmk custom notifications notify users for everything they can see?

Kind regards,
Dirk.

Edit: I probably should add, that we’re using a 5 minute notification delay and 1 minute check interval. So fake check results is no option, either.

[ttr (Tom Tretbar)]

Hi,

I use some simple script as local check to generate a service that goes crit once a week. This service can be treated as any other and can be used in notification rules, we use it to test our SMS gateways. You might adjust this to your needs.

Regards,
Tom

#!/bin/bash
#
# Description:  checkmk check that goes CRIT at a configurable date and time
#               and stays CRIT for a configurable count of seconds.
#               Useful for testing nofication subsystem on a regular base,
#               e.g. SMS gateways.
#
# Maintainer:   (ttr)
#
# Version:      0.1
#
# Created:      16.10.20
#
# Usage:        - checkmk local check
#               - discover service "SMS" and enable notification for that
#
# Changelog:    0.1   - 16.10.2020 - ttr
#                      - initial


# date of status change to crit
SMS_DATE="11:50 Fri"

# seconds to stay in CRIT state:
DURATION=300

# the messages, checkmk local check syntax
GOOD="0 SMS - SMS gateway, next check "$SMS_DATE
BAD="2 SMS - SMS gateway test message"


# calculate seconds to go
s=$(echo $(($(date -d "$SMS_DATE + $DURATION sec" +%s)-$(date +%s))))

# evaluate
[ $s -gt 0 -a $s -lt $DURATION ] && echo $BAD || echo $GOOD

[dmayer (Dirk Mayer)]

Thanks for your input! That might be an option. However, this would still not check the productive notification rules themselves. For example I can’t simulate a fault on a NetApp filer for storage oncall that way. I’ll keep that option in mind as a last resort, though. At least this could prove that SMS is active.

Custom notifications in the GUI work as expected and alert the right people, based on the notification rules. Is there any way to trigger custom notifications via API? Or some slightly dirty approach via livestatus, possibly?

[ttr (Tom Tretbar)]

@dmayer

… However, this would still not check the productive notification rules themselves. For example I can’t simulate a fault on a NetApp filer for storage oncall that way

We use different service groups for different notification rules. So we can can put this SMS service into desired group and notification would be exactly as for any other failed service within this group. In your example you would create a service group including NetApp stuff and the SMS thing. Works like a charm and you can be sure that correct notification rule is triggered.

For the API-stuff: sorry, I do not know a way to do so.

Regards,
Tom.

[dmayer (Dirk Mayer)]

Hi!

After lots of further tests and analysis of the logs in var/log/notify.log, I realized that the old external command file still works as expected in checkmk. The custom notifications are correctly evaluated by the notification rules in checkmk.

So as information to anyone facing the same task: One of the possible approaches for sending scheduled notifications is by using nagios.cmd, which is still supported in checkmk 2.0.0p1.

Example script:

#!/bin/bash

NAGIOS_CMDFILE="/opt/omd/sites/yoursite/tmp/run/nagios.cmd"
now=`date +%s`

# --  Test SMS to Linux oncall --
/bin/echo "[$now] SEND_CUSTOM_SVC_NOTIFICATION;yourlinuxhost;Check_MK;3;checkmk scheduled message;Test-SMS to Linux oncall"  >> $NAGIOS_CMDFILE

If there is a notification rule, which sends out notifications for service Check_MK for host “yourlinuxhost”, the contacts from this rule will get the scheduled notification. You can add further commands for notifying other teams, by using different hosts or service checks.

However, contact groups are a tricky thing. “Service contacts” only lists the intended contacts, but “Service contact groups” in our configuration includes “all” by default. So I had to change the rules to send to the specific contact groups only, not to “all contacts of this host or service”.

I still have a problem with a rule that won’t accept the custom notifications due to their event type. But I’ll post that to a new thread, as this is not a topic about scheduled notifications anymore.

Thanks for your input!
Bye, Dirk.

Edit: Just to add this information: My problem with one of the rules was that service state “OK → OK” was not selected, which is required for such custom alerts, where the state stays at OK. If you see log entries in notify.log that event type ‘rr’ is not allowed in a rule, that’s the option to activate! (I guess it stands for recovery → recovery.)