I’m running Check_MK 1.2.6p16 on Ubuntu 14.04.4 LTS.
In the Check_MK page “Host- and Service notifications”, I see the alerts I want to get notifications for (response time, cpu usage, etc) but I never get these emails. When I look at Notifications in WATO, none of these show up - only the Check_MK Discovery alerts show up in WATO Notifications and this is all I receive in my mailbox.
Here’s one of the errors I get in /var/log/nagios3/nagios.log:
[1477948258] Warning: Attempting to execute the command “NOTIFY_CONTACTNAME=‘check-mk-notify’ NOTIFY_CONTACTEMAIL=’’ NOTIFY_CONTACTPAGER=’’ NOTIFY_NOTIFICATIONTYPE=‘PROBLEM’ NOTIFY_HOSTNOTIFICATIONNUMBER=‘0’ NOTIFY_SERVICENOTIFICATIONNUMBER=‘1’ NOTIFY_HOSTPROBLEMID=‘0’ NOTIFY_SERVICEPROBLEMID=‘5535445’ NOTIFY_HOSTNAME=‘myserver’ NOTIFY_HOSTALIAS=‘myserver’ NOTIFY_HOSTADDRESS=‘myserver.mydomain.com’ NOTIFY_HOSTATTEMPT=‘1’ NOTIFY_LASTHOSTSTATE=‘UP’ NOTIFY_LASTHOSTSTATEID=‘0’ NOTIFY_LASTHOSTSTATECHANGE=‘1475280077’ NOTIFY_LASTHOSTUP=‘1477947963’ NOTIFY_HOSTSTATE=‘UP’ NOTIFY_HOSTSTATEID=‘0’ NOTIFY_HOSTCHECKCOMMAND=“check-mk-host-tcp!3389” NOTIFY_HOSTOUTPUT=‘TCP OK - 0.001 second response time on port 3389’ NOTIFY_HOSTPERFDATA=‘time=0.001080s;;;0.000000;10.000000’ NOTIFY_LONGHOSTOUTPUT=’’ NOTIFY_SERVICEDESC=‘CPU_Time’ NOTIFY_LASTSERVICESTATE=‘WARNING’ NOTIFY_LASTSERVICESTATEID=‘1’ NOTIFY_LASTSERVICESTATECHANGE=‘1477947654’ NOTIFY_LASTSERVICEOK=‘1477947354’ NOTIFY_SERVICEATTEMPT=‘3’ NOTIFY_SERVICESTATE=‘WARNING’ NOTIFY_SERVICESTATEID=‘1’ NOTIFY_SERVICEOUTPUT=‘WARN - CPU Time is 90.406 seconds which is greater than 70’ NOTIFY_LONGSERVICEOUTPUT=’’ NOTIFY_SERVICEPERFDATA=‘average(sec)=90.406;70;140;;’ NOTIFY_SERVICECHECKCOMMAND=“check_mk-local” NOTIFY_DATE=‘2016-10-31’ NOTIFY_SHORTDATETIME=‘2016-10-31 21:10:58’ NOTIFY_LONGDATETIME=‘Mon Oct 31 21:10:58 UTC 2016’ NOTIFY_HOSTDOWNTIME=‘0’ NOTIFY_NOTIFICATIONCOMMENT=’’ NOTIFY_NOTIFICATIONAUTHOR=’’ NOTIFY_NOTIFICATIONAUTHORNAME=’’ NOTIFY_NOTIFICATIONAUTHORALIAS=’’ NOTIFY_SERVICEACKAUTHOR=’’ NOTIFY_SERVICEACKCOMMENT=’’ NOTIFY_SERVICEGROUPNAMES=‘ADX_Services’ NOTIFY_HOSTACKAUTHOR=’’ NOTIFY_HOSTACKCOMMENT=’’ NOTIFY_HOSTGROUPNAMES=‘all,ADX_Proxies’ NOTIFY_HOSTTAGS=‘cmk-agent prod True lan tcp wato /wato/’ NOTIFY_HOST_SL=’$_HOSTEC_SL$’ NOTIFY_SVC_SL=’$_SERVICEEC_SL$’ NOTIFY_SERVICE_SL=’$_SERVICEEC_SL$’ NOTIFY_HOST_EC_CONTACT=’$_HOSTEC_CONTACT$’ NOTIFY_SERVICE_EC_CONTACT=’$_SERVICEEC_CONTACT$’ /usr/bin/check_mk --notify” resulted in a return code of 127. Make sure the script or binary you are trying to execute actually exists…
When I run:
$ which mail
I get:
/usr/bin/mail
I also receive email via:
echo
“This is a test from Check_MK server” | mail -s “Test from
CMK” myemail@addres.com
If I understand the notification workflow correctly:
I run this to see what notifications nagios is trying to send:
tail -f /var/log/nagios3/nagios.log | grep NOTIFICATION
I run this to see what my current notification rules allows:
tail -f /var/lib/check_mk/log/notify.log
I run this to actually see what is being email out of the server:
tail -f /var/log/mail.log
For the Check_MK Discovery notifications, I see it go from nagios.log to notify.log then mail.log and into my mailbox.
For other notifications, I only see it from nagios.log. So I suspect it’s my notification rule but I’ve tried all different settings and I still only get Check_MK Discovery notifications. This is the only entry I have for Global notification rules:
Description: Notify all contacts of a host/service via HTML email
Overriding by users: [checked] allow users to deactivate this notification
Notification Method: HTML_Email
All contacts of the notified object: [checked] Notify all contacts of the notified host or service.
I also set my email address up as:
Fallback email address for rule based notifications
I’m pretty sure I have rules set correctly to associate the services with the correct contact group and the contacts to the contact groups. From one of the services I’m expecting a notification:
Servicegroups the service is member of My_Services
Service contact groups check-mk-notify, My_CG
Service contacts check-mk-notify, myaccount
The weird thing is, I have another Check_MK server set up for our staging environment running the same Check_MK version and it’s running fine. And I’ve compared the two configs and haven’t found a difference between the two.
Any suggestions? Is there a step between Nagios and CMK Notification rule that I’m missing?