Can't get notifications via Slack webhook to work

Troubleshooting
1

I’ve created a notification rule, with “Slock or Mattermost” as the Notification Method.
There’s a dropdown below it, it’s blank. The only other option in there is ‘cancel previous notifications’. I’ve left it blank.

I’ve put in a “Webhook URL” as the next dropbox, then I’ve set the URL in the following text box.

I’ve added “URL Prefix for links to Check_MK” as checked, and gave it a specific URL prefix to my Check_MK site.

For contact selection, I told it to notify the automation account only. (As suggested by someone else on this forum)

I’ve enabled two notification conditions, “Match host event type” and “Match service event type”
I’ve check a couple events from those from the checkboxes, (namely up to down, ok to warn, ok to crit etc)

When I select these two conditions, it mentions that it must match events created by the core… Not exactly sure what that means, I don’t have rules in either of those links where it takes me to “Notified events for hosts/services” Which sounds like to me, any event will be matched by the notification (I could be wrong, but honestly, this isn’t where the issue lies, at least I don’t believe so.)

Anytime a notification needs to go out, I get a “failed notification” event. The output of which says the following:

Failed to send notification. Status: 400, Response: {“message”: “Cannot send an empty message”, “code”: 50006}

I’m pulling hair on this issue. I can’t get it to work. Any help would be greatly appreciated. Thank you.

Check_MK Enterprise, v1.6.0p5

2

I think you’re mixing up States and Events.

I recommend you to read this article: https://checkmk.com/cms_monitoring_basics.html

3

First step for further troubleshooting is - go to global settings - Notifications - raise the “Notification log level”
The full dump option give some more useful information.
The log itself can then be found inside your monitoring site under ~/var/log/

From your version to the actual there where only one change if i see it correctly and this change has nothing to do with an empty message.

4

Thank you, Simon-Meuller, I read that page you sent, it did have some insight for me, which was good, but I would argue that I am not mixing up the two, in this case. As the article says, “When a host’s condition changes, (e.g. from OK to CRIT ), Checkmk registers an event.” and I feel like I have configured the notifications to notify on such an event, like going from OK to CRIT. Unless I am still missing the point completely, I apologize.

Thank you Andreas-Doehler, I’ve changed the notification log level, and I searched all around for the appropriate log file, but I am having troubles finding it… There is nothing related to webhooks… The mail.log doesn’t reference anything in there either… Is there a specific file I could look for?

Thanks

5

You need to look inside your monitoring site. Not in the system /var/log.
Path is normally “/omd/sites//var/log”

6

The log file you’re searching for in the directory mentioned above is called mknotifyd.log

7

I found the log, thanks for both of your inputs… Sorry I’m new to linux and navigating around it is difficult for me.

Here’s the latest notification failure, I’ve added new lines to make it more readable…

2020-04-13 15:51:01 [6] running cmk --notify --log-to-stdout spoolfile /omd/sites/{OMDSITE}/var/check_mk/notify/spool/a1a1dd8c-1a01-4627-96f1-47c174100a2b
2020-04-13 15:51:03 [6] got exit code 2
2020-04-13 15:51:03 [4] processing spoolfile a1a1dd8c-1a01-4627-96f1-47c174100a2b failed (permanently) with code 2: Failed to send notification. Status: 400,
Response: {“message”: “Cannot send an empty message”,
“code”: 50006}
2020-04-13 15:51:03 [6]
{‘context’:{
‘LASTSERVICESTATECHANGE_REL’: u’0d 00:00:00’,
‘LASTSERVICESTATE’: u’OK’,
‘HOSTCHECKCOMMAND’: u’check-mk-host-smart’,
‘HOSTOUTPUT_HTML’: u’(This macro is deprecated: see Werk#7427) Packet received via smart PING’,
‘LONGSERVICEOUTPUT_HTML’: u’(This macro is deprecated: see Werk#7427) ‘,
‘LASTHOSTUP_REL’: u’0d 00:00:01’,
‘HOSTNOTESURL’: u’‘,
‘SERVICEDESC’: u’DotNet Memory Management Global’,
‘SERVICEPERFDATA’: u’percent=11.235258;10;15;0;100’,
‘LASTHOSTSTATEID’: u’0’,
‘HOSTPERFDATA’: u’‘,
‘SERVICEATTEMPT’: u’1’,
‘LASTHOSTSHORTSTATE’: u’UP’,
‘NOTIFICATIONCOMMENT’: u’‘,
‘SERVICESHORTSTATE’: u’WARN’,
‘MAXSERVICEATTEMPTS’: u’1’,
‘MAIL_COMMAND’: u"mail -s ‘$SUBJECT$’ ‘$CONTACTEMAIL$’",
‘HOSTNAME’: u’{host}‘,
‘LASTHOSTSTATECHANGE’: u’1586665165’,
‘SERVICEFORURL’: u’DotNet%20Memory%20Management%20_Global_‘,
‘SERVICEGROUPNAMES’: u’‘,
‘SERVICENOTESURL’: u’‘,
‘HOSTSTATE’: u’UP’,
‘LASTHOSTUP’: u’1586793057’,
‘SERVICEACKCOMMENT’: u’‘,
‘HOST_TAGS’: u’/wato/ auto-piggyback cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:{OMDSITE} tcp’,
‘SHORTDATETIME’: u’2020-04-13 15:50:58’,
‘CONTACTPAGER’: ‘’,
‘LASTSERVICESTATECHANGE’: u’1586793058’,
‘LONGSERVICEOUTPUT’: u’‘,
‘HOSTPROBLEMID’: u’7539’,
‘CONTACTNAME’: u’automation’,
‘LONGHOSTOUTPUT’: u’‘,
‘MONITORING_HOST’: u’{OMDHOST}‘,
‘HOSTATTEMPT’: u’1’,
‘SERVICESTATE’: u’WARNING’,
‘WHAT’: u’SERVICE’,
‘HOSTALIAS’: u’{host}‘,
‘SERVICE_EC_CONTACT’: u’‘,
‘SERVICEACKAUTHOR’: u’‘,
‘SERVICEOUTPUT_HTML’: u’(This macro is deprecated: see Werk#7427) WARN - 11.24% time in GC (warn/crit at 10.0/15.0)WARN‘,
‘HOST_FILENAME’: u’/wato/hosts.mk’,
‘SERVICECHECKCOMMAND’: u’check_mk-dotnet_clrmemory’,
‘LASTSERVICESTATEID’: u’0’,
‘LASTSERVICEOK’: u’1586792996’,
‘SERVICEPROBLEMID’: u’7734’,
‘HOST_SL’: u’‘,
‘NOTIFICATIONAUTHORALIAS’: u’‘,
‘HOST_ADDRESS_4’: u’192.168.1.101’,
‘HOST_ADDRESS_6’: u’‘,
‘SERVICEOUTPUT’: u’WARN - 11.24% time in GC (warn/crit at 10.0/15.0)(!)’,
‘CONTACTALIAS’: u’Check_MK Automation - used for calling web services’,
‘HOSTDOWNTIME’: u’0’,
‘SERVICENOTIFICATIONNUMBER’: u’1’,
‘SERVICEDOWNTIME’: u’0’,
‘NOTIFICATIONAUTHORNAME’: u’‘,
‘HOSTGROUPNAMES’: u’check_mk’,
‘HOSTSHORTSTATE’: u’UP’,
‘HOSTNOTIFICATIONNUMBER’: u’1’,
‘OMD_ROOT’: u’/omd/sites/{OMDSITE}‘,
‘LASTHOSTSTATECHANGE_REL’: u’1d 11:31:33’,
‘PREVIOUSHOSTHARDSTATEID’: u’1’,
‘LASTSERVICESHORTSTATE’: u’OK’,
‘CONTACTEMAIL’: u’‘,
‘PREVIOUSSERVICEHARDSHORTSTATE’: u’OK’,
‘HOST_ADDRESS_FAMILY’: u’4’,
‘HOSTACKAUTHOR’: u’‘,
‘HOSTURL’: u’/check_mk/index.py?start_url=view.py%3Fview_name%3Dhoststatus%26host%3D{host}%26site%3D{OMDSITE}‘,
‘HOSTSTATEID’: u’0’,
‘MICROTIME’: u’1586793058031297’,
‘LASTSERVICEPROBLEMID’: u’7734’,
‘PREVIOUSSERVICEHARDSTATE’: u’OK’,
‘SERVICEDISPLAYNAME’: u’DotNet Memory Management Global’,
‘NOTIFICATIONTYPE’: u’PROBLEM’,
‘HOSTNOTES’: u’‘,
‘LOGDIR’: u’/omd/sites/{OMDSITE}/var/check_mk/notify’,
‘MAXHOSTATTEMPTS’: u’1’,
‘OMD_SITE’: u’{OMDSITE}‘,
‘PARAMETER_WEBHOOK_URL’: ‘webhook_url\t{WEBHOOKADDRESS}’,
‘HOSTACKCOMMENT’: u’‘,
‘PREVIOUSSERVICEHARDSTATEID’: u’0’,
‘PARAMETER_URL_PREFIX_MANUAL’: ‘{OMDURL}’,
‘SERVICE_SL’: u’‘,
‘DATE’: u’2020-04-13’,
‘HOSTOUTPUT’: u’Packet received via smart PING’,
‘NOTIFICATIONAUTHOR’: u’‘,
‘HOSTFORURL’: u’{host}‘,
‘HOSTTAGS’: u’/wato/ auto-piggyback cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:{OMDSITE} tcp’,
‘SERVICENOTES’: u’‘,
‘SERVICESTATEID’: u’1’,
‘HOSTADDRESS’: u’{HOSTIP}‘,
‘PREVIOUSHOSTHARDSTATE’: u’DOWN’,
‘LASTSERVICEOK_REL’: u’0d 00:01:02’,
‘HOSTCONTACTGROUPNAMES’: u’all’,
‘HOST_EC_CONTACT’: u’‘,
‘SERVICECONTACTGROUPNAMES’: u’all’,
‘CONTACTS’: u’DatsiK96’,
‘LASTHOSTPROBLEMID’: u’7539’,
‘SVC_SL’: u’‘,
‘LASTHOSTSTATE’: u’UP’,
‘PREVIOUSHOSTHARDSHORTSTATE’: u’DOWN’,
‘LONGDATETIME’: u’Mon Apr 13 15:50:58 UTC 2020’,
‘SERVICEURL’: u’/check_mk/index.py?start_url=view.py%3Fview_name%3Dservice%26host%3D{host}%26service%3DDotNet%20Memory%20Management%20_Global_%26site%3D{OMDSITE}‘},
‘plugin’: u’slack’}
2020-04-13 15:51:03 [6] sending command LOG;SERVICE NOTIFICATION RESULT: automation;{host};DotNet Memory Management Global;CRITICAL;slack;Failed to send notification. Status: 400,
Response: {“message”: “Cannot send an empty message”,
“code”: 50006};Failed to send notification. Status: 400,
Response: {“message”: “Cannot send an empty message”,
“code”: 50006}

From what I can see, it’s there’s a call for a mail command, a couple of things are also stating the function is deprecated… But I’m not sure why it’s getting an empty message…

Any thoughts?

8

I got it working. I made an assumption about slack webhooks and discord webhooks being the same because I had seen it work in another application. I appended /slack at the end of the webhook URL and it’s working now.

Thanks to everyone who gave me pointers and helped point me in the right direction.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.