Pushover Notifications not working

Troubleshooting
,
1

CMK version:
2.0.0p19cre
OS version:
Debian 10
Error message:
I don’t know what is wrong. I have looked at other forms with no luck. And I have succesfully setup notifications using pushover in the past with no issues. But the logs say that it is properlly sending using pushover but it will never come through. I’ve done a curl -X POST command on the server cli and it sends just fine. So I don’t think it has anything to do with connectivity. Any ideas? (Logs below)

2022-02-10 11:27:00,086 [20] [cmk.base.notify] ----------------------------------------------------------------------
2022-02-10 11:27:00,086 [20] [cmk.base.notify] Got raw notification (Septic-Plug) context with 36 variables
2022-02-10 11:27:00,086 [10] [cmk.base.notify] Raw context:
                    CONTACTEMAIL=
                    CONTACTNAME=check-mk-notify
                    CONTACTPAGER=
                    DATE=2022-02-10
                    HOSTACKAUTHOR=
                    HOSTACKCOMMENT=
                    HOSTADDRESS=10.0.40.22
                    HOSTALIAS=Septic-Plug
                    HOSTATTEMPT=1
                    HOSTCHECKCOMMAND=check-mk-host-ping!-w 200.00,80.00% -c 500.00,100.00%
                    HOSTDOWNTIME=0
                    HOSTGROUPNAMES=check_mk
                    HOSTNAME=Septic-Plug
                    HOSTNOTIFICATIONNUMBER=1
                    HOSTOUTPUT=CRITICAL - 10.0.40.22: rta 809.979ms, lost 0%
                    HOSTPERFDATA=rta=809.979ms;200.000;500.000;0; pl=0%;80;100;; rtmax=981.585ms;;;; rtmin=646.694ms;;;;
                    HOSTPROBLEMID=490
                    HOSTSTATE=DOWN
                    HOSTSTATEID=1
                    HOSTTAGS=/wato/iot/ auto-piggyback ip-v4 ip-v4-only lan no-agent no-snmp ping prod site:cmk
                    HOST_ADDRESS_4=10.0.40.22
                    HOST_ADDRESS_6=
                    HOST_ADDRESS_FAMILY=4
                    LASTHOSTSTATE=UP
                    LASTHOSTSTATECHANGE=1644510418
                    LASTHOSTSTATEID=0
                    LASTHOSTUP=1644510357
                    LONGDATETIME=Thu Feb 10 11:26:58 EST 2022
                    LONGHOSTOUTPUT=
                    NOTIFICATIONAUTHOR=
                    NOTIFICATIONAUTHORALIAS=
                    NOTIFICATIONAUTHORNAME=
                    NOTIFICATIONCOMMENT=
                    NOTIFICATIONTYPE=PROBLEM
                    SERVICECHECKCOMMAND=$
                    SHORTDATETIME=2022-02-10 11:26:58
2022-02-10 11:27:00,088 [20] [cmk.base.events] Computed variables:
                    CONTACTS=
                    HOSTFORURL=Septic-Plug
                    HOSTSHORTSTATE=DOWN
                    HOSTURL=/check_mk/index.py?start_url=view.py%3Fview_name%3Dhoststatus%26host%3DSeptic-Plug%26site%3Dcmk
                    LASTHOSTSHORTSTATE=UP
                    LASTHOSTSTATECHANGE_REL=0d 00:00:02
                    LASTHOSTUP_REL=0d 00:01:03
                    MICROTIME=1644510420087297
                    MONITORING_HOST=1ea47fcec3d8
                    OMD_ROOT=/omd/sites/cmk
                    OMD_SITE=cmk
                    PREVIOUSHOSTHARDSHORTSTATE=UP
                    PREVIOUSHOSTHARDSTATE=UP
                    WHAT=HOST
2022-02-10 11:27:00,089 [10] [cmk.base.notify] Preparing rule based notifications
2022-02-10 11:27:00,089 [10] [cmk.base.notify] Found 1 user specific rules
2022-02-10 11:27:00,090 [20] [cmk.base.notify] Global rule 'PushOver'...
2022-02-10 11:27:00,090 [20] [cmk.base.notify]  -> matches!
2022-02-10 11:27:00,091 [20] [cmk.base.notify]    - adding notification of ccasteel, cmkadmin, automation via pushover
2022-02-10 11:27:00,092 [20] [cmk.base.notify] User ccasteel's rule ''...
2022-02-10 11:27:00,092 [20] [cmk.base.notify]  -> matches!
2022-02-10 11:27:00,093 [20] [cmk.base.notify]    - adding notification of ccasteel via pushover
2022-02-10 11:27:00,093 [20] [cmk.base.notify] Executing 2 notifications:
2022-02-10 11:27:00,093 [20] [cmk.base.notify]   * notifying ccasteel via pushover, parameters: api_key, recipient_key, priority, sound, bulk: no
2022-02-10 11:27:00,094 [20] [cmk.base.notify]      executing /omd/sites/cmk/share/check_mk/notifications/pushover
2

What is in the log after that line?

5

Actually this is what comes after if that helps:

2022-02-10 11:42:17,021 [20] [cmk.base.notify]      executing /omd/sites/cmk/share/check_mk/notifications/pushover
2022-02-10 11:42:52,876 [10] [cmk.store] Trying to acquire lock on /omd/sites/cmk/var/check_mk/notify/backlog.mk
2022-02-10 11:42:52,876 [10] [cmk.store] Got lock on /omd/sites/cmk/var/check_mk/notify/backlog.mk
2022-02-10 11:42:52,881 [10] [cmk.store] Releasing lock on /omd/sites/cmk/var/check_mk/notify/backlog.mk
2022-02-10 11:42:52,881 [10] [cmk.store] Released lock on /omd/sites/cmk/var/check_
6

You could try to debug the execution of the pushover notification plugin. For this you have to set environment variables from the “context” prefixed with NOTIFY_, e.g.:

export NOTIFY_HOSTALIAS="Septic-Plug"
export NOTIFY_HOSTSTATE=DOWN

etc.

7

Is that in a certain file?

8

You have to set these Environment variables in the shell before you execute /omd/sites/cmk/share/check_mk/notifications/pushover for debugging purposes.

9

Alright after running it again with tose Variable this is the output:

Traceback (most recent call last):
  File "/omd/sites/cmk/share/check_mk/notifications/pushover", line 12, in <module>
    from cmk.notification_plugins import pushover
ModuleNotFoundError: No module named 'cmk'
10

You have to run it as the site user. As root do omd su cmk before.

11

Thanks for the help! So after running it as the correct user this is now the error output:

Traceback (most recent call last):
  File "/omd/sites/cmk/share/check_mk/notifications/pushover", line 15, in <module>
    sys.exit(pushover.main())
  File "/omd/sites/cmk/lib/python3/cmk/notification_plugins/pushover.py", line 15, in main
    subject = get_subject(context)
  File "/omd/sites/cmk/lib/python3/cmk/notification_plugins/pushover.py", line 25, in get_subject
    s = context["HOSTNAME"]
KeyError: 'HOSTNAME'
12

You need to set all relevant notification variables. This error tells you that NOTIFY_HOSTNAME is missing.

13

Well I was able to run the command as the OMD user with all the proper variables you recommended. However, it keeps asking for more variables like:

what
servicesdesc
notificationtype
and now parameter api key.

So I don’t want to keep typing infinite variables. Is there something else we can test?

14

Actually I pursued the rest of the variables and it was able to now attempt to send the message. The problem now is this:

Failed to send notifications. Status: 400, Response {“message”:“cannot be blank”,“errors”:[“message cannot be blank”],“status”:0,

So is it possible that CheckMK is not attaching the alert to the notification?

15

That is not so easy - you can trigger a notification from inside the notification system with the replay button. But to get more output or error messages from the pushover script you need to modify the script itself i think.

16

Did you assign an email-address to your checkmk-user?

I had the same error. I only configured pushover. I have checked my pushover-keys with a curl command on my checkmk docker:

curl -s \
   --form-string "token=xyxyxy" \
   --form-string "user=abaabab" \
   --form-string "message=here is an image attachment" \
     https://api.pushover.net/1/messages.json

that worked. but checkmk was not able to notify via pushover.

so i checked the notify log im my site with name “cmk”

OMD[cmk]:~$ tail -f /omd/sites/cmk/var/log/notify.log

i found always:

2022-09-16 22:54:55,966 [20] [cmk.base.notify] ----------------------------------------------------------------------
2022-09-16 22:54:55,966 [20] [cmk.base.notify] Got raw notification (pi4;Alpha ESS Soc) context with 52 variables
2022-09-16 22:54:55,983 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via pushover'...
2022-09-16 22:54:55,983 [20] [cmk.base.notify]  -> matches!
2022-09-16 22:54:55,984 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2022-09-16 22:55:00,741 [20] [cmk.base.notify] ----------------------------------------------------------------------

then i found via google a tip to assign an email address to my checkmk-user, although i did not need an email-adress because i only wanted to notify via pushover. Anyway, after i assigned the email adress it worked:

2022-09-16 23:11:19,280 [20] [cmk.base.notify] Got raw notification (pi4;Alpha ESS EGridCharge) context with 52 variables
2022-09-16 23:11:19,295 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via pushover'...
2022-09-16 23:11:19,296 [20] [cmk.base.notify]  -> matches!
2022-09-16 23:11:19,296 [20] [cmk.base.notify]    - adding notification of ogaida via pushover
2022-09-16 23:11:19,299 [20] [cmk.base.notify] Executing 1 notifications:
2022-09-16 23:11:19,300 [20] [cmk.base.notify]   * notifying ogaida via pushover, parameters: api_key, recipient_key, sound, bulk: no
2022-09-16 23:11:19,301 [20] [cmk.base.notify]      executing /omd/sites/cmk/share/check_mk/notifications/pushover

i hope that will help. And for me it looks like a bug in checkmk.

Cheers
Olli

17

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.