Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
Initially, notifications were functioning seamlessly, and without any apparent modification, a set of recent notifications surfaced for analysis. Subsequently, Check MK ceased sending all notifications, and there is no indication of errors in the var/log/notify.log, even after adjusting the notification service in global settings to a full dump at the notification level.
Currently, notifications are only dispatched when I access one of the recent notifications in the analysis queue and manually click the reload button to replay the notification and send it again.
I seek your assistance as this issue arose without any discernible cause. Your help is greatly appreciated.
I changed the notifications to full debug and executed the command cmk --notify to troubleshoot why notifications are not working. However, I encountered the following error:
we have the same problem since upgrade from (enterprise) 2.1.x to 2.2.x
Example of notify log for a notification that could not be sent:
2024-01-15 09:47:13,315 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-01-15 09:47:13,316 [20] [cmk.base.notify] Got raw notification (USV-DAUN-BRZ) context with 44 variables
2024-01-15 09:47:13,316 [20] [cmk.base.notify] Global rule 'Standardregel HTML-Mail'...
2024-01-15 09:47:13,316 [20] [cmk.base.notify] -> matches!
2024-01-15 09:47:13,316 [40] [cmk.base.notify] ERROR:
Traceback (most recent call last):
File "/omd/sites/cdp/lib/python3/cmk/base/notify.py", line 352, in locally_deliver_raw_context
return notify_rulebased(raw_context, analyse=analyse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/cdp/lib/python3/cmk/base/notify.py", line 444, in notify_rulebased
notifications, rule_info = _create_notifications(
^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/cdp/lib/python3/cmk/base/notify.py", line 465, in _create_notifications
contacts = rbn_rule_contacts(rule, raw_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/cdp/lib/python3/cmk/base/notify.py", line 968, in rbn_rule_contacts
if disable_notifications_opts.get("disable", False):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'bool' object has no attribute 'get'
any ideas?
Edit: happened in 2.2.0p14 and did not change with 2.2.0p18.
Only affects notifications, reports are sent as scheduled.
Is there a (simple) way to downgrade back to 2.1?
Edit2: When replaying the notification, there is no error/waring. Just the same entry in notification log of checkmk (see above).
It appears that there is a recurring issue in the 2.x.x version of Check MK that remains unresolved. This issue is causing difficulties for users, as it disrupts the proper functioning of notifications and generates error messages on the GUI.
We have seen a few users with issue of the disabled notification key, all came from previous versions (incl. 1.5) and the steps from the werk solved the issue for them.
@Martin.B
Do you remember the first version you used? It has nothing to do with the upgrade from 2.1 - 2.2, more with the incompatible werk from 1.5. If you did not changed your config back then, you have this problems now. Are you sure all users have the correct format?
@fredaranha
Your logs don’t show any related error messages to the problem mentioned in this werk.
The logs show an empty context from the backlog.
It would be helpful if you could post logs of a real notification not being sent
I checked my old mails from the point where we started with cmk enterprise. It was 1.5.0p4.
I checked all users now (we only have 10 and 2 of them have notifications disabled).
in contacts.mk all users do either have disable_notifications': {'disable': True}, or disable_notifications': {},
2 minutes ago, we started to receive notifications again.
I don’t know what exactly did the trick, i checked and saved (even if i did not change anything) every setting/rule regarding notifications and also every user profile in the GUI.
