Trouble with notifications in 2.2.0.cre

Troubleshooting
, , ,
1

CMK version: Open Monitoring Distribution Version 2.2.0.cre
OS version: Ubuntu 20.04.6 LTS

Error message:

2023-06-06 14:42:26,723 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-06 14:42:26,723 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-06 14:42:26,724 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-06 14:42:26,724 [20] [cmk.base.notify]  -> matches!
2023-06-06 14:42:26,724 [40] [cmk.base.notify] ERROR:
Traceback (most recent call last):
  File "/omd/sites/sitename/lib/python3/cmk/base/notify.py", line 351, in locally_deliver_raw_context
    return notify_rulebased(raw_context, analyse=analyse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/sitename/lib/python3/cmk/base/notify.py", line 443, in notify_rulebased
    notifications, rule_info = _create_notifications(
                               ^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/sitename/lib/python3/cmk/base/notify.py", line 464, in _create_notifications
    contacts = rbn_rule_contacts(rule, raw_context)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/sitename/lib/python3/cmk/base/notify.py", line 967, in rbn_rule_contacts
    if disable_notifications_opts.get("disable", False):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'bool' object has no attribute 'get'

I used Checkmk in 2.1.0 for some time and performed recently an upgrade to 2.2.0. To do so, I first updateted to 2.1.0p28.cre and afterards to 2.2.0.cre.

Then I noticed, that notifications are not send by email anymore. I checked the logs an found that entry from above.
Now I checked the code and saw, that there was a condition block, checking, if the user blocks notifications.
I ended up, commecnting this conditional out, because non of the users are using this feature:

        if contact:
            disable_notifications_opts = contact.get("disable_notifications", {})
#            if disable_notifications_opts.get("disable", False):
#                start, end = disable_notifications_opts.get("timerange", (None, None))
#                if start is None or end is None:
#                    logger.info(
#                        "   - skipping contact %s: he/she has disabled notifications", contactname
#                    )
#                    continue
#                if start <= time.time() <= end:
#                    logger.info(
#                        "   - skipping contact %s: he/she has disabled notifications from %s to %s.",
#                        contactname,
#                        start,
#                        end,
#                    )
#                    continue

Now, notifications are send, by way too many.
If I trigger a test notification for a random check, every contact of this check receives more than 6 emails. This behavior is also logged in var/log/notify.log.

2023-06-06 14:52:05,407 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-06 14:52:05,407 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-06 14:52:05,408 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-06 14:52:05,408 [20] [cmk.base.notify]  -> matches!
2023-06-06 14:52:05,408 [20] [cmk.base.notify]    - adding notification of User1, User2, User3, User4, User5, User6, User7, User8, User9 via mail
2023-06-06 14:52:05,408 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-06 14:52:05,408 [20] [cmk.base.notify]   * notifying User1, User2, User3, User4, User5, User6, User7, User8, User9 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-06 14:52:05,408 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: User1, User2, User3, User4, User5, User6, User7, User8, User9;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-06 14:52:05,408 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-06 14:52:05,604 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-06 14:52:05,627 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: User1, User2, User3, User4, User5, User6, User7, User8, User9;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent

This block of lines is repeated in total for 6 times in var/log/notifiy.log

Any ideas how to fix the notification by mail, like one notification per user per triggered event?

2

Same issue here on debian 11 after upgrading our staging setup from 2.1.0p28. Interesting that no one else seems to face that problem? Commenting the part out fixes it halfway, like you said, but creates a mail flood instead. So any help appreciated, i also sent a report to feedback-2.2-beta@checkmk.com

3

Hi,

there seems to be an old config fomat for the disabled option.

Please see this werk for instructions how to adjust your config:

1 Like
4

That fixed it, thanks. Pretty odd that this wasn’t a problem until 2.2 now?

5

I changed to the new config fomat for the disabled option in ~/etc/check_mk/conf.d/wato/contacts.mk and removed the comments in /omd/sites/rh/lib/python3/cmk/base/notify.py.

As a result, I dont get any error messages.
But when triggering a custom notification, the problem of multiple notification emails persists.

Did the werk #6417 fix this issue of multiple messages for you @mho-nicat too?

6

No issues here, just a single notification for every contact. Does the notify.log contain any usefull information?

7

I dont see any error or something else, that would leed me to the root of the problem:

2023-06-12 13:04:29,453 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:29,453 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:29,454 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:29,454 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:29,454 [20] [cmk.base.notify]    - adding notification of user1, user2, user3, user4, user5, user6, user7, user8, user9 via mail
2023-06-12 13:04:29,454 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:29,454 [20] [cmk.base.notify]   * notifying user1, user2, user3, user4, user5, user6, user7, user8, user9 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:29,454 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user1,user2,user3,user4,user5,user6,user7,user8,user9;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:29,455 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:29,683 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:29,707 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user1,user2,user3,user4,user5,user6,user7,user8,user9;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:30,043 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:30,043 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:30,044 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:30,045 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:30,045 [20] [cmk.base.notify]    - adding notification of user8, user9, user6, user2, user5, user1, user4, user3, user7 via mail
2023-06-12 13:04:30,045 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:30,045 [20] [cmk.base.notify]   * notifying user8, user9, user6, user2, user5, user1, user4, user3, user7 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:30,045 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user8,user9,user6,user2,user5,user1,user4,user3,user7;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:30,045 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:30,252 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:30,275 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user8,user9,user6,user2,user5,user1,user4,user3,user7;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:30,580 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:30,580 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:30,581 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:30,581 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:30,581 [20] [cmk.base.notify]    - adding notification of user9, user7, user6, user1, user3, user8, user4, user5, user2 via mail
2023-06-12 13:04:30,581 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:30,581 [20] [cmk.base.notify]   * notifying user9, user7, user6, user1, user3, user8, user4, user5, user2 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:30,581 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user9,user7,user6,user1,user3,user8,user4,user5,user2;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:30,581 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:30,781 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:30,806 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user9,user7,user6,user1,user3,user8,user4,user5,user2;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:31,098 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:31,098 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:31,099 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:31,099 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:31,099 [20] [cmk.base.notify]    - adding notification of user3, user8, user2, user1, user7, user4, user9, user6, user5 via mail
2023-06-12 13:04:31,099 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:31,100 [20] [cmk.base.notify]   * notifying user3, user8, user2, user1, user7, user4, user9, user6, user5 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:31,100 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user3,user8,user2,user1,user7,user4,user9,user6,user5;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:31,100 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:31,309 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:31,332 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user3,user8,user2,user1,user7,user4,user9,user6,user5;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:31,621 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:31,622 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:31,623 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:31,623 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:31,623 [20] [cmk.base.notify]    - adding notification of user9, user1, user5, user3, user8, user6, user2, user7, user4 via mail
2023-06-12 13:04:31,623 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:31,623 [20] [cmk.base.notify]   * notifying user9, user1, user5, user3, user8, user6, user2, user7, user4 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:31,623 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user9,user1,user5,user3,user8,user6,user2,user7,user4;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:31,623 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:31,832 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:31,854 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user9,user1,user5,user3,user8,user6,user2,user7,user4;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:32,147 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:32,147 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:32,148 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:32,148 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:32,148 [20] [cmk.base.notify]    - adding notification of user5, user2, user4, user7, user9, user8, user6, user3, user1 via mail
2023-06-12 13:04:32,148 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:32,148 [20] [cmk.base.notify]   * notifying user5, user2, user4, user7, user9, user8, user6, user3, user1 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:32,149 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user5,user2,user4,user7,user9,user8,user6,user3,user1;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:32,149 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:32,354 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:32,377 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user5,user2,user4,user7,user9,user8,user6,user3,user1;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:32,669 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:32,669 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:32,670 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:32,670 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:32,670 [20] [cmk.base.notify]    - adding notification of user6, user3, user1, user5, user2, user7, user9, user4, user8 via mail
2023-06-12 13:04:32,670 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:32,670 [20] [cmk.base.notify]   * notifying user6, user3, user1, user5, user2, user7, user9, user4, user8 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:32,670 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user6,user3,user1,user5,user2,user7,user9,user4,user8;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:32,671 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:32,903 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:32,924 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user6,user3,user1,user5,user2,user7,user9,user4,user8;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:33,223 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:33,223 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:33,224 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:33,224 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:33,224 [20] [cmk.base.notify]    - adding notification of user1, user3, user5, user9, user6, user4, user2, user7, user8 via mail
2023-06-12 13:04:33,224 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:33,225 [20] [cmk.base.notify]   * notifying user1, user3, user5, user9, user6, user4, user2, user7, user8 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:33,225 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user1,user3,user5,user9,user6,user4,user2,user7,user8;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:33,225 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:33,441 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:33,465 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user1,user3,user5,user9,user6,user4,user2,user7,user8;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:33,760 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:33,760 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:33,761 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:33,761 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:33,761 [20] [cmk.base.notify]    - adding notification of user4, user5, user3, user6, user7, user1, user9, user8, user2 via mail
2023-06-12 13:04:33,761 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:33,761 [20] [cmk.base.notify]   * notifying user4, user5, user3, user6, user7, user1, user9, user8, user2 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:33,762 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user4,user5,user3,user6,user7,user1,user9,user8,user2;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:33,762 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:33,964 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:33,987 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user4,user5,user3,user6,user7,user1,user9,user8,user2;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2023-06-12 13:04:34,280 [20] [cmk.base.notify] ----------------------------------------------------------------------
2023-06-12 13:04:34,280 [20] [cmk.base.notify] Got raw notification (host.fqdn.de;Systemd Service Summary) context with 52 variables
2023-06-12 13:04:34,281 [20] [cmk.base.notify] Global rule 'basic notification'...
2023-06-12 13:04:34,281 [20] [cmk.base.notify]  -> matches!
2023-06-12 13:04:34,281 [20] [cmk.base.notify]    - adding notification of user6, user1, user2, user8, user4, user5, user7, user9, user3 via mail
2023-06-12 13:04:34,282 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-12 13:04:34,282 [20] [cmk.base.notify]   * notifying user6, user1, user2, user8, user4, user5, user7, user9, user3 via mail, parameters: from, host_subject, service_subject, elements, no_floating_graphs, graphs_per_notification, notifications_with_graphs, bulk: no
2023-06-12 13:04:34,282 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: user6,user1,user2,user8,user4,user5,user7,user9,user3;host.fqdn.de;Systemd Service Summary;OK;mail;Total: 148, Disabled: 4, Failed: 0
2023-06-12 13:04:34,282 [20] [cmk.base.notify]      executing /omd/sites/sitename/share/check_mk/notifications/mail
2023-06-12 13:04:34,550 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2023-06-12 13:04:34,572 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: user6,user1,user2,user8,user4,user5,user7,user9,user3;host.fqdn.de;Systemd Service Summary;OK;mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent

Settings in Notification configuration → basic notification

Rule properties:Description: basic notification
Notification method:Notification Method: HTML
Notification method:Notification Method:Create Notification with the following parameters:From:Email address: example@fqdn.de
Notification method:Notification Method:Create Notification with the following parameters:From:Display Name: Senders Name
Notification method:Notification Method:Create Notification with the following parameters:Subject for service notification: Check_MK: $HOSTNAME$ - $EVENT_TXT$
Notification method:Notification Method:Create Notification with the following parameters:Subject for service notifications: Check_MK: $HOSTNAME$/$SERVICEDESC$ $EVENT_TXT$
Notification method:Notification Method:Create Notification with the following parameters:Display additional information:
- IP Address of Host
- Absolute Time of Alert
- Additional Plugin Output
- Acknowledgement Author
- Acknowledgement Comment
- Notification Author
- Notification Comment
- Time series graph
Notification method:Notification Method:Create Notification with the following parameters:Display graphs among each other
Contact selection: Notify all contacts of the notified host or service.
8

I was not able to fix it. So I restored Check_MK 2.1.

9

Hello,

For additional informations, since upgrading in 2.2.0p2 we also get 4 times every notifications.
We did not have this behaviour in 2.1.0p26.
Basically it looks like every check that match a notification rule send a notification once the state of the service is hard.

10

Maybe I’m wrong, but it looked to me, that there where always n+1 mails send, where n=“number of all recipients for this check”.

11

I have the same issues.
I’m running two Installations. The first one has only one user which receives two notifications, the second has two users which receive three notifications every time.
In both cases the notification rule is set to only notify those specific users but in the notify.log i can see two/three identical messages.

I did a plain installation of cmk cre 2.2.0p3 on a test system. Notifications worked as expected.
So i guess it must be some setting in the upgraded installations causing the issue which isn’t very obvious.

12

If you talk about that you have got two installations, do you mean two seperated installation on different maschines or two sites on one system?
Did you had any problems while upgrading the package to 2.2.x, so you had to start the installation multiple times?

13

Hi,

two seperate Installations on different machines, monitoring different things :slight_smile:
I had no issues upgrading to the 2.2 Version and to later Patches.

14

Hell yeah i fixed it.
After Hours of comparing files between my productive Installation and the test Installation i found the following:

My ~/etc/check_mk/conf.d/wato/contacts.mk looked like this on the productive Installation:

contacts.update({'username': {'alias': 'username', 'notification_period': '24X7', 'service_notification_options': 'wucr', 'notifications_enabled': True, 'contactgroups': [''], 'disable_notifications': {}, 'notification_method': ('flexible', [{'parameters': [], 'disabled': False, 'service_events': ['w', 'u', 'c', 'r', 'f'], 'timeperiod': '24X7', 'host_events': ['d', 'u', 'r', 'f']}]), 'pager': '', 'email': 'username@example.com', 'host_notification_options': 'dur', 'force_authuser_webservice': True, 'user_scheme_serial': 1, 'fallback_contact': False, 'notification_rules': []}})

After deleting the user and recreating it it looked like this:

contacts.update({'username': {'alias': 'username', 'email': 'username@example.com', 'pager': '', 'contactgroups': ['everything'], 'fallback_contact': False, 'disable_notifications': {}, 'user_scheme_serial': 1}})

So it was definitly some legacy configuration which caused the issue in my case.

Hope that helps someone else figure out that issue.

15

Hello,

Thanks for your explanation about this.
However, I did delete my account and created it again, and as soon as I setup some notifications rules for my account I’m back with the same problem as before.
Although I could say that it managed to lower de number of notifications from 4 to 3, so it’s technically better.
Maybe I’m doing it wrong by setting mail notifications.

16

Did you use flexible notifications before?

17

Hello,

I’ve check and I dont see any logs regarding this werk so I think we were not using it.

It feels like every failed check before entering hard state triggers a notifications that is buffered and when it becomes hard state all of it is sent.
I’m saying that because we get 3 mail notifications and the timestamp is spaced by 5 secondes between each.
For example we have :

  • Mon Jul 17 09:18:53 CEST 2023
  • Mon Jul 17 09:18:58 CEST 2023
  • Mon Jul 17 09:19:03 CEST 2023

This is just speculation of course but it’s my best description of what’s happening to us.