Different contexts for test and real notifications (so rules does not work)

admin-linux-it · February 7, 2025, 10:37am

CMK version: Checkmk Raw Edition 2.3.0p21
OS version: Debian 12

Problem: Notification condition does not work

When I test notification it shows such information as:

SERVICELABEL_notify
SERVICECONTACTGROUPNAMES
HOSTCONTACTGROUPNAMES

Test also indicates that “Rule no. X matches” and collects valid notification contacts.

However when real notification fires this info is not visible in the context, thus “Rule no. X” does not match and notification is not sent:

I’ve checked “Host monitoring rules” and “Service monitoring rules” as well: labels and contact-groups are applied correctly (I attach screen for host only to no clutter this post too much):

Is it a bug or I missed something?

admin-linux-it · February 7, 2025, 11:43am

Info from notify.log

Test notification:

2025-02-07 10:17:00,410 [20] [cmk.base.notify] Analysing notification (gru**********;Alert docker-hub) context with 53 variables
2025-02-07 10:17:00,410 [15] [cmk.base.notify] Global rule 'Slack (hosts): logger-checkmk-microsoft-it'...
2025-02-07 10:17:00,410 [15] [cmk.base.notify]  -> does not match: This is a service notification, but the rule just matches host events
2025-02-07 10:17:00,411 [15] [cmk.base.notify] Global rule 'Slack (services): logger-checkmk-microsoft-it'...
2025-02-07 10:17:00,411 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-microsoft-it'} did not match {'notify': 'logger-team-sdlc'}
2025-02-07 10:17:00,411 [15] [cmk.base.notify] Global rule 'Slack (hosts): logger-admin-linux-it'...
2025-02-07 10:17:00,411 [15] [cmk.base.notify]  -> does not match: This is a service notification, but the rule just matches host events
2025-02-07 10:17:00,411 [15] [cmk.base.notify] Global rule 'Slack (services): logger-admin-linux-it'...
2025-02-07 10:17:00,411 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-linux-it'} did not match {'notify': 'logger-team-sdlc'}
2025-02-07 10:17:00,411 [15] [cmk.base.notify] Global rule 'Slack (services): alerts-vault-unity-pl'...
2025-02-07 10:17:00,411 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-vault'} did not match {'notify': 'logger-team-sdlc'}
2025-02-07 10:17:00,411 [20] [cmk.base.notify] Global rule 'Slack (services): logger-team-sdlc'...
2025-02-07 10:17:00,411 [20] [cmk.base.notify]  -> matches!
2025-02-07 10:17:00,412 [20] [cmk.base.notify]    - adding notification of ab***** via slack
2025-02-07 10:17:00,412 [15] [cmk.base.notify] Global rule 'Slack (services): logger-unicoit'...
2025-02-07 10:17:00,412 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'logger-unicoit'} did not match {'notify': 'logger-team-sdlc'}
2025-02-07 10:17:00,412 [15] [cmk.base.notify] Global rule 'Slack (services): logger-team-app'...
2025-02-07 10:17:00,412 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-app'} did not match {'notify': 'logger-team-sdlc'}
2025-02-07 10:17:00,412 [15] [cmk.base.notify] Global rule 'email HTML (services): admin-app'...
2025-02-07 10:17:00,412 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-app'} did not match {'notify': 'logger-team-sdlc'}
2025-02-07 10:17:00,412 [15] [cmk.base.notify] Global rule 'Suppress unwanted states (Slack)'...
2025-02-07 10:17:00,412 [15] [cmk.base.notify]  -> does not match: Event type 'rc' not handled by this rule. Allowed are: rr, rw, wr, cw, cu, uw, f, s, x, as, af
2025-02-07 10:17:00,413 [15] [cmk.base.notify] Global rule 'Suppress unwanted states (mail)'...
2025-02-07 10:17:00,413 [15] [cmk.base.notify]  -> does not match: Event type 'rc' not handled by this rule. Allowed are: rr, rw, wr, cw, cu, uw, f, s, x, as, af
2025-02-07 10:17:00,413 [20] [cmk.base.notify] Executing 1 notifications:
2025-02-07 10:17:00,413 [20] [cmk.base.notify]   * would notify ab***** via slack, parameters: webhook_url, bulk: no

Real notification:

2025-02-07 10:15:46,253 [20] [cmk.base.notify] Got raw notification (gru************;Alert docker-hub) context with 54 variables
2025-02-07 10:15:46,255 [20] [cmk.base.events] Previous service hard state not known. Allowing all states.
2025-02-07 10:15:46,255 [15] [cmk.base.notify] Global rule 'Slack (hosts): logger-checkmk-microsoft-it'...
2025-02-07 10:15:46,255 [15] [cmk.base.notify]  -> does not match: This is a service notification, but the rule just matches host events
2025-02-07 10:15:46,255 [15] [cmk.base.notify] Global rule 'Slack (services): logger-checkmk-microsoft-it'...
2025-02-07 10:15:46,255 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-microsoft-it'} did not match {}
2025-02-07 10:15:46,255 [15] [cmk.base.notify] Global rule 'Slack (hosts): logger-admin-linux-it'...
2025-02-07 10:15:46,256 [15] [cmk.base.notify]  -> does not match: This is a service notification, but the rule just matches host events
2025-02-07 10:15:46,256 [15] [cmk.base.notify] Global rule 'Slack (services): logger-admin-linux-it'...
2025-02-07 10:15:46,256 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-linux-it'} did not match {}
2025-02-07 10:15:46,256 [15] [cmk.base.notify] Global rule 'Slack (services): alerts-vault-unity-pl'...
2025-02-07 10:15:46,256 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-vault'} did not match {}
2025-02-07 10:15:46,256 [15] [cmk.base.notify] Global rule 'Slack (services): logger-team-sdlc'...
2025-02-07 10:15:46,256 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'logger-team-sdlc'} did not match {}
2025-02-07 10:15:46,256 [15] [cmk.base.notify] Global rule 'Slack (services): logger-unicoit'...
2025-02-07 10:15:46,256 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'logger-unicoit'} did not match {}
2025-02-07 10:15:46,256 [15] [cmk.base.notify] Global rule 'Slack (services): logger-team-app'...
2025-02-07 10:15:46,256 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-app'} did not match {}
2025-02-07 10:15:46,256 [15] [cmk.base.notify] Global rule 'email HTML (services): admin-app'...
2025-02-07 10:15:46,256 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-app'} did not match {}
2025-02-07 10:15:46,256 [15] [cmk.base.notify] Global rule 'Suppress unwanted states (Slack)'...
2025-02-07 10:15:46,256 [15] [cmk.base.notify]  -> does not match: Event type 'cr' not handled by this rule. Allowed are: rr, rw, wr, cw, cu, uw, f, s, x, as, af
2025-02-07 10:15:46,257 [15] [cmk.base.notify] Global rule 'Suppress unwanted states (mail)'...
2025-02-07 10:15:46,257 [15] [cmk.base.notify]  -> does not match: Event type 'cr' not handled by this rule. Allowed are: rr, rw, wr, cw, cu, uw, f, s, x, as, af
2025-02-07 10:15:46,257 [20] [cmk.base.notify] No rule matched, notifying fallback contacts
2025-02-07 10:15:46,257 [20] [cmk.base.notify]   Sending email to ['logger-checkmk-*************']
2025-02-07 10:15:46,257 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: mailto:logger-checkmk-*************;gru************;Alert docker-hub;OK;asciimail;Version: HTTP/2.0, Status: 401 Unauthorized
2025-02-07 10:15:46,257 [20] [cmk.base.notify]      executing /omd/sites/itw/share/check_mk/notifications/asciimail
2025-02-07 10:15:46,873 [20] [cmk.base.notify]      Output: Spooled mail to local mail transmission agent
2025-02-07 10:15:46,964 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: mailto:logger-checkmk-*************;gru************;Alert docker-hub;OK;asciimail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent

andreas-doehler · February 7, 2025, 11:52am

That is a notification context from a raw edition or enterprise edition with Nagios core.
The Nagios core does not support all the parameters from the CMC core. You need to modify your selection according to the available parameters from the core.
With test notification the system don’t use the core but “fake” the notification directly inside the GUI with livestatus data. This data is identical over the different editions. But the used cores gives different datasets back in real notification situations.

admin-linux-it · February 7, 2025, 1:41pm

Thanks @andreas-doehler, I was not aware about this differences in cores.

Nevertheless, the same host, the same settings but different service - context is exactly the same in GUI (so I do not post a screenshot) but in notify.log the label is passed

2025-02-07 14:24:15,946 [20] [cmk.base.notify] Got raw notification (gru********;Alert Rule FilesystemAlmostFull) context with 54 variables
2025-02-07 14:24:15,947 [20] [cmk.base.events] Previous service hard state not known. Allowing all states.
2025-02-07 14:24:15,948 [15] [cmk.base.notify] Global rule 'Slack (hosts): logger-checkmk-microsoft-it'...
2025-02-07 14:24:15,948 [15] [cmk.base.notify]  -> does not match: This is a service notification, but the rule just matches host events
2025-02-07 14:24:15,948 [15] [cmk.base.notify] Global rule 'Slack (services): logger-checkmk-microsoft-it'...
2025-02-07 14:24:15,948 [15] [cmk.base.notify]  -> does not match: The service labels {'notify': 'admin-microsoft-it'} did not match {'notify': 'logger-team-sdlc'}
(...)
2025-02-07 14:24:15,949 [20] [cmk.base.notify] Global rule 'Slack (services): logger-team-sdlc'...
2025-02-07 14:24:15,949 [20] [cmk.base.notify]  -> matches!
2025-02-07 14:24:15,949 [20] [cmk.base.notify]    - adding notification of ab***** via slack
(...)
2025-02-07 14:24:15,949 [20] [cmk.base.notify] Executing 1 notifications:
2025-02-07 14:24:15,950 [20] [cmk.base.notify]   * notifying ab***** via slack, parameters: webhook_url, bulk: no
2025-02-07 14:24:15,950 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: ab*****;gru********;Alert Rule FilesystemAlmostFull;OK;slack;Severity: None, Group name: Node
2025-02-07 14:24:15,950 [20] [cmk.base.notify]      executing /omd/sites/itw/share/check_mk/notifications/slack
2025-02-07 14:24:16,896 [20] [cmk.base.notify]      Output: 200: OK
2025-02-07 14:24:16,896 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION RESULT: ab*****;gru********;Alert Rule FilesystemAlmostFull;OK;slack;200: OK;200: OK

Does that mean that in RAW edition with Nagios core I should not relay on lables?

If so - can you advise me what to use to dispatch notifications between teams?
I tried “Contact groups” matching, but this is also not available in the raw context

admin-linux-it · September 23, 2025, 12:19pm

For anyone struggling with similar issue, I’ve found a working solution:

Create host and service groups using already existing labels
Reconfigure notification rules to use this host/service groups (instead of labels)

Service/Host group is always passed in context so notifications work as expected now.

Cheers!