Some time ago we set up notifications based with ms-teams plugin and used service label filter to decide notifiable services, it was working before but looks like recently it started not to work, not sure what caused it to not work. Following is log where I see rule not matched
-> does not match: The service labels {'super_important': 'yes'} did not match {}
while on ui I see the service label, any ideas? how we can better debug it
Update: notifications works for some services, but not some. Despite both having same labels, I wonder what could be reason to see service label on ui but on logs it is clear that service labels are empty (did not match {})
Update2: I just tried to create a service group using service labels, it is also not working properly. Despite having many service with the defined label, service group discovers only a few
Update3: as workaround created a contact group and with the help of “Assignment of services to contact groups” assigned services with defined labels to contact group. then used contact group of notifications rule match. But even this one not worked as expected, it alerts many other things not just services part of contact group, expected?
Hallo,
don t know if it is a problem but {‘super_important’: ‘yes’} contains an blank.
Otherwise controlle your rules if there are any conditions that surpress the expected notification.
Ralf
But it clearly states “did not match {}” so service labels seen as empty while from ui I see the label, it is also weird why it works for some. Any local commands to debug the issue would be helpful. Commands like listing service labels, where service labels kept so I can check locally, maybe clear caches of them. If they supposed to be passed to notification channel by ui then how I investigate what possible goes wrong on ui side. Generally I lack information how to debug/research the situation
Update: could it be that “custom notifications” feature not sends service labels along with notification?
if you need devs to look into it, you’ll need
a) a lot more people able to reproduce this
b) a support contract with checkmk enterprise.
for a) I guess reproducing it might be difficult, since you said it works for some services but not for all, but can you reproduce this in a minimal new environment where you would then be able to share the complete config?
I can confirm this issue too.
To investigate (2.2.0p6 CRE Ubuntu 22.04) I labeled all services from one host with the specific notification label that matches my notification rule.
Checks done by Check MK Agent work, active checks do not e.g. SSH or even the Check_MK check.
I am experiencing a similar issue. With an additional wrinkle.
Service label is applied to the service (verified in GUI), however notify.log shows ‘did not match {}’.
The wrinkle is that it only affects distributed CheckMK instances. This is where I can see the processing in the notify.log. I have tried instigating the notification from both the primay and remote instance of Check_MK.
All notifications for the primary CheckMK instance work for services with the same service label applied. One difference is that the remote instance services are applied with a separate rule.
Working Rule applies the label to a Folder and with a list of services. All of these services are monitored directly by the primary CheckMK instance.
Non-working Rule applies the label to a different folder with a list of exactly one service, Check_MK. All of these services are monitored by 1 of 4 remote Check_MK instances.
The same
I am mostly posting to provide further details in case that helps with a fix. I can provide more details if necesary.
I have the same problem where notification based on service label did not match on 2.2.0p1 RAW edition.
I updated to last minor 2.2.0p19 but the problem persists.
After many tests I fount that notification did not match only on active checks while passive checks over check mk agent match and work correctly.
I can’t reproduce a problem with service labels as condition in notifications on 2.2.p17.cce Cloud Edition.
For me service notifications using a service label as condition just work as expected.
I tried with an active HTTP check.
Even if it’s on a remote site and the remote forwards the notifications to a central site it works for me.
If you test your notifications with fake check results you have to make sure
you have Maximum number of check attempts for service set to 1:
Current check attempt 1/1
… otherwise you would have to fake a check result n-times until it enters the hard state.
The notification script even knows the service label as environment variable like this:
NOTIFY_SERVICELABEL_foobar=barfoo
Looks like it could be a issue only for the CRE Raw Edition.
Yes, looks like it could be a issue only for the Raw Edition.
For testing purposes I made a clean installation of check MK RAW edition on new server debian 12 with Checkmk Raw Edition 2.2.0p19 and the problem persists. Only passive check match the notification with service label.
Has anyone found a solution by now? I’m currently on 2.3.0 (raw) and still encounter this problem.
Agent based notifications (e.g. NetApp agent, checkmk agent), SNMP etc. seem to work fine with the service label notifications.
But active checks (such as HTTP) ignore the notification rule even though the labels are present and the configuration looks the same as on the working hosts/services.
Currently I use the workaround to create new notification rules for such services but that does not seem like a good solution to me.
On 2.3.0p26 if I use the test notification for active check with service label all work correctly and the correct rule was identified.
But for the same active check if I use the “fake check result” function to force the service in critical or warning status the correct notification was ignored.
2025-05-12 11:55:29,967 [20] [cmk.base.notify] Got raw notification (AMONTCENTRAL01;Plis OCR Azure 2 en pause) context with 54 variables
2025-05-12 11:55:29,969 [15] [cmk.base.notify] Global rule ‘Testing Astreinte’…
2025-05-12 11:55:29,969 [15] [cmk.base.notify] → does not match: The service labels {‘notification’: ‘astreinte’} did not match {}
