No Service Notifications in 2.3.0

k.fritsch · May 3, 2024, 8:54am

CMK version: 2.3.0 Enterprise
OS version: Ubuntu 20.04

Error message:
Hey there,
we log every event to a log file. This mechanism worked fine until 2.2.0p25. After the update, we don’t get any notifications for service events anymore.
Host events are delivered just fine.

What I did to test is use the “Test notification” tool. Using the tool, I get notifications. But when a service status changes, I don’t get a notification.

As shown in the screenshot, there is no Service notification, just the alert.

Is there a new setting in the new checkmk version, preventing the notifications from being sent?

_rb · May 3, 2024, 9:09am

With a quick look i would say the host was down before, so you will not get any service notification for this host.

k.fritsch · May 3, 2024, 9:17am

It was up, when I tested. The Host was perfectly green and fine when I switched one service to CRIT.

I just triggered the state again and the host is up since 30mins. Again the Alert is triggered but not the notification.

The last notification triggered is the host notification.

_rb · May 3, 2024, 9:53am

In your first example, first the host went to state DOWN.

from what version did you update to 2.2.0p25
did you already check ~/var/log/notify.log for hints

k.fritsch · May 6, 2024, 5:45am

Yeah I know, this came from my testing what notifications work and which ones doesn’t. The notify log shows every log that gets delivered, so it shows the host notifications and the test notifications but the actual service notifications are missing.

k.fritsch · May 6, 2024, 9:49am

I updated from 2.2.0p25 to 2.3.0.

_rb · May 6, 2024, 10:07am

Did you checked the log file i mentioned?

k.fritsch · May 6, 2024, 11:02am

Yes, I did. There is no entry for the service notifications. There should have been different notifications today but none appears in the log.

OMD[spcm]:~/var/log$ tail -100 notify.log
2024-05-06 06:24:03,298 [20] [cmk.base.events] Starting in keepalive mode with PID 2701152
2024-05-06 06:40:49,870 [20] [cmk.base.events] Configuration has changed. Restarting myself.
2024-05-06 06:40:50,575 [20] [cmk.base.events] We are back after a restart.
2024-05-06 06:40:50,580 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 06:40:50,581 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-06 06:40:50,581 [20] [cmk.base.notify] Global rule 'Send Events to Logfile'...
2024-05-06 06:40:50,581 [20] [cmk.base.notify]  -> matches!
2024-05-06 06:40:50,581 [20] [cmk.base.notify]    - adding notification of cmkadmin via log
2024-05-06 06:40:50,581 [15] [cmk.base.notify] Global rule 'Send Events to Logfile'...
2024-05-06 06:40:50,581 [15] [cmk.base.notify]  -> does not match: This is a host notification, but the rule just matches service events
2024-05-06 06:40:50,582 [20] [cmk.base.notify] Executing 1 notifications:
2024-05-06 06:40:50,582 [20] [cmk.base.notify]   * notifying cmkadmin via log, parameters: (no parameters), bulk: no
2024-05-06 06:40:50,582 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/ee085541-b8f1-4199-a9cd-0a57c572a929
2024-05-06 06:40:54,383 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 06:40:54,384 [20] [cmk.base.notify] Got spool file ee085541 (deagstest010) for local delivery via log
2024-05-06 06:40:54,384 [20] [cmk.utils.notify] sending command LOG;HOST NOTIFICATION: cmkadmin;deagstest010;DOWN;log;[agent] Data is encrypted but no secret is known(!!), [piggyback] Successfully processed from source 'deags00030', Missing monitoring data for plugins, apt(!), apt_update(!), certificates(!), checkmk_agent(!), cpu_loads(!), cpu_threads(!), df(!), diskstat(!), dpkg_packages(!), entropy_avail(!), haproxy_frontend(!), haproxy_server(!), http_remote(!), ipv6_status(!), kernel_performance(!), kernel_util(!), lnx_if(!), mem_linux(!), mounts(!), open_ports(!), splunk_read(!), systemd_units_services_summary(!), systemd_units_sockets_summary(!), tcp_conn_stats(!), timesyncd(!), uptime(!), execution time 4.1 sec
2024-05-06 06:40:54,384 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 06:41:54,324 [20] [cmk.base.events] Configuration has changed. Restarting myself.
2024-05-06 06:41:55,035 [20] [cmk.base.events] We are back after a restart.
2024-05-06 06:41:55,040 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 06:41:55,040 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-06 06:41:55,041 [20] [cmk.base.notify] Global rule 'Send Events to Logfile'...
2024-05-06 06:41:55,041 [20] [cmk.base.notify]  -> matches!
2024-05-06 06:41:55,041 [20] [cmk.base.notify]    - adding notification of cmkadmin via log
2024-05-06 06:41:55,041 [15] [cmk.base.notify] Global rule 'Send Events to Logfile'...
2024-05-06 06:41:55,041 [15] [cmk.base.notify]  -> does not match: This is a host notification, but the rule just matches service events
2024-05-06 06:41:55,041 [20] [cmk.base.notify] Executing 1 notifications:
2024-05-06 06:41:55,041 [20] [cmk.base.notify]   * notifying cmkadmin via log, parameters: (no parameters), bulk: no
2024-05-06 06:41:55,041 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/40225213-2091-45d6-b45f-56ea181d224d
2024-05-06 06:41:59,413 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 06:41:59,414 [20] [cmk.base.notify] Got spool file 40225213 (deagstest010) for local delivery via log
2024-05-06 06:41:59,414 [20] [cmk.utils.notify] sending command LOG;HOST NOTIFICATION: cmkadmin;deagstest010;UP;log;[agent] Success, [piggyback] Successfully processed from source 'deags00030', execution time 4.7 sec
2024-05-06 06:41:59,415 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log

_rb · May 6, 2024, 11:29am

Quick test looks good after updating (i would assume we would have get multiple reports that this is not working any more in 2.3)

Only thing i can think of is that service notification were disabled. But this would need the permission for the command (disabled by default).

Do you have such an icon (the last one) in a service view for a service you would have expected a notification?

k.fritsch · May 6, 2024, 1:05pm

I cannot find any service with the notifcations disabled button.

I just rechecked the update log and there is no warning or problem at all.

k.fritsch · May 6, 2024, 1:38pm

I can expand my obersavtions a little bit.

I found out, that the event console service generates its events just fine. But only those, every other notification is missing. Do I miss something?

See this Nofity.log file:

2024-05-06 13:19:01,803 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:19:01,803 [20] [cmk.base.notify] Got raw notification (deagstest058;OMD spcm Event Console) context with 73 variables
2024-05-06 13:19:01,804 [20] [cmk.base.notify] Global rule 'Send Events to Logfile'...
2024-05-06 13:19:01,804 [20] [cmk.base.notify]  -> matches!
2024-05-06 13:19:01,804 [20] [cmk.base.notify]    - adding notification of automation, cmkadmin via log
2024-05-06 13:19:01,804 [20] [cmk.base.notify] Executing 1 notifications:
2024-05-06 13:19:01,804 [20] [cmk.base.notify]   * notifying automation, cmkadmin via log, parameters: (no parameters), bulk: no
2024-05-06 13:19:01,804 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/0c7bb5c9-c741-404d-8371-4dff9e9a490a
2024-05-06 13:19:01,807 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/3d190515-60c7-49c7-9d2f-98183d6c834d
2024-05-06 13:19:01,809 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/a9401483-8a47-48fc-ba04-67a9fec4942e
2024-05-06 13:19:01,811 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/4cd8f620-4581-48ce-ac82-e7993245db10
2024-05-06 13:19:01,813 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/8218a60b-731c-4b8b-a03f-64137747e910
2024-05-06 13:19:01,815 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/73897eb5-3c47-4f49-a128-057ff8e06d3b
2024-05-06 13:19:05,627 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:19:05,628 [20] [cmk.base.notify] Got spool file 3d190515 (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:19:05,628 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: automation;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:19:05,629 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:19:06,395 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:19:06,396 [20] [cmk.base.notify] Got spool file 0c7bb5c9 (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:19:06,396 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: fritska;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:19:06,397 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:19:07,183 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:19:07,184 [20] [cmk.base.notify] Got spool file a9401483 (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:19:07,184 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: cmkadmin;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:19:07,185 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:19:07,960 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:19:07,961 [20] [cmk.base.notify] Got spool file 4cd8f620 (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:19:07,961 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: negreda;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:19:07,961 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:19:08,801 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:19:08,802 [20] [cmk.base.notify] Got spool file 8218a60b (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:19:08,802 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: fandrse;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:19:08,802 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:19:09,944 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:19:09,945 [20] [cmk.base.notify] Got spool file 73897eb5 (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:19:09,945 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: metzglu;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:19:09,946 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:28:42,674 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:28:42,675 [20] [cmk.base.notify] Got raw notification (deagstest058;OMD spcm Event Console) context with 73 variables
2024-05-06 13:28:42,675 [20] [cmk.base.notify] Global rule 'Send Events to Logfile'...
2024-05-06 13:28:42,675 [20] [cmk.base.notify]  -> matches!
2024-05-06 13:28:42,675 [20] [cmk.base.notify]    - adding notification of fritska, automation, cmkadmin, negreda, fandrse, metzglu via log
2024-05-06 13:28:42,676 [20] [cmk.base.notify] Executing 1 notifications:
2024-05-06 13:28:42,676 [20] [cmk.base.notify]   * notifying cmkadmin via log, parameters: (no parameters), bulk: no
2024-05-06 13:28:42,676 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/5b969158-d453-45e7-ab95-5970cf0a709e
2024-05-06 13:28:42,678 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/d84e54fa-d562-4f95-a1ec-a64fef093f80
2024-05-06 13:28:42,680 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/754ac2bc-e9f7-473b-9d9b-47ae53032370
2024-05-06 13:28:42,695 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/7a2aeffe-e19e-452a-b2b1-7ceebc9de6aa
2024-05-06 13:28:42,697 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/0d2633a3-a683-4573-beb0-f8e945fe6de1
2024-05-06 13:28:42,700 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/e671bc97-3431-4679-97fe-ab86d209e0a3
2024-05-06 13:28:46,505 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:28:46,506 [20] [cmk.base.notify] Got spool file d84e54fa (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:28:46,506 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: automation;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:28:46,506 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:28:47,278 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:28:47,279 [20] [cmk.base.notify] Got spool file 5b969158 (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:28:47,279 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: fritska;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:28:47,280 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:28:48,050 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:28:48,051 [20] [cmk.base.notify] Got spool file 754ac2bc (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:28:48,051 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: cmkadmin;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:28:48,051 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:28:48,831 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:28:48,833 [20] [cmk.base.notify] Got spool file 7a2aeffe (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:28:48,833 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: negreda;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:28:48,833 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:28:49,656 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:28:49,657 [20] [cmk.base.notify] Got spool file 0d2633a3 (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:28:49,657 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: fandrse;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:28:49,658 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log
2024-05-06 13:28:50,727 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-06 13:28:50,728 [20] [cmk.base.notify] Got spool file e671bc97 (deagstest058;OMD spcm Event Console) for local delivery via log
2024-05-06 13:28:50,728 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: metzglu;deagstest058;OMD spcm Event Console;UNKNOWN;log;Item not found in monitoring data
2024-05-06 13:28:50,729 [20] [cmk.base.notify]      executing /omd/sites/spcm/local/share/check_mk/notifications/log

And this screenshot:

k.fritsch · May 7, 2024, 11:57am

I did just setup a completly new instance with a new site. I configured just one host only to the the notifications.

The only rules configured are

symmetric encryption
host status via CheckMK Agent
Always notify cmkadmin

No notifications for services are sent out. How did you test the notifications on your side? Did I forget a setting for service alerts to trigger notifications?

The Warn Notififcation is triggered via “Test notifications” and works fine, i can find the entry of the tool in the nofify.log.
The crit alert is triggered via an actual event on the host, here I don’t see any notifcation and no entry in the log at all.

Notify.log:

OMD[spcm]:~/var/log$ cat notify.log 
2024-05-07 11:10:48,019 [20] [cmk.base.events] Starting in keepalive mode with PID 1096709
2024-05-07 11:11:42,556 [20] [cmk.base.events] CMC has closed the connection. Shutting down.
2024-05-07 11:12:57,529 [20] [cmk.base.events] Starting in keepalive mode with PID 1099878
2024-05-07 11:17:41,339 [20] [cmk.base.events] Configuration has changed. Restarting myself.
2024-05-07 11:17:42,070 [20] [cmk.base.events] We are back after a restart.
2024-05-07 11:17:42,072 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:17:42,072 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-07 11:17:42,072 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:17:42,072 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:17:42,072 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2024-05-07 11:18:24,489 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:18:24,489 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-07 11:18:24,489 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:18:24,489 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:18:24,490 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2024-05-07 11:18:41,490 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:18:41,490 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-07 11:18:41,491 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:18:41,491 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:18:41,491 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2024-05-07 11:19:29,764 [20] [cmk.base.events] Configuration has changed. Restarting myself.
2024-05-07 11:19:30,450 [20] [cmk.base.events] We are back after a restart.
2024-05-07 11:19:30,452 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:19:30,452 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-07 11:19:30,453 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:19:30,453 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:19:30,453 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2024-05-07 11:19:47,489 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:19:47,490 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-07 11:19:47,490 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:19:47,490 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:19:47,490 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2024-05-07 11:20:41,494 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:20:41,494 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-07 11:20:41,495 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:20:41,495 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:20:41,495 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2024-05-07 11:31:45,836 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:31:45,836 [20] [cmk.base.notify] Analysing notification (deagstest011;HAProxy Server ca-adm-stateful/ca-1) context with 57 variables
2024-05-07 11:31:45,836 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:31:45,836 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:31:45,836 [20] [cmk.base.notify] Warning: Contacts of deagstest011;HAProxy Server ca-adm-stateful/ca-1 cannot be determined. Using fallback contacts
2024-05-07 11:31:45,836 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2024-05-07 11:32:00,576 [20] [cmk.base.events] Configuration has changed. Restarting myself.
2024-05-07 11:32:01,337 [20] [cmk.base.events] We are back after a restart.
2024-05-07 11:32:01,340 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:32:01,340 [20] [cmk.base.notify] Got raw notification (deagstest010) context with 44 variables
2024-05-07 11:32:01,341 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:32:01,341 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:32:01,341 [20] [cmk.base.notify] 1 rules matched, but no notification has been created.
2024-05-07 11:34:39,400 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:34:39,400 [20] [cmk.base.notify] Analysing notification (deagstest011) context with 45 variables
2024-05-07 11:34:39,400 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:34:39,400 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:34:39,400 [20] [cmk.base.notify]    - adding notification of cmkadmin via mail
2024-05-07 11:34:39,400 [20] [cmk.base.notify] Executing 1 notifications:
2024-05-07 11:34:39,400 [20] [cmk.base.notify]   * notifying cmkadmin via mail, parameters: graphs_per_notification, notifications_with_graphs, bulk: no
2024-05-07 11:34:39,400 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/4df27c1c-facc-4f28-a6dd-49c5f010fa08
2024-05-07 11:34:43,318 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:34:43,319 [20] [cmk.base.notify] Got spool file 4df27c1c (deagstest011) for local delivery via mail
2024-05-07 11:34:43,319 [20] [cmk.utils.notify] sending command LOG;HOST NOTIFICATION: cmkadmin;deagstest011;DOWN;mail;
2024-05-07 11:34:43,319 [20] [cmk.base.notify]      executing /omd/sites/spcm/share/check_mk/notifications/mail
2024-05-07 11:34:43,687 [20] [cmk.base.notify]      Output: Cannot send HTML email: empty destination email address
2024-05-07 11:34:43,735 [20] [cmk.base.notify]      Plugin exited with code 2
2024-05-07 11:35:13,521 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:35:13,521 [20] [cmk.base.notify] Analysing notification (deagstest011;HAProxy Server ca-adm-stateful/ca-1) context with 57 variables
2024-05-07 11:35:13,522 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2024-05-07 11:35:13,522 [20] [cmk.base.notify]  -> matches!
2024-05-07 11:35:13,522 [20] [cmk.base.notify]    - adding notification of cmkadmin via mail
2024-05-07 11:35:13,522 [20] [cmk.base.notify] Executing 1 notifications:
2024-05-07 11:35:13,522 [20] [cmk.base.notify]   * notifying cmkadmin via mail, parameters: graphs_per_notification, notifications_with_graphs, bulk: no
2024-05-07 11:35:13,522 [20] [cmk.base.notify] Creating spoolfile: /omd/sites/spcm/var/check_mk/notify/spool/0430d621-3637-4dc2-bec4-f86091597e47
2024-05-07 11:35:17,654 [20] [cmk.base.notify] ----------------------------------------------------------------------
2024-05-07 11:35:17,654 [20] [cmk.base.notify] Got spool file 0430d621 (deagstest011;HAProxy Server ca-adm-stateful/ca-1) for local delivery via mail
2024-05-07 11:35:17,655 [20] [cmk.utils.notify] sending command LOG;SERVICE NOTIFICATION: cmkadmin;deagstest011;HAProxy Server ca-adm-stateful/ca-1;WARNING;mail;
2024-05-07 11:35:17,655 [20] [cmk.base.notify]      executing /omd/sites/spcm/share/check_mk/notifications/mail
2024-05-07 11:35:18,088 [20] [cmk.base.notify]      Output: Cannot send HTML email: empty destination email address
2024-05-07 11:35:18,140 [20] [cmk.base.notify]      Plugin exited with code 2

aeckstein · May 7, 2024, 12:02pm

You have not configured an e-mail address for your contact…

k.fritsch · May 7, 2024, 12:02pm

I now, but there should be a event in the log file. But there isn’t. The behavior is excactly the same when I configure a mail or use any other notification plugin.

k.fritsch · May 7, 2024, 12:07pm

To clarify: I do get notifcations when using the “Test notifcation” button and this is what you can see in the notify.log. But on a real event there is no notification and no entry in the Log

aeckstein · May 7, 2024, 12:11pm

If you have not configured one of the following rules

Maximum number of check attempts for Hosts|Services
delay host|service notifications
Notified Events for Hosts|Services

the notification should be instantly created and be be visible in the analysis in setup - notifications.
What is “Service Notifications” of that service showing after the event when you click on the affected service and is there a “Notification postponement reason” in the table of that service ?

k.fritsch · May 7, 2024, 12:22pm

This is what the events table looks like:

And this is the nofitifaction table:

As you can see, only the “Test Event” triggered the notification.

k.fritsch · May 8, 2024, 6:38am

Just updated to 2.3.0p1 because there was something about notify.log in the changes but it did not help solve the notifcation problem.

CFriedrich · May 8, 2024, 7:52am

Hi,
can you please post your notification rule and, if you have several, an overview of the rules?

k.fritsch · May 8, 2024, 9:35am

Here you go:

And this is the plugin:

#!/bin/bash
# Log Event to Logfile

LOGFILE=/var/log/checkmk/checkmk.log
echo "New Event" >>  $LOGFILE
if [ $NOTIFY_WHAT == SERVICE ]
then
    echo $NOTIFY_LONGDATETIME type=\"SERVICE\" address=\"$NOTIFY_HOST_ADDRESS_4\" alias=\"$NOTIFY_HOSTALIAS\" hostname=\"$NOTIFY_HOSTNAME\" service=\"$NOTIFY_SERVICEDESC\" state=\"$NOTIFY_SERVICESTATE\" output=\"$NOTIFY_SERVICEOUTPUT\" >> $LOGFILE
elif [ $NOTIFY_WHAT == HOST ]
then
    echo $NOTIFY_LONGDATETIME type=\"HOST\" address=\"$NOTIFY_HOSTADDRESS\" alias=\"$NOTIFY_HOSTALIAS\" hostname=\"$NOTIFY_HOSTNAME\" state=\"$NOTIFY_HOSTSTATE\" output=\"$NOTIFY_HOSTOUTPUT\" >> $LOGFILE
fi

exit 0