Good Morning,
I can provide another update.
I did setup a completely new OS and installed just checkmk-enterprise 2.3.0p1, made the minimal setup steps to connect to one agent and still do not get any service notifications.
Did you also check ~/var/log/cmc.log ?
Otherwise it may be helpful to enable debugging for core and notifications in the global settings and check the logs again
Yes I did, there is also no entry for the servie notification but for the host notification.
The service notifications should be between 6:55 and 7:00. Afterwards I checked if the host notification works.
2024-05-13 06:33:56 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK Discovery;1715582036;TRY
2024-05-13 06:33:57 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK Discovery;1715582037;TRY
2024-05-13 06:35:01 [5] [core 1222] Reloading configuration on-the-fly...
2024-05-13 06:35:01 [5] [main] 0 cached RRD infos migrated, 0 objects vanished, 0 objects changed their RRD format
2024-05-13 06:35:01 [5] [core 1222] migrated 0 hosts and 0 services, created 1 hosts and 31 services
2024-05-13 06:35:01 [5] [core 1222] migrated 0 replies
2024-05-13 06:35:01 [5] [core 1222] loaded configuration 3 (0x1a87940) from 2024-05-13 06:35:01 with 1 hosts and 31 services in 13.7166 ms
2024-05-13 06:35:01 [5] [main] [RRD helper 1246] reload configuration
2024-05-13 06:35:01 [5] [core 1222] deleting configuration 2 (0x18f7360) from 2024-05-08 12:54:25 with 0 hosts and 0 services
2024-05-13 06:35:01 [5] [rrdcached] [rrdcached at "/omd/sites/spcm/tmp/run/rrdcached.sock"] settings unchanged
2024-05-13 06:35:53 [5] [core 1222] [config cleaner] removed dead configuration directory "/omd/sites/spcm/var/check_mk/core/helper_config/2"
2024-05-13 06:36:11 [5] [rrdcached] [rrdcached at "/omd/sites/spcm/tmp/run/rrdcached.sock"] successfully connected
2024-05-13 06:51:37 [5] [core 1222] Reloading configuration on-the-fly...
2024-05-13 06:51:37 [5] [main] 32 cached RRD infos migrated, 0 objects vanished, 0 objects changed their RRD format
2024-05-13 06:51:37 [5] [core 1222] migrated 1 hosts and 31 services, created 0 hosts and 0 services
2024-05-13 06:51:37 [5] [core 1222] migrated 0 replies
2024-05-13 06:51:37 [5] [core 1222] loaded configuration 4 (0x1acea90) from 2024-05-13 06:51:37 with 1 hosts and 31 services in 0.225664 ms
2024-05-13 06:51:37 [5] [main] [RRD helper 1246] reload configuration
2024-05-13 06:51:37 [5] [core 1222] deleting configuration 3 (0x1a87940) from 2024-05-13 06:35:01 with 1 hosts and 31 services
2024-05-13 06:51:37 [5] [rrdcached] [rrdcached at "/omd/sites/spcm/tmp/run/rrdcached.sock"] settings unchanged
2024-05-13 06:51:54 [5] [core 1222] [config cleaner] removed dead configuration directory "/omd/sites/spcm/var/check_mk/core/helper_config/3"
2024-05-13 07:02:23 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583743
2024-05-13 07:02:23 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:03:08 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583788
2024-05-13 07:03:08 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:03:11 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583791
2024-05-13 07:03:11 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:03:14 [5] [core 1222] Executing external command: LOG;HOST NOTIFICATION: cmkadmin;deagstest010;DOWN;log;[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec
2024-05-13 07:03:14 [5] [core 1222] Executing external command: LOG;HOST NOTIFICATION RESULT: cmkadmin;deagstest010;OK;log;;
2024-05-13 07:03:54 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583834
2024-05-13 07:03:54 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:03:58 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583838
2024-05-13 07:03:58 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:04:44 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583884
2024-05-13 07:04:44 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:04:47 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583887
2024-05-13 07:04:47 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:04:52 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583892
2024-05-13 07:04:52 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:04:54 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583894
2024-05-13 07:04:54 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:04:57 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583897
2024-05-13 07:04:57 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:05:09 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583909
2024-05-13 07:05:09 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:05:19 [5] [core 1222] Executing external command: SCHEDULE_FORCED_SVC_CHECK;deagstest010;Check_MK;1715583919
2024-05-13 07:05:19 [5] [core 1222] Manually scheduling check for service "deagstest010;Check_MK"
2024-05-13 07:05:19 [5] [core 1222] Executing external command: LOG;HOST NOTIFICATION: cmkadmin;deagstest010;UP;log;[agent] Success, [piggyback] Success (but no data found for this host), execution time 30.9 sec
2024-05-13 07:05:19 [5] [core 1222] Executing external command: LOG;HOST NOTIFICATION RESULT: cmkadmin;deagstest010;OK;log;;
I switched notifications and core to debug and there is the entry for the service problem and no notification either.
2024-05-13 07:54:48 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:48 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:48 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:49 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:49 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:49 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:50 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:50 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:50 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:51 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:51 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:51 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": hard state change to OK
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": notification number 0 is already pending
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": cancelled: state changed back to OK
2024-05-13 07:54:52 [7] [core 1222] sending check result from service "deagstest010;Check_MK" to host "deagstest010"
2024-05-13 07:54:52 [7] [core 1222] released SerialToken{Request[deagstest010],5} => SerialTokenFactory{5:1}
2024-05-13 07:54:52 [7] [core 1222] scheduling service "deagstest010;Check_MK" at 2024-05-13 07:55:52 with commandline [@cmk deagstest010 10.19.68.10]
2024-05-13 07:54:52 [7] [core 1222] [fetcher pool scheduler] scheduling service "deagstest010;Check_MK" at 2024-05-13 07:55:52
2024-05-13 07:54:53 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:53 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:54 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
If I do the same for the whole host, I get the log entries I wanted to see:
2024-05-13 07:56:58 [7] [core 1222] [fetcher pool scheduler] unscheduling service "deagstest010;Check_MK" at 2024-05-13 07:56:58
2024-05-13 07:56:58 [7] [core 1222] obtained SerialToken{Request[deagstest010],5} => SerialTokenFactory{5:2}
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;Check_MK": hard state change to CRITICAL
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;Check_MK": setting up, problem ID 11, delay 0sec
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, last host check not recent enough
2024-05-13 07:56:58 [7] [core 1222] sending check result from service "deagstest010;Check_MK" to host "deagstest010"
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": hard state change to DOWN
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": setting up, problem ID 12, delay 0sec
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": sending PROBLEM notification to its contacts
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": spooling notification to rule based notifications
2024-05-13 07:56:58 [7] spooled 1362 byte event: 'CONTACTS=
NOTIFICATIONTYPE=PROBLEM
NOTIFICATIONCOMMENT=
NOTIFICATIONAUTHOR=
NOTIFICATIONAUTHORNAME=
NOTIFICATIONAUTHORALIAS=
HOSTACKAUTHOR=
HOSTACKCOMMENT=
MICROTIME=1715587018173149
HOSTNOTIFICATIONNUMBER=1
LASTHOSTPROBLEMID=12
HOSTPROBLEMID=12
HOSTATTEMPT=1
MAXHOSTATTEMPTS=1
HOSTNAME=deagstest010
HOSTALIAS=dev-ca-1
HOSTADDRESS=10.19.68.10
HOSTOUTPUT=[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec
HOSTPERFDATA= execution_time=0.010 user_time=0.020 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000
HOSTSTATE=DOWN
HOSTSTATEID=1
LASTHOSTSTATE=UP
LASTHOSTSTATEID=0
PREVIOUSHOSTHARDSTATE=UP
PREVIOUSHOSTHARDSTATEID=0
LONGHOSTOUTPUT=
HOSTCHECKCOMMAND=check-mk-host-service
LASTHOSTSTATECHANGE=1715587018
LASTHOSTUP=1715586958
HOSTDOWNTIME=0
HOSTGROUPNAMES=check_mk
HOSTCONTACTGROUPNAMES=all
HOSTTAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_SL=
HOST_EC_CONTACT=
HOSTNOTESURL=
HOSTNOTES=
HOST_FILENAME=/wato/hosts.mk
HOST_ADDRESS_6=
HOST_TAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_ADDRESSES_6=
HOST_ADDRESS_4=10.19.68.10
HOST_ADDRESS_FAMILY=4
HOST_ADDRESSES_4=
'
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": cancelled: no more notifications necessary
2024-05-13 07:56:58 [7] [core 1222] released SerialToken{Request[deagstest010],5} => SerialTokenFactory{5:1}
2024-05-13 07:56:58 [7] [core 1222] scheduling service "deagstest010;Check_MK" at 2024-05-13 07:57:58 with commandline [@cmk deagstest010 10.19.68.10]
2024-05-13 07:56:58 [7] [core 1222] [fetcher pool scheduler] scheduling service "deagstest010;Check_MK" at 2024-05-13 07:57:58
2024-05-13 07:56:58 [7] unspooled 1362 byte event: 'CONTACTS=
NOTIFICATIONTYPE=PROBLEM
NOTIFICATIONCOMMENT=
NOTIFICATIONAUTHOR=
NOTIFICATIONAUTHORNAME=
NOTIFICATIONAUTHORALIAS=
HOSTACKAUTHOR=
HOSTACKCOMMENT=
MICROTIME=1715587018173149
HOSTNOTIFICATIONNUMBER=1
LASTHOSTPROBLEMID=12
HOSTPROBLEMID=12
HOSTATTEMPT=1
MAXHOSTATTEMPTS=1
HOSTNAME=deagstest010
HOSTALIAS=dev-ca-1
HOSTADDRESS=10.19.68.10
HOSTOUTPUT=[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec
HOSTPERFDATA= execution_time=0.010 user_time=0.020 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000
HOSTSTATE=DOWN
HOSTSTATEID=1
LASTHOSTSTATE=UP
LASTHOSTSTATEID=0
PREVIOUSHOSTHARDSTATE=UP
PREVIOUSHOSTHARDSTATEID=0
LONGHOSTOUTPUT=
HOSTCHECKCOMMAND=check-mk-host-service
LASTHOSTSTATECHANGE=1715587018
LASTHOSTUP=1715586958
HOSTDOWNTIME=0
HOSTGROUPNAMES=check_mk
HOSTCONTACTGROUPNAMES=all
HOSTTAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_SL=
HOST_EC_CONTACT=
HOSTNOTESURL=
HOSTNOTES=
HOST_FILENAME=/wato/hosts.mk
HOST_ADDRESS_6=
HOST_TAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_ADDRESSES_6=
HOST_ADDRESS_4=10.19.68.10
HOST_ADDRESS_FAMILY=4
HOST_ADDRESSES_4=
'
2024-05-13 07:56:58 [7] [notification helper 1275] successfully sent event
2024-05-13 07:56:59 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:56:59 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:56:59 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:00 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:00 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:57:00 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:01 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:01 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:57:01 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:02 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:02 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:57:02 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:03 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:03 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:57:03 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:03 [7] [core 1222] [livestatus external] spooled command 'LOG;HOST NOTIFICATION: cmkadmin;deagstest010;DOWN;log;[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec'
2024-05-13 07:57:03 [5] [core 1222] Executing external command: LOG;HOST NOTIFICATION: cmkadmin;deagstest010;DOWN;log;[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec
2024-05-13 07:57:03 [7] [core 1222] [livestatus external] spooled command 'LOG;HOST NOTIFICATION RESULT: cmkadmin;deagstest010;OK;log;;'
2024-05-13 07:57:03 [5] [core 1222] Executing external command: LOG;HOST NOTIFICATION RESULT: cmkadmin;deagstest010;OK;log;;
2024-05-13 07:57:04 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:04 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
The host is not up. That’s the same problem as of the beginning of this thread
That is the problem! Finally “solved”! The notification only works when the host is reachable via ping.
This is how it looks, when the “Host Check Command” Rule is not set:
As we have restriced ping we usually use the agent state as host state and not ping.
2024-05-13 08:15:54 [5] [core 1222] Executing external command: LOG;SERVICE NOTIFICATION: cmkadmin;deagstest010;Check_MK Agent;WARNING;log;Version: 2.3.0p1, OS: linux, TLS is not activated on monitored host (see details)(!), Agent plug-ins: 9, Local checks: 0
2024-05-13 08:15:54 [7] [core 1222] [livestatus external] spooled command 'LOG;SERVICE NOTIFICATION RESULT: cmkadmin;deagstest010;Check_MK Agent;OK;log;;'
2024-05-13 08:15:54 [5] [core 1222] Executing external command: LOG;SERVICE NOTIFICATION RESULT: cmkadmin;deagstest010;Check_MK Agent;OK;log;;
2024-05-13 08:15:55 [6] [core 1222] [state saver] saved state to "/omd/sites/spcm/var/check_mk/core/state.pb" in 0.938977 ms
2024-05-13 08:15:57 [5] [core 1222] [config cleaner] removed dead configuration directory "/omd/sites/spcm/var/check_mk/core/helper_config/5"
2024-05-13 08:16:53 [7] [core 1222] [fetcher pool scheduler] unscheduling service "deagstest010;Check_MK" at 2024-05-13 08:16:53
2024-05-13 08:16:53 [7] [core 1222] obtained SerialToken{Request[deagstest010],6} => SerialTokenFactory{6:2}
2024-05-13 08:16:58 [7] [core 1222] released SerialToken{Request[deagstest010],6} => SerialTokenFactory{6:1}
2024-05-13 08:16:58 [7] [core 1222] scheduling service "deagstest010;Check_MK" at 2024-05-13 08:17:58 with commandline [@cmk deagstest010 10.19.68.10]
2024-05-13 08:16:58 [7] [core 1222] [fetcher pool scheduler] scheduling service "deagstest010;Check_MK" at 2024-05-13 08:17:58
2024-05-13 08:17:58 [7] [core 1222] [fetcher pool scheduler] unscheduling service "deagstest010;Check_MK" at 2024-05-13 08:17:58
2024-05-13 08:17:58 [7] [core 1222] obtained SerialToken{Request[deagstest010],6} => SerialTokenFactory{6:2}
2024-05-13 08:18:04 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": hard state change to CRITICAL
2024-05-13 08:18:04 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": setting up, problem ID 16, delay 0sec
2024-05-13 08:18:04 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:04 [7] [core 1222] released SerialToken{Request[deagstest010],6} => SerialTokenFactory{6:1}
2024-05-13 08:18:04 [7] [core 1222] scheduling service "deagstest010;Check_MK" at 2024-05-13 08:19:04 with commandline [@cmk deagstest010 10.19.68.10]
2024-05-13 08:18:04 [7] [core 1222] [fetcher pool scheduler] scheduling service "deagstest010;Check_MK" at 2024-05-13 08:19:04
2024-05-13 08:18:04 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:05 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:06 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:07 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:08 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:09 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:10 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:11 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:12 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:13 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:14 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:15 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:16 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:17 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:18 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:19 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:20 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:21 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:22 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:23 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:24 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:25 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:26 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:27 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:28 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:29 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:30 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:31 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:32 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:33 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:34 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:35 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:36 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:37 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:38 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:39 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:40 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:41 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:42 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:43 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:44 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:45 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:46 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:47 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:48 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:49 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:50 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:51 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:53 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:54 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:55 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:56 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:57 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:58 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:18:59 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:19:00 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:19:01 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:19:02 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:19:03 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 08:19:04 [7] [core 1222] [fetcher pool scheduler] unscheduling service "deagstest010;Check_MK" at 2024-05-13 08:19:04
2024-05-13 08:19:04 [7] [core 1222] obtained SerialToken{Request[deagstest010],6} => SerialTokenFactory{6:2}
2024-05-13 08:19:04 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": sending PROBLEM notification to its contacts
2024-05-13 08:19:04 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": spooling notification to rule based notifications
2024-05-13 08:19:04 [7] spooled 1885 byte event: 'CONTACTS=
NOTIFICATIONTYPE=PROBLEM
NOTIFICATIONCOMMENT=
NOTIFICATIONAUTHOR=
NOTIFICATIONAUTHORNAME=
NOTIFICATIONAUTHORALIAS=
HOSTACKAUTHOR=
HOSTACKCOMMENT=
SERVICEACKAUTHOR=
SERVICEACKCOMMENT=
MICROTIME=1715588344755339
HOSTNOTIFICATIONNUMBER=0
LASTHOSTPROBLEMID=12
HOSTPROBLEMID=12
HOSTATTEMPT=1
MAXHOSTATTEMPTS=1
HOSTNAME=deagstest010
HOSTALIAS=dev-ca-1
HOSTADDRESS=10.19.68.10
HOSTOUTPUT=Packet received via smart PING
HOSTPERFDATA=
HOSTSTATE=UP
HOSTSTATEID=0
LASTHOSTSTATE=UP
LASTHOSTSTATEID=0
PREVIOUSHOSTHARDSTATE=DOWN
PREVIOUSHOSTHARDSTATEID=1
LONGHOSTOUTPUT=
HOSTCHECKCOMMAND=check-mk-host-smart
LASTHOSTSTATECHANGE=1715587090
LASTHOSTUP=1715588344
HOSTDOWNTIME=0
HOSTGROUPNAMES=check_mk
HOSTCONTACTGROUPNAMES=all
HOSTTAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_SL=
HOST_EC_CONTACT=
HOSTNOTESURL=
HOSTNOTES=
HOST_FILENAME=/wato/hosts.mk
HOST_ADDRESS_6=
HOST_TAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_ADDRESSES_6=
HOST_ADDRESS_4=10.19.68.10
HOST_ADDRESS_FAMILY=4
HOST_ADDRESSES_4=
SERVICEDESC=HAProxy Server ca-adm-stateful/ca-1
SERVICEOUTPUT=Status: DOWN(!!), Active, Layer Check: L7STS, Up since 15 seconds
SERVICEPERFDATA=
SERVICENOTIFICATIONNUMBER=1
SERVICEPROBLEMID=16
LASTSERVICEPROBLEMID=16
SERVICEATTEMPT=1
MAXSERVICEATTEMPTS=1
SERVICESTATE=CRITICAL
SERVICESTATEID=2
LASTSERVICESTATE=OK
LASTSERVICESTATEID=0
PREVIOUSSERVICEHARDSTATE=OK
PREVIOUSSERVICEHARDSTATEID=0
LASTSERVICESTATECHANGE=1715588284
LASTSERVICEOK=1715588218
SERVICEDOWNTIME=0
LONGSERVICEOUTPUT=Status: DOWN(!!)\\nActive\\nLayer Check: L7STS\\nUp since 15 seconds
SERVICECHECKCOMMAND=check_mk-haproxy_server
SERVICEGROUPNAMES=
SERVICECONTACTGROUPNAMES=all
SVC_SL=
SERVICE_SL=
SERVICE_EC_CONTACT=
SERVICENOTESURL=
SERVICENOTES=
SERVICEDISPLAYNAME=HAProxy Server ca-adm-stateful/ca-1
'
2024-05-13 08:19:04 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": cancelled: no more notifications necessary
2024-05-13 08:19:05 [7] unspooled 1885 byte event: 'CONTACTS=
NOTIFICATIONTYPE=PROBLEM
NOTIFICATIONCOMMENT=
NOTIFICATIONAUTHOR=
NOTIFICATIONAUTHORNAME=
NOTIFICATIONAUTHORALIAS=
HOSTACKAUTHOR=
HOSTACKCOMMENT=
SERVICEACKAUTHOR=
SERVICEACKCOMMENT=
MICROTIME=1715588344755339
HOSTNOTIFICATIONNUMBER=0
LASTHOSTPROBLEMID=12
HOSTPROBLEMID=12
HOSTATTEMPT=1
MAXHOSTATTEMPTS=1
HOSTNAME=deagstest010
HOSTALIAS=dev-ca-1
HOSTADDRESS=10.19.68.10
HOSTOUTPUT=Packet received via smart PING
HOSTPERFDATA=
HOSTSTATE=UP
HOSTSTATEID=0
LASTHOSTSTATE=UP
LASTHOSTSTATEID=0
PREVIOUSHOSTHARDSTATE=DOWN
PREVIOUSHOSTHARDSTATEID=1
LONGHOSTOUTPUT=
HOSTCHECKCOMMAND=check-mk-host-smart
LASTHOSTSTATECHANGE=1715587090
LASTHOSTUP=1715588344
HOSTDOWNTIME=0
HOSTGROUPNAMES=check_mk
HOSTCONTACTGROUPNAMES=all
HOSTTAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_SL=
HOST_EC_CONTACT=
HOSTNOTESURL=
HOSTNOTES=
HOST_FILENAME=/wato/hosts.mk
HOST_ADDRESS_6=
HOST_TAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_ADDRESSES_6=
HOST_ADDRESS_4=10.19.68.10
HOST_ADDRESS_FAMILY=4
HOST_ADDRESSES_4=
SERVICEDESC=HAProxy Server ca-adm-stateful/ca-1
SERVICEOUTPUT=Status: DOWN(!!), Active, Layer Check: L7STS, Up since 15 seconds
SERVICEPERFDATA=
SERVICENOTIFICATIONNUMBER=1
SERVICEPROBLEMID=16
LASTSERVICEPROBLEMID=16
SERVICEATTEMPT=1
MAXSERVICEATTEMPTS=1
SERVICESTATE=CRITICAL
SERVICESTATEID=2
LASTSERVICESTATE=OK
LASTSERVICESTATEID=0
PREVIOUSSERVICEHARDSTATE=OK
PREVIOUSSERVICEHARDSTATEID=0
LASTSERVICESTATECHANGE=1715588284
LASTSERVICEOK=1715588218
SERVICEDOWNTIME=0
LONGSERVICEOUTPUT=Status: DOWN(!!)\\nActive\\nLayer Check: L7STS\\nUp since 15 seconds
SERVICECHECKCOMMAND=check_mk-haproxy_server
SERVICEGROUPNAMES=
SERVICECONTACTGROUPNAMES=all
SVC_SL=
SERVICE_SL=
SERVICE_EC_CONTACT=
SERVICENOTESURL=
SERVICENOTES=
SERVICEDISPLAYNAME=HAProxy Server ca-adm-stateful/ca-1
'
2024-05-13 08:19:05 [7] [notification helper 1275] successfully sent event
2024-05-13 08:19:09 [7] [core 1222] [livestatus external] spooled command 'LOG;SERVICE NOTIFICATION: cmkadmin;deagstest010;HAProxy Server ca-adm-stateful/ca-1;CRITICAL;log;Status: DOWN(!!), Active, Layer Check: L7STS, Up since 15 seconds'
2024-05-13 08:19:09 [5] [core 1222] Executing external command: LOG;SERVICE NOTIFICATION: cmkadmin;deagstest010;HAProxy Server ca-adm-stateful/ca-1;CRITICAL;log;Status: DOWN(!!), Active, Layer Check: L7STS, Up since 15 seconds
2024-05-13 08:19:09 [7] [core 1222] [livestatus external] spooled command 'LOG;SERVICE NOTIFICATION RESULT: cmkadmin;deagstest010;HAProxy Server ca-adm-stateful/ca-1;OK;log;;'
2024-05-13 08:19:09 [5] [core 1222] Executing external command: LOG;SERVICE NOTIFICATION RESULT: cmkadmin;deagstest010;HAProxy Server ca-adm-stateful/ca-1;OK;log;;
2024-05-13 08:19:09 [7] [core 1222] released SerialToken{Request[deagstest010],6} => SerialTokenFactory{6:1}
2024-05-13 08:19:09 [7] [core 1222] scheduling service "deagstest010;Check_MK" at 2024-05-13 08:20:09 with commandline [@cmk deagstest010 10.19.68.10]
2024-05-13 08:19:09 [7] [core 1222] [fetcher pool scheduler] scheduling service "deagstest010;Check_MK" at 2024-05-13 08:20:09
2024-05-13 08:20:09 [7] [core 1222] [fetcher pool scheduler] unscheduling service "deagstest010;Check_MK" at 2024-05-13 08:20:09
2024-05-13 08:20:09 [7] [core 1222] obtained SerialToken{Request[deagstest010],6} => SerialTokenFactory{6:2}
2024-05-13 08:20:15 [7] [core 1222] released SerialToken{Request[deagstest010],6} => SerialTokenFactory{6:1}
2024-05-13 08:20:15 [7] [core 1222] scheduling service "deagstest010;Check_MK" at 2024-05-13 08:21:15 with commandline [@cmk deagstest010 10.19.68.10]
2024-05-13 08:20:15 [7] [core 1222] [fetcher pool scheduler] scheduling service "deagstest010;Check_MK" at 2024-05-13 08:21:15
Might this be a bug?
I reevaluated the behavior and yes this seems to be a bug. When I switch to “Smart Ping” for “Host Check Command” I get the notifications back as expected. But we need some already existing check to be used for Host Availability. Is there an easy fix with still using the “Use the status of the Checkmk Agent” option or do I have to hope for a fix in some upcoming version?
Might be a bug, but careful, mind this:
There is a mechanism to not notify a service problem if the host is down and
a mechanism to suppress a service notification until there is a fresh host stated available.
Therefore it’s important to recognize a host down as quick as possible and this is one of the features of smartping because it runs with a check interval of 6s by default.
But, if you change the host check command from smartping to something else also the normal host check interval will change from 6s to 60s.
In other words, if you do your test you perhaps have be more patient.
And the other pitfall is, if the Service Check_MK is CRIT, notification for that will be suppressed if the Host is DOWN at the same time and if you use the Check_MK Service as your your custom Host check command, then you will only get the Host DOWN notification but never the Service Check_MK CRIT notification, or?
Yeah I know of that mechanism and it’s quiet useful. I have been testing with several minutes between “Host UP” state in CheckMK and then triggering a service to go to CRIT.
And the monitoring is running contiuously so there should have been some notification up to now, as the Hosts are all UP.
We are currently looking into it. May still take some time but i will post here if there is a solution.
1 Like