I switched notifications and core to debug and there is the entry for the service problem and no notification either.
2024-05-13 07:54:48 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:48 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:48 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:49 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:49 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:49 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:50 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:50 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:50 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:51 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:51 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:51 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": postponing, last host check not recent enough
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": hard state change to OK
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": notification number 0 is already pending
2024-05-13 07:54:52 [7] [notification helper 1275] service "deagstest010;HAProxy Server ca-adm-stateful/ca-1": cancelled: state changed back to OK
2024-05-13 07:54:52 [7] [core 1222] sending check result from service "deagstest010;Check_MK" to host "deagstest010"
2024-05-13 07:54:52 [7] [core 1222] released SerialToken{Request[deagstest010],5} => SerialTokenFactory{5:1}
2024-05-13 07:54:52 [7] [core 1222] scheduling service "deagstest010;Check_MK" at 2024-05-13 07:55:52 with commandline [@cmk deagstest010 10.19.68.10]
2024-05-13 07:54:52 [7] [core 1222] [fetcher pool scheduler] scheduling service "deagstest010;Check_MK" at 2024-05-13 07:55:52
2024-05-13 07:54:53 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:54:53 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:54:54 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
If I do the same for the whole host, I get the log entries I wanted to see:
2024-05-13 07:56:58 [7] [core 1222] [fetcher pool scheduler] unscheduling service "deagstest010;Check_MK" at 2024-05-13 07:56:58
2024-05-13 07:56:58 [7] [core 1222] obtained SerialToken{Request[deagstest010],5} => SerialTokenFactory{5:2}
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, last host check not recent enough
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, last host check not recent enough
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;Check_MK": hard state change to CRITICAL
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;Check_MK": setting up, problem ID 11, delay 0sec
2024-05-13 07:56:58 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, last host check not recent enough
2024-05-13 07:56:58 [7] [core 1222] sending check result from service "deagstest010;Check_MK" to host "deagstest010"
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": hard state change to DOWN
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": setting up, problem ID 12, delay 0sec
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": sending PROBLEM notification to its contacts
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": spooling notification to rule based notifications
2024-05-13 07:56:58 [7] spooled 1362 byte event: 'CONTACTS=
NOTIFICATIONTYPE=PROBLEM
NOTIFICATIONCOMMENT=
NOTIFICATIONAUTHOR=
NOTIFICATIONAUTHORNAME=
NOTIFICATIONAUTHORALIAS=
HOSTACKAUTHOR=
HOSTACKCOMMENT=
MICROTIME=1715587018173149
HOSTNOTIFICATIONNUMBER=1
LASTHOSTPROBLEMID=12
HOSTPROBLEMID=12
HOSTATTEMPT=1
MAXHOSTATTEMPTS=1
HOSTNAME=deagstest010
HOSTALIAS=dev-ca-1
HOSTADDRESS=10.19.68.10
HOSTOUTPUT=[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec
HOSTPERFDATA= execution_time=0.010 user_time=0.020 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000
HOSTSTATE=DOWN
HOSTSTATEID=1
LASTHOSTSTATE=UP
LASTHOSTSTATEID=0
PREVIOUSHOSTHARDSTATE=UP
PREVIOUSHOSTHARDSTATEID=0
LONGHOSTOUTPUT=
HOSTCHECKCOMMAND=check-mk-host-service
LASTHOSTSTATECHANGE=1715587018
LASTHOSTUP=1715586958
HOSTDOWNTIME=0
HOSTGROUPNAMES=check_mk
HOSTCONTACTGROUPNAMES=all
HOSTTAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_SL=
HOST_EC_CONTACT=
HOSTNOTESURL=
HOSTNOTES=
HOST_FILENAME=/wato/hosts.mk
HOST_ADDRESS_6=
HOST_TAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_ADDRESSES_6=
HOST_ADDRESS_4=10.19.68.10
HOST_ADDRESS_FAMILY=4
HOST_ADDRESSES_4=
'
2024-05-13 07:56:58 [7] [notification helper 1275] host "deagstest010": cancelled: no more notifications necessary
2024-05-13 07:56:58 [7] [core 1222] released SerialToken{Request[deagstest010],5} => SerialTokenFactory{5:1}
2024-05-13 07:56:58 [7] [core 1222] scheduling service "deagstest010;Check_MK" at 2024-05-13 07:57:58 with commandline [@cmk deagstest010 10.19.68.10]
2024-05-13 07:56:58 [7] [core 1222] [fetcher pool scheduler] scheduling service "deagstest010;Check_MK" at 2024-05-13 07:57:58
2024-05-13 07:56:58 [7] unspooled 1362 byte event: 'CONTACTS=
NOTIFICATIONTYPE=PROBLEM
NOTIFICATIONCOMMENT=
NOTIFICATIONAUTHOR=
NOTIFICATIONAUTHORNAME=
NOTIFICATIONAUTHORALIAS=
HOSTACKAUTHOR=
HOSTACKCOMMENT=
MICROTIME=1715587018173149
HOSTNOTIFICATIONNUMBER=1
LASTHOSTPROBLEMID=12
HOSTPROBLEMID=12
HOSTATTEMPT=1
MAXHOSTATTEMPTS=1
HOSTNAME=deagstest010
HOSTALIAS=dev-ca-1
HOSTADDRESS=10.19.68.10
HOSTOUTPUT=[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec
HOSTPERFDATA= execution_time=0.010 user_time=0.020 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000
HOSTSTATE=DOWN
HOSTSTATEID=1
LASTHOSTSTATE=UP
LASTHOSTSTATEID=0
PREVIOUSHOSTHARDSTATE=UP
PREVIOUSHOSTHARDSTATEID=0
LONGHOSTOUTPUT=
HOSTCHECKCOMMAND=check-mk-host-service
LASTHOSTSTATECHANGE=1715587018
LASTHOSTUP=1715586958
HOSTDOWNTIME=0
HOSTGROUPNAMES=check_mk
HOSTCONTACTGROUPNAMES=all
HOSTTAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_SL=
HOST_EC_CONTACT=
HOSTNOTESURL=
HOSTNOTES=
HOST_FILENAME=/wato/hosts.mk
HOST_ADDRESS_6=
HOST_TAGS=/wato/ auto-piggyback checkmk-agent cmk-agent ip-v4 ip-v4-only lan no-snmp prod site:spcm tcp
HOST_ADDRESSES_6=
HOST_ADDRESS_4=10.19.68.10
HOST_ADDRESS_FAMILY=4
HOST_ADDRESSES_4=
'
2024-05-13 07:56:58 [7] [notification helper 1275] successfully sent event
2024-05-13 07:56:59 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:56:59 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:56:59 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:00 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:00 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:57:00 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:01 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:01 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:57:01 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:02 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:02 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:57:02 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:03 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:03 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up
2024-05-13 07:57:03 [7] [notification helper 1275] service "deagstest010;Check_MK": postponing, host of this service is not up
2024-05-13 07:57:03 [7] [core 1222] [livestatus external] spooled command 'LOG;HOST NOTIFICATION: cmkadmin;deagstest010;DOWN;log;[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec'
2024-05-13 07:57:03 [5] [core 1222] Executing external command: LOG;HOST NOTIFICATION: cmkadmin;deagstest010;DOWN;log;[agent] Communication failed: [Errno 111] Connection refused(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec
2024-05-13 07:57:03 [7] [core 1222] [livestatus external] spooled command 'LOG;HOST NOTIFICATION RESULT: cmkadmin;deagstest010;OK;log;;'
2024-05-13 07:57:03 [5] [core 1222] Executing external command: LOG;HOST NOTIFICATION RESULT: cmkadmin;deagstest010;OK;log;;
2024-05-13 07:57:04 [7] [notification helper 1275] service "deagstest010;Check_MK Agent": postponing, host of this service is not up
2024-05-13 07:57:04 [7] [notification helper 1275] service "deagstest010;APT Updates": postponing, host of this service is not up