I still do not see why the host was marked up, the pagerduty logs and notify.log both concur that at 5:00:24AM the host was marked as down then at 5:31:29AM it was marked as up. The host was down the entire time and showed as down in the checkmk WebUI.
Here is the notify.log for that host and a screenshot from pagerduty.
2020-04-04 05:00:24 Got raw notification (example.hostname) context with 42 variables
2020-04-04 05:00:24 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:00:24 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:00:30 Got spool file 72ec65b0 (example.hostname) for local delivery via pagerduty-agent
2020-04-04 05:00:30 Output: Event processed. Incident Key: (‘event_source=host;host_name=example.hostname’, [0])
2020-04-04 05:02:58 Got raw notification (example.hostname;CPU utilization) context with 71 variables
2020-04-04 05:02:58 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:02:58 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:02:58 Got raw notification (example.hostname;Disk IO SUMMARY) context with 71 variables
2020-04-04 05:02:58 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:02:58 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:02:58 Got raw notification (example.hostname;Memory used) context with 71 variables
2020-04-04 05:02:58 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:02:58 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:02:58 Got raw notification (example.hostname;Overall state) context with 71 variables
2020-04-04 05:02:58 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:02:58 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:31:22 Got raw notification (example.hostname) context with 42 variables
2020-04-04 05:31:22 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:31:22 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:31:29 Got spool file 68c1085a (example.hostname) for local delivery via pagerduty-agent
2020-04-04 05:31:29 Output: Event processed. Incident Key: (‘event_source=host;host_name=example.hostname’, [0])
2020-04-04 05:32:03 Got raw notification (example.hostname;CPU utilization) context with 71 variables
2020-04-04 05:32:03 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:32:03 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:32:03 Got raw notification (example.hostname;Disk IO SUMMARY) context with 71 variables
2020-04-04 05:32:03 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:32:03 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:32:03 Got raw notification (example.hostname;Memory used) context with 71 variables
2020-04-04 05:32:03 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:32:03 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:33:05 Got raw notification (example.hostname;Overall state) context with 71 variables
2020-04-04 05:33:05 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:33:05 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:44:00 Got raw notification (example.hostname) context with 42 variables
2020-04-04 05:44:00 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:44:00 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:44:07 Got spool file 8a7ac482 (example.hostname) for local delivery via pagerduty-agent
2020-04-04 05:44:07 Output: Event processed. Incident Key: (‘event_source=host;host_name=example.hostname’, [0])
2020-04-04 05:46:32 Got raw notification (example.hostname;CPU utilization) context with 71 variables
2020-04-04 05:46:32 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:46:32 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:46:32 Got raw notification (example.hostname;Disk IO SUMMARY) context with 71 variables
2020-04-04 05:46:32 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:46:32 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:46:32 Got raw notification (example.hostname;Memory used) context with 71 variables
2020-04-04 05:46:32 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:46:32 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 05:46:32 Got raw notification (example.hostname;Overall state) context with 71 variables
2020-04-04 05:46:32 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 05:46:32 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)
2020-04-04 21:41:52 Got raw notification (example.hostname) context with 42 variables
2020-04-04 21:41:52 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (others hosts)
2020-04-04 21:41:52 → does not match: The host’s name ‘example.hostname’ is not on the list of allowed hosts (another host)