Active Checks Not sending Notifications to PagerDuty

Troubleshooting
, ,
#1

CMK version: Checkmk Raw Edition 2.1.0p20
OS version: RHEL 7

Notification for process checks are sending notifications to PagerDuty as expected. However notifications for TCP Ports or HTTPD are not sending notifications to PagerDuty (active checks check_mk_active-http and check_mk_active-tcp) both had the same symptoms.

When manually setting the active check to Critical to test the Notifications seem to fire, but call to PagerDuty using service labels seems to not occur.

Anyone see this? Is there some different setting/configuration for active checks and notifications that I’m missing?

#2

Sorry but I’m not clear on the behaviour you are seeing.

Are you saying that if you fake a critical check result the notification is sent to PagerDuty (PD), but notifications generated in response to normal monitoring are not sent to PD?

#3

Sorry for not being more clear.

Normal checks are working as expected. Active checks are not.

We have seen the same if we force a fake notification or stop a service and let checkmk detect the issue.

#4

This sounds like it could be an issue with the way the PagerDuty notification rule is set up.

Can you share your configuration?

Regards,
Jason

#5

Sure, Here is the notification configuration:

notify_config_example_1
notify_config_example_11015×908 33.3 KB

We just label the services we want to notify.

#6

Here is the rest of that notification - the post would only allow me to have one image:

notify_config_example_2
notify_config_example_2727×657 11.9 KB

#7

Have you tried removing the Contact Selection constraint?

If I remember correctly from when I set up PD integration in my test environment, PD notification in Checkmk 2+ is not associated with Checkmk users. Rather, PD integration is controlled by the Integration Key, which is linked to a specific service in PD.

This means that a Checkmk rule that defines PD integration says “Tell PagerDuty that Service1 is down.” PD then decides who to notify based on its configuration. This is fundamentally different from Checkmk rules that define which Checkmk user(s) to notify of an event.

Hope this helps,
Jason