Multiple Opsgenie integrations, only 1 notification is send

Hi,

Currently in our setup we have multiple global notification rules. So for different host/service groups, we have different OpsGenie integrations as in our Opsgenie setup we have check_mk integrations per team.

For example:
Service group A, any alerts generated are being forwarded to OpsGenie via the rules we have for this service group. In OpsGenie the integration for Service Group A then alerts team A which is responsible for this Service group.
Same is doen for Service group B, in OpsGenie we have the integration for Service Group B (so a different OpsGenie API endpoint) and those alerts are handled by team B.

This is all working just fine.

Now we have the situation that Team B is for some reason moving to a different OpsGenie account. They have created an integration for the Service group B and this integration has a different API endpoint to be configured.
So basically we copied the existing global notification rule for B in CheckMK and updated this with the new API endpoint resulting in 2 global notification rules for team B. 1 using the OpsGenie API endpoint that we already had, the 2nd using the new API endpoint. So the only difference between the two rules is the API endpoint and actual name of the rule (Team B vs Team B-new). All other settings are the same.

Before removing the old rule, we want to make sure all alerts are being received at the new OpsGenie, so both rules are currently active. What we noticed however is that when a service goes critical and a notification needs to be send out, both rules match, but only 1 notification is actually send to the API endpoint in the old rule.
If we disable this old rule, the alerts are send to the new rule and the new API endpoint is used. If both rules are active, only 1 notification is send and only to the oldest rule.

Here is what I’m seeing in the notify.log (replaced sensitive info with xxxx):

2020-07-01 11:55:42 Got raw notification (tiara.xxxxxxx.xx;HTTPS Tiara Business Webservice [SLA]) context with 71 variables
2020-07-01 11:55:42 Global rule 'Team B-New'...
2020-07-01 11:55:42  -> matches!
2020-07-01 11:55:42 Warning: cannot get information about contact mailto:xxxxxxxx: ignoring restrictions
2020-07-01 11:55:42    - adding notification of mailto:xxxxxxxx via opsgenie
2020-07-01 11:55:42 Global rule 'Team B'...
2020-07-01 11:55:42  -> matches!
2020-07-01 11:55:42 Warning: cannot get information about contact mailto:xxxxxxx: ignoring restrictions
2020-07-01 11:55:42    - modifying notification of mailto:xxxxxxx via opsgenie
2020-07-01 11:55:42 Executing 1 notifications:
2020-07-01 11:55:42   * notifying mailto:xxxxxxx via opsgenie, parameters: https://api.opsgenie.com/v1/json/checkmk?apiKey=xxxxx-xxxx-xxxx-xxxx-xxxxxxx, bulk: no
2020-07-01 11:55:42 Creating spoolfile: /omd/sites/sla/var/check_mk/notify/spool/xxxxx-xxxx-xxxx-xxxx-xxxxxxx
2020-07-01 11:55:47 ----------------------------------------------------------------------
2020-07-01 11:55:47 Got spool file 42841af3 (tiara.xxxxxxx.xx;HTTPS Tiara Business Webservice [SLA]) for local delivery via opsgenie

The “cannot get information about contact…” message is something we see on all of our rules as we’re using some placeholder email addresses here, this isn’t intervering with the notifications (as far as I can tell).

I was actually expecting that a notification to be send to BOTH as both match. So 1 notification using the old API endpoint and one notification using the new API endpoint.
Now maybe I’m just missing something here and is this how it is supposed to be or maybe I’m misunderstanding the use of Global rules.

I’ve also been looking at notifications for users… I could create a specific user with a notification rule that used OpsGenie (with the new endpoint) and that way have 2 OpsGenie endpoint active for the same service group?

I’m unsure on the best approach here and if I’m missing something/misunderstanding the functionality of the notification rules (user vs global).

Any advise here? :slight_smile:

So I found a work around by creating a separate user and setting a notifcation rule on that use using the new opsgenie endpoint. This seems to work.
Feels like a work around, but as said, I could be totally misunderstanding the way Global notification rules are used :slight_smile:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.