Weird behavior with notifications after upgrade to 2.0

Consider this: After the upgrade of our CME environment from 1.6.0p22 to 2.0.0p3 we discovered after some days, that notifications seem missing. Well at least in the connected ticket system which receives emails from our satellite. The weird thing about it is, that I can see successful notifications in Checkmk and the nullmailer log reports that mails have been send. But somehow the ticket system does not show tickets, although a manual test mail worked as expected. I am not using any custom notification plugins, just HTML Mail and ASCII Mail.

So much for the weird part. On to the technical stuff: In all cmc.log files on my Checkmk Servers I repeatedly see the following messages in direct succession:

2021-04-28 06:55:34 [2] [notification helper 49722] invalid signal byte ASCII 76, expected '*'	
2021-04-28 06:55:34 [2] [notification helper 49722] read 104 further bytes: 'oading autochecks from /omd/sites/$SITE/var/check_mk/autochecks/$HOSTNAME.mk

Also on the notification analysis page we tun into the following error when analyzing any rule:

Error running automation call notification-analyse, output:
Loading autochecks from /omd/sites/$SITE/var/check_mk/autochecks/$HOSTNAME.mk

<A shitload of (json?) data that seems to contain all the notification rules and their analysis output>

: invalid syntax (<unknown>, line 1)

I am really at a loss here: Does anyone have the faintest idea where I can look for the cause? Or at least can help me sorting out the parts that can’t be it?

Thanks in advance!

What is your current setting in the global settings option “Notification log level”?

image

I don’t recall changing that setting, must have been a colleague.

Also is this setting relevant?
image

Can you please change that to “Normal logging”.
The messages in cmc.log should disappear after activating the change.

Would be good to know if the missing notification problem is also fixed with this.

1 Like

Changes are activating right now, will keep you posted on my observation.

Big shout out thanks for the quick and helpful response!

Update: The messages stopped which is nice and I also could pinpoint the issue of the missing mails: They are stuck on a mail gateway because the sender address is not right.

Let me elaborate: Because of some awesome ticket system we need to set a lot of different email sender addresses in our notification rules. For some reason Checkmk sends these mails with the fallback sender address ($SITE@$HOSTNAME) of the appliance, rather than the specified address. What is even weirder is that some rules seem to work correctly. This behavior started right after the upgrade to 2.0.

I am going to enable better logging and try to understand why this happens.
Will update here as soon as I got information.

Sounds like this problem but should be resolved in p2

Two reasons why that cannot be my fix:

  1. I am not using bulk notifications
  2. I am already on p3

UPDATE: When I change the notification plugin from asciimail to mail it seems to work. Still verifying, but the issue seems to be in the asciimail plugin.

P.S.: I clarified my former post regarding the sender address: It is not the address I set in the appliance as fallback, but rather the (builtin?) fallback, that goes like $SITE@$HOSTNAME. In my use case that is a major problem as the host has an internal domain which cannot be routed on the internet.

To wrap this up if anyone stumbles through here: I “fixed” the issue with the wrong sender address by switching all asciimail notifications to mail. That is not really a fix but at least my notifications work again and the HTML mails actually improve my use case.

Anyway a solution to the underlying problem would be appreciated.

P.S.: I am marking @_rb’s answer as the solution, as it answers the question in my original post.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.