Email notification doesn't work

Hello everyone,
We have the problem that our CheckMK does not send an email notification.

CMK version:
2.1.0p33.cre

OS version:
CentOS Linux release 7.9.2009 (Core)

Error message:
unfortunately none available

There are neither error messages in the log
/opt/omd/sites/$SiteName$/var/log/web.log

There is also no output or error message when I enter the following
cmk --notify

We have configured global notification rules and contact groups and we have assigned these users with valid email addresses

When I send emails from the server’s console with sendmail or mailx, they arrive immediately
mailx -s “Testmail with MailX” mail.address@firma.com
echo “das ist ein Test” | sendmail -f mail.address@firma.com -t

where or how can I continue to look for the error here, or is it not possible to use the email notification in the CRE (RAW) edition?

Its a myth that Checkmk sends email messages (unless you use the appliance) its your MTA that sends emails, meaning your OS…

/var/log/mail is a good start and verify your MTA like postfix.

Hello Anders,

Thanks for your answer.
Unfortunately, the log file /var/log/maillog is empty, and the operating system itself sends the emails without any problems, as I wrote.
My colleague has now set the logging to debug, so we might see a little more now.
We would be grateful for further helpful tips.

best regards
Sebastian

We found the following in the crash.log (/omd/sites/$SiteName$/var/check_mk/notify/crash.log ).

Traceback (most recent call last):
   File "/omd/sites/$SiteName$/lib/python3/cmk/base/notify.py", line 279, in do_notify
     notify_notify(raw_context_from_env(os.environ))
   File "/omd/sites/$SiteName$/lib/python3/cmk/base/notify.py", line 320, in notify_notify

Maybe that will help pinpoint the error

how does the OS send logs if your maillog is empty? Checkmk would require a mail transport agent on the OS, sendmail, postfix etc. They normally write to the above file (or some similar)

if you run mail to send you a test mail, do you get it?

Hello Anders,

we use postfix old MTA.
The log was previously empty because the log level was set to warning and there appeared to be no errors or warnings.
As written, we set the log level to debug, and now messages arrive in the log.
However, there are always only the following messages and no errors.

Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: qmgr_scan_start: start deferred queue scan
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: done deferred queue scan
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: trigger_server_accept_local: trigger arrived
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: master_notify: status 0
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: request: 87 (W)
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: request: 0 (?)
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: request ignored
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: qmgr_scan_start: start incoming queue scan
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: master_notify: status 1
Sep 15 09:14:35 $SERVERNAME$ postfix/qmgr[98890]: done incoming queue scan
Sep 15 09:14:37 $SERVERNAME$ postfix/pickup[70192]: trigger_server_accept_local: trigger arrived
Sep 15 09:14:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 0
Sep 15 09:14:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 1
Sep 15 09:15:37 $SERVERNAME$ postfix/pickup[70192]: trigger_server_accept_local: trigger arrived
Sep 15 09:15:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 0
Sep 15 09:15:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 1
Sep 15 09:16:37 $SERVERNAME$ postfix/pickup[70192]: trigger_server_accept_local: trigger arrived
Sep 15 09:16:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 0
Sep 15 09:16:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 1
Sep 15 09:17:37 $SERVERNAME$ postfix/pickup[70192]: trigger_server_accept_local: trigger arrived
Sep 15 09:17:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 0
Sep 15 09:17:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 1
Sep 15 09:18:37 $SERVERNAME$ postfix/pickup[70192]: trigger_server_accept_local: trigger arrived
Sep 15 09:18:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 0
Sep 15 09:18:37 $SERVERNAME$ postfix/pickup[70192]: master_notify: status 1

As written above, when I send emails with sendmail or mailx on the operating system console, I receive them immediately.

Best regards

Does anyone know how to get notification via email?
i.e. it is generated by, then passed to, …, until the MTA of the OS then sends the email.

Does anyone else know whether I will get this error if I install a completely new server and then import the export from the current environment?

Hi @sebastian.rohde,

can you please share the notification rule you’ve created?

A Screenshot without sensitive information would help to understand your problem better.

Thanks in advance!
Norm

Hello Norm,

Here are the screenshots of my test roll.



image

Best regards
Sebastian

Hi @sebastian.rohde,

thanks for sharing the rule.

So far nothing special I can see there, but could you please make a small adjustment to the marked things:

If you use an explicit email, you don’t have to use “Notify all contacts…”
Since you only have a single site, you don’t have to match it there, but that’s just a minor thing.

Can you also please check your notify.log and see if your notification is getting matched when you expect it?

Thanks in advance!
Norm

Depending on the OS you want to upgrade to ( i use Rocky Linux 9.x) : mailx has been replaced by package s-nail

This was taken from the rockyLinux docs:

… and i also use postfix :slight_smile:

The only things i have set/configured in the /etc/postfix/main.cf:

# Set myhostname to the FQDN of the machine
myhostname = monitor.mydomain.com

# Set external SMTP relay host here IP or hostname accepted along with a port number. 
# The portnumber is only needed if you are not using the default port 25
relayhost = [YOUR-SMTP-SERVER-IP-HERE]:587

# Only accept email from our web-server (running on this box)
inet_interfaces = 127.0.0.1

Ofcourse this is very rudimentary, no certificate or other things configured, just plain SMTP

Tested it with sending a fake notification from CMK as described in the https://docs.checkmk.com/latest/en/notifications.html#smtp

And works.

  • Glowsome
1 Like

Hello Norm,

Thank you for your quick response.

I made both adjustments once.
the /omd/sites/$MySite$/var/log/notify.log is still empty.

Best regards
Sebastian

Hello Glowsome,

thanks for your tips.
We use CentOS 7.9 and sending emails on the command line works without any problems.
It looks as if CheckMK does not transfer the emails to the OS or does not receive them.

Best regards
Sebastian

Mhh, that shouldn’t be the case.

Can you please try the troubleshooting from the documentation for the Nagios core, since you are on the CRE. →

Thanks in advance.
Norm

Hi @sebastian.rohde,

A question, as this is unclear to me, has it worked before ?
Or have you started setting up notifications just now with this version and bumped into this issue ?

  • Glowsome

… just my 2 cents…

The only case where i can think of a logfile not being filled is when the rights to the file are off.

So is the notify.log owner/group correct ?

The owner/group should be sitename/sitename

  • Glowsome

Hello Norm,

when I set the debug_level to 32 as described in the error diagnosis, I get messages like this about Notification.

[1695804080.081279] [032.0] [pid=76081] ** Service Notification Attempt ** Host: '$SERVERNAME$', Service: 'YUM Updates', Type: 0, Options: 0, Current State: 2, Last Notification: Thu Aug  3 17:18:34 2023
[1695804080.081291] [001.0] [pid=76081] check_service_notification_viability()
[1695804080.081296] [001.0] [pid=76081] check_time_against_period()
[1695804080.081312] [001.0] [pid=76081] check_service_dependencies()
[1695804080.081323] [001.0] [pid=76081] check_host_dependencies()
[1695804080.081329] [032.0] [pid=76081] Notification viability test failed.  No notification will be sent out.
[1695804080.081347] [001.0] [pid=76081] run_service_performance_data_command()
[1695804080.081352] [001.0] [pid=76081] update_service_performance_data_file()
[1695804080.081365] [001.0] [pid=76081] handle_async_service_check_result()


[1695804414.163222] [032.0] [pid=76081] ** Service Notification Attempt ** Host: '$SERVERNAME$', Service: 'Memory', Type: 0, Options: 0, Current State: 1, Last Notification: Thu Aug  3 17:18:26 2023
[1695804414.163227] [001.0] [pid=76081] check_service_notification_viability()
[1695804414.163230] [001.0] [pid=76081] check_time_against_period()
[1695804414.163242] [001.0] [pid=76081] check_service_dependencies()
[1695804414.163251] [001.0] [pid=76081] check_host_dependencies()
[1695804414.163255] [032.0] [pid=76081] Notification viability test failed.  No notification will be sent out.
[1695804414.163261] [001.0] [pid=76081] run_service_performance_data_command()
[1695804414.163264] [001.0] [pid=76081] update_service_performance_data_file()
[1695804414.163273] [001.0] [pid=76081] handle_async_service_check_result()

Best regards
Sebastian

Hello Glowsome,

We had not previously used the email notification.
Unfortunately, a colleague who is no longer with us once activated the notification for MS Teams.
However, he deactivated this again after a few weeks.

Best regards
Sebastian

Hello Glowsome,

The permissions on the notify.log file are set to Owner and Group of the site user and to rw rw r.

Best regards
Sebastian