Checkmk email notifications not working anymore

Troubleshooting
1

CMK version:2.3.0.p28 cre
OS version:Debian 11

Error message:

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Hello all,

The problem I am having is that notifications are not working anymore on my RAW checkmk server. I guess a patch messed it up.

If I run echo echo “test” | /usr/sbin/sendmail notifications@mydomain.org I get the email, which is telling me that the underling email system is functioning correctly just checkmk does not send the notifications to it.

If someone has any idea how can I troubleshoot this, I will much appreciate it.

Thanks,

1 Like
2

Look at $OMD_ROOT/var/log/notify.log and $OMD_ROOT/var/nagios/nagios.log and maybe /var/log/mail.log.

3 
~/var/log/notify.log
2026-02-17 06:30:21,725 [10] [cmk.store] Trying to acquire lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 06:30:21,725 [10] [cmk.store] Got lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 06:30:21,725 [10] [cmk.store] Releasing lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 06:30:21,725 [10] [cmk.store] Released lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 06:31:21,920 [10] [cmk.store] Trying to acquire lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 06:31:21,920 [10] [cmk.store] Got lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 06:31:21,921 [10] [cmk.store] Releasing lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 06:31:21,921 [10] [cmk.store] Released lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 10:16:55,722 [10] [cmk.store] Trying to acquire lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 10:16:55,723 [10] [cmk.store] Got lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 10:16:55,723 [10] [cmk.store] Releasing lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk
2026-02-17 10:16:55,723 [10] [cmk.store] Released lock on /omd/sites/MYSITE/var/check_mk/notify/backlog.mk


~/var/nagios/nagios.log
[1771320614] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771320630] Auto-save of retention data completed successfully.
[1771320674] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771320735] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771320795] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771320855] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771320915] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771320975] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321035] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321096] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321156] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321216] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321276] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321336] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321397] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321457] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321517] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321577] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321637] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321698] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1771321758] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)


/var/log/mail.log
Feb 15 18:16:38 ROCLJVKRCMCK nullmailer-send[443]: Rescanning queue.
Feb 16 18:16:39 ROCLJVKRCMCK nullmailer-send[443]: Rescanning queue.
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Trigger pulled.
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Rescanning queue.
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Starting delivery, 1 message(s) in queue.
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Starting delivery: host: mymailserver.org protocol: smtp file: 1771316167.428959
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: From: <MYSITE@mymailserver.org> to: <your@email.com>
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Message-Id: <1771316167.706493.428958.nullmailer@mymailserver.org>
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[428960]: smtp: Failed: 550 Relay denied for <email.com>
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Sending failed: Permanent error in sending the message
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Moving message 1771316167.428959 into failed
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Generating bounce for 1771316167.428959
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Delivery complete, 0 message(s) remain.
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Trigger pulled.
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Rescanning queue.
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Starting delivery, 1 message(s) in queue.
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Starting delivery: host: mymailserver.org protocol: smtp file: 1771316167.428961
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: From: <> to: <MYSITE@mymailserver.org>
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Message-Id: <1771316167.738361.428962.nullmailer@mymailserver.org>
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[428963]: smtp: Failed: 550 User not found
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Sending failed: Permanent error in sending the message
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Moving message 1771316167.428961 into failed
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Not generating double bounce for 1771316167.428961
Feb 17 10:16:07 ROCLJVKRCMCK nullmailer-send[443]: Delivery complete, 0 message(s) remain.
Feb 17 10:16:34 ROCLJVKRCMCK nullmailer-send[443]: Trigger pulled.
Feb 17 10:16:34 ROCLJVKRCMCK nullmailer-send[443]: Rescanning queue.
Feb 17 10:16:34 ROCLJVKRCMCK nullmailer-send[443]: Starting delivery, 1 message(s) in queue.
Feb 17 10:16:34 ROCLJVKRCMCK nullmailer-send[443]: Starting delivery: host: mymailserver.org protocol: smtp file: 1771316194.429127
Feb 17 10:16:34 ROCLJVKRCMCK nullmailer-send[443]: From: <MYSITE@mymailserver.org> to: <notifications@mymailserver.org>
Feb 17 10:16:34 ROCLJVKRCMCK nullmailer-send[443]: Message-Id: <1771316194.807098.429126.nullmailer@mymailserver.org>
Feb 17 10:16:35 ROCLJVKRCMCK nullmailer-send[429128]: smtp: Succeeded: 250 Mail queued for delivery
Feb 17 10:16:35 ROCLJVKRCMCK nullmailer-send[443]: Sent file.
Feb 17 10:16:35 ROCLJVKRCMCK nullmailer-send[443]: Delivery complete, 0 message(s) remain.

The strange thing is that no log is being created if I generate a failure on a host. The email server I know for sure it works and does not deny relaying since I have more services in the network that are using it and which have no problem sending emails.

If I manually send a test email then I get the process in the mail.log

Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[443]: Trigger pulled.
Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[443]: Rescanning queue.
Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[443]: Starting delivery, 1 message(s) in queue.
Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[443]: Starting delivery: host: mymailserver.org protocol: smtp file: 1771323457.505577
Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[443]: From: <MYSITE@mymailserver.org> to: <notifications@mymailserver.org>
Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[443]: Message-Id: <1771323457.767489.505576.nullmailer@mymailserver.org>
Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[505578]: smtp: Succeeded: 250 Mail queued for delivery
Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[443]: Sent file.
Feb 17 12:17:37 ROCLJVKRCMCK nullmailer-send[443]: Delivery complete, 0 message(s) remain.
4

According to this log the from address is empty, on my server it was a no no to send mail from checkmk.

Perhaps check in global settings and specific notification rules if the from address is properly set.

image
image593×280 9.04 KB

image
image595×337 9.83 KB

5

We have been running into quite some issues with nullmailer stopping working from time to time to work again for a while after cleaning all queues. Internally, many of us thus have switched to dma (DragonFly Mail Agent). It’s as simple to configure as nullmailer and even more robust as nullmailer ever was.

1 Like
6

Already checked, since nothing changed here. Just happened to not work.

image
image830×887 18.6 KB

7

Thank you. I will put a note for a future change, but right now I really want to fix this since it is driving me crazy.

1 Like
8

When you say not working it seems that it stopped suddenly, if you can track the latest email maybe you will be able to correlate that with a recent change like an update in the server. When testing notifications I like to create a test rule just to validate if its being triggered and also review the content of the variables that will be used:

1 Like