Hello there!
I just want to share my recent experience because I’ve struggled to find info on the web about this problem, so maybe this could help someone in future.
I recently migrated my original installation from 2.1.0 Raw to 2.2.0 Enterprise (first from 2.1.0 Raw to 2.1.0 Enterprise, then to 2.2.0 Enterprise) and also changed the underlying physical server along the way (OS is AlmaLinux 8.8 on both machines). What I’m about to tell happened just after the upgrade from 2.1.0 to 2.2.0.
I unplugged a switch from my infrastructure, without setting downtime on checkmk, this triggered notifications about smartping service being down on that host (and the host itself obviously).
Later I removed the switch also from checkmk config, but the latest notification about smartping service got “stuck” and kept appearing in Monitor > History > Host & service events, every 40-50 seconds or so, though hard notification was not triggered due to other rules.
[User notification] 58 m HOST NOTIFICATION switch_hostname NOTIFY (DOWN) No IP packet received for 15.900722 s (deadline is 15.000000 s)
Basically nothing was happening except for this notification that has spammed the Host & Service events console for 2 days even while the host itself was not present/monitored anymore in checkmk.
Looking through the logs, I finally found something in /omd/sites/*mysite*/var/log/notify.log :
2023-11-27 00:01:51,359 [20] [cmk.base.notify] Got spool file e3825948 (switch_hostname) for local delivery via mail
Looking for a file named e3825948, I found it in /omd/sites/*mysite*/var/check_mk/notify/spool/ and after removed the file e3825948-d0ec-401a-977d-23fd56895228 in that directory and restarted the site, the notification finally has stopped and not flooding the console anymore.
I think it’s possible this file would have been removed by WATO’s Automatic disk space cleanup setting, but I did not want to mess with it while also waiting 30 days and maybe find that notifications again on the 31st day.
Hope this could be helpful to anyone in future.
Cheers!