How is mknotifyd working ? No such file or directory ..../spool/**ID**.new***

Troubleshooting
1

I’ve trouble with mknotifyd on one site, it’s crashing several time since 2 weeks…

Enterprise 1.6.0p11; multisite environment.

The crashed site has 2.5k hosts and 70k services and was running smooth untill now.

In mknotifyd.log we can see following:

2021-11-09 09:30:45 [0] FATAL ERROR: [Errno 2] No such file or directory: ‘/omd/sites/site_name/var/check_mk/notify/spool/.e755ddad-7177-4a44-a752-de0e79e9359c.newTOcnKU’
2021-11-09 09:30:45 [0] Traceback (most recent call last):
File “/omd/sites/site_name/lib/python/cmk/cee/mknotifyd/main.py”, line 1564, in main
run_notifyd(paths)
File “/omd/sites/site_name/lib/python/cmk/cee/mknotifyd/main.py”, line 1309, in run_notifyd
handle_spool_files()
File “/omd/sites/site_name/lib/python/cmk/cee/mknotifyd/main.py”, line 535, in handle_spool_files
process_spool_directory(g_spool_dir, 3, not_after=now + 2)
File “/omd/sites/site_name/lib/python/cmk/cee/mknotifyd/main.py”, line 578, in process_spool_directory
for spoolfile, last_modified in get_sorted_spool_files(dir_path):
File “/omd/sites/site_name/lib/python/cmk/cee/mknotifyd/main.py”, line 657, in get_sorted_spool_files
modified_age = now - os.stat(spoolfile_path).st_mtime
OSError: [Errno 2] No such file or directory: ‘/omd/sites/site_name/var/check_mk/notify/spool/.e755ddad-7177-4a44-a752-de0e79e9359c.newTOcnKU’

2021-11-09 09:41:34 [5] -----------------------------------------------------------------

why is mknotifyd searching for a spool fiel with.new*** at the end… ?

hopefully that are enough information

2

Hi,

did you check if your filesystem is full or was full?

Cheers,
Christian

3

Hi,

yes :wink:

10% of file system (mounted for OMD) is available.
inodes are also available.

Host status is also OK:

Cpu(s): 4.2%us, 1.7%sy, 0.0%ni, 92.0%id, 1.7%wa, 0.0%hi, 0.4%si, 0.0%st
Mem: 65971932k total, 60961756k used, 5010176k free, 280076k buffers
Swap: 2105340k total, 53668k used, 2051672k free, 43647996k cached

CheckMK helper usage is between 55 and 75 %

Maximum concurrent Check_MK checks is set to 30
Maximum concurrent active checks is set to 20

Thorsten

4

To state the obvious: You might want to upgrade to the latest patch release.
I understand you did not have the problem before and it just started to happen, but up to date software is never a bad starting point.

5

for sure it’s a good starting point to update the environment, but befor this I’ve to solve some other issues ;-). The important question is, where the .new*** comes from, is it the normal case that every new spool file is named for the first action with .new*** ? I’ve never saw such a file in spool folder…

This environment was stable for month…

BR

6

Any update on this @Thorsten? I am curious, if you were able to fix it.

7

not from my site, I did a restart of mknotifyd ervery 10 min, but this is not the solution.
I can see in my “restart-log” that mknotifyd crashed round about 2 times a day…

first open point from my site is: how is mknotifyd working, is it ok that mknotifyd is searching for a spool file with “.new***”

8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.