CMK version:2.2.0p14
OS version:Check MK virtual Appliance FW1.6.7
I have a problem with the event console. For some time now my cmc.log has been full with the following message.
After restarting Check MK it runs briefly, but then this message comes back.
** [client 0] error: [mkeventd at /omd/sites/XXXXXXXX/tmp/run/mkeventd/status] cannot send request: Transport endpoint is not connected:**
Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
What does omd status say?
Is there anything in cmc.log, liveproxyd.log, or mkeventd.log?
Hello Robin,
this is the output from omd status
Overall state: running
This entry occurs most frequently in the cmc.log.
2023-11-30 08:01:52 [4] [client 0] error: [mkeventd at /omd/sites/xxxxxxx/tmp/run/mkeventd/status] cannot send request: Transport endpoint is not connected
The others are service notification logs.
This is the last entry in liveproxyd.log.1
2023-11-27 15:10:57,889 [20] [cmk.liveproxyd] ----------------------------------------------------------
2023-11-27 15:10:57,890 [20] [cmk.liveproxyd] Livestatus Proxy-Daemon (2.2.0p14) starting...
2023-11-27 15:10:57,895 [20] [cmk.liveproxyd] Configured 0 sites
These are the entrys from mkeventd.log
2023-11-27 08:11:46,458 [20] [cmk.mkeventd] -----------------------------------------------------------------
2023-11-27 08:11:46,458 [20] [cmk.mkeventd] mkeventd version 2.2.0p14 starting
2023-11-27 08:11:46,460 [20] [cmk.mkeventd.EventServer] Created FIFO '/omd/sites/xxxxxxx/tmp/run/mkeventd/events' for receiving events
2023-11-27 08:11:46,460 [20] [cmk.mkeventd.EventServer] Opened UNIX socket '/omd/sites/xxxxxxx/tmp/run/mkeventd/eventsocket' for receiving events
2023-11-27 08:11:46,460 [20] [cmk.mkeventd.EventServer] Opened builtin syslog server on inherited filedescriptor 3
2023-11-27 08:11:46,460 [20] [cmk.mkeventd.EventServer] Opened builtin syslog-tcp server on inherited filedescriptor 4
2023-11-27 08:11:46,460 [20] [cmk.mkeventd.EventServer] Opened builtin snmptrap server on inherited filedescriptor 5
2023-11-27 08:11:46,494 [20] [cmk.mkeventd.EventServer.snmp] adding SNMPv1 system: communityIndex=snmpv2-1
2023-11-27 08:11:46,589 [20] [cmk.mkeventd.EventServer.snmp] Loaded 39 SNMP MIB modules
2023-11-27 08:11:46,608 [20] [cmk.mkeventd.EventStatus] Loaded event state from /omd/sites/xxxxxxx/var/mkeventd/status.
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] Compiled 1 active rules (ignoring 0 disabled rules)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] Rule hash: 1 rules - 0 hashed, 1 unspecific
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] kern : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] user : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] mail : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] daemon : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] auth : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] syslog : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] lpr : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] news : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] uucp : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] cron : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] authpriv : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] ftp : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] ntp : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] logaudit : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] logalert : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] clock : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] local0 : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] local1 : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] local2 : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] local3 : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] local4 : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] local5 : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,609 [20] [cmk.mkeventd.EventServer] local6 : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,610 [20] [cmk.mkeventd.EventServer] snmptrap : emerg(1) alert(1) crit(1) err(1) warning(1) notice(1) info(1) debug(1)
2023-11-27 08:11:46,617 [20] [cmk.mkeventd] Daemonized with PID 21923.
2023-11-27 08:11:46,621 [20] [cmk.mkeventd.StatusServer] Starting up
2023-11-27 08:11:46,622 [20] [cmk.mkeventd.EventServer] Starting up
2023-11-27 08:11:59,321 [40] [cmk.mkeventd.EventServer] Exception in EventServer server
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 226, in run
self.serve()
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 735, in serve
self.process_raw_lines(message, address)
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 785, in process_raw_lines
if line := scrub_string(line_bytes.rstrip().decode("utf-8")):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 1: invalid start byte
2023-11-27 08:12:00,323 [40] [cmk.mkeventd.EventServer] Exception in EventServer server
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 226, in run
self.serve()
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 735, in serve
self.process_raw_lines(message, address)
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 785, in process_raw_lines
if line := scrub_string(line_bytes.rstrip().decode("utf-8")):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 1: invalid start byte
2023-11-27 08:16:58,267 [40] [cmk.mkeventd.EventServer] Exception in EventServer server
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 226, in run
self.serve()
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 735, in serve
self.process_raw_lines(message, address)
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 785, in process_raw_lines
if line := scrub_string(line_bytes.rstrip().decode("utf-8")):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 589, in open_pipe
return os.open(str(self.settings.paths.event_pipe.value), os.O_RDWR | os.O_NONBLOCK)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 24] Too many open files: '/omd/sites/xxxxxxx/tmp/run/mkeventd/events'
2023-11-27 09:17:38,234 [40] [cmk.mkeventd] Exception in main thread
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 2561, in run_eventd
event_status.save_status()
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 2740, in save_status
with path_new.open(mode="wb") as f:
^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/xxxxxxx/lib/python3.11/pathlib.py", line 1044, in open
OSError: [Errno 24] Too many open files: '/omd/sites/xxxxxxx/var/mkeventd/status.new'
2023-11-27 09:17:38,396 [40] [cmk.mkeventd.StatusServer] Error handling client None: [Errno 24] Too many open files
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 2204, in serve
client_socket, addr_info = s.accept()
^^^^^^^^^^
File "/omd/sites/xxxxxxx/lib/python3.11/socket.py", line 294, in accept
OSError: [Errno 24] Too many open files
2023-11-27 09:17:38,597 [40] [cmk.mkeventd.StatusServer] Error handling client None: [Errno 24] Too many open files
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 2204, in serve
client_socket, addr_info = s.accept()
^^^^^^^^^^
File "/omd/sites/xxxxxxx/lib/python3.11/socket.py", line 294, in accept
OSError: [Errno 24] Too many open files
2023-11-27 09:17:38,798 [40] [cmk.mkeventd.StatusServer] Error handling client None: [Errno 24] Too many open files
Traceback (most recent call last):
File "/omd/sites/OTNCAMPUS/lib/python3/cmk/ec/main.py", line 2204, in serve
client_socket, addr_info = s.accept()
^^^^^^^^^^
File "/omd/sites/xxxxxxx/lib/python3.11/socket.py", line 294, in accept
OSError: [Errno 24] Too many open files
2023-11-27 09:17:38,999 [40] [cmk.mkeventd.StatusServer] Error handling client None: [Errno 24] Too many open files
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 2204, in serve
client_socket, addr_info = s.accept()
^^^^^^^^^^
File "/omd/sites/xxxxxxx/lib/python3.11/socket.py", line 294, in accept
OSError: [Errno 24] Too many open files
2023-11-27 09:17:39,200 [40] [cmk.mkeventd.StatusServer] Error handling client None: [Errno 24] Too many open files
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 2204, in serve
client_socket, addr_info = s.accept()
^^^^^^^^^^
File "/omd/sites/xxxxxxx/lib/python3.11/socket.py", line 294, in accept
OSError: [Errno 24] Too many open files
2023-11-27 09:17:39,206 [40] [cmk.mkeventd.EventServer] Exception in EventServer server
Traceback (most recent call last):
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 226, in run
self.serve()
File "/omd/sites/xxxxxxx/lib/python3/cmk/ec/main.py", line 596, in serve
pipe = self.open_pipe()
Thanks in advance
Looks like there are too many open files. Too many open files is a limit imposed by the kernel and can be adapted if necessary on OS level. But as you are using the appliance, it is not allowed for you to change these values.
I recommend you open a support ticket, so our support engineers can look into this matter more in-depth.
Thanks for watching. I’ll open a ticket.
Have a nice day.
Greetings Sven