Check_mk agent not working anymore

Troubleshooting
1

Version: 2.0.0p16
Ubuntu 20.04.3 LTS

Check_mk agent already worked on the server, but now it doesn`t any more:

systemctl status check_mk.socket
● check_mk.socket - Check_MK Agent Socket
     Loaded: loaded (/etc/systemd/system/check_mk.socket; enabled; vendor preset: enabled)
     Active: active (listening) since Thu 2021-12-09 04:00:16 UTC; 1 weeks 4 days ago
     Listen: [::]:6556 (Stream)
   Accepted: 18321; Connected: 0;
      Tasks: 0 (limit: 38433)
     Memory: 752.0K
     CGroup: /system.slice/check_mk.socket

systemctl status check-mk-agent.socket
● check-mk-agent.socket
     Loaded: not-found (Reason: Unit check-mk-agent.socket not found.)
     Active: failed (Result: resources)

Dez 20 14:03:47 bbelobuild systemd[2353493]: check-mk-agent.socket: Failed to create listening socket ([::]:6556): Address already in use
Dez 20 14:03:47 bbelobuild systemd[1]: check-mk-agent.socket: Failed to receive listening socket ([::]:6556): Input/output error
Dez 20 14:03:47 bbelobuild systemd[1]: check-mk-agent.socket: Failed to listen on sockets: Input/output error
Dez 20 14:03:47 bbelobuild systemd[1]: check-mk-agent.socket: Failed with result 'resources'.
Dez 20 14:03:47 bbelobuild systemd[1]: Failed to listen on Checkmk agent socket.
Dez 20 14:04:21 bbelobuild systemd[2353699]: check-mk-agent.socket: Failed to create listening socket ([::]:6556): Address already in use
Dez 20 14:04:21 bbelobuild systemd[1]: check-mk-agent.socket: Failed to receive listening socket ([::]:6556): Input/output error
Dez 20 14:04:21 bbelobuild systemd[1]: check-mk-agent.socket: Failed to listen on sockets: Input/output error
Dez 20 14:04:21 bbelobuild systemd[1]: check-mk-agent.socket: Failed with result 'resources'.
Dez 20 14:04:21 bbelobuild systemd[1]: Failed to listen on Checkmk agent socket.

netstat -lpn | grep 6556
tcp6       0      0 :::6556                 :::*                    LISTEN      1/init

Agent test from check mk server:

Communication failed: [Errno 111] Connection refused<<<esx_vsphere_vm:cached(1640011866,90)>>>
time_reference 1640008256
config.guestFullName Ubuntu Linux (64 Bit)
config.hardware.device virtualDeviceType VirtualCdrom|label CD-/DVD-Laufwerk 1|summary Remote-ATAPI|startConnected false|allowGuestControl true|connected false|status ok@@virtualDeviceType VirtualVmxnet3|label Netzwerkadapter 1|summary VM Network|startConnected true|allowGuestControl true|connected true|status ok
config.hardware.memoryMB 32768
config.hardware.numCPU 10
config.hardware.numCoresPerSocket 2
config.template false
config.uuid 4237e84c-71df-4fd1-90c7-b9de7937b208
config.version vmx-14
guest.toolsVersion 11360
guest.toolsVersionStatus guestToolsUnmanaged
guestHeartbeatStatus green
name olo
runtime.host 172.16.100.219
runtime.powerState poweredOn
summary.guest.hostName olo
summary.quickStats.balloonedMemory 0
summary.quickStats.compressedMemory 14028
summary.quickStats.consumedOverheadMemory 149
summary.quickStats.distributedCpuEntitlement 86
summary.quickStats.distributedMemoryEntitlement 4303
summary.quickStats.guestMemoryUsage 327
summary.quickStats.hostMemoryUsage 13823
summary.quickStats.overallCpuDemand 86
summary.quickStats.overallCpuUsage 86
summary.quickStats.privateMemory 13242
summary.quickStats.sharedMemory 19376
summary.quickStats.staticCpuEntitlement 2762
summary.quickStats.staticMemoryEntitlement 33207
summary.quickStats.swappedMemory 136
summary.quickStats.uptimeSeconds 18774182
<<<labels:sep(0)>>>
{"cmk/piggyback_source_vmcenter": "yes"}
2

There might be residual xinetd configuration that is blocking the systemd socket.
Check for /etc/xinetd.d/check_mk or similar. If it exists delete it and restart xinetd.
After that, restart the checkmk socket.

3

Should not be the problem - netstat shows that systemd(init) has the port.

@horst how does the netstat looks like after you stop the socket manually?

4

when I stop the socket, the netstat command “netstat -lpn | grep 6556” doesn`t give any output back.

the /etc/xinetd.d/ folder does contain a check_mk file with following content:

# +------------------------------------------------------------------+
# |             ____ _               _        __  __ _  __           |
# |            / ___| |__   ___  ___| | __   |  \/  | |/ /           |
# |           | |   | '_ \ / _ \/ __| |/ /   | |\/| | ' /            |
# |           | |___| | | |  __/ (__|   <    | |  | | . \            |
# |            \____|_| |_|\___|\___|_|\_\___|_|  |_|_|\_\           |
# |                                                                  |
# | Copyright Mathias Kettner 2014             mk@mathias-kettner.de |
# +------------------------------------------------------------------+
#
# This file is part of Check_MK.
# The official homepage is at http://mathias-kettner.de/check_mk.
#
# check_mk is free software;  you can redistribute it and/or modify it
# under the  terms of the  GNU General Public License  as published by
# the Free Software Foundation in version 2.  check_mk is  distributed
# in the hope that it will be useful, but WITHOUT ANY WARRANTY;  with-
# out even the implied warranty of  MERCHANTABILITY  or  FITNESS FOR A
# PARTICULAR PURPOSE. See the  GNU General Public License for more de-
# tails. You should have  received  a copy of the  GNU  General Public
# License along with GNU Make; see the file  COPYING.  If  not,  write
# to the Free Software Foundation, Inc., 51 Franklin St,  Fifth Floor,
# Boston, MA 02110-1301 USA.

service check_mk
{
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/bin/check_mk_agent

        # listen on IPv4 AND IPv6 when available on this host
        #flags          = IPv6

        # If you use fully redundant monitoring and poll the client
        # from more then one monitoring servers in parallel you might
        # want to use the agent cache wrapper:
        #server         = /usr/bin/check_mk_caching_agent

        # configure the IP address(es) of your Nagios server here:
        #only_from      = 127.0.0.1 10.0.20.1 10.0.20.2

        # Don't be too verbose. Don't log every check. This might be
        # commented out for debugging. If this option is commented out
        # the default options will be used for this service.
        log_on_success =

        disable        = no
}

the strange thing is that “systemctl status x → tab” does not give xinetd back, only this commands are available:
x11-common.service
xfs_scrub_all.service
xfs_scrub_all.timer

could this be the problem?
how do I get xinetd back running?

5

For myself it looks like there is no xinetd installed on this machine.

6

I agree, xinetd is not the reason for your trouble, but the file can be cleaned up anyway.

I hate to say this, but did you reboot the server already?

7

ok, shame on me. After the server reboot the check MK worked quite well again.
I dont know how and why I got this problem, but I´m happy its working again.

Thanks for your help!

8

Sounds like a kernel related issue, maybe the port was not freed properly, but I am not as deep into the Linux kernel as I would like to be.

Also, I hate it, when ‘Windows-style troubleshooting’ works, but hey, at least your agent works properl now. :wink:

9

just restart the check_mk and it will work. At least it was the case on my side
systemctl restart check_mk.socket

10

Sometimes the OS keeps file descriptors and ports open, so a restart of the service might not suffice in all cases, but it should in most.

11

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.