CRIT - no unmonitored services found, no vanished services found, no new host labels, [agent] Communication failed: [Errno 111] Connection refusedCRIT

Alexander · December 30, 2021, 12:40pm

CMK version: 1.6.0p13
OS version: centos 7
I have been trying to get the Check_MK agent into our Linux
systems. I have added Host using WATO configuration from Browser and installed agent into the host. After this Host status is Up and getting the below error

CRIT - no unmonitored services found, no vanished services found, no new host labels, [agent] Communication failed: [Errno 111] Connection refusedCRIT

I have run a /bin/systemctl status xinetd.service to check the status of xinetd in the host and got the below error

xinetd[28923]: bind failed (Address already in use (errno = 98)). service = check_mk
xinetd[28923]: Service check_mk failed to start and is deactivated.

tosch · December 30, 2021, 12:45pm

Hi @Alexander,

seems like something is already using the port checkmk agent uses normally (default TCP 6556).

Alexander · December 30, 2021, 12:51pm

hi, @tosch thanks a lot for joining. I have checked it. Port number is used by
xinetd only. Any way to fix it? Please help me to resolve

tosch · December 30, 2021, 12:53pm

Can you check what happens if you call the agent from the local machine via telnet localhost 6556? Is it returning the agent output or does it fail with an timeout or connection reset by peer?

Alexander · December 30, 2021, 12:56pm

Hi @toschi After issuing telnet command getting agent output with a lot of **loaded failed failed ** status

tosch · December 30, 2021, 1:01pm

Seems like there is something not running properly at all. Can you please run the agent directly via /usr/bin/check_mk_agent? This should return the agent output. If not can you run the agent with option -d (debug) and post the output (please censor sensible data before)? (and please use the preformatted text function from the forum for the output)

Alexander · December 30, 2021, 1:10pm

hi @tosch please check the output i have shared the output partially

check-mk-agent@0-xxx-xx-x.143:6556-xxx-xx-x.188:45558.service               loaded    failed   failed    CheckMK
check-mk-agent@1-xxx-xx-x.143:6556-xxx-xx-x.188:55468.service               loaded    failed   failed    CheckMK
check-mk-agent@10-xxx-xx-x.143:6556-xxx-xx-x.188:59174.service              loaded    failed   failed    CheckMK
check-mk-agent@100-xxx-xx-x.143:6556-xxx-xx-x.188:52026.service             loaded    failed   failed    CheckMK
check-mk-agent@1000-xxx-xx-x.143:6556-xxx-xx-x.188:57546.service            loaded    failed   failed    CheckMK (xxx-xx-x.188:57546)
check-mk-agent@1001-xxx-xx-x.143:6556-xxx-xx-x.188:57706.service            loaded    failed   failed    CheckMK (xxx-xx-x.188:57706)
check-mk-agent@1002-xxx-xx-x.143:6556-xxx-xx-x.188:58138.service            loaded    failed   failed    CheckMK (xxx-xx-x.188:58138)
check-mk-agent@1003-xxx-xx-x.143:6556-xxx-xx-x.188:58454.service            loaded    failed   failed    CheckMK (xxx-xx-x.188:58454)

Alexander · December 30, 2021, 1:15pm

Hi @tosch These are begining of log of the above

<<<check_mk>>>
Version: 1.6.0p13
AgentOS: linux
Hostname: ip-xxx-xx-x-143.ec2.internal
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
<<<df>>>
tmpfs          tmpfs      940152       0    940152       0% /dev/shm
tmpfs          tmpfs      940152   25092    915060       3% /run
tmpfs          tmpfs      940152       0    940152       0% /sys/fs/cgroup
/dev/xvda1     xfs       8377344 1858608   6518736      23% /
tmpfs          tmpfs      188032       0    188032       0% /run/user/1000
<<<df>>>
[df_inodes_start]
tmpfs          tmpfs  235038     1  235037    1% /dev/shm
tmpfs          tmpfs  235038   385  234653    1% /run
tmpfs          tmpfs  235038    16  235022    1% /sys/fs/cgroup
/dev/xvda1     xfs   4193792 33817 4159975    1% /
tmpfs          tmpfs  235038     1  235037    1% /run/user/1000
[df_inodes_end]
<<<systemd_units>>>
[list-unit-files]
UNIT FILE                                     STATE   
proc-sys-fs-binfmt_misc.automount             static  
dev-hugepages.mount                           static  
dev-mqueue.mount                              static  
proc-fs-nfsd.mount                            static  
proc-sys-fs-binfmt_misc.mount                 static  
sys-fs-fuse-connections.mount                 static  
sys-kernel-config.mount                       static  
sys-kernel-debug.mount                        static  
tmp.mount                                     masked  
var-lib-nfs-rpc_pipefs.mount                  static  
brandbot.path                                 disabled
systemd-ask-password-console.path             static  
systemd-ask-password-wall.path                static  
session-231.scope                             static  
session-233.scope                             static  
arp-ethers.service                            disabled
auditd.service                                enabled 
auth-rpcgss-module.service                    static  
autovt@.service                               enabled 
blk-availability.service                      disabled
brandbot.service                              static  
check-mk-agent@.service                       static  
chrony-dnssrv@.service                        static  
chrony-wait.service                           disabled
chronyd.service                               enabled 
cloud-config.service                          enabled 
cloud-final.service                           enabled 
cloud-init-local.service                      enabled 
cloud-init.service                            enabled 
console-getty.service                         disabled
console-shell.service                         disabled
container-getty@.service                      static  
cpupower.service                              disabled
crond.service                                 enabled 
dbus-org.freedesktop.hostname1.service        static  
dbus-org.freedesktop.import1.service          static  
dbus-org.freedesktop.locale1.service          static  
dbus-org.freedesktop.login1.service           static  
dbus-org.freedesktop.machine1.service         static  
dbus-org.freedesktop.timedate1.service        static  
dbus.service                                  static  
debug-shell.service                           disabled
dracut-cmdline.service                        static  
dracut-initqueue.service                      static  
dracut-mount.service                          static  
dracut-pre-mount.service                      static  
dracut-pre-pivot.service                      static  
dracut-pre-trigger.service                    static  
dracut-pre-udev.service                       static  
dracut-shutdown.service                       static  
emergency.service                             static  
fstrim.service                                static  
getty@.service                                enabled 
gssproxy.service                              disabled
halt-local.service                            static  
htcacheclean.service                          static  
httpd.service                                 enabled 
initrd-cleanup.service                        static  
initrd-parse-etc.service                      static  
initrd-switch-root.service                    static  
initrd-udevadm-cleanup-db.service             static  
irqbalance.service                            enabled 
kdump.service                                 enabled 
kmod-static-nodes.service                     static  
messagebus.service                            static  
microcode.service                             enabled 
nfs-blkmap.service                            disabled
nfs-config.service                            static  
nfs-idmap.service                             static  
nfs-idmapd.service                            static  
nfs-lock.service                              static  
nfs-mountd.service                            static  
nfs-rquotad.service                           disabled
nfs-secure.service                            static  
nfs-server.service                            disabled
nfs-utils.service                             static  
nfs.service                                   disabled
nfslock.service                               static  
polkit.service                                static  
postfix.service                               enabled 
qemu-guest-agent.service                      enabled 
quotaon.service                               static  
rc-local.service                              static  
rdisc.service                                 disabled
rescue.service                                static  
rhel-autorelabel-mark.service                 disabled
rhel-autorelabel.service                      enabled 
rhel-configure.service                        enabled 
rhel-dmesg.service                            enabled 
rhel-domainname.service                       enabled 
rhel-import-state.service                     enabled 
rhel-loadmodules.service                      enabled 
rhel-readonly.service                         enabled 
rpc-gssd.service                              static  
rpc-rquotad.service                           disabled
rpc-statd-notify.service                      static  
rpc-statd.service                             static  
rpcbind.service                               enabled 
rpcgssd.service                               static  
rpcidmapd.service                             static  
rsyncd.service                                disabled
rsyncd@.service                               static  
rsyslog.service                               enabled 
selinux-policy-migrate-local-changes@.service static  
serial-getty@.service                         disabled
sshd-keygen.service                           static  
sshd.service                                  enabled 
sshd@.service                                 static  
systemd-ask-password-console.service          static  
systemd-ask-password-wall.service             static  
systemd-backlight@.service                    static  
systemd-binfmt.service                        static  
systemd-bootchart.service                     disabled
systemd-firstboot.service                     static  
systemd-fsck-root.service                     static  
systemd-fsck@.service                         static  
systemd-halt.service                          static  
systemd-hibernate-resume@.service             static  
systemd-hibernate.service                     static  
systemd-hostnamed.service                     static  
systemd-hwdb-update.service                   static  
systemd-hybrid-sleep.service                  static  
systemd-importd.service                       static  
systemd-initctl.service                       static  
systemd-journal-catalog-update.service        static  
systemd-journal-flush.service                 static  
systemd-journald.service                      static  
systemd-kexec.service                         static  
systemd-localed.service                       static  
systemd-logind.service                        static  
systemd-machine-id-commit.service             static  
systemd-machined.service                      static  
systemd-modules-load.service                  static  
systemd-nspawn@.service                       disabled
systemd-poweroff.service                      static  
systemd-quotacheck.service                    static  
systemd-random-seed.service                   static  
systemd-readahead-collect.service             enabled 
systemd-readahead-done.service                indirect
systemd-readahead-drop.service                enabled 
systemd-readahead-replay.service              enabled 
systemd-reboot.service                        static  
systemd-remount-fs.service                    static  
systemd-rfkill@.service                       static  
systemd-shutdownd.service                     static  
systemd-suspend.service                       static  
systemd-sysctl.service                        static  
systemd-timedated.service                     static  
systemd-tmpfiles-clean.service                static  
systemd-tmpfiles-setup-dev.service            static  
systemd-tmpfiles-setup.service                static  
systemd-udev-settle.service                   static  
systemd-udev-trigger.service                  static  
systemd-udevd.service                         static  
systemd-update-done.service                   static  
systemd-update-utmp-runlevel.service          static  
systemd-update-utmp.service                   static  
systemd-user-sessions.service                 static  
systemd-vconsole-setup.service                static  
teamd@.service                                static  
tuned.service                                 enabled 
wpa_supplicant.service                        disabled
xinetd.service                                enabled 
-.slice                                       static  
machine.slice                                 static  
system.slice                                  static  
user-1000.slice                               static  
user.slice                                    static  
check-mk-agent.socket                         enabled 
dbus.socket                                   static  
rpcbind.socket                                enabled 
rsyncd.socket                                 disabled
sshd.socket                                   disabled
syslog.socket                                 static  
systemd-initctl.socket                        static  
systemd-journald.socket                       static  
systemd-shutdownd.socket                      static  
systemd-udevd-control.socket                  static  
systemd-udevd-kernel.socket                   static  
basic.target                                  static  
bluetooth.target                              static  
cloud-config.target                           static  
cloud-init.target                             static  
cryptsetup-pre.target                         static  
cryptsetup.target                             static  
ctrl-alt-del.target                           disabled
default.target                                enabled 
emergency.target                              static  
final.target                                  static  
getty-pre.target                              static  
getty.target                                  static  
graphical.target                              static  
halt.target                                   disabled
hibernate.target                              static  
hybrid-sleep.target                           static  
initrd-fs.target                              static  
initrd-root-fs.target                         static  
initrd-switch-root.target                     static  
initrd.target                                 static  
kexec.target                                  disabled
local-fs-pre.target                           static  
local-fs.target                               static  
machines.target                               disabled
multi-user.target                             enabled 
network-online.target                         static  
network-pre.target                            static  
network.target                                static  
nfs-client.target                             enabled 
nss-lookup.target                             static  
nss-user-lookup.target                        static  
paths.target                                  static  
poweroff.target                               disabled
printer.target                                static  
reboot.target                                 disabled
remote-cryptsetup.target                      disabled
remote-fs-pre.target                          static  
remote-fs.target                              enabled 
rescue.target                                 disabled
rpc_pipefs.target                             static  
rpcbind.target                                static  
runlevel0.target                              disabled
runlevel1.target                              disabled
runlevel2.target                              enabled 
runlevel3.target                              enabled 
runlevel4.target                              enabled 
runlevel5.target                              static  
runlevel6.target                              disabled
shutdown.target                               static  
sigpwr.target                                 static  
sleep.target                                  static  
slices.target                                 static  
smartcard.target                              static  
sockets.target                                static  
sound.target                                  static  
suspend.target                                static  
swap.target                                   static  
sysinit.target                                static  
system-update.target                          static  
time-sync.target                              static  
timers.target                                 static  
umount.target                                 static  
chrony-dnssrv@.timer                          disabled
fstrim.timer                                  disabled
systemd-readahead-done.timer                  indirect
systemd-tmpfiles-clean.timer                  static  

245 unit files listed.
[all]
  UNIT                                                                        LOAD      ACTIVE   SUB       
  proc-sys-fs-binfmt_misc.automount                                           loaded    active   waiting   Arbitrary Executable File Formats File System Automount Point
  dev-disk-by\x2duuid-f41e390f\x2d835b\x2d4223\x2da9bb\x2d9b45984ddf8d.device loaded    active   plugged   /dev/disk/by-uuid/f41e390f-835b-4223-a9bb-9b45984ddf8d
  dev-ttyS0.device                                                            loaded    active   plugged   /dev/ttyS0
  dev-ttyS1.device                                                            loaded    active   plugged   /dev/ttyS1
  dev-ttyS2.device                                                            loaded    active   plugged   /dev/ttyS2
  dev-ttyS3.device                                                            loaded    active   plugged   /dev/ttyS3
  dev-xvda.device                                                             loaded    active   plugged   /dev/xvda
  dev-xvda1.device                                                            loaded    active   plugged   /dev/xvda1
  sys-devices-platform-serial8250-tty-ttyS1.device                            loaded    active   plugged   /sys/devices/platform/serial8250/tty/ttyS1
  sys-devices-platform-serial8250-tty-ttyS2.device                            loaded    active   plugged   /sys/devices/platform/serial8250/tty/ttyS2
  sys-devices-platform-serial8250-tty-ttyS3.device                            loaded    active   plugged   /sys/devices/platform/serial8250/tty/ttyS3
  sys-devices-pnp0-00:06-tty-ttyS0.device                                     loaded    active   plugged   /sys/devices/pnp0/00:06/tty/ttyS0
  sys-devices-vbd\x2d768-block-xvda-xvda1.device                              loaded    active   plugged   /sys/devices/vbd-768/block/xvda/xvda1
  sys-devices-vbd\x2d768-block-xvda.device                                    loaded    active   plugged   /sys/devices/vbd-768/block/xvda
  sys-devices-vif\x2d0-net-eth0.device                                        loaded    active   plugged   /sys/devices/vif-0/net/eth0
  sys-module-configfs.device                                                  loaded    active   plugged   /sys/module/configfs
  sys-subsystem-net-devices-eth0.device                                       loaded    active   plugged   /sys/subsystem/net/devices/eth0
  -.mount                                                                     loaded    active   mounted   /
  dev-hugepages.mount                                                         loaded    active   mounted   Huge Pages File System
  dev-mqueue.mount                                                            loaded    active   mounted   POSIX Message Queue File System
  proc-fs-nfsd.mount                                                          loaded    inactive dead      NFSD configuration filesystem
  proc-sys-fs-binfmt_misc.mount                                               loaded    inactive dead      Arbitrary Executable File Formats File System
  run-user-1000.mount                                                         loaded    active   mounted   /run/user/1000
  sys-fs-fuse-connections.mount                                               loaded    inactive dead      FUSE Control File System
  sys-kernel-config.mount                                                     loaded    active   mounted   Configuration File System
  sys-kernel-debug.mount                                                      loaded    active   mounted   Debug File System
* sysroot.mount                                                               not-found inactive dead      sysroot.mount
* tmp.mount                                                                   masked    inactive dead      tmp.mount
  var-lib-nfs-rpc_pipefs.mount                                                loaded    active   mounted   RPC Pipe File System
  systemd-ask-password-console.path                                           loaded    active   waiting   Dispatch Password Requests to Console Directory Watch
  systemd-ask-password-wall.path                                              loaded    active   waiting   Forward Password Requests to Wall Directory Watch
  session-231.scope                                                           loaded    active   running   Session 231 of user centos
  session-233.scope                                                           loaded    active   running   Session 233 of user centos
  auditd.service                                                              loaded    active   running   Security Auditing Service
  auth-rpcgss-module.service                                                  loaded    inactive dead      Kernel Module supporting RPCSEC_GSS

tosch · December 30, 2021, 1:23pm

Seems like it’s a problem persisting between all versions and no final fix available. You can work around by running systemctl reset-failed check-mk-agent@*. But’s only a temporary fix if one of the agents fails.
Reason behind this issue is a not properly ended agent process and systemd keeps the unit active but failed.
Related posts i found (partially in german):

Alexander · December 30, 2021, 1:35pm

Thanks @tosch But Still the issue exists after tried sudo systemctl reset-failed and sudo check_mk_agent -v

bmst · January 5, 2022, 3:14am

@tosch TCP 6556, not UDP.

@Alexander do a netstat -tnlp (TCP, Numbers, Listening, Program/PID) to see what’s listening at 6556 already. Then kill it I suspect it will be the systemd listener, since I see check-mk-agent.socket enabled in you output there.

FWIW the only time I had this sort problem was also that the systemd listener was also installed (and for whatever reason wasn’t working - I’ve always had bad luck with it…).

reset-failed won’t fix anything, it just clears the failed flag on dead services so that systemctl status says the system state is clean. You’ll wanna systemctl stop check-mk-agent.socket; systemctl disable check-mk-agent.socket (and possibly that check-mk-agent@.service also, I don’t think I’ve seen that before, doubt you’ll need it using xinetd). You should then be able to start xinetd just fine.

Let me know how that goes.

tosch · January 5, 2022, 7:36am

Fixed my post, thanks for correcting.

Alexander · January 7, 2022, 8:42am

Thanks a lot @bmst This helped me to resolve the issue. And now I can monitor the VM. Also really thank you @tosch for Jump in to issue

Alexander · January 10, 2022, 6:22am

Hi, @tosch @bmst After disabling check-mk-agent.socket I have started xinetd after a few minutes Getting CRIT - [agent] keepalive timed outCRIT, Got no information from host, execution time 60.0 sec. I have checked the status of checkmk agent by issuing systemctl status check-mk-agent@.services and got Stopped. what should we do? Thanks in advance

bmst · January 11, 2022, 7:21am

If you’re running with xinetd, you shouldn’t need any service about checkmk besides xinetd itself.

Alexander · January 21, 2022, 4:48pm

hi, @tosch @bmst still the issue persisted… after a few moments Keep alive agent stopped working. and the monitoring does not work further

system · January 21, 2023, 4:49pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.