CMK Agent Controller not working (CentOS 7) - CMK 2.1

scoopex · May 25, 2022, 8:19pm

New thread as suggested from Agent Problems after update to 2.1.0 - #27 by scoopex

The cmk-agent-ctl daemon is not running at all and relevant services got disabled during installation of the rpm package since systemd v219 has been found and xinetd has been selected:

[root@jumphost ~]# rpm -ivh check-mk-agent-2.1.0-1.noarch.rpm --force
Preparing… ################################# [100%]

Updating / installing…
1:check-mk-agent-2.1.0-1 ################################# [100%]

The Checkmk agent may require features that are either buggy,
or not even supported in systemd versions prior to 220.
We recommend using xinetd on such old systems.
Deployed xinetd
Creating/updating cmk-agent user account …
Deactivating systemd unit ‘check-mk-agent-async.service’…
Deactivating systemd unit ‘check-mk-agent.socket’…
Deactivating systemd unit ‘cmk-agent-ctl-daemon.service’…
Reloading xinetd
Redirecting to /bin/systemctl reload xinetd.service
[root@jumphost ~]#

[root@jumphost /]# systemctl stop cmk-agent-ctl-daemon.service
Failed to stop cmk-agent-ctl-daemon.service: Unit cmk-agent-ctl-daemon.service not loaded.
[root@jumphost /]#

Agent Controller is not running, no config files can be found in the systemd directory and within xinetd.d, only the check-mk-agent can be found:

[root@jumphost ~]# cd /etc/systemd/system
[root@jumphost system]# ls -l
total 48
drwxr-xr-x. 2 root root 4096 Apr 29 2021 basic.target.wants
drwxr-xr-x. 2 root root 4096 Feb 24 20:59 cloud-init.target.wants
lrwxrwxrwx. 1 root root 57 Apr 29 2021 dbus-org.freedesktop.nm-dispatcher.service → /usr/lib/systemd/system/NetworkManager-dispatcher.service
lrwxrwxrwx. 1 root root 37 Apr 29 2021 default.target → /lib/systemd/system/multi-user.target
drwxr-xr-x. 2 root root 4096 Apr 29 2021 default.target.wants
drwxr-xr-x. 2 root root 4096 Apr 29 2021 dev-virtio\x2dports-org.qemu.guest_agent.0.device.wants
drwxr-xr-x. 2 root root 4096 Apr 29 2021 getty.target.wants
drwxr-xr-x. 2 root root 4096 Apr 29 2021 local-fs.target.wants
drwxr-xr-x. 2 root root 4096 May 24 14:54 multi-user.target.wants
drwxr-xr-x. 2 root root 4096 Apr 29 2021 network-online.target.wants
drwxr-xr-x. 2 root root 4096 May 16 15:37 sockets.target.wants
drwxr-xr-x. 2 root root 4096 Apr 29 2021 sysinit.target.wants
drwxr-xr-x. 2 root root 4096 Apr 29 2021 system-update.target.wants
drwxr-xr-x. 2 root root 4096 Apr 29 2021 timers.target.wants
[root@jumphost system]# cd /etc/xinetd.d/
[root@jumphost xinetd.d]# ls -l
total 48
-rw------- 1 root root 1157 Apr 1 2020 chargen-dgram
-rw------- 1 root root 1159 Apr 1 2020 chargen-stream
-rw-r–r-- 1 root root 1938 May 25 22:15 check-mk-agent
-rw------- 1 root root 1157 Apr 1 2020 daytime-dgram
-rw------- 1 root root 1159 Apr 1 2020 daytime-stream
-rw------- 1 root root 1157 Apr 1 2020 discard-dgram
-rw------- 1 root root 1159 Apr 1 2020 discard-stream
-rw------- 1 root root 1148 Apr 1 2020 echo-dgram
-rw------- 1 root root 1150 Apr 1 2020 echo-stream
-rw------- 1 root root 1212 Apr 1 2020 tcpmux-server
-rw------- 1 root root 1149 Apr 1 2020 time-dgram
-rw------- 1 root root 1150 Apr 1 2020 time-stream
[root@jumphost xinetd.d]#

The check-mk-agent is running in the legacy xinetd method:

[root@jumphost /]# ss -tulpn | grep 6556
tcp LISTEN 0 64 [::]:6556 [::]:* users:((“xinetd”,pid=932,fd=5))
[root@jumphost /]#

Trying to register the client fails with:

[root@jumphost]# cmk-agent-ctl register --hostname myclient \

--server cmkserver --site mysite \

ERROR [cmk_agent_ctl] Something seems wrong with the agent socket (/run/check-mk-agent.socket), aborting
[root@jumphost xinetd.d]#

Regards

mschlenker · May 25, 2022, 8:30pm

OK, let’s figure out who is doing what. All commands to be executed on the host to be monitored. First, is someone claiming port 6556, and who is doing so?

ss -tulpn | grep 6556

Second: Did someone try registration and what is the state?

cmk-agent-ctl status

Please also provide

systemctl --version

We all all knew that what works in 98% of all cases might be some hard work in the remaining 2%. Let’s face it.

scoopex · May 25, 2022, 8:33pm

All commands were executed on the host to be monitored.

1.) ss -tulpn | grep 6556

[root@jumphost /]# ss -tulpn | grep 6556
tcp    LISTEN     0      64     [::]:6556               [::]:*                   users:(("xinetd",pid=932,fd=5))
[root@jumphost /]#

2.) cmk-agent-ctl status

[root@jumphost /]# cmk-agent-ctl status
Version: 2.1.0
Agent socket: inoperational (!!)
IP allowlist: any
Legacy mode: enabled
No connections
[root@jumphost /]#

3.) systemctl --version

[root@jumphost /]# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
[root@jumphost /]#

mschlenker · May 25, 2022, 8:41pm

So please stop and disable the services for the new agent controller:

systemctl stop cmk-agent-ctl-daemon.service
systemctl stop check-mk-agent.socket
systemctl disable cmk-agent-ctl-daemon.service
systemctl disable check-mk-agent.socket

Then restart your xinetd (or just reboot). Afterwards, port 6556 should be claimed by xinetd and

nd localhost 6556

should give you full agent output.

scoopex · May 25, 2022, 8:47pm

Output:

[root@jumphost /]# systemctl stop cmk-agent-ctl-daemon.service
Failed to stop cmk-agent-ctl-daemon.service: Unit cmk-agent-ctl-daemon.service not loaded.
[root@jumphost /]# systemctl stop check-mk-agent.socket
Failed to stop check-mk-agent.socket: Unit check-mk-agent.socket not loaded.
[root@jumphost /]# systemctl disable cmk-agent-ctl-daemon.service
Failed to execute operation: No such file or directory
[root@jumphost /]# systemctl disable check-mk-agent.socket
Failed to execute operation: No such file or directory
[root@jumphost /]# service xinetd restart
Redirecting to /bin/systemctl restart xinetd.service
[root@jumphost /]# nd localhost 6556
-bash: nd: command not found
[root@jumphost /]# ss -tulpn | grep 6556
tcp    LISTEN     0      64     [::]:6556               [::]:*                   users:(("xinetd",pid=4729,fd=5))
[root@jumphost /]#

mschlenker · May 25, 2022, 8:54pm

Ooopsy, sorry, command should be nc. So now you must de-register, on the host:

cmk-agent-ctl delete-all --enable-insecure-connections

Then on the CMK server: Properties of host , then menu entry Host > Remove TLS registration
Afterwards connections should work albeit insecure.

andreas-doehler · May 25, 2022, 8:57pm

I think here is a small miss understanding.

The first post shows a fully normal installation of an agent on a legacy system (CentOS 7).
There is no real problem from my viewpoint.
As long as the CentOS 7 has an older systemd as version 220 only the legacy agent will work.

I don’t see any information that the agent is not working on this system.

mschlenker · May 25, 2022, 9:01pm

Yes, but there might be some edge conditions where a client is able to register for TLS on such hosts. This has to be reverted to go back to old unencrypted communication.

scoopex · May 25, 2022, 9:02pm

OK just for the record: the new CMK Agent Controller (+ registration process - regardless if encrypted via TLS or unencryped) is expected not to work on CentOS 7 systems (using xinetd because systemd is V219) ?

Thanks

andreas-doehler · May 25, 2022, 9:04pm

That’s exactly what the message from the install package says. Old system with systemd < 220 → only xinetd is supported and no agent controller

mschlenker · May 25, 2022, 9:07pm

Yes. Systemd lower than 220 might or might not work so just assume it does not work. So just install xinetd and get together everything else afterwards. Your message says xinetd is already installed, so clean up.

I am sorry, but that’s the way life is. And my job is just documenting the whole thing.

scoopex · May 25, 2022, 9:08pm

OK spot on - probably the message is not clear enough - the message states Checkmk agent only - it´s not obvious at all that the new Checkmk agent controller is not supported on legacy systems/systems with systemd lower than 220 …

Thanks anyway.

mschlenker · May 25, 2022, 9:12pm

Exacly. The old style agent will always work: Monitoring Linux in legacy mode

The new Agent Controller probably will not in some edge cases. We will help when this is the case and hint on the way to go.

Blisk · June 7, 2022, 4:37pm

THANK YOU! This helps my Centos 7 server to connect to checkmk

system · June 7, 2023, 4:38pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.