2.1.0p2: agent is not registered

Troubleshooting
#1

I cannot make agents working with the new “registration” feature.
Ubuntu 18.04 with systemd 237

The register command exists without giving any output after:

Do you want to establish this connection? [Y/n]
Y

and the cmk-agent-ctl status says:

Version: 2.1.0p2
Agent socket: operational
IP allowlist: any

Connection: [...]:8000/mk
        UUID: b6947f24-[...]
        Local:
                Connection type: pull-agent
                Certificate issuer: Site 'mk' local CA
                Certificate validity: Wed, 08 Jun 2022 14:28:28 +0000 - Mon, 09 Oct 3020 14:28:28 +0000
        Remote:
                Connection type: pull-agent
                Registration state: operational
                Host name: [...]

But the same was happening in p1 even if there was a permission issue, after updating to p2 I saw that error and fixed it by giving more permissions until the error has disappeared, I tried with admin login as well, but it’s the same…

The connection test on the host says:

Agent
Agent controller not registered

What can be the issue?

#2

In my case I had the xinetd services (from the old agents) still running. I had to uninstall xinetd (removing the configuration files first) and then uninstall and reinstall the agent. Since than the connection is established, stable and TLS secured.

#3

thanks, unfortunately I already removed xinetd time ago, replaced with systemd first and now I’m trying with the new agent… I correctly have cmk-agent-ctl listening on 6556:

# ss -tulpn | grep 6556
tcp    LISTEN   0        128                     *:6556                 *:*      users:(("cmk-agent-ctl",pid=30716,fd=9))

if I telnet from the server to the host on port 6556 I get only 16 as an answer:

$  telnet **** 6556
Trying ****::...
Connected to ****.
Escape character is '^]'.
16

Which means the connection is ok, there are no firewalls blocking it

#4

Also another issue, the agent installed on the server to monitor itself is not working anymore:

image
image978×202 16.1 KB

even if I telnet 127.0.0.1 6556 I get the agent output. Here I cannot remove xinetd, because if I try it says it will remove check-mk-raw-2.1.0p2 and all the dependencies as well.

Here is the connection test:

image
image1287×485 24.7 KB

#5

Same issues persist after updating both client and server to 2.1.0p3

#6

On the server you can remove xinetd (with check-mk) and immidate reinstall check-mk after that.
Be sure, that you have a good backup, first :crazy_face:.

On linux hosts the agent deactivates xinetd configuration automatically, but we saw still xinetd services running which prevent the agent-service from starting.

systemctl

should list the services in “running”

cmk-agent-ctl-daemon.service loaded active running Checkmk agent controller daemon
check-mk-agent-async.service loaded active running Checkmk agent - Asynchronous background tasks

Did you tried the steps descriped here https://docs.checkmk.com/latest/en/agent_linux.html#test ?

1 Like
#7
#8

Thank you, unfortunately it didn’t work, after remove + reinstall I still have xinetd, I also spawned a new instance with Ubuntu 22, but I have xinetd installed:

$ apt list --installed | grep xinetd
xinetd/jammy,now 1:2.3.15.3-1 amd64 [installed,automatic]

#9

on the clients, I removed xinetd with apt purge xinetd so it’s not running (systemctl | grep xine shows nothing), I have these services running:

check-mk-agent-async.service loaded active running Checkmk agent - Asynchronous background tasks
system-check\x2dmk\x2dagent.slice loaded active active system-check\x2dmk\x2dagent.slice
check-mk-agent.socket loaded active listening Local Checkmk agent socket
cmk-agent-ctl-daemon.service loaded active running Checkmk agent controller daemon

Since I’m still struggling with this issue, I have spawned a brand new server with Ubuntu 22, installed checkmk raw, switched the public IP from the old server to the new one (got hundreds of monitored servers with a firewall rule to allow 6556 to that IP only), configured apache to have SSL, then imported the backup from the old server (with omd backup + omd restore)

Then I installed the agent on the server, and registered, and this part finally worked, the server can monitor itself (even if I don’t understand why cmk-agent-ctl register --hostname 127.0.0.1 gave 404 Not Found: Host 127.0.0.1 does not exist., I had to put the public hostname).

However, all the other hosts doesn’t work, I see the same issue as before… :frowning:

To recap, the issue is the following:

  • ss -tulpn says cmk-agent-ctl is listening on 6556
  • register command is successful (status says Remote: Connection type: pull-agent, Registration state: operational) but the monitoring host doesn’t get any output (no unmonitored services found, 18 vanished services)
  • If I do cmk-agent-ctl delete-all --enable-insecure-connections it works again but I have the annoying WARN TLS is not activated on monitored host

I think this whole TLS thing should be optional without showing any warning, since there are much simpler ways to secure the connection, like a simple firewall rule: ufw allow from 1.2.3.4 to any port 6556 proto tcp EDIT: I found I can do this in Setup>Services>Service monitoring rules>Checkmk Agent installation auditing>New rule: Checkmk Agent installation auditing