Check-mk-agent-2.2.0_raw_edition_systemd_issue

General
,
1

Hi, i had a hard time to get the new 2.2.0 Agent working on our debian systems (buster).

After upgrading from 2.1.p28 the new agent does not listening on port 6556 anymore.
After digging around i found that the cmk-agent-ctl-daemon.service showing the following entries:

Started Checkmk agent controller daemon.
/lib/systemd/system/cmk-agent-ctl-daemon.service:22: Unknown lvalue 'RestrictSUIDSGID' in section 'Service', ignoring
/lib/systemd/system/cmk-agent-ctl-daemon.service:23: Unknown lvalue 'ProtectKernelLogs' in section 'Service', ignoring
/lib/systemd/system/cmk-agent-ctl-daemon.service:24: Unknown lvalue 'ProtectClock' in section 'Service', ignoring
/lib/systemd/system/cmk-agent-ctl-daemon.service:26: Unknown lvalue 'ProtectHostname' in section 'Service', ignoring
/lib/systemd/system/cmk-agent-ctl-daemon.service:22: Unknown lvalue 'RestrictSUIDSGID' in section 'Service', ignoring
/lib/systemd/system/cmk-agent-ctl-daemon.service:23: Unknown lvalue 'ProtectKernelLogs' in section 'Service', ignoring
/lib/systemd/system/cmk-agent-ctl-daemon.service:24: Unknown lvalue 'ProtectClock' in section 'Service', ignoring
/lib/systemd/system/cmk-agent-ctl-daemon.service:26: Unknown lvalue 'ProtectHostname' in section 'Service', ignoring

After removing these lines everything works again.
Can someone explain why these lines should be in there??

Thanks in advance

Oliver

2

Hi Oliver,

I see the same messages on a Debian buster in journald logs.
But as the log entries state out, they should just be ignored, so they shouldn’t prevent the controller from running properly

Have opened an internal ticket on the same.

3

…thanks for the comment.
All i can say is, that my “old” service File from 2.1p28 is working and does not contain all the entries like this:

# added v214
ProtectHome=yes
ProtectSystem=full

# added v231
RestrictRealtime=yes

# added v232
ProtectControlGroups=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RemoveIPC=yes

# added v235
LockPersonality=yes

# added v242
RestrictSUIDSGID=yes

# added v244
ProtectKernelLogs=yes

# added v245
ProtectClock=yes

# added v239
PrivateMounts=yes

# added v242
ProtectHostname=yes

[Install]
WantedBy=multi-user.target
pe or paste code here

The log message indicates that it is ignored, but it seems that it’s not!

Regards

Oliver

For Reference: previous service file looks like this:

[Unit]
Description=Checkmk agent controller daemon
After=network.target

[Service]
ExecStart=/usr/bin/cmk-agent-ctl daemon
Type=simple
Restart=on-failure

User=cmk-agent

[Install]
WantedBy=multi-user.target
4

You see at the comments what version of systemd you need that the option is supported.
Debian buster as example has only v241 and that means all the options for newer systemd will not work on this version.

5

…thanks that explains it clearly!
But i think that’s not working like expected - at least on our systems:

I have:

systemd --version
systemd 241 (241)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

So everything below v242 should work without issues!

Even when activate the single “ProtectedHome=yes” feature introduced in v214 the agent is not listening anymore!
If i comment out everything except the install section - everything works and i get:

tcp    LISTEN  0       128      *:6556       *:*    users:(("cmk-agent-ctl",pid=14798,fd=9))
6

Hi Oliver!

Thanks for sharing this!
We should take this problem seriously, as it may affect many users.

I’d like to know which of the parameters actually cause a problem here.
You already found out that ProtectHome=yes seems to break something.
Would it be possible for you to enable other parameters one-by-one and to find out whether you can track this down all problem-causing ones?

You can also increase the log-level of cmk-agent-ctl by adding -vv to the ExecStart parameter:
(Then see the log with journalctl -u cmk-agent-ctl-daemon)

...
[Service]
ExecStart=/usr/bin/cmk-agent-ctl -vv daemon
Type=simple
Restart=on-failure
...

If we’re lucky, cmk-agent-ctl may tell us why it’s failing.

Cheers
Andi

7

…hello Andi,

i found out that only the option “ProtectedHome=yes” causes the agent not listening.
with all the other options activated - there is no problem! Here is the log:

INFO [cmk_agent_ctl] starting
INFO [cmk_agent_ctl] Loaded config from '"/home/cmk-agent/cmk-agent-ctl.toml"', connection registry from '"/home/cmk-agent/registered_connections.js
INFO [cmk_agent_ctl::modes::daemon] Could not load pre-configured connections from "/home/cmk-agent/pre_configured_connections.json": No such file o
Jun 02 13:33:58 wg cmk-agent-ctl[30636]: DEBUG [cmk_agent_ctl::misc] Sleeping 55s to avoid DDOSing of sites
DEBUG [cmk_agent_ctl::misc] Sleeping 3s to avoid DDOSing of sites
INFO [cmk_agent_ctl::modes::pull] Start listening for incoming pull requests
INFO [cmk_agent_ctl::modes::pull] Listening on [::]:6556 for incoming pull connections (IPv6 & IPv4 if activated)
DEBUG [cmk_agent_ctl::modes::renew_certificate] Checking registered connections for certificate expiry.
INFO [cmk_agent_ctl::modes::pull] [::ffff:10.1.0.3]:55088: Handling pull request.
DEBUG [cmk_agent_ctl::modes::pull] [::ffff:10.1.0.3]:55088: Handling pull request DONE (Task detached).
DEBUG [cmk_agent_ctl::modes::pull] handle_request starts
DEBUG [rustls::server::hs] decided upon suite TLS13_AES_256_GCM_SHA384
DEBUG [cmk_agent_ctl::modes::pull] processed task!
INFO [cmk_agent_ctl::modes::pull] [::ffff:10.1.0.3]:40258: Handling pull request.
DEBUG [cmk_agent_ctl::modes::pull] [::ffff:10.1.0.3]:40258: Handling pull request DONE (Task detached).
DEBUG [cmk_agent_ctl::modes::pull] handle_request starts
DEBUG [rustls::server::hs] decided upon suite TLS13_AES_256_GCM_SHA384
DEBUG [cmk_agent_ctl::modes::pull] processed task!

With ProtectedHome activated the log looks like this:

Stopping Checkmk agent controller daemon...
cmk-agent-ctl-daemon.service: Main process exited, code=killed, status=15/TERM
cmk-agent-ctl-daemon.service: Succeeded.
Stopped Checkmk agent controller daemon.
Started Checkmk agent controller daemon.
INFO [cmk_agent_ctl] starting
INFO [cmk_agent_ctl] Loaded config from '"/home/cmk-agent/cmk-agent-ctl.toml"', connection registry from '"/home/cmk-agent/registered_connections.js
INFO [cmk_agent_ctl::modes::daemon] Could not load pre-configured connections from "/home/cmk-agent/pre_configured_connections.json": Permission den
DEBUG [cmk_agent_ctl::misc] Sleeping 39s to avoid DDOSing of sites
DEBUG [cmk_agent_ctl::misc] Sleeping 56s to avoid DDOSing of sites

So it seems that ProtectedHome=yes ist the problem!

Oliver

8

Thanks for tracking this down!

This explains the problem:

INFO [cmk_agent_ctl] Loaded config from '"/home/cmk-agent/cmk-agent-ctl.toml"', connection registry from '"/home/cmk-agent/registered_connections.json
INFO [cmk_agent_ctl::modes::daemon] Could not load pre-configured connections from "/home/cmk-agent/pre_configured_connections.json": Permission denied

ProtectHome=yes prevents the systemd service from accessing anything under /home, in order to protect data of other users of the system.

But the cmk-agent’s home directory should be located at /var/lib/cmk-agent, rather than at /home/cmk-agent

Did you initlially create the cmk-agent user manually?

9

…hello Andi,

i checked our systems and there is indeed already a cmk-agent user under
/home
This fact causes the cmk-agent-useradd.sh script to fail!

/var/lib/cmk-agent/scripts/cmk-agent-useradd.sh 
Creating/updating cmk-agent user account ...
cmk-agent user found:  expected 'Checkmk agent system user:/var/lib/cmk-agent:/bin/false'
                      but found ':/home/cmk-agent:/bin/false'

So all this is a local problem on our systems and has nothing to do with the agent or systemd!

Thank you very much for pointing me to the right direction. I modified the install scripts and removed
the cmk user under /home - everything is working fine now.

thx again

Oliver