I was considering setting up automatic agent updates.
While looking into that, I hade some security concerns.
Classically, the master gets read-only access to all hosts with monitored agents.
But with automatic agent updates, the master essentially gets remote code execution (usually as root) on all those hosts.
Essentially, this turns the CheckMK master into a security chokepoint, akin to something like an internal artifact repository or a configuration management system.
You don’t expect that from a monitoring solution.
The updates are already signed, but the signing key, as far as I understand, is stored on the master.
A way to increase security would be to separate out the signing process, so the master never gets to know the signing key.
Is that possible?
Is there anything else that can be done to alleviate these security concerns?
But the passphrase for the signing process is in your hand and not stored on the monitoring server.
This passphrase is the most important secret in your environment.
I agree, that there are a few things to consider when using the agent updater, as checkmk takes on the role of software distribution.
Because of that, checkmk has the same security relevance as the most important servers you are monitoring. If an attacker gets access to the checkmk site user, he can easily modify scripts so that in the next agent baking process (innocently executed by a regular checkmk administrator) scripts are distributed to all servers.
So you need to make sure that security best practices are applied to checkmk server:
checkmks operating system is being updated regularly (monitor security updates with e.g. the apt check)
checkmk is being patched regularly
Firewalling is enabled
Encryption is enabled whereever possible (Frontend Apache, checkmk agent, Services like mknotifyd, livestatus)
Public/Private Key Authentication and ssh hardending is in place
External Logging (syslog) is used
Kernel Security Modules like selinux or apparmor
Agent Updater is configured for https and has a signing key assigend
Besides of this
Use the checkmk password store
only create read only accounts on the stuff you are monitoring, if possible
Other disadvantages of the agent bakery are that you don´t have a real staging process like in real Software Distribution Software and that changes in the packages are not very clear / visible.
Sometimes you see the orange bake sign and wonder - what will happen if I click that Button ?
Nevertheless, you have to weigh up the pros and cons and with a proper security strategy, there is nothing to be said against using the agent updater in my view.
@andreas-doehler
As long as the signing takes place on the CheckMK master, the CheckMK master necessarily has access to the unwrapped signing key at some point.
And if the instance is already compromised at that point, the attacker has RCE access to our entire infrastructure.
@aeckstein
I’m not confident everyone involved will intuitively make that connection.
They’ll see it as a monitoring system and not as a software distribution system and not treat it with the priority that requires.
We do not run the Agent as root, with or without automatic updates the baked Agent can be compromised and necessary protection on the checkmk master server should be taken. We do normal things like send audit logs to a remote log server, use privilege access etc.
With 2.3 the Agent will be able to update itself without the need to run as root.
Agent compromise (as in, RCE) didn’t use to be possible due to it not reading anything from the socket.
It was literally just a script that outputs data through a socket opened by xinetd/systemd.
Then the agent got more complicated and actually processed input from the socket.
Am I correct in that this change happened between 2.0 and 2.1?
By not reading enough of the manual, it seems
Thanks for clearing that up.
The agent actually does read from the socket, though, to do the TLS handshake.
So there is some very minor potential for a remote attack on an agent via that route.
