Agent output is publicly readable without any authentication

Troubleshooting
,
1

Hi everyone,

we deployed two separate CheckMK Instances for our customers. These two are completely separated.

Now, one customer representative contacted us, that it came to his attention, that he can grab CheckMK agent outputs without any authentication from arbitrary public IPs:

❯ netcat <CHECKMK_IP> 6556 | head
<<<check_mk>>>
Version: 2.4.0p11
AgentOS: linux
Hostname: **REDACTED**
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
OSType: linux
...

I redacted and cut the output here, but as you certainly can imagine, the full output contains very sensitive information about the node, including processes (and their commandline args, including secrets), filesystems and their state, systemd units and their status, … you know what’s all in there … nothing you’d like to publicly share with the world.

The other Instance doesn’t have this behavior: Agent port is open there as well, but accessing that one with netcat, it takes a few seconds of just anwering by what looks like “16” and then disconnects with a timeout:

❯ netcat <OTHER_CHECKMK_IP> 6556
16%

# From journald of the destination machine
Jun 03 11:33:06 XXX systemd[1]: Started check-mk-agent@18969-3496566-997.service - Checkmk agent (PID 3496566/UID 997).
Jun 03 11:33:06 XXX kernel: [TTM] Buffer eviction failed
Jun 03 11:33:08 XXX systemd[1]: check-mk-agent@18969-3496566-997.service: Deactivated successfully.
Jun 03 11:33:26 XXX cmk-agent-ctl[3496566]: WARN [cmk_agent_ctl::modes::pull] [::ffff:188.111.4.162]:63714: Request failed. (deadline has elapsed)

In other words: Exactly as it is explained at Monitoring Linux - The new agent for Linux in detail in Section 7.3. Restricting access via IP addresses:

Since only authorized Checkmk servers can retrieve data and unauthorized servers fail after a few bytes of handshake, the risk of a Denial of Service (DoS) attack is very low.

In both cases, we are running:

CMK version: 2.4.0p11
OS version: Debian 11

Executing cmk-agent-ctl status provides something odd:

On the “noisy” instance:

# cmk-agent-ctl status
Version: 2.4.0p11
Agent socket: operational
IP allowlist: any
Legacy mode: enabled
No connections
#

versus on the machine that does not provide a metric-dump to the world:

# cmk-agent-ctl status
Version: 2.4.0p29
Agent socket: operational
IP allowlist: any


Connection: 10.110.0.151/cmk
	UUID: **REDACTED**
	Local:
		Connection mode: pull-agent
		Connecting to receiver port: 8000
		Certificate issuer: Site 'cmk' agent signing CA
		Certificate validity: Mon, 16 Jun 2025 12:43:14 +0000 - Sun, 16 Jun 2030 12:43:14 +0000
	Remote:
		Connection mode: pull-agent
		Hostname: XXX

In the CheckMK Agent Configuration, I can’t find anything to configure authentication or IP limitation. This conforms with what’s written at Monitoring Linux - The new agent for Linux in detail in section 7.3. Restricting access via IP addresses:

[…] For this reason, no further access restriction is currently planned. Of course you can block port 6556 against unauthorized access via iptables. Any rule that may exist and which has been transferred to clients via the Agent Bakery to restrict access to certain IP addresses is ignored by the Agent Controller.

My question here is: What is happening here? What did we possibly do wrong and how (other than enrolling iptables rules on all agent-nodes restricting agent port access manually) can we stop the noisy instance to provide the whole world with our node insights?

2

Please enable TLS encryption on your agents then the agent only speaks with the registered monitoring instance.

Forget to register the TLS. → cmk-agent-ctl register …

It is not the instance that is noise but your monitored servers.

3

The first question that comes to my mind is: Why are both the server and the agent exposed to the public internet in the first place? To me, this points to a more fundamental security issue than the agent output itself being accessible.

From a security best practice perspective, nothing should be exposed directly to the internet unless there is a specific and well justified business requirement. Access should be restricted at multiple layers:

  • Restrict access on the perimeter firewall to the Checkmk server’s source IP address(es).
  • Restrict access on the host’s local firewall as an additional layer of protection.
  • Configure the Checkmk agent to only accept connections from authorized Checkmk servers.
  • Enable TLS for agent communication to provide encryption and authentication.

Limiting access to trusted monitoring systems should be the primary security control. TLS provides an additional layer of protection.