Docker monitoring: mk_docker.py tries to run check_mk_agent inside all containers; how to enable it only for one container?

Troubleshooting
,
1

Hi,

I run Checkmk Community (formerly Raw) 2.4.0p32 in a Docker container on an Ubuntu 22.04 host. The Checkmk container is named check-mk-raw. With this instance I monitor both the Ubuntu host and all the Docker containers running on it.

The Ubuntu host itself is monitored as host.docker.internal. Docker containers are monitored as separate piggyback hosts. All container hosts are configured as:

  • address_family: no-ip

  • agent: no-agent

  • piggyback: auto-piggyback

  • parent: host.docker.internal

The Docker agent plugin is installed on the Ubuntu host:

/etc/check_mk/docker.cfg
/usr/lib/check_mk_agent/plugins/mk_docker.py

My /etc/check_mk/docker.cfg used:

[DOCKER]
skip_sections: docker_node_disk_usage,docker_node_images
container_id: name
base_url: unix://var/run/docker.sock
persist_period_node_disk_usage: 9

This way I saw tons of the following errors in the Docker daemon logs:

exec: "check_mk_agent": executable file not found in $PATH

All of my containers (but the check-mk-raw container) do not have the Checkmk agent installed, so this error makes sense technically.

So I added docker_container_agent to skip_sections in /etc/check_mk/docker.cfg. After that, a manual agent fetch for host.docker.internal succeeded and no new check_mk_agent Docker exec errors appeared in the Docker journal during the immediate test.

However, this also created a side effect: After skipping docker_container_agent globally, most services in the check-mk-raw host in Checkmk became stale/unknown, which seems logical because Checkmk no longer receives the container-internal agent output via mk_docker.py.

My question:

What is the recommended Checkmk way to handle this situation?

I would like:

  • docker_container_agent skipped for all normal application containers, because they do not contain check_mk_agent.

  • But still monitor the check-mk-raw container with its internal Checkmk agent output.

  • Ideally without causing mk_docker.py to try docker exec check_mk_agent inside every other container every time the Docker plugin runs.

Is there an official or recommended solution for this? In particular, is there a supported way to enable docker_container_agent only for selected containers? Or any other idea?

Thanks!

2

I don’t think there is an official way to do this. In your case i would modify the mk_docker.py to only call this “check_mk_agent” for some specific container names.

3

@andreas-doehler Thank you again for your quick reply. Really appreciated!

I didn’t think about this, but yes, that is a good solution for me. I patched mk_docker.py like this:

- 702:    if not is_disabled_section(config, "docker_container_agent"):
+ 702:    if not is_disabled_section(config, "docker_container_agent") and container_id == "check-mk-raw":

The only downside of this approach for me is that I need to do this repeatedly when updating the plugin. Maybe you could consider introducing a configuration option for this purpose in the Docker plugin? Or check whether the agent is available in a container before trying to execute it?

1 Like