Running Legacy Agent to check Docker Swarm container

Troubleshooting
, ,
1

Hello,

CMK version: Checkmk Community (formerly Raw) 2.5.0p11
OS version: Docker Container checkmk/check-mk-community:2.5.0-latest

I’m running a Checkmk Container on host1. The CheckMk instance is working fine and monitoring successfully other Hosts/Services. I want to monitor Docker Services on host2.

Therefore I setup the Legacy Agent via SSH on host2, installed the Docker Plugin and the python3-docker dependencies. I further configured the /etc/check_mk/docker.cfg as follows:

[DOCKER] 
skip_sections: docker_node_disk_usage 
container_id: name

The integration seems to work, I get

Docker containers - Containers: 98, Running: 26, Paused: 0, Stopped: 72

The container names in Docker Swarm are formatted like <container-name>.<instance-id>.<random> so it will become my-container.1.2ts5bccy9vkd1b4kspkuo0x6t. So I added a Host name translation for piggybacked hosts , as a regex ([a-z_\-]*)(.*) → \1. So I get only the container name.

I also configured the piggyback service and I get the running containers on my host1 in tmp/check_mk/piggyback/<container-name>. That’s working fine.

However, if I add now the container name as a new host into checkmk, I’ll get the services discovered, but the host stays DOWN.

Found data from multiple Docker nodes - see service details for more information

As the Services are tied to one specific node, I can exclude, that there are many Docker nodes.

If I change the the setting in docker.cfg to use the container id it works. However, this is not a feasable solution, as I have to change the container name, everytime I restart the service.

If I don’t use the Host name translation for piggybacked hosts the the folders will be named again my-container.1.2ts5bccy9vkd1b4kspkuo0x6t which does not bring any advantage over the container ID.

I assume, all old containers of this services are all merged together in one output file. Which is my-container.aow738t5fgaw7o3gt5, my-container.1.2ts5bccy9vkd1b4kspkuo0x6t and my-container.9y8736g4aowwo3a5f are all merged to my-container leading to confusion within checkmk.

I learned, that the Enterprise Version has the feature DND. Of course this would be probably a solution, but I’m using checkmk for my homelab and therefore no budget for enterprise grade :stuck_out_tongue: .

Is there no other way, to monitor Docker containers which are running in a Swarm with checkmk or am I missing something?
Google and this forum didn’t bring any results. Happy for any good hints, help or idea, how to solve this. Thanks a lot!

2

Unfortunately, I cannot edit the post. Some further diagnostics:

The Check_MK Service of the container host shows:

[piggyback] Successfully processed from source 'host2', execution time 0.1 sec

The Details of the Container Service Docker container status, my-container are:

This docker container apparently exists on multiple parent hosts. This should be reflected in the fact that this host has multiple piggyback sources, see the output of the Check_MK service. Hence, no definitive information on the container can be displayed. To resolve this situation, you have two options: 1. configure the docker agent plug-in to use the container IDs as host names, 2. use the ruleset 'Host name translation for piggybacked hosts' to create unique host names for the affected containers.
3

For the host state CMK normally uses this rule for docker containers
image
That means if the “Docker container status” service is not OK the docker container will be shown as down.
And with your message that there are data from multiple nodes the service will be minimum warning and the container shown as down.

Exactly. You need to find a piggyback renaming rule that leaves every container with a unique name.

This will not help with the renaming and unique names. If you have not more than 750 services in your homelab you can use the pro/ultimate version without a licencse.
For a test if your setup can be done this should be enough.

In the end the problem is the renaming rule. I think you don’t know from the numbering what container is the currently running one?

4

Thanks Andreas, you indeed understood the problem.

At the time, where I translate the name, I need to have the information, which of these containers are currently in a running state from the Docker containers Service on host2. Otherwise I only see a list of containers with a homogenous name with no attribute where I can distinguish the state. And I am not aware, that Docker is able to give running containers a different name.

In my opinion, possible solutions would be:

  • filtering that only for running containers of a Swarm service piggyback data is carried
  • defining somewhere a global variable, injecting the state, which can be used for the renaming rule

But for both of them, I didn’t find a way to do it.

It’s hard for me to believe, that I should be the first one, trying to monitor container on a Docker Swarm node. So I thought, there should be already a solution for this.

5

I do this - but i remove stopped container automatically on my system :wink: