UP/DOWN Status of Docker Containers not Updating

Troubleshooting
1

I have an Ubuntu 22.04 host on which I run Docker. Check_MK Raw Edition (v2.3.0p13) runs as a Docker container. On the Host, a Check_MK agent runs. Also the Docker plugin runs. In Check_MK I see the host plus all running Docker containers, like this:

image
image914×522 13.7 KB

When I stop a container, I observe the following:

  • Docker Container Host state stays in (green) “UP” state
  • Docker Container Service “Check_MK“ switches to WARN
  • Docker Container Service “Check_MK Discovery“ switches to WARN
  • Other Docker Container Services switch to stale mode

The “Check_MK” service reports something like:
[piggyback] Successfully processed from source ‘host.docker.internal’, Missing monitoring data for plugins, cpu_utilization_os WARN, diskstat WARN, docker_container_status WARN, docker_container_status_uptime WARN, mem_used WARN, execution time 0.0 sec

Details:

image
image1196×538 44.1 KB

When I go to Setup > Hosts > Host monitoring rules > Host check command > Edit rule: Host check command, I see this rule:

image
image1041×781 29.7 KB

This lets me assume that the Docker Container Host state should depend on the “Docker container status”. However, as mentioned above, this service does not switch to WARN/ERROR state, but switches to stale mode when the container is stopped/removed and still reports “Container running on node …”.

So if I click a stopped/removed Container host, and then click the container name, I see this page:

image
image1187×724 27.5 KB

I would like that a stopped/removed Docker Container Host switches to (yellow or red) “DOWN” host state. How to achieve this?

Any help is appreciated.

2

Additional information:

I noticed, that if the container fails to start and is configured to automatically restart, the “Docker container status” service reports “restarting” instead of “running”. Thus the container host state switches to “DOWN”, which is fine.

However, as described above, if the container is completely stopped/removed, the “Docker container status” service switches to stale mode (still reporting the last status of “running”), and (probably) thus the container host state stays in (green) UP state.

Any idea how to solve/fix this?

3

Any feedback?
Please let me know if the issue is not described clear enough. I would happily provide more information …