SW Inventory missing Name, Creation, Labels, Status, Image for Docker containers (only ID is shown)

gregor.hoffleit · January 14, 2025, 4:28pm

CMK version: 2.3.0p23.cme
OS version: Debian 11

On our CME site, the SW inventory shows only the ID of containers. The fields “Name”, “Create”, “Labels”, “Status” and “Image” are empty.

The nodes run these versions:

Docker Engine 27.5.1 or 27.5.0 or 27.4.1
Ubuntu 22.04 / Debian 11
mk_inventory.linux 2.3.0p23
mk_docker.py 2.3.0p23
python3-docker 5.0.3-1 resp. 4.1.0-1.2 (deb package)

Only options in docker.cfg are “skip_sections: docker_container_agent” and “container_id: name”.

“Search docker images” has full data for the images on the nodes. Except the column “# Containers”, which is empty…

robin.gierse · January 23, 2025, 9:56am

Are the hosts monitored on a central or remote site?
If it is monitored on a remote site, is the information available there?
Did you restart all involved sites already?

gregor.hoffleit · January 23, 2025, 10:44am

Some hosts are monitored on the central site, others are on remote sites. But the information is missing on the remote sites as well. Problem persists after restarts of all sites…

I just noticed that for the containers on the Docker nodes on the central site, all information is available in the inventory.

For all nodes monitored from remote sites, the information for the containers is missing. It’s missing on the central site as well as on the remote sites.

When I download the agent output of one of the Docker nodes, all the missing information pieces about the containers seems to be present.

There’s a section “<<<docker_node_images:sep(0)>>>” followed by a subsection “[[[images]]]” with appearently directories about the images and after that there’s a subsection “[[[container]]]” with (very long) directories about the containers. These directories seem to contain all the missing information. The directories have the full id of the containers, though (as “Id”), whereas other parts of the agent output only have the short ids.

In the agent output, I don’t see a difference between the containers with full information (from the central site) and the containers where the information is not shown in the inventory (i.e. those monitored on remote sites).

robin.gierse · January 24, 2025, 3:16pm

Is the Livestatus Proxy Daemon enabled for all sites?

gregor.hoffleit · January 24, 2025, 4:37pm

Yes, LIVEPROXYD is “on” on all sites. On the remote sites, LIVESTATUS_TCP is configured as well, on the central site LIVESTATUS_TCP is off.

jodok.glabasna · January 28, 2025, 9:20am

This is how a docker container looks in my site in Check_MK HW/SW Inventory

If you are using distributed monitoring, first check this data on the site that
actually monitors the docker hosts and has got the piggyback data of the containers,
because sometimes the sync of inventory data from remote to central is sometimes broken.
And this is another topic.

Also make sure that you are using the mk_docker.py on your docker host that matches your checkmk agent and server version.

Third, make sure you are using a recent required docker python module.
I am using 7.0.0 installed with pip as root since my Ubuntu 22.04 only provides 5.0.x

recent enough

root@dockerhost:/$ pip3 freeze | grep docker
docker==7.0.0

too old

root@dockerhost:/$ apt search python3-docker
...
python3-docker/jammy 5.0.3-1 al

(pip3 install docker) I think 6.0.x is the recent one?

You can also actually look into the inventory data under the hood:
Use cat <file> | ruff format - or black -q - < <file> to make it more human readable

E.g.:
OMD[central23]:~$ cat var/check_mk/inventory/graylog-opensearch-1 | ruff format -

You can even do funny things like converting the python data to json to be able to filter it with jq

Check if the name can be found in the datastructure:

OMD[central23]:~$ python3 -c 'import sys, json; print(json.dumps( (eval( sys.stdin.read() ) ), indent=4, sort_keys=True ) )' < var/check_mk/inventory/graylog-opensearch-1 | jq -r '.Nodes.software.Nodes.applications.Nodes.docker.Nodes.container.Nodes.networks.Table.Rows[0].name'

graylog_graylog

The idea is to follow the data.
If it is already missing in the ~/var/check_mk/inventory/ file than the problem is most likely mk_docker.py not creating this data which could be due to an to old or even to recent pip docker python module.

gregor.hoffleit · January 28, 2025, 12:21pm

mk_docker.py is uptodate (2.3.0p23, see OP).

Docker python module is 5.0.3-1 resp. 4.1.0-1.2 (package python3-docker from Ubuntu/Debian).

According to the documentation, this should be fine: “Note, that the docker Python library is required (not docker-py). At least Version 2.6.1 is necessary.”

Are you sure the information in the documentation is no longer valid?

My inventory (Docker section) looks quite different from yours:

Well, as I wrote: I tried to follow the data. The agent output, i.e. the data collected by mk_docker.py and sent by the agent on the node to Checkmk. That agent output contains (very detailed) information about the containers.

As far as I understand, that agent output is parsed by Checkmk and then it goes into the inventory and the checks.

I looked at the ~/var/check_mk/inventory/ files and detailed information about the containers is missing (although detailed information about the images is present).

Therefore it seems that Checkmk is not able to parse the information sent by the agent.

jodok.glabasna · January 28, 2025, 1:32pm

Do you have this rule active with “Do status data inventory” ?

if yes, then it could help to try to update the python docker module to 7.0.x

gregor.hoffleit · January 28, 2025, 2:48pm

Thanks for the hint, “data inventory” was not enabled. I enabled it for the Docker nodes and upgraded the python docker module to version 7.1.0.

It didn’t help, though. I guess I’ll wait a few hours.

Still, why does your Docker inventory look so different? For me, it’s (see above):

Docker
[table with information about Docker]
- Docker containers
  - [table of containers: Columns ID, Creation, …]
- Docker images
  - [table of images: Columns ID, Creation, …]
- Networks
  - network1: “Bridge”
    - [table with properties of the network]

Yours is (see your posting):

Docker
- Container
  Labels: [list of labels]
  - Networks
    - [table of networks]
  - Ports
    - [table of ports]

Is this a real screenshot from Checkmk or a mockup? What Checkmk version are you running?

jodok.glabasna · January 28, 2025, 4:25pm

The screenshots are from Checkmk Enterprise Edition 2.3.0p24
running on my laptop on Ubuntu 22.04 Jammy.

I set up the docker monitoring via backery rule:

docker.cfg looks like this:

root@dockernode:~$ cat /etc/check_mk/docker.cfg 
# Created by Check_MK Agent Bakery.
# This file is managed via WATO, do not edit manually or you
# lose your changes next time when you update the agent.

[DOCKER]
# skip_sections: no sections skipped
container_id: name

I can send you the output of mk_docker.py on my system if you want, perhaps you can spot a difference.

I am using this docker flavour:

root@dockernode:~$ docker info
Client: Docker Engine - Community
 Version:    27.5.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4
    Path:     /usr/libexec/docker/cli-plugins/docker-compose


root@dockernode:~$ which docker
/usr/bin/docker

root@dockernode:~$ dpkg -S /usr/bin/docker
docker-ce-cli: /usr/bin/docker

root@dockernode:~$ apt-cache show docker-ce-cli | head
Package: docker-ce-cli
Architecture: amd64
Version: 5:27.5.1-1~ubuntu.22.04~jamm

andreas-doehler · January 28, 2025, 6:45pm

That’s not needed and also actively discouraged by the actual distributions → system wide pip install.
The version 5.0.3 works perfectly fine on all of my docker hosts (Ubuntu 24.04 or Debian 12)
Ubuntu 24.04 / Python docker 7.1.0 system wide pip

Debian 12 / Python docker 5.0.3 deb package looks the same.

All the problems look more like a execution problem of the mk_docker.py than a sync problem between master and slave sites.
What happens if you execute the “mk_docker.py --debug -v -c /etc/check_mk/docker.cfg” on a machine monitored by the remote site?

gregor.hoffleit · January 29, 2025, 10:32am

Output looks nice and verbose. This is what goes to stderr (“-v”):

 INFO: read configration file(s): ['/etc/check_mk/docker.cfg']
 INFO: 'set_version_info' took 0.009522199630737305s
 INFO: 'section_node_info' took 0.0002760887145996094s
 INFO: 'section_node_disk_usage' took 1.1028037071228027s
 INFO: 'section_node_images' took 0.05076432228088379s
 INFO: 'section_node_network' took 0.003683805465698242s
 INFO: container id: grafana-alloy
 INFO: container id: nginx_1
 INFO: container id: drupal_1
 INFO: container id: redis_1
 INFO: container id: solr_1
 INFO: skipped section: docker_container_agent
 INFO: container id: micro_service_1
 INFO: skipped section: docker_container_agent
 INFO: skipped section: docker_container_agent
 INFO: container id: micro_helper_1
 INFO: container id: micro_server_1
 INFO: skipped section: docker_container_agent
 INFO: skipped section: docker_container_agent
 INFO: skipped section: docker_container_agent
 INFO: skipped section: docker_container_agent
 INFO: skipped section: docker_container_agent

I’m not able to post the output. But I confirmed that in the section “[[[containers]]]” there’s full information about each container. E.g. (pretty-printed):

[[[containers]]]
{"Id": "3e4b290c61aa7c350804350716ac5881efa69c705cfa14034cb1e97976c81348",  
     "Created": "2025-01-28T18:06:57.645034264Z",                               
     "Path": "/bin/alloy",                                                      
     "Args": ["run", "--disable-reporting", "--storage.path=/data-alloy", "--stability.level=public-preview", "--server.http.listen-addr=0.0.0.0:12345", "/etc/alloy/my.alloy"],
     "State": {"Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1691171, "ExitCode": 0, "Error": "", "StartedAt": "2025-01-28T18:06:57.822586441Z", "FinishedAt": "0001-01-01T00:00:00Z"},
     "Image": "sha256:52f26bdcc72c9e988dc41ca48d2845cb5a562a3f372c1d06e8ee5f58f03538a8",
...
     "Config":
         {"Hostname": "3e4b290c61aa", "Domainname": "", "User": "",
          ...
          "Image": "grafana/alloy:latest",
          ...
          "Labels": {"com.docker.compose.config-hash": "ab16fbbc241975eceab435d28b623c7010ae5c889c725304cf8dc4c85291e12e", "com.d    ocker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:52f26bdcc72c9e988dc41ca48d2845cb5a562a3f372c1d06e8ee5f58f03538a8", "    com.docker.compose.oneoff": "False", "com.docker.compose.project": "alloy", "com.docker.compose.project.config_files": "/home/my/alloy/docker-compose.yml,/home/my/alloy/docker-compose.my.yml", "com.docker.compose.project.working_dir": "/home/my/alloy", "com.docker.compose.service": "alloy", "com.docker.compose.version": "2.32.1"    , "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.source": "https://github.com/grafana/alloy", "org.opencontainers.image.version": "24.04", "otel.deployme    nt.environment.name": "my", "otel.service.name": "alloy", "otel.service.namespace": "alloy-my", "otel.service.version": "latest"}
     },

That’s also what I found in the agent output (“Download agent output” in Checkmk ), so I’m pretty sure mk_docker.py is reporting the data to the Checkmk server.

system · January 29, 2026, 10:32am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.