CPU Load not reported (cmk raw 2.2.0, Debian 11, LXC)

I am provisioning a small instance for testing purposes. Currently this includes a couple of Debian LXCs and Centos VM target hosts. The Centos hosts list “CPU Load” as a monitored service, but it is absent from the Debian hosts. Running a discovery against the Debian hosts finds nothing new.

Agent installed using .deb downloaded from monitoring host.

CMK version: 2.2.0p12.cre (raw)
OS version: Debian 11

Error message: n/a (CPU load is not reported in services)

Output of “cmk --debug -vvn hostname”:

Checkmk version 2.2.0p12
+ FETCHING DATA
  Source: SourceInfo(hostname='cmk-satellite.southwold.net', ipaddress='10.0.0.104', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f33836dea50]
Read from cache: AgentFileCache(cmk-satellite.southwold.net, path_template=/omd/sites/southwold/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 14 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 10.0.0.104:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Reading data from agent
Closing TCP connection to 10.0.0.104:6556
Write data to cache file /omd/sites/southwold/tmp/check_mk/cache/cmk-satellite.southwold.net
Trying to acquire lock on /omd/sites/southwold/tmp/check_mk/cache/cmk-satellite.southwold.net
Got lock on /omd/sites/southwold/tmp/check_mk/cache/cmk-satellite.southwold.net
Releasing lock on /omd/sites/southwold/tmp/check_mk/cache/cmk-satellite.southwold.net
Released lock on /omd/sites/southwold/tmp/check_mk/cache/cmk-satellite.southwold.net
[cpu_tracking] Stop [7f33836dea50 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.6900000013411045))]
  Source: SourceInfo(hostname='cmk-satellite.southwold.net', ipaddress='10.0.0.104', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f3383574790]
Read from cache: NoCache(cmk-satellite.southwold.net, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for 'cmk-satellite.southwold.net'. Skip processing.
No piggyback files for '10.0.0.104'. Skip processing.
[cpu_tracking] Stop [7f3383574790 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<lxc_container_cpu_cgroupv2>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_status:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_ssl_certs:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<mkeventd_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cmk_site_statistics:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_status:cached(1699272727,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<mknotifyd:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_apache:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_diskusage:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_info:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
  HostKey(hostname='cmk-satellite.southwold.net', source_type=<SourceType.HOST: 1>)  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'cifsmounts', 'cmk_agent_ctl_status', 'cmk_site_statistics', 'df_v2', 'diskstat', 'job', 'labels', 'livestatus_ssl_certs', 'livestatus_status', 'lnx_bonding', 'lnx_if', 'local', 'lxc_container_cpu_cgroupv2', 'md', 'mem', 'mkeventd_status', 'mknotifyd', 'mounts', 'nfsmounts_v2', 'omd_apache', 'omd_diskusage', 'omd_info', 'omd_status', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  HostKey(hostname='cmk-satellite.southwold.net', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
[cpu_tracking] Start [7f3382a03d90]
value store: synchronizing
Trying to acquire lock on /omd/sites/southwold/tmp/check_mk/counters/cmk-satellite.southwold.net
Got lock on /omd/sites/southwold/tmp/check_mk/counters/cmk-satellite.southwold.net
value store: loading from disk
Releasing lock on /omd/sites/southwold/tmp/check_mk/counters/cmk-satellite.southwold.net
Released lock on /omd/sites/southwold/tmp/check_mk/counters/cmk-satellite.southwold.net
CPU utilization      Total CPU: 5.07%
Check_MK Agent       Version: 2.2.0p12, OS: linux, TLS is not activated on monitored host (see details), Agent plugins: 0, Local checks: 0
Disk IO SUMMARY      Read: 106 kB/s, Write: 140 kB/s, Latency: 803 microseconds
Filesystem /         Used: 23.31% - 2.73 GiB of 11.7 GiB, trend per 1 day 0 hours: -13.3 MiB, trend per 1 day 0 hours: -0.11%
Interface 2          [eth0], (up), MAC: DA:46:B0:73:EB:90, Speed: 10 GBit/s, In: 175 B/s (<0.01%), Out: 5.47 kB/s (<0.01%)
Memory               Total virtual memory: 17.89% - 733 MiB of 4.00 GiB, 8 additional details available
Mount options of /   Mount options exactly as expected
OMD satellite Event Console Current events: 0, Virtual memory: 191 MiB, Overall event limit inactive, No hosts event limit active, No rules event limit active, Received messages: 0.00/s, Rule hits: 0.00/s, Rule tries: 0.00/s, Message drops: 0.00/s, Created events: 0.00/s, Client connects: 0.06/s, Rule hit ratio: -, Processing time per message: -, Time per client request: 0.21 ms
OMD satellite apache No activity since last check
OMD satellite disk usage Total: 9.76 MiB, Agents: 0 B, Core: 24.8 KiB, History: 0 B, Inventory: 0 B, Local: 296 KiB, Logs: 972 KiB, PNP4Nagios: 2.60 MiB, RRDs: 0 B, Tmp: 180 KiB
OMD satellite performance Livestatus version: 2.2.0p12, Host checks: 0.0/s, Service checks: 0.0/s, Notifications are disabled(!!)
OMD satellite status running
Postfix Queue        Deferred queue length: 0, Active queue length: 0
Postfix status       Status: the Postfix mail system is running, PID: 910
Site satellite statistics Total hosts: 0, Problem hosts: 0, Total services: 0, Problem services: 0
Systemd Service Summary Total: 105, Disabled: 7, Failed: 0
Systemd Socket Summary Total: 14, Disabled: 0, Failed: 1, 1 static socket failed (systemd-journald-audit)(!!)
TCP Connections      Established: 1
Uptime               Up since Oct 31 2023 15:03:23, Uptime: 5 days 21 hours
+ EXECUTING INVENTORY PLUGINS
 allnet_ip_sensoric: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 check_mk: ok
 check_mk: skipped (no data)
 checkmk_agent_plugins: ok
 checkmk_agent_plugins: skipped (no data)
 cisco_meraki_org_device_info: skipped (no data)
 cisco_meraki_org_device_info: skipped (no data)
 dell_hw_info: skipped (no data)
 dell_hw_info: skipped (no data)
 docker_node_info: skipped (no data)
 docker_node_info: skipped (no data)
 esx_systeminfo: skipped (no data)
 esx_systeminfo: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fritz: skipped (no data)
 fritz: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ibm_mq_queues: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_if: ok
 inv_if: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_packages: skipped (no data)
 aix_packages: skipped (no data)
 aix_service_packs: skipped (no data)
 aix_service_packs: skipped (no data)
 azure_load_balancer: skipped (no data)
 azure_load_balancer: skipped (no data)
 inventory_checkmk: ok
 inventory_checkmk: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_state: skipped (no data)
 citrix_state: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 dmidecode: skipped (no data)
 dmidecode: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_network: skipped (no data)
 inventory_docker_container_node_name: skipped (no data)
 inventory_docker_container_node_name: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_network: skipped (no data)
 docker_node_network: skipped (no data)
 enviromux_micro_information: skipped (no data)
 enviromux_micro_information: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortigate_ha: skipped (no data)
 fortigate_ha: skipped (no data)
 fortimail_system: skipped (no data)
 fortimail_system: skipped (no data)
 fortinet_firewall_network_interfaces: skipped (no data)
 fortinet_firewall_network_interfaces: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_system: skipped (no data)
 fortisandbox_system: skipped (no data)
 ipmi_firmware: skipped (no data)
 ipmi_firmware: skipped (no data)
 k8s_service_info: skipped (no data)
 k8s_service_info: skipped (no data)
 kube_cluster: skipped (no data)
 kube_cluster: skipped (no data)
 kube_cronjob: skipped (no data)
 kube_cronjob: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_deployment: skipped (no data)
 kube_deployment: skipped (no data)
 kube_namespace: skipped (no data)
 kube_namespace: skipped (no data)
 kube_node: skipped (no data)
 kube_node: skipped (no data)
 kube_pod: skipped (no data)
 kube_pod: skipped (no data)
 kube_statefulset: skipped (no data)
 kube_statefulset: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_video: skipped (no data)
 lnx_video: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_clusters: skipped (no data)
 oracle_systemparameter: skipped (no data)
 oracle_systemparameter: skipped (no data)
 primekey: skipped (no data)
 primekey: skipped (no data)
 snmp_extended_info: skipped (no data)
 snmp_extended_info: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_routes: skipped (no data)
 statgrab_net: skipped (no data)
 statgrab_net: skipped (no data)
 win_bios: skipped (no data)
 win_bios: skipped (no data)
 win_disks: skipped (no data)
 win_disks: skipped (no data)
 win_exefiles: skipped (no data)
 win_exefiles: skipped (no data)
 win_ip_r: skipped (no data)
 win_ip_r: skipped (no data)
 win_networkadapter: skipped (no data)
 win_networkadapter: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_video: skipped (no data)
 win_video: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_updates: skipped (no data)
 win_wmi_updates: skipped (no data)
 juniper_info: skipped (no data)
 juniper_info: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_pod_container: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_if: ok
 lnx_if: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_uname: skipped (no data)
 lparstat_aix: skipped (no data)
 lparstat_aix: skipped (no data)
 mem: ok
 mem: skipped (no data)
 mem_used: skipped (no data)
 mem_used: skipped (no data)
 mssql_instance: skipped (no data)
 mssql_instance: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_info: skipped (no data)
 netapp_api_info: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_tablespaces: skipped (no data)
 oracle_tablespaces: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_psmu: skipped (no data)
 perle_psmu: skipped (no data)
 prtconf: skipped (no data)
 prtconf: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_info: skipped (no data)
 snmp_info: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_uname: skipped (no data)
 solaris_uname: skipped (no data)
 suseconnect: skipped (no data)
 suseconnect: skipped (no data)
 win_computersystem: skipped (no data)
 win_computersystem: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_os: skipped (no data)
 win_os: skipped (no data)
 win_system: skipped (no data)
 win_system: skipped (no data)
 winperf_if: skipped (no data)
 winperf_if: skipped (no data)
Trying to acquire lock on /omd/sites/southwold/tmp/check_mk/status_data/cmk-satellite.southwold.net
Got lock on /omd/sites/southwold/tmp/check_mk/status_data/cmk-satellite.southwold.net
Releasing lock on /omd/sites/southwold/tmp/check_mk/status_data/cmk-satellite.southwold.net
Released lock on /omd/sites/southwold/tmp/check_mk/status_data/cmk-satellite.southwold.net
Trying to acquire lock on /omd/sites/southwold/tmp/check_mk/status_data/cmk-satellite.southwold.net.gz
Got lock on /omd/sites/southwold/tmp/check_mk/status_data/cmk-satellite.southwold.net.gz
Releasing lock on /omd/sites/southwold/tmp/check_mk/status_data/cmk-satellite.southwold.net.gz
Released lock on /omd/sites/southwold/tmp/check_mk/status_data/cmk-satellite.southwold.net.gz
No piggyback files for 'cmk-satellite.southwold.net'. Skip processing.
No piggyback files for '10.0.0.104'. Skip processing.
[cpu_tracking] Stop [7f3382a03d90 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.019999999552965164))]
[agent] Success, [piggyback] Success (but no data found for this host), execution time 0.7 sec | execution_time=0.710 user_time=0.010 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.690

For some reason, the agent skips load reporting on LXCs (?)

    if [ -z "${IS_DOCKERIZED}" ] && [ -z "${IS_LXC_CONTAINER}" ]; then
        echo '<<<cpu>>>'
        echo "$(cat /proc/loadavg) ${NUM_CPUS}"

That’s correct as the LXC shows only the load from the host system.
There is no real load information for inside the LXC.

I have new information on this topic. LXC reports the actual container load for a few years now:

Since the check is hardcoded into the agent, I have trouble implementing the load check on my lxc containers. Is there a way to manually enable the check, despite the above code?

The linked article is only relevant for cgroup-v1 - with the current cgroup-v2 this will not work.
Pls read the linke issues here.

Thank you for your reply. I can assure you, the lxc load virtualization works flawlessly with lxd 5.0.3 and cgroup-v2, and has been for at least 2 years now. We’re running Ubuntu 18.04, 20.04 and 22.04 containers without any issues. The linked issue never occured to us.

Then can you give some information how do you get the internal load from the container?

It almost works out of the box:

snap install lxd
snap refresh lxd
snap set lxd lxcfs.loadavg=true
snap restart lxd
lxc launch ubuntu:24.04 test
lxc shell test
uptime

To return to my original question, is there a simple way to override the agent default?

You can modify the agent code - or you write a own small shell script as local check or plugin.

PS: your procedure gives me the same load values on all the LXC running on one host

Thank you, I will try that route first, but I opened an idea as well: https://ideas.checkmk.com/suggestions/565770/cpu-load-for-lxc-container

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.