Description of problem
Hey
I noticed that 4 machines which are LXC containers stopped working with CheckMK
In the CheckMK gui I see the message [agent] Communication failed: [Errno 111] Connection refusedCRIT, [piggyback] Success (but no data found for this host), Missing monitoring data for all pluginsWARN, execution time 0.0 sec
Interestingly, the communication test passes
Telnet returns only this
root@chmk-sk:/tmp# telnet 127.0.0.1 6556
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
16
And nothing more…
I’ve already tried uninstalling the agent, reinstalling it, re-registering it with TLS also again nothing helped.
Where to look for the problem?
CMK version:
Checkmk Raw Edition 2.2.0p24
OS version:
Debian 12
Error message:
sudo systemctl status cmk-agent-ctl-daemon.service
Request failed. (received corrupt message)
Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
OMD[sienkiewicza]:~$ cmk --debug -vvn CHECK_MK_SIENKIEWICZA
Checkmk version 2.2.0p24
- FETCHING DATA
Source: SourceInfo(hostname=‘CHECK_MK_SIENKIEWICZA’, ipaddress=‘127.0.0.1’, ident=‘agent’, fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fb54777d590]
Read from cache: AgentFileCache(CHECK_MK_SIENKIEWICZA, path_template=/omd/sites/sienkiewicza/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 262 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 127.0.0.1:6483 (5.0s timeout)
Closing TCP connection to 127.0.0.1:6483
[cpu_tracking] Stop [7fb54777d590 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
Source: SourceInfo(hostname=‘CHECK_MK_SIENKIEWICZA’, ipaddress=‘127.0.0.1’, ident=‘piggyback’, fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fb546a95790]
Read from cache: NoCache(CHECK_MK_SIENKIEWICZA, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for ‘CHECK_MK_SIENKIEWICZA’. Skip processing.
No piggyback files for ‘127.0.0.1’. Skip processing.
[cpu_tracking] Stop [7fb546a95790 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))] - PARSE FETCHER RESULTS
HostKey(hostname=‘CHECK_MK_SIENKIEWICZA’, source_type=<SourceType.HOST: 1>) → Not adding sections: ConnectionRefusedError(111, ‘Connection refused’)
HostKey(hostname=‘CHECK_MK_SIENKIEWICZA’, source_type=<SourceType.HOST: 1>) → Add sections:
Received no piggyback data
[cpu_tracking] Start [7fb546ab8d50]
value store: synchronizing
Trying to acquire lock on /omd/sites/sienkiewicza/tmp/check_mk/counters/CHECK_MK_SIENKIEWICZA
Got lock on /omd/sites/sienkiewicza/tmp/check_mk/counters/CHECK_MK_SIENKIEWICZA
value store: loading from disk
Releasing lock on /omd/sites/sienkiewicza/tmp/check_mk/counters/CHECK_MK_SIENKIEWICZA
Released lock on /omd/sites/sienkiewicza/tmp/check_mk/counters/CHECK_MK_SIENKIEWICZA
CPU utilization PEND Check plugin received no monitoring data
Check_MK Agent PEND Check plugin received no monitoring data
Disk IO SUMMARY PEND Check plugin received no monitoring data
Filesystem / PEND Check plugin received no monitoring data
Filesystem /opt/omd/sites/sienkiewicza/tmp PEND Check plugin received no monitoring data
Interface 2 PEND Check plugin received no monitoring data
MD Softraid md0 PEND Check plugin received no monitoring data
MD Softraid md1 PEND Check plugin received no monitoring data
MD Softraid md2 PEND Check plugin received no monitoring data
MD Softraid md3 PEND Check plugin received no monitoring data
MD Softraid md4 PEND Check plugin received no monitoring data
MD Softraid md5 PEND Check plugin received no monitoring data
Memory PEND Check plugin received no monitoring data
Mount options of / PEND Check plugin received no monitoring data
OMD sienkiewicza Event Console PEND Check plugin received no monitoring data
OMD sienkiewicza apache PEND Check plugin received no monitoring data
OMD sienkiewicza backup Backup_to_TMP PEND Check plugin received no monitoring data
OMD sienkiewicza disk usage PEND Check plugin received no monitoring data
OMD sienkiewicza performance PEND Check plugin received no monitoring data
OMD sienkiewicza status PEND Check plugin received no monitoring data
Site sienkiewicza statistics PEND Check plugin received no monitoring data
Systemd Service Summary PEND Check plugin received no monitoring data
Systemd Socket Summary PEND Check plugin received no monitoring data
TCP Connections PEND Check plugin received no monitoring data
Uptime PEND Check plugin received no monitoring data - EXECUTING INVENTORY PLUGINS
allnet_ip_sensoric: skipped (no data)
allnet_ip_sensoric: skipped (no data)
aruba_wlc_aps: skipped (no data)
aruba_wlc_aps: skipped (no data)
check_mk: skipped (no data)
check_mk: skipped (no data)
checkmk_agent_plugins: skipped (no data)
checkmk_agent_plugins: skipped (no data)
cisco_meraki_org_device_info: skipped (no data)
cisco_meraki_org_device_info: skipped (no data)
dell_hw_info: skipped (no data)
dell_hw_info: skipped (no data)
docker_node_info: skipped (no data)
docker_node_info: skipped (no data)
esx_systeminfo: skipped (no data)
esx_systeminfo: skipped (no data)
fireeye_sys_status: skipped (no data)
fireeye_sys_status: skipped (no data)
fritz: skipped (no data)
fritz: skipped (no data)
hp_proliant_da_phydrv: skipped (no data)
hp_proliant_da_phydrv: skipped (no data)
hp_proliant_mem: skipped (no data)
hp_proliant_mem: skipped (no data)
hp_proliant_systeminfo: skipped (no data)
hp_proliant_systeminfo: skipped (no data)
ibm_mq_channels: skipped (no data)
ibm_mq_channels: skipped (no data)
ibm_mq_managers: skipped (no data)
ibm_mq_managers: skipped (no data)
ibm_mq_queues: skipped (no data)
ibm_mq_queues: skipped (no data)
infoblox_osinfo: skipped (no data)
infoblox_osinfo: skipped (no data)
infoblox_systeminfo: skipped (no data)
infoblox_systeminfo: skipped (no data)
inv_cisco_vlans: skipped (no data)
inv_cisco_vlans: skipped (no data)
inv_esx_vsphere_hostsystem: skipped (no data)
inv_esx_vsphere_hostsystem: skipped (no data)
inv_if: skipped (no data)
inv_if: skipped (no data)
aix_baselevel: skipped (no data)
aix_baselevel: skipped (no data)
aix_lparstat_inventory: skipped (no data)
aix_lparstat_inventory: skipped (no data)
aix_packages: skipped (no data)
aix_packages: skipped (no data)
aix_service_packs: skipped (no data)
aix_service_packs: skipped (no data)
azure_load_balancer: skipped (no data)
azure_load_balancer: skipped (no data)
inventory_checkmk: skipped (no data)
inventory_checkmk: skipped (no data)
checkpoint_vpn_tunnels: skipped (no data)
checkpoint_vpn_tunnels: skipped (no data)
citrix_controller: skipped (no data)
citrix_controller: skipped (no data)
citrix_state: skipped (no data)
citrix_state: skipped (no data)
couchbase_nodes_ports: skipped (no data)
couchbase_nodes_ports: skipped (no data)
dmidecode: skipped (no data)
dmidecode: skipped (no data)
docker_container_labels: skipped (no data)
docker_container_labels: skipped (no data)
docker_container_network: skipped (no data)
docker_container_network: skipped (no data)
inventory_docker_container_node_name: skipped (no data)
inventory_docker_container_node_name: skipped (no data)
docker_node_images: skipped (no data)
docker_node_images: skipped (no data)
docker_node_network: skipped (no data)
docker_node_network: skipped (no data)
enviromux_micro_information: skipped (no data)
enviromux_micro_information: skipped (no data)
inventory_esx_vsphere_clusters: skipped (no data)
inventory_esx_vsphere_clusters: skipped (no data)
inventory_esx_vsphere_virtual_machines: skipped (no data)
inventory_esx_vsphere_virtual_machines: skipped (no data)
fortiauthenticator_system: skipped (no data)
fortiauthenticator_system: skipped (no data)
fortigate_ha: skipped (no data)
fortigate_ha: skipped (no data)
fortimail_system: skipped (no data)
fortimail_system: skipped (no data)
fortinet_firewall_network_interfaces: skipped (no data)
fortinet_firewall_network_interfaces: skipped (no data)
fortisandbox_software: skipped (no data)
fortisandbox_software: skipped (no data)
fortisandbox_system: skipped (no data)
fortisandbox_system: skipped (no data)
ipmi_firmware: skipped (no data)
ipmi_firmware: skipped (no data)
k8s_service_info: skipped (no data)
k8s_service_info: skipped (no data)
kube_cluster: skipped (no data)
kube_cluster: skipped (no data)
kube_cronjob: skipped (no data)
kube_cronjob: skipped (no data)
kube_daemonset: skipped (no data)
kube_daemonset: skipped (no data)
kube_deployment: skipped (no data)
kube_deployment: skipped (no data)
kube_namespace: skipped (no data)
kube_namespace: skipped (no data)
kube_node: skipped (no data)
kube_node: skipped (no data)
kube_pod: skipped (no data)
kube_pod: skipped (no data)
kube_statefulset: skipped (no data)
kube_statefulset: skipped (no data)
lnx_block_devices: skipped (no data)
lnx_block_devices: skipped (no data)
lnx_ip_r: skipped (no data)
lnx_ip_r: skipped (no data)
lnx_packages: skipped (no data)
lnx_packages: skipped (no data)
lnx_sysctl: skipped (no data)
lnx_sysctl: skipped (no data)
lnx_video: skipped (no data)
lnx_video: skipped (no data)
mobileiron_inventory: skipped (no data)
mobileiron_inventory: skipped (no data)
mssql_clusters: skipped (no data)
mssql_clusters: skipped (no data)
oracle_systemparameter: skipped (no data)
oracle_systemparameter: skipped (no data)
primekey: skipped (no data)
primekey: skipped (no data)
snmp_extended_info: skipped (no data)
snmp_extended_info: skipped (no data)
solaris_addresses: skipped (no data)
solaris_addresses: skipped (no data)
solaris_pkginfo: skipped (no data)
solaris_pkginfo: skipped (no data)
solaris_cpus: skipped (no data)
solaris_cpus: skipped (no data)
solaris_routes: skipped (no data)
solaris_routes: skipped (no data)
statgrab_net: skipped (no data)
statgrab_net: skipped (no data)
win_bios: skipped (no data)
win_bios: skipped (no data)
win_disks: skipped (no data)
win_disks: skipped (no data)
win_exefiles: skipped (no data)
win_exefiles: skipped (no data)
win_ip_r: skipped (no data)
win_ip_r: skipped (no data)
win_networkadapter: skipped (no data)
win_networkadapter: skipped (no data)
win_reg_uninstall: skipped (no data)
win_reg_uninstall: skipped (no data)
win_video: skipped (no data)
win_video: skipped (no data)
win_wmi_software: skipped (no data)
win_wmi_software: skipped (no data)
win_wmi_updates: skipped (no data)
win_wmi_updates: skipped (no data)
juniper_info: skipped (no data)
juniper_info: skipped (no data)
k8s_pod_container: skipped (no data)
k8s_pod_container: skipped (no data)
lnx_cpuinfo: skipped (no data)
lnx_cpuinfo: skipped (no data)
lnx_distro: skipped (no data)
lnx_distro: skipped (no data)
lnx_if: skipped (no data)
lnx_if: skipped (no data)
lnx_uname: skipped (no data)
lnx_uname: skipped (no data)
lparstat_aix: skipped (no data)
lparstat_aix: skipped (no data)
mem: skipped (no data)
mem: skipped (no data)
mem_used: skipped (no data)
mem_used: skipped (no data)
mssql_instance: skipped (no data)
mssql_instance: skipped (no data)
netapp_api_disk: skipped (no data)
netapp_api_disk: skipped (no data)
netapp_api_info: skipped (no data)
netapp_api_info: skipped (no data)
netapp_ontap_disk: skipped (no data)
netapp_ontap_disk: skipped (no data)
netapp_ontap_info: skipped (no data)
netapp_ontap_info: skipped (no data)
oracle_dataguard_stats: skipped (no data)
oracle_dataguard_stats: skipped (no data)
oracle_instance: skipped (no data)
oracle_instance: skipped (no data)
oracle_performance: skipped (no data)
oracle_performance: skipped (no data)
oracle_recovery_area: skipped (no data)
oracle_recovery_area: skipped (no data)
oracle_tablespaces: skipped (no data)
oracle_tablespaces: skipped (no data)
perle_chassis: skipped (no data)
perle_chassis: skipped (no data)
perle_chassis_slots: skipped (no data)
perle_chassis_slots: skipped (no data)
perle_psmu: skipped (no data)
perle_psmu: skipped (no data)
prtconf: skipped (no data)
prtconf: skipped (no data)
snmp_quantum_storage_info: skipped (no data)
snmp_quantum_storage_info: skipped (no data)
snmp_info: skipped (no data)
snmp_info: skipped (no data)
solaris_prtdiag: skipped (no data)
solaris_prtdiag: skipped (no data)
solaris_uname: skipped (no data)
solaris_uname: skipped (no data)
suseconnect: skipped (no data)
suseconnect: skipped (no data)
win_computersystem: skipped (no data)
win_computersystem: skipped (no data)
win_cpuinfo: skipped (no data)
win_cpuinfo: skipped (no data)
win_os: skipped (no data)
win_os: skipped (no data)
win_system: skipped (no data)
win_system: skipped (no data)
winperf_if: skipped (no data)
winperf_if: skipped (no data)
No piggyback files for ‘CHECK_MK_SIENKIEWICZA’. Skip processing.
No piggyback files for ‘127.0.0.1’. Skip processing.
[cpu_tracking] Stop [7fb546ab8d50 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.010000001639127731))]
[agent] ConnectionRefusedError(111, ‘Connection refused’)(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 0.0 sec | execution_time=0.010 user_time=0.010 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000
