CMK version: 2.0.0p37
OS version: Ubuntu Server 20.04
Error message: Item not found in monitoring data
After updating from 2.0.0p35 to 2.0.0p37 some local checks get ignored and Check_MK states their data wouldn’t be in the agent output even though it is. Except updating the server and agent nothing was changed.
Agent output:
<<<local:sep(0)>>>
0 "3CX Trunks" - 2 of 2 trunks are online.
0 "VPN Certificate server.pem" - expires in 122 days
P "ClamAV Scan Result" Viruses=0;;1
Only the VPN Certificate check isn’t found by CMK, all others are.
The check script is located in /usr/lib/check_mk_agent/local. As it’s clearly working on the agent side (agent output in the CMK-GUI is correct), it seems like a server bug and it actively doesn’t care.
Debug output:
Checkmk version 2.0.0p37
Try license usage history update.
Trying to acquire lock on /omd/sites/hv2/var/check_mk/license_usage/next_run
Got lock on /omd/sites/hv2/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/hv2/var/check_mk/license_usage/history.json
Got lock on /omd/sites/hv2/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/hv2/var/check_mk/license_usage/history.json
Released lock on /omd/sites/hv2/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/hv2/var/check_mk/license_usage/next_run
Released lock on /omd/sites/hv2/var/check_mk/license_usage/next_run
Loading autochecks from /omd/sites/hv2/var/check_mk/autochecks/hv2-checkmk.mk
+ FETCHING DATA
Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f6f98df0a00]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(base_path=PosixPath('/omd/sites/hv2/tmp/check_mk/cache/hv2-checkmk'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 31 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 172.16.240.19:6556 (5.0s timeout)
Reading data from agent
Output is not encrypted
Write data to cache file /omd/sites/hv2/tmp/check_mk/cache/hv2-checkmk
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/cache/hv2-checkmk
Got lock on /omd/sites/hv2/tmp/check_mk/cache/hv2-checkmk
Releasing lock on /omd/sites/hv2/tmp/check_mk/cache/hv2-checkmk
Released lock on /omd/sites/hv2/tmp/check_mk/cache/hv2-checkmk
Closing TCP connection to 172.16.240.19:6556
[cpu_tracking] Stop [7f6f98df0a00 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.0, children_user=0.0, children_system=0.0, elapsed=13.389999996870756))]
Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f6f98df0940]
[PiggybackFetcher] Fetch with cache settings: NoCache(base_path=PosixPath('/omd/sites/hv2/tmp/check_mk/data_source_cache/piggyback/hv2-checkmk'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
[PiggybackFetcher] Execute data source
No piggyback files for 'hv2-checkmk'. Skip processing.
No piggyback files for '172.16.240.19'. Skip processing.
[cpu_tracking] Stop [7f6f98df0940 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7f6f98df0940]
+ PARSE FETCHER RESULTS
Source: SourceType.HOST/FetcherType.TCP
No persisted sections loaded
-> Add sections: ['apt', 'check_mk', 'cifsmounts', 'cmk_site_statistics', 'cpu', 'df', 'diskstat', 'job', 'kernel', 'labels', 'livestatus_ssl_certs', 'livestatus_status', 'lnx_if', 'local', 'logins', 'md', 'mem', 'mkbackup', 'mkeventd_status', 'mknotifyd', 'mounts', 'multipath', 'nfsmounts', 'omd_apache', 'omd_info', 'omd_status', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'sshd_config', 'systemd_units', 'tcp_conn_stats', 'timesyncd', 'uptime', 'vbox_guest']
Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections loaded
-> Add sections: []
Received no piggyback data
Loading item states
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/counters/hv2-checkmk
Got lock on /omd/sites/hv2/tmp/check_mk/counters/hv2-checkmk
Releasing lock on /omd/sites/hv2/tmp/check_mk/counters/hv2-checkmk
Released lock on /omd/sites/hv2/tmp/check_mk/counters/hv2-checkmk
3CX Trunks 2 of 2 trunks are online.
APT Updates No updates pending for installation
CPU load 15 min load: 1.23 at 8 cores (0.15 per core)
CPU utilization Total CPU: 28.48%
ClamAV Scan Result Viruses: 0.00
Disk IO SUMMARY Read: 474 kB/s, Write: 5.15 MB/s, Latency: 856 microseconds
Filesystem / 74.53% used (14.55 of 19.52 GB), trend: +802.43 kB / 24 hours
Filesystem /boot 28.29% used (275.41 of 973.42 MB), trend: 0.00 B / 24 hours
Filesystem /opt/omd/sites/hv2/tmp 0.52% used (12.70 MB of 2.40 GB), trend: +1.81 MB / 24 hours
Interface 2 [ens160], (up), MAC: 00:0C:29:B0:D3:FB, Speed: 10 GBit/s, In: 91.1 kB/s (<0.01%), Out: 7.96 kB/s (<0.01%)
Kernel Performance Process Creations: 39.32/s, Context Switches: 1536.98/s, Major Page Faults: 2.07/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Logins On system: 1
Memory Total virtual memory: 26.83% - 2.32 GB of 8.65 GB
Mount options of / Mount options exactly as expected
Mount options of /boot Mount options exactly as expected
Number of threads Count: 547 threads, Usage: 1.42%
OMD hv2 Event Console Current events: 51, Virtual memory: 191.25 MB, Overall event limit inactive, No hosts event limit active, No rules event limit active, Received messages: 0.00/s, Rule hits: 0.00/s, Rule tries: 0.00/s, Message drops: 0.00/s, Created events: 0.00/s, Client connects: 0.02/s, Rule hit ratio: -, Processing time per message: -, Time per client request: 2.62 ms
OMD hv2 apache 0.20 Requests/s, 0.01 Seconds serving/s, 43.48 B Sent/s
OMD hv2 backup backup Backup completed, it was running for 4 m from 2023-07-17 00:00:02 till 2023-07-17 00:04:02, Size: 423.26 MB, Next run: 2023-07-18 00:00:00
OMD hv2 performance Livestatus version: 2.0.0p37, Host checks: 1.8/s, Service checks: 20.0/s
OMD hv2 status running
SSH daemon configuration Ports: 22, Use pluggable authentication module: yes, Permit root login: key-based, Allow password authentication: yes, Allow keyboard-interactive authentication: no, Allow challenge-response authentication: no, Permit X11 forwarding: yes, Permit empty passwords: no, Ciphers: aes128-ctr, aes128-gcm@openssh.com, aes192-ctr, aes256-ctr, aes256-gcm@openssh.com, chacha20-poly1305@openssh.com
Site hv2 statistics Total hosts: 111, Problem hosts: 0, Total services: 1398, Problem services: 14
System Reboot required A system reboot is not required.
Systemd Service Summary Total: 157, Disabled: 2, Failed: 0
Systemd Service node_exporter Status: active, Prometheus Node Exporter Service
Systemd Timesyncd Time Offset: 7 microseconds, Time since last sync: 9 minutes 24 seconds, Stratum: 2.00, Jitter: 6.92 µs, synchronized on 91.189.91.157
TCP Connections Established: 10
Uptime Up since Jul 17 2023 13:47:42, Uptime: 36 minutes 53 seconds
VPN Certificate server.pem Item not found in monitoring data
+ EXECUTING INVENTORY PLUGINS
hp_proliant_mem: skipped (no data)
hp_proliant_mem: skipped (no data)
ipmi_firmware: skipped (no data)
ipmi_firmware: skipped (no data)
inventory_checkmk: ok
inventory_checkmk: skipped (no data)
inv_cisco_vlans: skipped (no data)
inv_cisco_vlans: skipped (no data)
inv_esx_vsphere_hostsystem: skipped (no data)
inv_esx_vsphere_hostsystem: skipped (no data)
inv_if: ok
inv_if: skipped (no data)
inventory_oracle_tablespaces: skipped (no data)
inventory_oracle_tablespaces: skipped (no data)
dmidecode: skipped (no data)
dmidecode: skipped (no data)
docker_node_network: skipped (no data)
docker_node_network: skipped (no data)
inventory_esx_vsphere_clusters: skipped (no data)
inventory_esx_vsphere_clusters: skipped (no data)
inventory_esx_vsphere_virtual_machines: skipped (no data)
inventory_esx_vsphere_virtual_machines: skipped (no data)
k8s_endpoint_info: skipped (no data)
k8s_endpoint_info: skipped (no data)
k8s_ingress_infos: skipped (no data)
k8s_ingress_infos: skipped (no data)
juniper_info: skipped (no data)
juniper_info: skipped (no data)
mem: ok
mem: skipped (no data)
mem_used: skipped (no data)
mem_used: skipped (no data)
snmp_info: skipped (no data)
snmp_info: skipped (no data)
aix_baselevel: skipped (no data)
aix_baselevel: skipped (no data)
aix_lparstat_inventory: skipped (no data)
aix_lparstat_inventory: skipped (no data)
aix_packages: skipped (no data)
aix_packages: skipped (no data)
aix_service_packs: skipped (no data)
aix_service_packs: skipped (no data)
allnet_ip_sensoric: skipped (no data)
allnet_ip_sensoric: skipped (no data)
aruba_wlc_aps: skipped (no data)
aruba_wlc_aps: skipped (no data)
check_mk: ok
check_mk: skipped (no data)
checkpoint_inv_tunnels: skipped (no data)
checkpoint_inv_tunnels: skipped (no data)
citrix_controller: skipped (no data)
citrix_controller: skipped (no data)
citrix_state: skipped (no data)
citrix_state: skipped (no data)
couchbase_nodes_ports: skipped (no data)
couchbase_nodes_ports: skipped (no data)
dell_hw_info: skipped (no data)
dell_hw_info: skipped (no data)
docker_container_labels: skipped (no data)
docker_container_labels: skipped (no data)
docker_container_network: skipped (no data)
docker_container_network: skipped (no data)
docker_container_node_name: skipped (no data)
docker_container_node_name: skipped (no data)
docker_node_images: skipped (no data)
docker_node_images: skipped (no data)
docker_node_info: skipped (no data)
docker_node_info: skipped (no data)
esx_systeminfo: skipped (no data)
esx_systeminfo: skipped (no data)
fireeye_sys_status: skipped (no data)
fireeye_sys_status: skipped (no data)
fritz: skipped (no data)
fritz: skipped (no data)
hp_proliant_da_phydrv: skipped (no data)
hp_proliant_da_phydrv: skipped (no data)
hp_proliant_systeminfo: skipped (no data)
hp_proliant_systeminfo: skipped (no data)
ibm_mq_channels: skipped (no data)
ibm_mq_channels: skipped (no data)
ibm_mq_managers: skipped (no data)
ibm_mq_managers: skipped (no data)
ibm_mq_queues: skipped (no data)
ibm_mq_queues: skipped (no data)
infoblox_osinfo: skipped (no data)
infoblox_osinfo: skipped (no data)
infoblox_systeminfo: skipped (no data)
infoblox_systeminfo: skipped (no data)
k8s_assigned_pods: skipped (no data)
k8s_assigned_pods: skipped (no data)
k8s_daemon_pod_containers: skipped (no data)
k8s_daemon_pod_containers: skipped (no data)
k8s_job_container: skipped (no data)
k8s_job_container: skipped (no data)
k8s_nodes: skipped (no data)
k8s_nodes: skipped (no data)
k8s_pod_container: skipped (no data)
k8s_pod_container: skipped (no data)
k8s_pod_info: skipped (no data)
k8s_pod_info: skipped (no data)
k8s_roles: skipped (no data)
k8s_roles: skipped (no data)
k8s_selector: skipped (no data)
k8s_selector: skipped (no data)
k8s_service_info: skipped (no data)
k8s_service_info: skipped (no data)
lnx_block_devices: skipped (no data)
lnx_block_devices: skipped (no data)
lnx_cpuinfo: skipped (no data)
lnx_cpuinfo: skipped (no data)
lnx_distro: skipped (no data)
lnx_distro: skipped (no data)
lnx_if: ok
lnx_if: skipped (no data)
lnx_ip_r: skipped (no data)
lnx_ip_r: skipped (no data)
lnx_packages: skipped (no data)
lnx_packages: skipped (no data)
lnx_sysctl: skipped (no data)
lnx_sysctl: skipped (no data)
lnx_uname: skipped (no data)
lnx_uname: skipped (no data)
lnx_video: skipped (no data)
lnx_video: skipped (no data)
lparstat_aix: skipped (no data)
lparstat_aix: skipped (no data)
mssql_clusters: skipped (no data)
mssql_clusters: skipped (no data)
mssql_versions: skipped (no data)
mssql_versions: skipped (no data)
netapp_api_disk: skipped (no data)
netapp_api_disk: skipped (no data)
netapp_api_info: skipped (no data)
netapp_api_info: skipped (no data)
oracle_dataguard_stats: skipped (no data)
oracle_dataguard_stats: skipped (no data)
oracle_instance: skipped (no data)
oracle_instance: skipped (no data)
oracle_performance: skipped (no data)
oracle_performance: skipped (no data)
oracle_recovery_area: skipped (no data)
oracle_recovery_area: skipped (no data)
oracle_systemparameter: skipped (no data)
oracle_systemparameter: skipped (no data)
perle_chassis: skipped (no data)
perle_chassis: skipped (no data)
perle_chassis_slots: skipped (no data)
perle_chassis_slots: skipped (no data)
perle_psmu: skipped (no data)
perle_psmu: skipped (no data)
prtconf: skipped (no data)
prtconf: skipped (no data)
snmp_extended_info: skipped (no data)
snmp_extended_info: skipped (no data)
snmp_os: skipped (no data)
snmp_os: skipped (no data)
snmp_quantum_storage_info: skipped (no data)
snmp_quantum_storage_info: skipped (no data)
solaris_addresses: skipped (no data)
solaris_addresses: skipped (no data)
solaris_pkginfo: skipped (no data)
solaris_pkginfo: skipped (no data)
solaris_prtdiag: skipped (no data)
solaris_prtdiag: skipped (no data)
solaris_prtpicl: skipped (no data)
solaris_prtpicl: skipped (no data)
solaris_psrinfo: skipped (no data)
solaris_psrinfo: skipped (no data)
solaris_routes: skipped (no data)
solaris_routes: skipped (no data)
solaris_uname: skipped (no data)
solaris_uname: skipped (no data)
statgrab_net: skipped (no data)
statgrab_net: skipped (no data)
suseconnect: skipped (no data)
suseconnect: skipped (no data)
win_bios: skipped (no data)
win_bios: skipped (no data)
win_computersystem: skipped (no data)
win_computersystem: skipped (no data)
win_cpuinfo: skipped (no data)
win_cpuinfo: skipped (no data)
win_disks: skipped (no data)
win_disks: skipped (no data)
win_exefiles: skipped (no data)
win_exefiles: skipped (no data)
win_ip_r: skipped (no data)
win_ip_r: skipped (no data)
win_networkadapter: skipped (no data)
win_networkadapter: skipped (no data)
win_os: skipped (no data)
win_os: skipped (no data)
win_reg_uninstall: skipped (no data)
win_reg_uninstall: skipped (no data)
win_system: skipped (no data)
win_system: skipped (no data)
win_video: skipped (no data)
win_video: skipped (no data)
win_wmi_software: skipped (no data)
win_wmi_software: skipped (no data)
win_wmi_updates: skipped (no data)
win_wmi_updates: skipped (no data)
winperf_if: skipped (no data)
winperf_if: skipped (no data)
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/status_data/hv2-checkmk
Got lock on /omd/sites/hv2/tmp/check_mk/status_data/hv2-checkmk
Releasing lock on /omd/sites/hv2/tmp/check_mk/status_data/hv2-checkmk
Released lock on /omd/sites/hv2/tmp/check_mk/status_data/hv2-checkmk
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/status_data/hv2-checkmk.gz
Got lock on /omd/sites/hv2/tmp/check_mk/status_data/hv2-checkmk.gz
Releasing lock on /omd/sites/hv2/tmp/check_mk/status_data/hv2-checkmk.gz
Released lock on /omd/sites/hv2/tmp/check_mk/status_data/hv2-checkmk.gz
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/status_data/.last
Got lock on /omd/sites/hv2/tmp/check_mk/status_data/.last
Releasing lock on /omd/sites/hv2/tmp/check_mk/status_data/.last
Released lock on /omd/sites/hv2/tmp/check_mk/status_data/.last
No piggyback files for 'hv2-checkmk'. Skip processing.
No piggyback files for '172.16.240.19'. Skip processing.
[cpu_tracking] Stop [7f6f98df0940 - Snapshot(process=posix.times_result(user=0.16000000000000014, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.18000000342726707))]
[agent] Version: 2.0.0p37, OS: linux, execution time 13.6 sec | execution_time=13.570 user_time=0.170 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=13.380