Intermittent agent error, Cached plugin reached timeout: chrony

Hi everyone, I’m getting a notification for this error about 6 times each day, and I can’t work out what’s causing it. I’ll get a Warning status notification, then back to OK 1 minute later.

CMK version:
2.4.0p8

OS version:
SLES15-SP7

Error message:
Notification email says:
Service: Check_MK Agent
Summary: Version: 2.4.0p8, OS: linux, Agent plug-ins: 0, Local checks: 0, Timed out plugin(s): chrony WARN

Service details:
Version: 2.4.0p8
OS: linux
Agent plug-ins: 0
Local checks: 0
Time until controller certificate for <HOSTNAME>/main, issued by Site 'main' agent signing CA, expires: 4 years 360 days
Cached plugins(s) that reached timeout: chrony (Timeout: 20s, PID: 65235) - Corresponding output is outdated and/or dropped. WARN

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

 # omd su main
OMD[main]:~$ cmk --debug -vvn <HOSTNAME>
value store: loading from disk
Checkmk version 2.4.0p8
+ FETCHING DATA
  Source: SourceInfo(hostname='<HOSTNAME>', ipaddress='127.0.0.2', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f64e0981c40]
Read from cache: AgentFileCache(path_template=/omd/sites/main/tmp/check_mk/cache/<HOSTNAME>, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 38 sec, allowed is 0 sec)
Connecting via TCP to 127.0.0.2:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN
Closing TCP connection to 127.0.0.2:6556
Write data to cache file /omd/sites/main/tmp/check_mk/cache/<HOSTNAME>
Trying to acquire lock on /omd/sites/main/tmp/check_mk/cache/<HOSTNAME>
Got lock on /omd/sites/main/tmp/check_mk/cache/<HOSTNAME>
Releasing lock on /omd/sites/main/tmp/check_mk/cache/<HOSTNAME>
Released lock on /omd/sites/main/tmp/check_mk/cache/<HOSTNAME>
[cpu_tracking] Stop [7f64e0981c40 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.0, children_user=0.0, children_system=0.0, elapsed=1.3499999977648258))]
  Source: SourceInfo(hostname='<HOSTNAME>', ipaddress='127.0.0.2', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f64e0e5bb90]
Read from cache: NoCache(path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
0 piggyback files for '<HOSTNAME>'.
0 piggyback files for '127.0.0.2'.
Get piggybacked data
[cpu_tracking] Stop [7f64e0e5bb90 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7f64e0e5bb90]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<multipath>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_status:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_ssl_certs:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<mkeventd_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cmk_site_statistics:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1758181402,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_status:cached(1758181343,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_broker_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_broker_queues:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_broker_shovels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<mknotifyd:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_apache:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_diskusage:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_info:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
  HostKey(hostname='<HOSTNAME>', source_type=<SourceType.HOST: 1>)  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cmk_site_statistics', 'cpu', 'df_v2', 'diskstat', 'job', 'kernel', 'labels', 'livestatus_ssl_certs', 'livestatus_status', 'lnx_if', 'local', 'md', 'mem', 'mkeventd_status', 'mknotifyd', 'mounts', 'multipath', 'nfsmounts_v2', 'omd_apache', 'omd_broker_queues', 'omd_broker_shovels', 'omd_broker_status', 'omd_diskusage', 'omd_info', 'omd_status', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  HostKey(hostname='<HOSTNAME>', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
CPU load             15 min load: 0.09, 15 min load per core: 0.04 (2 cores)
CPU utilization      Total CPU: 6.36%
Check_MK Agent       Version: 2.4.0p8, OS: linux, Agent plug-ins: 0, Local checks: 0
Disk IO SUMMARY      Read: 599 B/s, Write: 11.4 kB/s, Latency: 163 microseconds
Filesystem /         Used: 30.92% - 6.10 GiB of 19.7 GiB, trend per 1 day 0 hours: +16.0 MiB, trend per 1 day 0 hours: +0.08%, Time left until disk full: 2 years 143 days
Filesystem /boot/efi Used: 0.12% - 316 KiB of 256 MiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Filesystem /sys/firmware/efi/efivars Used: 12.11% - 31.0 KiB of 256 KiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Interface 2          [eth0], (up), MAC: 00:50:56:AA:2F:92, Speed: 10 GBit/s, In: 3.97 kB/s (<0.01%), Out: 1.87 kB/s (<0.01%)
Kernel Performance   Process Creations: 8.80/s, Context Switches: 636.37/s, Major Page Faults: 0.15/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Memory               Total virtual memory: 41.34% - 3.18 GiB of 7.69 GiB, 8 additional details available
Mount options of /   Mount options exactly as expected
Mount options of /boot/efi Mount options exactly as expected
NTP Time             Offset: 0.1023 ms, Stratum: 2, Time since last sync: 15 minutes 39 seconds
Number of threads    402, Usage: 0.64%
OMD main Event Console Current events: 0, Virtual memory: 207 MiB, Overall event limit inactive, No hosts event limit active, No rules event limit active, Received messages: 0.00/s, Rule hits: 0.00/s, Rule tries: 0.00/s, Message drops: 0.00/s, Created events: 0.00/s, Client connects: 0.02/s, Rule hit ratio: -, Processing time per message: -, Time per client request: 0.75 ms
OMD main Notification Spooler Version: 2.4.0p8, Spooler running
OMD main apache      No activity since last check
OMD main disk usage  Total: 481 MiB, Agents: 60.1 MiB, Core: 1.07 MiB, History: 24.0 KiB, Inventory: 2.15 KiB, Local: 1.61 KiB, Logs: 7.57 MiB, PNP4Nagios: 83 B, RRDs: 401 MiB, Tmp: 718 KiB
OMD main performance Livestatus version: 2.4.0p8, Host checks: 0.9/s, Service checks: 2.0/s
OMD main status      running
Postfix Queue default Deferred queue length: 0, Active queue length: 0
Postfix status default Status: the Postfix mail system is running, PID: 1863
Site main statistics Total hosts: 5, Problem hosts: 0, Total services: 104, Problem services: 4
Systemd Service Summary Total: 165, Disabled: 22, Failed: 0
Systemd Socket Summary Total: 16, Disabled: 3, Failed: 0
TCP Connections      Established: 3
Uptime               Up since 2025-09-12 15:58:52, Uptime: 5 days 17 hours
+ EXECUTING INVENTORY PLUGINS
 allnet_ip_sensoric: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 check_mk: ok
 check_mk: skipped (no data)
 checkmk_agent_plugins: ok
 checkmk_agent_plugins: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 dell_hw_info: skipped (no data)
 dell_hw_info: skipped (no data)
 docker_node_info: skipped (no data)
 docker_node_info: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_channels: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fireeye_sys_status: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ibm_mq_queues: skipped (no data)
 win_disks: skipped (no data)
 win_disks: skipped (no data)
 win_exefiles: skipped (no data)
 win_exefiles: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 win_ip_r: skipped (no data)
 win_ip_r: skipped (no data)
 win_networkadapter: skipped (no data)
 win_networkadapter: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_block_devices: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_if: ok
 inv_if: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_reg_uninstall: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_baselevel: skipped (no data)
 win_video: skipped (no data)
 win_video: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_software: skipped (no data)
 aix_packages: skipped (no data)
 aix_packages: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_ip_r: skipped (no data)
 aix_service_packs: skipped (no data)
 aix_service_packs: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_packages: skipped (no data)
 azure_load_balancer: skipped (no data)
 azure_load_balancer: skipped (no data)
 win_wmi_updates: skipped (no data)
 win_wmi_updates: skipped (no data)
 inventory_checkmk: ok
 inventory_checkmk: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_sysctl: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 lnx_video: skipped (no data)
 lnx_video: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_state: skipped (no data)
 citrix_state: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mobileiron_inventory: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 dmidecode: skipped (no data)
 dmidecode: skipped (no data)
 kube_node: skipped (no data)
 kube_node: skipped (no data)
 lnx_if: ok
 lnx_if: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_labels: skipped (no data)
 kube_namespace: skipped (no data)
 kube_namespace: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_network: skipped (no data)
 kube_pod: skipped (no data)
 kube_pod: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_uname: skipped (no data)
 inventory_docker_container_node_name: skipped (no data)
 inventory_docker_container_node_name: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_clusters: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_images: skipped (no data)
 oracle_systemparameter: skipped (no data)
 oracle_systemparameter: skipped (no data)
 docker_node_network: skipped (no data)
 docker_node_network: skipped (no data)
 primekey: skipped (no data)
 primekey: skipped (no data)
 enviromux_micro_information: skipped (no data)
 enviromux_micro_information: skipped (no data)
 snmp_extended_info: skipped (no data)
 snmp_extended_info: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 kube_statefulset: skipped (no data)
 kube_statefulset: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortigate_ha: skipped (no data)
 fortigate_ha: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_addresses: skipped (no data)
 fortimail_system: skipped (no data)
 fortimail_system: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_pkginfo: skipped (no data)
 fortinet_firewall_network_interfaces: skipped (no data)
 fortinet_firewall_network_interfaces: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_cpus: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_software: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_routes: skipped (no data)
 fortisandbox_system: skipped (no data)
 fortisandbox_system: skipped (no data)
 ipmi_firmware: skipped (no data)
 ipmi_firmware: skipped (no data)
 kube_cluster: skipped (no data)
 kube_cluster: skipped (no data)
 juniper_info: skipped (no data)
 juniper_info: skipped (no data)
 kube_cronjob: skipped (no data)
 kube_cronjob: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_daemonset: skipped (no data)
 win_bios: skipped (no data)
 win_bios: skipped (no data)
 kube_deployment: skipped (no data)
 kube_deployment: skipped (no data)
 kyocera_printer: skipped (no data)
 kyocera_printer: skipped (no data)
 mem: ok
 mem: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_psmu: skipped (no data)
 perle_psmu: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_distro: skipped (no data)
 lparstat_aix: skipped (no data)
 lparstat_aix: skipped (no data)
 mem_used: skipped (no data)
 mem_used: skipped (no data)
 prtconf: skipped (no data)
 prtconf: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 suseconnect: skipped (no data)
 suseconnect: skipped (no data)
 snmp_info: skipped (no data)
 snmp_info: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_uname: skipped (no data)
 solaris_uname: skipped (no data)
 win_computersystem: skipped (no data)
 win_computersystem: skipped (no data)
 win_computersystemproduct: skipped (no data)
 win_computersystemproduct: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_os: skipped (no data)
 win_os: skipped (no data)
 win_system: skipped (no data)
 win_system: skipped (no data)
 winperf_if: skipped (no data)
 winperf_if: skipped (no data)
 fritz: skipped (no data)
 fritz: skipped (no data)
 cisco_meraki_org_device_info: skipped (no data)
 cisco_meraki_org_device_info: skipped (no data)
 cisco_meraki_power_supplies: skipped (no data)
 cisco_meraki_power_supplies: skipped (no data)
 mssql_instance: skipped (no data)
 mssql_instance: skipped (no data)
 netapp_ontap_disk: skipped (no data)
 netapp_ontap_disk: skipped (no data)
 netapp_ontap_info: skipped (no data)
 netapp_ontap_info: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_tablespaces: skipped (no data)
 oracle_tablespaces: skipped (no data)
 redfish_firmware: skipped (no data)
 redfish_firmware: skipped (no data)
 redfish_firmware_hpe_ilo4: skipped (no data)
 redfish_firmware_hpe_ilo4: skipped (no data)
 robotmk: skipped (no data)
 robotmk: skipped (no data)
 esx_systeminfo: skipped (no data)
 esx_systeminfo: skipped (no data)
Trying to acquire lock on /omd/sites/main/tmp/check_mk/status_data/<HOSTNAME>
Got lock on /omd/sites/main/tmp/check_mk/status_data/<HOSTNAME>
Releasing lock on /omd/sites/main/tmp/check_mk/status_data/<HOSTNAME>
Released lock on /omd/sites/main/tmp/check_mk/status_data/<HOSTNAME>
0 piggyback files for '<HOSTNAME>'.
[cpu_tracking] Stop [7f64e0e5bb90 - Snapshot(process=posix.times_result(user=0.040000000000000036, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.06000000238418579))]
[agent] Success, [piggyback] Success (but no data found for this host), execution time 1.4 sec | execution_time=1.410 user_time=0.050 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=1.340
OMD[main]:~$

The host being checked here is my check_mk server itself.

I can’t find any evidence of a plugin for chrony:

<HOSTNAME>:~ # find /opt/omd/sites/main/share/check_mk/agents/plugins/ | grep chrony
<HOSTNAME>:~ #

Assumption, chrony was detected on your system (maybe used for NTP) at discovery and added as check.

Your debug shows the following:

<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
  HostKey(hostname='<HOSTNAME>', 
source_type=<SourceType.HOST: 1>)  -> Add sections: 
['check_mk', 'checkmk_agent_plugins_lnx', 'chrony', ...]

Untested solution, in case you want to remove chrony:
In my 2.1.0p5.cre there is an example file exclude_sections.cfg that can be downloaded from Setup > Agents > Linux, Solaris, AIX files

In that file it gives options to exclude sections, also chrony:

# List of sample sections which will be excluded in the next Checkmk agent run
#MK_SKIP_BONDING_IF=yes
#MK_SKIP_CPU=yes
#MK_SKIP_CHRONY=yes

I assume it is enough to remove the comment # in front of MK_SKIP_CHRONY=yes, save the file in /etc/check_mk/ (according to Monitoring Linux - The new agent for Linux in detail) and let the agent run again.

Thanks, Yggy. That appears to have suppressed the error message, which is fine by me. The NTP service is still being monitored.

1 Like