Hi all,
Suddenly, I get my docker server’s monitoring plugin’s output as Unknown, with the errors below. I cannot to seem to find a solution to this issue… Is there a way to make it work again?
CMK version: RAW 2.1.0p36
OS version: Ubuntu 18.04.6 LTS
Error message: Plugin exception in MKDockerClient.init: Error while fetching server API version: (‘Connection aborted.’, ConnectionRefusedError(111, ‘Connection refused’))
Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
OMD[mysite]:~$ cmk --debug -vvn SRV08
Checkmk version 2.1.0p36
Try license usage history update.
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Got lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Got lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Released lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Released lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
+ FETCHING DATA
Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f7e33309d90]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(SRV08, base_path=/omd/sites/mysite/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=90, inventory=90), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 25 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 10.10.10.8:6556 (30.0s timeout)
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Reading data from agent
Write data to cache file /omd/sites/mysite/tmp/check_mk/cache/SRV08
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/cache/SRV08
Got lock on /omd/sites/mysite/tmp/check_mk/cache/SRV08
Releasing lock on /omd/sites/mysite/tmp/check_mk/cache/SRV08
Released lock on /omd/sites/mysite/tmp/check_mk/cache/SRV08
Closing TCP connection to 10.10.10.8:6556
[cpu_tracking] Stop [7f7e33309d90 - Snapshot(process=posix.times_result(user=0.06000000000000005, system=0.0, children_user=0.0, children_system=0.0, elapsed=1.3999999985098839))]
Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f7e33327220]
[PiggybackFetcher] Fetch with cache settings: NoCache(SRV08, base_path=/omd/sites/mysite/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=90, inventory=90), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'SRV08'. Skip processing.
No piggyback files for '10.10.10.8'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f7e33327220 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<multipath>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<timesyncd>>> / Transition HostSectionParser -> HostSectionParser
<<<timesyncd_ntpmessage:sep(10)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<docker_node_info:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<docker_node_info:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
-> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df_v2', 'diskstat', 'docker_node_info', 'job', 'kernel', 'labels', 'lnx_if', 'local', 'md', 'mem', 'mounts', 'multipath', 'nfsmounts', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'timesyncd', 'timesyncd_ntpmessage', 'uptime', 'vbox_guest']
Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
-> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7f7e332dc190]
value store: synchronizing
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/counters/SRV08
Got lock on /omd/sites/mysite/tmp/check_mk/counters/SRV08
value store: loading from disk
Releasing lock on /omd/sites/mysite/tmp/check_mk/counters/SRV08
Released lock on /omd/sites/mysite/tmp/check_mk/counters/SRV08
CPU utilization Total CPU: 1.74%
Check_MK Agent Version: 2.1.0p36, OS: linux, Agent plugins: 1, Local checks: 0
Docker containers Containers: count not present in agent output(?), Running: count not present in agent output(?), Paused: count not present in agent output(?), Stopped: count not present in agent output(?)
Docker node info Plugin exception in MKDockerClient.__init__: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
Filesystem / 15.92% used (39.01 of 245.02 GB), trend: +604.66 kB / 24 hours
Interface 2 [docker0], (up), MAC: 02:42:04:34:5D:13, Speed: unknown, In: 14.0 B/s, Out: 15.7 B/s
Interface 3 [ens160], (up), MAC: 00:50:56:A6:42:DD, Speed: 10 GBit/s, In: 1.02 kB/s (<0.01%), Out: 22.4 kB/s (<0.01%)
Memory Total virtual memory: 13.2% - 1.55 GB of 11.76 GB, 9 additional details available
TCP Connections Established: 7
Uptime Up since Feb 02 2024 12:00:05, Uptime: 3 days 3 hours
No piggyback files for 'SRV08'. Skip processing.
No piggyback files for '10.10.10.8'. Skip processing.
[cpu_tracking] Stop [7f7e332dc190 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
[agent] Success, execution time 1.4 sec | execution_time=1.410 user_time=0.070 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=1.340