CMK version:
2.1.0p20
OS version:
Ubuntu 18.04
Error message:
[agent] Communication failed: timed outCRIT
Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
I do have vcenter configured and its working. but i want to add each host individually so i can pull more info. So most if not all of the info is from piggy backing off vcenter, when i add the host the agent never connects. I am using the root account on the ESX server for now just to make sure there arent permission issues. I can connect to the host via curl (below), but i get a 301 redirect as the output and im not sure why or if that is the reason im having issues. I have to use the -k so it ignores the cert and i also have that checked under the conf for the esxi service otherwise i get cert errors. Ive went through the troubleshooting in the documentation and it was not able to resolve my issue. Ive tried to include as much info as possible, you ive missed something please let me know. Thanks for the help!
(i had to add spaces to the URLs to break them otherwise i couldnt post this as a new account)
OMD[bwb]:~$ cmk --debug -vvn vm02.fvbnet. com
Checkmk version 2.1.0p20
Try license usage history update.
Trying to acquire lock on /omd/sites/bwb/var/check_mk/license_usage/next_run
Got lock on /omd/sites/bwb/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/bwb/var/check_mk/license_usage/history.json
Got lock on /omd/sites/bwb/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/bwb/var/check_mk/license_usage/history.json
Released lock on /omd/sites/bwb/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/bwb/var/check_mk/license_usage/next_run
Released lock on /omd/sites/bwb/var/check_mk/license_usage/next_run
- FETCHING DATA
Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f7ade6ea190]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(vm02.fvbnet. com, base_path=/omd/sites/bwb/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Does not exist)
[TCPFetcher] Execute data source
Connecting via TCP to 172.31.21.38:6556 (5.0s timeout)
Closing TCP connection to 172.31.21.38:6556
[cpu_tracking] Stop [7f7ade6ea190 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=5.0))]
Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f7ade720970]
[PiggybackFetcher] Fetch with cache settings: NoCache(vm02.fvbnet .com, base_path=/omd/sites/bwb/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
Piggyback file ‘/omd/sites/bwb/tmp/check_mk/piggyback/vm02.fvbnet.com/vcenter.fvb-nt.fvbnet. com’: Successfully processed from source ‘vcenter.fvb-nt.fvbnet. com’
No piggyback files for ‘172.31.21.38’. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f7ade720970 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))] - PARSE FETCHER RESULTS
Source: SourceType.HOST/FetcherType.TCP
→ Not adding sections: timeout(‘timed out’)
Source: SourceType.HOST/FetcherType.PIGGYBACK
<<<esx_vsphere_counters:cached(1675355656,90):sep(124)>>> / Transition NOOPParser → HostSectionParser
<<<esx_vsphere_hostsystem:cached(1675355656,90)>>> / Transition HostSectionParser → HostSectionParser
<<<esx_vsphere_sensors:cached(1675355656,90):sep(59)>>> / Transition HostSectionParser → HostSectionParser
<<labels:sep(0)>> / Transition HostSectionParser → HostSectionParser
No persisted sections
→ Add sections: [‘esx_vsphere_counters’, ‘esx_vsphere_hostsystem’, ‘esx_vsphere_sensors’, ‘labels’]
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7f7ade6ea910]
value store: synchronizing
Trying to acquire lock on /omd/sites/bwb/tmp/check_mk/counters/vm02.fvbnet.com
Got lock on /omd/sites/bwb/tmp/check_mk/counters/vm02.fvbnet.com
value store: loading from disk
Releasing lock on /omd/sites/bwb/tmp/check_mk/counters/vm02.fvbnet.com
Released lock on /omd/sites/bwb/tmp/check_mk/counters/vm02.fvbnet.com
CPU utilization Total CPU: 1.27%
Datastore IO SUMMARY Read: 25.00 kB/s, Write: 86.00 kB/s, Latency: 0.00 ms, Read latency: 1.00 ms, Write latency: 0.00 ms, Read operations: 0.00 1/s, Write operations: 0.00 1/s
Disk IO SUMMARY Read: 25.6 kB/s, Write: 88.7 kB/s, Latency: 0 seconds
Hardware Sensors All sensors are in normal state
Interface 1 [vmnic0], (up), Speed: unknown, In: 1.02 kB/s, Out: 3.07 kB/s
Interface 2 [vmnic1], (up), Speed: unknown, In: 0.00 B/s, Out: 0.00 B/s
Maintenance Mode System not in Maintenance mode
Memory Usage: 31.09% - 9.93 GiB of 32.0 GiB
Multipath 303132333435363738393031 1 active, 0 dead, 0 disabled, 0 standby, 0 unknown
Multipath 6d094660779f730023193e292b08c80f 1 active, 0 dead, 0 disabled, 0 standby, 0 unknown
Multipath 766d6862 1 active, 0 dead, 0 disabled, 0 standby, 0 unknown
Overall state Entity state: green, Power state: poweredOn
Uptime Up since Tue Nov 8 23:29:14 2022, uptime: 85 days, 12:05:49
VMKernel Swap Swap in: 0.00 KB, Swap out: 0.00 KB, Swap used: 0.00 KB
Piggyback file ‘/omd/sites/bwb/tmp/check_mk/piggyback/vm02.fvbnet.com/vcenter.fvb-nt.fvbnet .com’: Successfully processed from source ‘vcenter.fvb-nt.fvbnet .com’
No piggyback files for ‘172.31.21.38’. Skip processing.
[cpu_tracking] Stop [7f7ade6ea910 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
[agent] timeout(‘timed out’)(!!), [piggyback] Successfully processed from source ‘vcenter. fvb-nt.fvbnet. com’, execution time 5.0 sec | execution_time=5.020 user_time=0.020 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=5.000
OMD[bwb]:~$
OMD[bwb]:~$ curl -kv vm02 .fvbnet.com
- Rebuilt URL to: vm02 .fvbnet.com/
- Trying 172.31.21.38…
- TCP_NODELAY set
- Connected to vm02 .fvbnet.com (172.31.21.38) port 80 (#0)
GET / HTTP/1.1
Host: vm02 .fvbnet.com
User-Agent: curl/7.58.0
Accept: /
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 2 Feb 2023 16:43:33 GMT
< Location: https:// vm02 .fvbnet.com/
< Connection: close
< Content-Type: text/html
< Content-Length: 56
<
- Closing connection 0
301 Moved Permanently
OMD [bwb]:~$OMD[bwb]:~$ curl -Ik https:// vm02 .fvbnet.com
HTTP/1.1 200 OK
Date: Thu, 2 Feb 2023 16:59:00 GMT
Connection: Keep-Alive
Content-Security-Policy: upgrade-insecure-requests
Content-Type: text/html
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1
Content-Length: 258
OMD[bwb]:~$
OMD[bwb]:~$ cmk -Dv vm02 .fvbnet.com
vm02 .fvbnet.com
Addresses: 172.31.21.38
Tags: [address_family:ip-v4-only], [agent:all-agents], [checkmk-agent:checkmk-agent], [criticality:prod], [ip-v4:ip-v4], [networking:wan], [piggyback:auto-piggyback], [site:bwb], [snmp_ds:no-snmp], [tcp:tcp]
Labels: [cmk/piggyback_source_vcenter.fvb-nt.fvbnet.com:yes], [cmk/site:bwb], [cmk/vsphere_object:server]
Host groups: check_mk
Contact groups: check-mk-notify, all
Agent mode: Normal Checkmk agent, all configured special agents
Type of agent:
TCP: 172.31.21.38:6556
Process piggyback data from /omd/sites/bwb/tmp/check_mk/piggyback/vm02.fvbnet.com
Services:
checktype item params description groups
esx_vsphere_hostsystem_cpu_usage None Timespecific parameters at 2023-02-02 12:09:40: {} CPU utilization
esx_vsphere_counters SUMMARY Timespecific parameters at 2023-02-02 12:09:40: {} Datastore IO SUMMARY
esx_vsphere_counters_diskio SUMMARY Timespecific parameters at 2023-02-02 12:09:40: {} Disk IO SUMMARY
esx_vsphere_sensors None Timespecific parameters at 2023-02-02 12:09:40: {} Hardware Sensors
esx_vsphere_counters_if 1 Timespecific parameters at 2023-02-02 12:09:40: {‘errors’: {‘both’: (‘perc’, (0.01, 0.1))}, ‘discovered_oper_status’: [‘1’], ‘discovered_speed’: 0} Interface 1
esx_vsphere_counters_if 2 Timespecific parameters at 2023-02-02 12:09:40: {‘errors’: {‘both’: (‘perc’, (0.01, 0.1))}, ‘discovered_oper_status’: [‘1’], ‘discovered_speed’: 0} Interface 2
esx_vsphere_hostsystem_maintenance None Timespecific parameters at 2023-02-02 12:09:40: {‘target_state’: ‘false’} Maintenance Mode
esx_vsphere_hostsystem_mem_usage None Timespecific parameters at 2023-02-02 12:09:40: {‘levels_upper’: (80.0, 90.0)} Memory
esx_vsphere_hostsystem_multipath 303132333435363738393031 Timespecific parameters at 2023-02-02 12:09:40: {} Multipath 303132333435363738393031
esx_vsphere_hostsystem_multipath 6d094660779f730023193e292b08c80f Timespecific parameters at 2023-02-02 12:09:40: {} Multipath 6d094660779f730023193e292b08c80f
esx_vsphere_hostsystem_multipath 766d6862 Timespecific parameters at 2023-02-02 12:09:40: {} Multipath 766d6862
esx_vsphere_hostsystem_state None Timespecific parameters at 2023-02-02 12:09:40: {} Overall state
esx_vsphere_counters_uptime None Timespecific parameters at 2023-02-02 12:09:40: {} Uptime
esx_vsphere_counters_swap None Timespecific parameters at 2023-02-02 12:09:40: {} VMKernel Swap
OMD[bwb]:~$ cmk -d vm02 .fvbnet.com
ERROR [agent]: Communication failed: timed out
OMD[bwb]:~$ ping vm02 .fvbnet.com
PING vm02 .fvbnet.com (172.31.21.38) 56(84) bytes of data.
64 bytes from vm02 .fvbnet.com (172.31.21.38): icmp_seq=1 ttl=62 time=3.81 ms
64 bytes from vm02 .fvbnet.com (172.31.21.38): icmp_seq=2 ttl=62 time=2.36 ms
64 bytes from vm02 .fvbnet.com (172.31.21.38): icmp_seq=3 ttl=62 time=4.22 ms
^C
— vm02 .fvbnet.com ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.364/3.469/4.227/0.802 ms
OMD[bwb]:~$ nslookup vm02 .fvbnet.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: vm02 .fvbnet.com
Address: 172.31.21.38