Cannot add ESXi 7.x Hosts, agent never connects

CMK version:
2.1.0p20

OS version:
Ubuntu 18.04

Error message:
[agent] Communication failed: timed outCRIT

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

I do have vcenter configured and its working. but i want to add each host individually so i can pull more info. So most if not all of the info is from piggy backing off vcenter, when i add the host the agent never connects. I am using the root account on the ESX server for now just to make sure there arent permission issues. I can connect to the host via curl (below), but i get a 301 redirect as the output and im not sure why or if that is the reason im having issues. I have to use the -k so it ignores the cert and i also have that checked under the conf for the esxi service otherwise i get cert errors. Ive went through the troubleshooting in the documentation and it was not able to resolve my issue. Ive tried to include as much info as possible, you ive missed something please let me know. Thanks for the help!
(i had to add spaces to the URLs to break them otherwise i couldnt post this as a new account)

OMD[bwb]:~$ cmk --debug -vvn vm02.fvbnet. com
Checkmk version 2.1.0p20
Try license usage history update.
Trying to acquire lock on /omd/sites/bwb/var/check_mk/license_usage/next_run
Got lock on /omd/sites/bwb/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/bwb/var/check_mk/license_usage/history.json
Got lock on /omd/sites/bwb/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/bwb/var/check_mk/license_usage/history.json
Released lock on /omd/sites/bwb/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/bwb/var/check_mk/license_usage/next_run
Released lock on /omd/sites/bwb/var/check_mk/license_usage/next_run

  • FETCHING DATA
    Source: SourceType.HOST/FetcherType.TCP
    [cpu_tracking] Start [7f7ade6ea190]
    [TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(vm02.fvbnet. com, base_path=/omd/sites/bwb/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
    Not using cache (Does not exist)
    [TCPFetcher] Execute data source
    Connecting via TCP to 172.31.21.38:6556 (5.0s timeout)
    Closing TCP connection to 172.31.21.38:6556
    [cpu_tracking] Stop [7f7ade6ea190 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=5.0))]
    Source: SourceType.HOST/FetcherType.PIGGYBACK
    [cpu_tracking] Start [7f7ade720970]
    [PiggybackFetcher] Fetch with cache settings: NoCache(vm02.fvbnet .com, base_path=/omd/sites/bwb/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
    Not using cache (Cache usage disabled)
    [PiggybackFetcher] Execute data source
    Piggyback file ‘/omd/sites/bwb/tmp/check_mk/piggyback/vm02.fvbnet.com/vcenter.fvb-nt.fvbnet. com’: Successfully processed from source ‘vcenter.fvb-nt.fvbnet. com’
    No piggyback files for ‘172.31.21.38’. Skip processing.
    Not using cache (Cache usage disabled)
    [cpu_tracking] Stop [7f7ade720970 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
  • PARSE FETCHER RESULTS
    Source: SourceType.HOST/FetcherType.TCP
    → Not adding sections: timeout(‘timed out’)
    Source: SourceType.HOST/FetcherType.PIGGYBACK
    <<<esx_vsphere_counters:cached(1675355656,90):sep(124)>>> / Transition NOOPParser → HostSectionParser
    <<<esx_vsphere_hostsystem:cached(1675355656,90)>>> / Transition HostSectionParser → HostSectionParser
    <<<esx_vsphere_sensors:cached(1675355656,90):sep(59)>>> / Transition HostSectionParser → HostSectionParser
    <<labels:sep(0)>> / Transition HostSectionParser → HostSectionParser
    No persisted sections
    → Add sections: [‘esx_vsphere_counters’, ‘esx_vsphere_hostsystem’, ‘esx_vsphere_sensors’, ‘labels’]
    Received no piggyback data
    Received no piggyback data
    [cpu_tracking] Start [7f7ade6ea910]
    value store: synchronizing
    Trying to acquire lock on /omd/sites/bwb/tmp/check_mk/counters/vm02.fvbnet.com
    Got lock on /omd/sites/bwb/tmp/check_mk/counters/vm02.fvbnet.com
    value store: loading from disk
    Releasing lock on /omd/sites/bwb/tmp/check_mk/counters/vm02.fvbnet.com
    Released lock on /omd/sites/bwb/tmp/check_mk/counters/vm02.fvbnet.com
    CPU utilization Total CPU: 1.27%
    Datastore IO SUMMARY Read: 25.00 kB/s, Write: 86.00 kB/s, Latency: 0.00 ms, Read latency: 1.00 ms, Write latency: 0.00 ms, Read operations: 0.00 1/s, Write operations: 0.00 1/s
    Disk IO SUMMARY Read: 25.6 kB/s, Write: 88.7 kB/s, Latency: 0 seconds
    Hardware Sensors All sensors are in normal state
    Interface 1 [vmnic0], (up), Speed: unknown, In: 1.02 kB/s, Out: 3.07 kB/s
    Interface 2 [vmnic1], (up), Speed: unknown, In: 0.00 B/s, Out: 0.00 B/s
    Maintenance Mode System not in Maintenance mode
    Memory Usage: 31.09% - 9.93 GiB of 32.0 GiB
    Multipath 303132333435363738393031 1 active, 0 dead, 0 disabled, 0 standby, 0 unknown
    Multipath 6d094660779f730023193e292b08c80f 1 active, 0 dead, 0 disabled, 0 standby, 0 unknown
    Multipath 766d6862 1 active, 0 dead, 0 disabled, 0 standby, 0 unknown
    Overall state Entity state: green, Power state: poweredOn
    Uptime Up since Tue Nov 8 23:29:14 2022, uptime: 85 days, 12:05:49
    VMKernel Swap Swap in: 0.00 KB, Swap out: 0.00 KB, Swap used: 0.00 KB
    Piggyback file ‘/omd/sites/bwb/tmp/check_mk/piggyback/vm02.fvbnet.com/vcenter.fvb-nt.fvbnet .com’: Successfully processed from source ‘vcenter.fvb-nt.fvbnet .com’
    No piggyback files for ‘172.31.21.38’. Skip processing.
    [cpu_tracking] Stop [7f7ade6ea910 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
    [agent] timeout(‘timed out’)(!!), [piggyback] Successfully processed from source ‘vcenter. fvb-nt.fvbnet. com’, execution time 5.0 sec | execution_time=5.020 user_time=0.020 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=5.000
    OMD[bwb]:~$

OMD[bwb]:~$ curl -kv vm02 .fvbnet.com

  • Rebuilt URL to: vm02 .fvbnet.com/
  • Trying 172.31.21.38…
  • TCP_NODELAY set
  • Connected to vm02 .fvbnet.com (172.31.21.38) port 80 (#0)

GET / HTTP/1.1
Host: vm02 .fvbnet.com
User-Agent: curl/7.58.0
Accept: /

< HTTP/1.1 301 Moved Permanently
< Date: Thu, 2 Feb 2023 16:43:33 GMT
< Location: https:// vm02 .fvbnet.com/
< Connection: close
< Content-Type: text/html
< Content-Length: 56
<

  • Closing connection 0

301 Moved Permanently

OMD [bwb]:~$

OMD[bwb]:~$ curl -Ik https:// vm02 .fvbnet.com
HTTP/1.1 200 OK
Date: Thu, 2 Feb 2023 16:59:00 GMT
Connection: Keep-Alive
Content-Security-Policy: upgrade-insecure-requests
Content-Type: text/html
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1
Content-Length: 258

OMD[bwb]:~$

OMD[bwb]:~$ cmk -Dv vm02 .fvbnet.com

vm02 .fvbnet.com
Addresses: 172.31.21.38
Tags: [address_family:ip-v4-only], [agent:all-agents], [checkmk-agent:checkmk-agent], [criticality:prod], [ip-v4:ip-v4], [networking:wan], [piggyback:auto-piggyback], [site:bwb], [snmp_ds:no-snmp], [tcp:tcp]
Labels: [cmk/piggyback_source_vcenter.fvb-nt.fvbnet.com:yes], [cmk/site:bwb], [cmk/vsphere_object:server]
Host groups: check_mk
Contact groups: check-mk-notify, all
Agent mode: Normal Checkmk agent, all configured special agents
Type of agent:
TCP: 172.31.21.38:6556
Process piggyback data from /omd/sites/bwb/tmp/check_mk/piggyback/vm02.fvbnet.com
Services:
checktype item params description groups


esx_vsphere_hostsystem_cpu_usage None Timespecific parameters at 2023-02-02 12:09:40: {} CPU utilization
esx_vsphere_counters SUMMARY Timespecific parameters at 2023-02-02 12:09:40: {} Datastore IO SUMMARY
esx_vsphere_counters_diskio SUMMARY Timespecific parameters at 2023-02-02 12:09:40: {} Disk IO SUMMARY
esx_vsphere_sensors None Timespecific parameters at 2023-02-02 12:09:40: {} Hardware Sensors
esx_vsphere_counters_if 1 Timespecific parameters at 2023-02-02 12:09:40: {‘errors’: {‘both’: (‘perc’, (0.01, 0.1))}, ‘discovered_oper_status’: [‘1’], ‘discovered_speed’: 0} Interface 1
esx_vsphere_counters_if 2 Timespecific parameters at 2023-02-02 12:09:40: {‘errors’: {‘both’: (‘perc’, (0.01, 0.1))}, ‘discovered_oper_status’: [‘1’], ‘discovered_speed’: 0} Interface 2
esx_vsphere_hostsystem_maintenance None Timespecific parameters at 2023-02-02 12:09:40: {‘target_state’: ‘false’} Maintenance Mode
esx_vsphere_hostsystem_mem_usage None Timespecific parameters at 2023-02-02 12:09:40: {‘levels_upper’: (80.0, 90.0)} Memory
esx_vsphere_hostsystem_multipath 303132333435363738393031 Timespecific parameters at 2023-02-02 12:09:40: {} Multipath 303132333435363738393031
esx_vsphere_hostsystem_multipath 6d094660779f730023193e292b08c80f Timespecific parameters at 2023-02-02 12:09:40: {} Multipath 6d094660779f730023193e292b08c80f
esx_vsphere_hostsystem_multipath 766d6862 Timespecific parameters at 2023-02-02 12:09:40: {} Multipath 766d6862
esx_vsphere_hostsystem_state None Timespecific parameters at 2023-02-02 12:09:40: {} Overall state
esx_vsphere_counters_uptime None Timespecific parameters at 2023-02-02 12:09:40: {} Uptime
esx_vsphere_counters_swap None Timespecific parameters at 2023-02-02 12:09:40: {} VMKernel Swap

OMD[bwb]:~$ cmk -d vm02 .fvbnet.com
ERROR [agent]: Communication failed: timed out

OMD[bwb]:~$ ping vm02 .fvbnet.com
PING vm02 .fvbnet.com (172.31.21.38) 56(84) bytes of data.
64 bytes from vm02 .fvbnet.com (172.31.21.38): icmp_seq=1 ttl=62 time=3.81 ms
64 bytes from vm02 .fvbnet.com (172.31.21.38): icmp_seq=2 ttl=62 time=2.36 ms
64 bytes from vm02 .fvbnet.com (172.31.21.38): icmp_seq=3 ttl=62 time=4.22 ms
^C
— vm02 .fvbnet.com ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.364/3.469/4.227/0.802 ms
OMD[bwb]:~$ nslookup vm02 .fvbnet.com
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: vm02 .fvbnet.com
Address: 172.31.21.38

More info, it seems the agent can connect if im ready the output of this command correctly. Any ideas why i cant add this and have the agent connect?

share/check_mk/agents/special/agent_vsphere --no-cert-check --debug --user root --secret Password -D vm02.fvbnet. com | grep -A4 esx_vsphere_objects
<<<esx_vsphere_objects:sep(9)>>>
virtualmachine ZW02_-Hixson vm02.fvbnet. com poweredOn
virtualmachine NW02
-_Hixson vm02.fvbnet. com poweredOn
<<<<>>>>
<<<<>>>>
<<<esx_vsphere_objects:sep(9)>>>
hostsystem vm02.fvbnet. com poweredOn
<<<<>>>>
<<>>
1675366363.284179 1675366363.2875783
OMD[bwb]:~$

Your esxi is configured to use the checkmk agent, you need to edit the host property to use “API, no checkmk Agent”

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.