After Upgrading Vmware ESXi to 7.0.2 no more Services found

Hi,

i just updatet an ESXi 6.7 to 7.0 U2 after that i can’t see any services on Host.
Also deleted the Host and readded.
We have other ESXi on 7.0 U2 with the same build, these Host are working fine.

Check MK Managed 1.6.0p24 (problem ist also on 1.6.0p14)

Here are the Debug message:

OMD[MSupport]:~$ cmk --debug -vvI ESXI
Discovering services on: ESXI
ESXI:

  • FETCHING DATA
    [special_vsphere] No persisted sections loaded
    [special_vsphere] Not using cache (Does not exist)
    [special_vsphere] Execute data source
    [special_vsphere] Calling external program “/omd/sites/MSupport/share/check_mk/agents/special/agent_vsphere -u ‘root’ -s ‘PW’ -i hostsystem,virtualmachine,datastore,counters --direct --hostname ‘ESXI’ -P --spaces underscore --no-cert-check ‘IP ADRESS’”
    [special_vsphere] ERROR: Agent exited with code 1: Error while processing received data

Traceback (most recent call last):
File “/omd/sites/MSupport/bin/cmk”, line 94, in
exit_status = modes.call(mode_name, mode_args, opts, args)
File “/omd/sites/MSupport/lib/python/cmk_base/modes/init.py”, line 72, in call
return mode.handler_function(*handler_args)
File “/omd/sites/MSupport/lib/python/cmk_base/modes/check_mk.py”, line 1498, in mode_discover
discovery.do_discovery(hostnames, options.get(“checks”), options[“discover”] == 1)
File “/omd/sites/MSupport/lib/python/cmk_base/discovery.py”, line 130, in do_discovery
multi_host_sections = _get_host_sections_for_discovery(sources, use_caches=use_caches)
File “/omd/sites/MSupport/lib/python/cmk_base/discovery.py”, line 894, in _get_host_sections_for_discovery
return sources.get_host_sections(max_cachefile_age)
File “/omd/sites/MSupport/lib/python/cmk_base/data_sources/init.py”, line 279, in get_host_sections
host_sections_from_source = source.run()
File “/omd/sites/MSupport/lib/python/cmk_base/data_sources/abstract.py”, line 134, in run
raw_data, is_cached_data = self._get_raw_data()
File “/omd/sites/MSupport/lib/python/cmk_base/data_sources/abstract.py”, line 183, in _get_raw_data
raw_data = self._execute()
File “/omd/sites/MSupport/lib/python/cmk_base/data_sources/programs.py”, line 61, in _execute
return self._get_agent_info_program(command_line, command_stdin)
File “/omd/sites/MSupport/lib/python/cmk_base/data_sources/programs.py”, line 109, in _get_agent_info_program
raise MKAgentError(“Agent exited with code %d: %s” % (exitstatus, stderr))
cmk_base.exceptions.MKAgentError: Agent exited with code 1: Error while processing received data

Full Scan message:
Starting job…

  • FETCHING DATA
    [special_vsphere] Execute data source
    [special_vsphere] ERROR: Agent exited with code 1: Error while processing received data

[piggyback] Execute data source
No piggyback files for ‘ESXI’. Skip processing.
No piggyback files for ‘IP ADRESS’. Skip processing.
Completed.

Thx Stefan

As the agent brings already the error, you should test the agent standalone.

→ /omd/sites/MSupport/share/check_mk/agents/special/agent_vsphere -u ‘root’ -s ‘PW’ -i hostsystem,virtualmachine,datastore,counters --direct --hostname ‘ESXI’ -P --spaces underscore --no-cert-check ‘IP ADRESS

It is also possible to add an command line switch to create a trace file with the complete communication.
Please don’t use the root user - create an read only account on the machine for the monitoring.

will replace root user in future.

here is the infos from the agent:

OMD[MSupport]:~$ /omd/sites/MSupport/share/check_mk/agents/special/agent_vsphere  -u 'root' -s 'PW' -i hostsystem,virtualmachine,datastore,counters --direct --hostname 'ESXI' -P --spaces underscore --no-cert-check --debug 'IP ADRESS'
Connecting IP ADRESS:443...Traceback (most recent call last):
  File "/omd/sites/MSupport/share/check_mk/agents/special/agent_vsphere", line 1883, in <module>
    fetch_data(esx_connection)
  File "/omd/sites/MSupport/share/check_mk/agents/special/agent_vsphere", line 1634, in fetch_data
    hostsystems = fetch_host_systems(connection)
  File "/omd/sites/MSupport/share/check_mk/agents/special/agent_vsphere", line 1353, in fetch_host_systems
    hostsystems_response = connection.query_server(telegram_list["hostsystems"])
  File "/omd/sites/MSupport/share/check_mk/agents/special/agent_vsphere", line 943, in query_server
    _respose_object, response_text = self.send_soapdata(soapdata, payload_params)
  File "/omd/sites/MSupport/share/check_mk/agents/special/agent_vsphere", line 913, in send_soapdata
    response = self._server_handle.getresponse()
  File "/omd/sites/MSupport/lib/python2.7/httplib.py", line 1148, in getresponse
    response.begin()
  File "/omd/sites/MSupport/lib/python2.7/httplib.py", line 448, in begin
    version, status, reason = self._read_status()
  File "/omd/sites/MSupport/lib/python2.7/httplib.py", line 412, in _read_status
    raise BadStatusLine("No status line received - the server has closed the connection")
httplib.BadStatusLine: No status line received - the server has closed the connection

Tracefile:

Tracefile 2021-07-15 17:03 Host address: IP ADRESS
####   Sent  ####
<SOAP-ENV:Envelope xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ZSI="http://www.zolera.com/schemas/ZSI/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><SOAP-ENV:Header></SOAP-ENV:Header><SOAP-ENV:Body xmlns:ns1="urn:vim25"><ns1:RetrieveServiceContent xsi:type="ns1:RetrieveServiceContentRequestType"><ns1:_this type="ServiceInstance">ServiceInstance</ns1:_this></ns1:RetrieveServiceContent></SOAP-ENV:Body></SOAP-ENV:Envelope>
#### Received #### Response took: 0.154529
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
 xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<RetrieveServiceContentResponse xmlns="urn:vim25"><returnval><rootFolder type="Folder">ha-folder-root</rootFolder><propertyCollector type="PropertyCollector">ha-property-collector</propertyCollector><viewManager type="ViewManager">ViewManager</viewManager><about><name>VMware ESXi</name><fullName>VMware ESXi 7.0.2 build-17867351</fullName><vendor>VMware, Inc.</vendor><version>7.0.2</version><build>17867351</build><localeVersion>INTL</localeVersion><localeBuild>000</localeBuild><osType>vmnix-x86</osType><productLineId>embeddedEsx</productLineId><apiType>HostAgent</apiType><apiVersion>7.0.2.0</apiVersion><licenseProductName>VMware ESX Server</licenseProductName><licenseProductVersion>7.0</licenseProductVersion></about><setting type="OptionManager">HostAgentSettings</setting><userDirectory type="UserDirectory">ha-user-directory</userDirectory><sessionManager type="SessionManager">ha-sessionmgr</sessionManager><authorizationManager type="AuthorizationManager">ha-authmgr</authorizationManager><perfManager type="PerformanceManager">ha-perfmgr</perfManager><eventManager type="EventManager">ha-eventmgr</eventManager><taskManager type="TaskManager">ha-taskmgr</taskManager><accountManager type="HostLocalAccountManager">ha-localacctmgr</accountManager><diagnosticManager type="DiagnosticManager">ha-diagnosticmgr</diagnosticManager><licenseManager type="LicenseManager">ha-license-manager</licenseManager><searchIndex type="SearchIndex">ha-searchindex</searchIndex><fileManager type="FileManager">ha-nfc-file-manager</fileManager><virtualDiskManager type="VirtualDiskManager">ha-vdiskmanager</virtualDiskManager><ovfManager type="OvfManager">ha-ovf-manager</ovfManager><dvSwitchManager type="DistributedVirtualSwitchManager">ha-dvsmanager</dvSwitchManager><localizationManager type="LocalizationManager">ha-l10n-manager</localizationManager><storageResourceManager type="StorageResourceManager">ha-storage-resource-manager</storageResourceManager><guestOperationsManager type="GuestOperationsManager">ha-guest-operations-manager</guestOperationsManager></returnval></RetrieveServiceContentResponse>
</soapenv:Body>
</soapenv:Envelope>
####   Sent  ####
<SOAP-ENV:Envelope xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ZSI="http://www.zolera.com/schemas/ZSI/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><SOAP-ENV:Header></SOAP-ENV:Header><SOAP-ENV:Body xmlns:ns1="urn:vim25"><ns1:RetrievePropertiesEx xsi:type="ns1:RetrievePropertiesExRequestType"><ns1:_this type="PropertyCollector">ha-property-collector</ns1:_this><ns1:specSet><ns1:propSet><ns1:type>HostSystem</ns1:type><ns1:pathSet>name</ns1:pathSet></ns1:propSet><ns1:objectSet><ns1:obj type="Folder">ha-folder-root</ns1:obj><ns1:skip>false</ns1:skip><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>visitFolders</ns1:name><ns1:type>Folder</ns1:type><ns1:path>childEntity</ns1:path><ns1:skip>false</ns1:skip><ns1:selectSet><ns1:name>visitFolders</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>dcToHf</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>dcToVmf</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>crToH</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>crToRp</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>dcToDs</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>hToVm</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>rpToVm</ns1:name></ns1:selectSet></ns1:selectSet><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>dcToVmf</ns1:name><ns1:type>Datacenter</ns1:type><ns1:path>vmFolder</ns1:path><ns1:skip>false</ns1:skip><ns1:selectSet><ns1:name>visitFolders</ns1:name></ns1:selectSet></ns1:selectSet><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>dcToDs</ns1:name><ns1:type>Datacenter</ns1:type><ns1:path>datastore</ns1:path><ns1:skip>false</ns1:skip><ns1:selectSet><ns1:name>visitFolders</ns1:name></ns1:selectSet></ns1:selectSet><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>dcToHf</ns1:name><ns1:type>Datacenter</ns1:type><ns1:path>hostFolder</ns1:path><ns1:skip>false</ns1:skip><ns1:selectSet><ns1:name>visitFolders</ns1:name></ns1:selectSet></ns1:selectSet><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>crToH</ns1:name><ns1:type>ComputeResource</ns1:type><ns1:path>host</ns1:path><ns1:skip>false</ns1:skip></ns1:selectSet><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>crToRp</ns1:name><ns1:type>ComputeResource</ns1:type><ns1:path>resourcePool</ns1:path><ns1:skip>false</ns1:skip><ns1:selectSet><ns1:name>rpToRp</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>rpToVm</ns1:name></ns1:selectSet></ns1:selectSet><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>rpToRp</ns1:name><ns1:type>ResourcePool</ns1:type><ns1:path>resourcePool</ns1:path><ns1:skip>false</ns1:skip><ns1:selectSet><ns1:name>rpToRp</ns1:name></ns1:selectSet><ns1:selectSet><ns1:name>rpToVm</ns1:name></ns1:selectSet></ns1:selectSet><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>hToVm</ns1:name><ns1:type>HostSystem</ns1:type><ns1:path>vm</ns1:path><ns1:skip>false</ns1:skip><ns1:selectSet><ns1:name>visitFolders</ns1:name></ns1:selectSet></ns1:selectSet><ns1:selectSet xsi:type="ns1:TraversalSpec"><ns1:name>rpToVm</ns1:name><ns1:type>ResourcePool</ns1:type><ns1:path>vm</ns1:path><ns1:skip>false</ns1:skip></ns1:selectSet></ns1:objectSet></ns1:specSet><ns1:options></ns1:options></ns1:RetrievePropertiesEx></SOAP-ENV:Body></SOAP-ENV:Envelope>

Please use the preformatted text field for the output. In the current form it is not readable.
But if you get no response after the last sent then there is the problem.

In the git history of the agent_vsphere i see no changes that reference the compatibility to vSphere 7.
You can make a quick check. Install an actual CMK 2.0 and create a test site with this version. Only add one of the ESX hosts as an object and assign the ESX special agent. Test if the connection is working there.

1 Like

we have 4 or more ESXi which are running the same version as this one. and there is the monitoring running fine!
So i think a test with cmk 2.0 will not help.

Then you should first reboot your ESX as there is something wrong with the output of the XML.
If you have some ESX servers why not use the vCenter as central datasource?
This is way better than create monitoring users on every machine.

ok came on the same idea with reboot of ESXi. These are Costumers ESXi Servers and not on the same site, because of this there is no vcenter.

Update:

After Reboot of the ESX the error is the same as before.