Citrix Farm Plugin issues incorrectly report VM tools not running

Troubleshooting
1

Hi. We are using the shipped Citrix Farm plugin to monitor our non-persistent VDA’s in Citrix, which pulls in the VDA information from our 2x Delivery Controllers, and then has a dynamic host rule to add the VDA’s it finds and monitor them.

We have always had an issue from day 1, where random VDA’s constantly report that the VMTools are not running - but when I check vSphere, they are very definitely running just fine, and we dont see any issues with them in Citrix Studio either.

Really stuck trying to figure out why they keep reporting back with VMTools not running. I am assuming this is actual data coming back from the plugin script, but why? What is it not seeing that makes it think the tools are off? And why does it fail completely at random in the middle of the day, when nothing has happened to the VDA?

E.g. today, all 64 of our Citrix VDA servers suddenly for no explicable reason started throwing up warnings that VMTools wasnt running - but it is. I can see in vSphere the tools are running and logging on to the VDA’s directly, the services are all running and working just fine.

Any ideas? Detail and host debug output from one of the VDA’s below - although the setup is a bit complex so happy to provide more detail if needed!

CMK version: 2.3p6

Output of “cmk --debug -vvn hostname”:

OMD[companyname]:~$ cmk --debug -vvn desktop-vda-host
value store: synchronizing
Trying to acquire lock on /omd/sites/companyname/tmp/check_mk/counters/desktop-vda-host
Got lock on /omd/sites/companyname/tmp/check_mk/counters/desktop-vda-host
value store: loading from disk
Releasing lock on /omd/sites/companyname/tmp/check_mk/counters/desktop-vda-host
Released lock on /omd/sites/companyname/tmp/check_mk/counters/desktop-vda-host
Checkmk version 2.3.0p6
+ FETCHING DATA
  Source: SourceInfo(hostname='desktop-vda-host', ipaddress=None, ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fe0d0eb8440]
Read from cache: NoCache(desktop-vda-host, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
Piggyback file '/omd/sites/companyname/tmp/check_mk/piggyback/desktop-vda-host/ddc01': Successfully processed from source 'ddc01'
Piggyback file '/omd/sites/companyname/tmp/check_mk/piggyback/desktop-vda-host/ddc02': Successfully processed from source 'ddc02'
Get piggybacked data
[cpu_tracking] Stop [7fe0d0eb8440 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7fe0d0b95a00]
+ PARSE FETCHER RESULTS
<<<citrix_state:cached(1745576806,90)>>> / Transition NOOPParser -> HostSectionParser
<<<citrix_serverload:cached(1745576806,90)>>> / Transition HostSectionParser -> HostSectionParser
<<<citrix_sessions:cached(1745576806,90)>>> / Transition HostSectionParser -> HostSectionParser
<<<citrix_state:cached(1745576765,90)>>> / Transition HostSectionParser -> HostSectionParser
<<<citrix_serverload:cached(1745576765,90)>>> / Transition HostSectionParser -> HostSectionParser
<<<citrix_sessions:cached(1745576765,90)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
  HostKey(hostname='desktop-vda-host', source_type=<SourceType.HOST: 1>)  -> Add sections: ['citrix_serverload', 'citrix_sessions', 'citrix_state', 'labels']
Received no piggyback data
Citrix Controller    ddc01
Citrix Hosting Server esx-host
Citrix Instance State FaultState None, MaintenanceMode False, PowerState On, RegistrationState Registered, VMToolsState NotStarted(!)
Citrix Serverload    Current Citrix Load: 24.54%
Citrix Sessions      Total: 6, Active: 6, Inactive: 0
Piggyback file '/omd/sites/companyname/tmp/check_mk/piggyback/desktop-vda-host/ddc01': Successfully processed from source 'ddc01'
Piggyback file '/omd/sites/companyname/tmp/check_mk/piggyback/desktop-vda-host/ddc02': Successfully processed from source 'ddc02'
[cpu_tracking] Stop [7fe0d0b95a00 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[piggyback] Successfully processed from source 'ddc01', Successfully processed from source 'ddc02', execution time 0.0 sec | execution_time=0.000 user_time=0.010 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000
2

It seems this is an issue with the Citrix powershell cmdlet “Get-BrokerMachine” incorrectly reporting the tools as not started.

VMTools are definitely running so I dont know why this command is incorrectly reporting it. Going to have to disable these checks as its spamming the crap out of CMK while I investigate this further. Seems to be a common issue with Citrix.

3

If anyone is interested, I managed to fix it by hashing out the VMToolsState section of the citrix_farm.ps1 plugin script on the DDC plugins folder.

Specifically this section:

# Column VMToolsState / Gets machines with a specific VM tools state. Valid values are NotPresent, Unknown, NotStarted, and Running.
	#$VMToolsState  = $XAmachine | %{ $_.VMToolsState }
	#"VMToolsState $VMToolsState"

I did this on both of our delivery controllers, removed the VDA hosts, and reran the dynamic host discovery rule. This then pulled in the hosts correctly minus the dodgy/broken VMTools check.

Not sure if CMK guys can review the script to update this part? The Citrix community forums indicate that you should pull the VMTools info from the “Get-Service” cmdlet on each VDA, but that does mean a fairly significant change to the script. I could give it a go and let you know how it goes, but I’m really not interested in the VMTools state, so probably not going to bother. They’ve literally never failed, but if they did, vSphere would alert me.