Wrong Values from Windows Agent

[DanielNunesFerreira (Daniel Ferreira)]

Hi there,

We recently bought Check_MK Enterprise and one of the uses we want to give it is to spread the agent throughout many machines (Microsoft) within the infrastructure with Powershell scripts in order to monitor a few services from all those machines (their physical location/network are different from eachother) to get the perception of how the service is doing from all the different locations.

The following script, if ran locally, always returns values under 10ms:
$response = (Measure-Command { Invoke-WebRequest -Uri “SOMEURL” -UseBasicParsing }).TotalMilliseconds

$status = 2 #critical
$crit = 200
$warn = 120

if ($response -gt $crit) {
$status = 2 #critical
}
elseif ($response -gt $warn) {
$status = 1 #warning
}
else {
$status = 0 #ok
}
“$status HTTPResponseTime-Exchange response_time=$response;$warn;$crit;; HTTP response time $response ms”

Result: 0 HTTPResponseTime-Exchange response_time=7.9582;120;200;; HTTP response time 7.9582 ms

But the same script will always return values above 300ms when ran via agent:
2 HTTPResponseTime-Exchange response_time=3763.0363;120;200;; HTTP response time 3763.0363 ms

The script is located at C:\ProgramData\checkmk\agent\local

I’ve been running some tests over the past days and I can assure you that the MS is under the influence of how many local scripts we’re running on the machine: If I copy this very same script 50 times and run the agent output test, the MS response I’ll get will go from 200 (which is already a wrong value) up to 3500+.

I’ve done this before some years ago in different CMK builds and I never had this issue so I’m not sure where to go from here.

Thank you,
Daniel

[DanielNunesFerreira (Daniel Ferreira)]

Anyone has an idea to help me out?

[MarsellusWallace (Marcel S.)]

Hi Daniel,

maybe this could be caused by a difference in used users?
Usually the agent runs as LocalSystem account. Could be this user (or the used one) does web requests differently to the one you tested the PS script (maybe proxy yes/no diff)…

Could you please check this?

BR,
Marsellus

[DanielNunesFerreira (Daniel Ferreira)]

Hi Marsellus,

Thank you so much for your reply!

I just tried running the test under the privilegies of the LocalSystem account and the output was still the same.

Regarding the proxy, there’s no proxy in the way. This is a closed network and we’re contacting the exchange server directly.

Best regards,
Daniel

[DanielNunesFerreira (Daniel Ferreira)]

Also keep in mind that the MS result tend to get bigger as I add more scripts to the local folder so I really think this is a matter of how the agent is processing them.

[gulaschcowboy (AlexW)]

Hi Daniel,
this directly raises some concerns:

@DanielNunesFerreira

This is a closed network

Do network packets that are trying to go out to the non-reachable Internet get dropped/not responded or are they properly rejected by a firewall with “ICMP admin prohibited”?

I’m asking, because I saw it too many times, that DNS or other traffic is generated by e.g. certificate validation of the OS. Then this traffic gets dropped (not responded) and therefore the program/script/powershell needs to wait for timeouts. This easily adds a couple of seconds to the total runtime - per request.

Just my 2 cent
Alex

[MarsellusWallace (Marcel S.)]

Hi Daniel,

our Windows devs are already in christmas vacation and neither Alex nor me are experts in Powershell development…

But are you really sure you get always the right result? No http errors, always right content et al?
Could it be your script returns that fast because used invoke-webrequest does not work like expected for this user?

Please try this script as your test user and check the output:

$response = Measure-Command { $request = Invoke-WebRequest -Uri “SOMEURL” -UseBasicParsing }
write-host "=============="
write-host "REQUEST object"
write-host "=============="
$request
write-host "==============="
write-host "RESPONSE object"
write-host "==============="
$response

If you want to check this back using with agent you should either redirect the output to some logfile or create it as plugin file instead of local check, outputting something like <<<webrequest_debug>>> as first line…

BR,
Marsellus W.

[DanielNunesFerreira (Daniel Ferreira)]

Hi Marsellus,

First of all, I’d like you thank you for your help in this case, alongside with gulaschcowboy. This is a really urgent matter for me and having some feedback from you guys is awesome!

I’ve tested your script both in the plugins and in the local folder and, although the MS are still different, here are the differences:
Output of the plugin ran with LocalSystem Account Permissions:

==============
REQUEST object
==============


StatusCode        : 200
StatusDescription : OK
Content           : {50, 48, 48, 32...}
RawContent        : HTTP/1.1 200 OK
                    Strict-Transport-Security: max-age=31536000; includeSubDomains; preload; redirectHttpToHttps=true
                    X-FEServer: EXCHANGESERVER
                    Content-Length: 37
                    Date: Mon, 23 Dec 2019 10:08:10 GMT
                    S...
Headers           : {[Strict-Transport-Security, max-age=31536000; includeSubDomains; preload;
                    redirectHttpToHttps=true], [X-FEServer, EXCHANGESERVER], [Content-Length, 37], [Date, Mon, 23 Dec
                    2019 10:08:10 GMT]...}
RawContentLength  : 37

===============
RESPONSE object
===============

Ticks             : 1907668
Days              : 0
Hours             : 0
Milliseconds      : 190
Minutes           : 0
Seconds           : 0
TotalDays         : 2,20794907407407E-06
TotalHours        : 5,29907777777778E-05
TotalMilliseconds : 190,7668
TotalMinutes      : 0,00317944666666667
TotalSeconds      : 0,1907668

Output of the Agent:

<<<local>>>
==============
REQUEST object
==============
===============
RESPONSE object
===============


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 145
Ticks             : 11459153
TotalDays         : 1,32629085648148E-05
TotalHours        : 0,000318309805555556
TotalMinutes      : 0,0190985883333333
TotalSeconds      : 1,1459153
TotalMilliseconds : 1145,9153

So, even tho the MS response is still different, the $request variable isn’t being printed in the Agent Output. (Keep in mind that I’m using the Local System Account to run the script directly/test the agent output).

Does this tell us anything?

Thank you very much once again.

Daniel

[DanielNunesFerreira (Daniel Ferreira)]

Hi Alex,

That’s a good question actually, I’ll investigate this and come back to you.

Thank you!

Daniel

[DanielNunesFerreira (Daniel Ferreira)]

It turns out that the REQUEST object wasn’t working due to a certificate problem. I’ve fixed that but the problem with the different response in MS is still here.

[DanielNunesFerreira (Daniel Ferreira)]

Any news regarding this matter?

[tavanez (Pedro Tavanez)]

@DanielNunesFerreira

$response = (Measure-Command { Invoke-WebRequest -Uri “SOMEURL” -UseBasicParsing }).TotalMilliseconds

$status = 2 #critical
$crit = 200
$warn = 120

if ($response -gt $crit) {
$status = 2 #critical
}
elseif ($response -gt $warn) {
$status = 1 #warning
}
else {
$status = 0 #ok
}
“$status HTTPResponseTime-Exchange response_time=$response;$warn;$crit;; HTTP response time $response ms”

Hi Daniel,

We have tested your script and in fact the response time will increase due to other scripts and their execution time.

Maybe you could try this and check if the results are more similiar.

$URL='Inser URL here'
$Request = New-Object System.Net.WebClient
$Start = Get-Date
$PageRequest = $Request.DownloadString($URL)
$TimeTaken = ((Get-Date) - $Start).TotalMilliseconds
$Request.Dispose()
$status = 2 #critical
$crit = 200
$warn = 120
if ($TimeTaken -gt $crit) {
$status = 2 #critical
}
elseif ($TimeTaken -gt $warn) {
$status = 1 #warning
}
else {
$status = 0 #ok
}
“$status HTTPResponseTime-Exchange response_time=$TimeTaken;$warn;$crit;; HTTP response time $TimeTaken ms”

First print is your script running with another that takes long time to execute. The second is the new code with the same script that takes long time.!

Invoke-WebRequest|638x45

Ficas a dever duas cervejas, para mim e para o Ricardo
Cheers

[DanielNunesFerreira (Daniel Ferreira)]

@tavanez

$URL=‘Inser URL here’ $Request = New-Object System.Net.WebClient $Start = Get-Date $PageRequest = $Request.DownloadString($URL) $TimeTaken = ((Get-Date) - $Start).TotalMilliseconds $Request.Dispose() $status = 2 #critical $crit = 200 $warn = 120 if ($TimeTaken -gt $crit) { $status = 2 #critical } elseif ($TimeTaken -gt $warn) { $status = 1 #warning } else { $status = 0 #ok } “$status HTTPResponseTime-Exchange response_time=$TimeTaken;$warn;$crit;; HTTP response time $TimeTaken ms”

Hi,

Tried it, still doesn’t work.

Thank you anyway palhaços

[tavanez (Pedro Tavanez)]

@DanielNunesFerreira try to setup cache on ini and repeat.
I had something similiar for some exchange plugins and cache sorted it out

[DanielNunesFerreira (Daniel Ferreira)]

I’m using version 1.6.0p6 so the ini file got replaced by a YAML file located in C:\ProgramData\checkmk\agent\check_mk.user.yml.

I’m also using cache but this doesn’t really solve the problem because everytime I get actual new data, it will still have the high ms value that I’m getting.

[BH2005 (Bernd Holzhauer)]

any updates or solved ???

[DanielNunesFerreira (Daniel Ferreira)]

No solution yet. Still the same.

[SergejKipnis (Sergej)]

Agent starts plugins using different, i.e. system account.
This is potential reason why our results are different.
Second potential reason is CPU load, normally when we are running checks, CPU load is a bit high
Third - session. Windows service are running in different sessions.

Can you try ‘async and runas’?