Windows Agents Intermittently Failing

Tralin
Wed, 29 Jun 2022 13:19:01 +0000

Thanks, for the responses, Andreas and Mike.

I’m not sure the answer is in the agent configuration. The agents were functioning entirely normally for over two years with no configuration changes, updates, or otherwise. The only reason I’ve switched some of these over to the Bakery was to try to fix agents just not working. Nevertheless, here’s the agent Bakery config from the server I posted yesterday:

# Created by Check_MK Agent Bakery.
# This file is managed via WATO, do not edit manually or you
# lose your changes next time when you update the agent.

global:
  disabled_sections:
  - msexch
  - openhardwaremonitor
  - skype
  enabled: true
  install: true
  only_from:
  - ---checkmk---
  port: 6556
plugins:
  enabled: true
  execution:
  - async: true
    cache_age: 14400
    pattern: $CUSTOM_PLUGINS_PATH$\windows_updates.vbs
    timeout: 600
  - cache_age: 3600
    pattern: $CUSTOM_PLUGINS_PATH$\cmk_update_agent.checkmk.py
ps:
  enabled: true
  full_path: true
  use_wmi: true
system:
  enabled: true
  firewall:
    mode: configure
    port: auto
  service:
    error_mode: log
    restart_on_crash: 'yes'
    start_mode: auto

Going back through the log, I do see Veeam was taking a very long time to return data, but so was the MSSQL check.

2022-06-28 14:40:16.463 [srv 5408] [Err  ] Timeout [5] seconds broken  when query WMI
2022-06-28 14:40:16.464 [srv 5408] [Warn ] Object 'Win32_PerfRawData_NETFramework_NETCLRMemory' in 4994ms sends NO DATA
2022-06-28 14:40:16.465 [srv 5408] [Warn ] On timeout in section 'dotnet_clrmemory' try reuse cache
2022-06-28 14:40:16.467 [srv 5408] [Trace] Sending data 'dotnet_clrmemory' id is [967921619262512] length [8519]
2022-06-28 14:40:16.468 [srv 5408] perf: Section 'dotnet_clrmemory' took [4997] milliseconds
2022-06-28 14:40:16.479 [srv 5408] Received [8647] bytes from 'dotnet_clrmemory'
2022-06-28 14:41:27.030 [srv 5408] [Warn ] perf:  In [75605] milliseconds process 'powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File "C:\ProgramData\checkmk\agent\plugins\veeam_backup_status.ps1"' pid:[13332] FAILED - generated [378] bytes of data in [2] blocks
2022-06-28 14:41:27.045 [srv 5408] [Warn ] Sync Plugin stopped 'C:\ProgramData\checkmk\agent\plugins\veeam_backup_status.ps1' Stopped: false Failed: true
2022-06-28 14:41:27.752 [srv 5408] [Warn ] perf:  In [76324] milliseconds process 'cscript.exe //Nologo "C:\ProgramData\checkmk\agent\plugins\mssql.vbs"' pid:[10572] FAILED - generated [191880] bytes of data in [71] blocks
2022-06-28 14:41:27.776 [srv 5408] [Warn ] Sync Plugin stopped 'C:\ProgramData\checkmk\agent\plugins\mssql.vbs' Stopped: false Failed: true
2022-06-28 14:41:27.794 [srv 5408] [Trace] Provider 'plugins' is about to be started, id '967921619262512' port [mail:\\.\mailslot\Global\WinAgent_0]
2022-06-28 14:41:27.796 [srv 5408] [Trace] Sending data 'plugins' id is [967921619262512] length [2820]

While I can certainly see Veeam and MSSQL tying each other up, it’s still strange that suddenly after years of everything working fine it suddenly breaks. It also doesn’t explain the other servers that have had the same issue.

I’ll try to keep an eye on the monitor today and see if I can catch a non-Veeam server having the issue, or if a Veeam server does have the issue I’ll try manually running the Veeam check to see if it works, and try removing the Veeam check from the agent to test.

Thanks,
Tralin