CheckMK Agent Service restarting after Agent Update

Noah · November 8, 2023, 1:39pm

CMK version: 2.2.0p4
OS version:Windows Server 2016 and 2019

Error message:
Windows Agent Service restarts after automatic Agent Update, Info from the Agents Log:

Summary

2023-11-08 14:28:22.013 [srv 10268] [Trace] Enabled Base
2023-11-08 14:28:22.018 [srv 10268] [Trace] Setting root. service: ‘CheckMkService’, preset: ‘’
2023-11-08 14:28:22.018 [srv 10268] [Trace] Try service: ‘CheckMkService’
2023-11-08 14:28:22.019 [srv 10268] [Trace] Try registry ‘CheckMkService’
2023-11-08 14:28:22.019 [srv 10268] [Trace] Service is found ‘C:\Program Files (x86)\checkmk\service\check_mk_agent.exe’
2023-11-08 14:28:22.019 [srv 10268] Set root ‘C:\Program Files (x86)\checkmk\service’ from registry ‘CheckMkService’
2023-11-08 14:28:22.019 [srv 10268] [Trace] Try registry ‘CheckMkService’
2023-11-08 14:28:22.019 [srv 10268] Protect file from User write ‘C:\Program Files (x86)\checkmk\service\check_mk_agent.exe’
2023-11-08 14:28:22.020 [srv 10268] Protect path from User access ‘C:\Program Files (x86)\checkmk\service’
2023-11-08 14:28:22.023 [srv 10268] [Trace] Using root = ‘C:\Program Files (x86)\checkmk\service’ and data = ‘C:\ProgramData\checkmk\agent’ folders
2023-11-08 14:28:22.025 [srv 10268] COM Initialized
2023-11-08 14:28:22.025 [srv 10268] COM initialized
2023-11-08 14:28:22.025 [srv 10268] Found root config on path C:\Program Files (x86)\checkmk\service\check_mk.yml
2023-11-08 14:28:22.036 [srv 10268] [Trace] Enabled Debug
2023-11-08 14:28:22.036 [srv 10268] Loaded start config ‘C:\Program Files (x86)\checkmk\service\check_mk.yml’,‘C:\ProgramData\checkmk\agent\bakery’,‘C:\ProgramData\checkmk\agent\check_mk.user.yml’
2023-11-08 14:28:22.037 [srv 10268] service to run
2023-11-08 14:28:22.039 [srv 10268] Service Main
2023-11-08 14:28:22.039 [srv 10268] Service handlers registered
2023-11-08 14:28:22.039 [srv 10268] [Trace] Installing cap file ‘C:\Program Files (x86)\checkmk\service\install\plugins.cap’
2023-11-08 14:28:22.039 [srv 10268] Timestamp OK, checking file content…
2023-11-08 14:28:22.041 [srv 10268] [Trace] Installing of CAP file is not required
2023-11-08 14:28:22.041 [srv 10268] [Trace] Installing yml file ‘C:\Program Files (x86)\checkmk\service\install\check_mk.install.yml’
2023-11-08 14:28:22.042 [srv 10268] Timestamp OK, checking file content…
2023-11-08 14:28:22.042 [srv 10268] [Trace] Installing of YML file is not required
2023-11-08 14:28:22.042 [srv 10268] [Trace] Copy file ‘C:\Program Files (x86)\checkmk\service\install\checkmk.dat’ to ‘C:\ProgramData\checkmk\agent\install\checkmk.dat’
2023-11-08 14:28:22.042 [srv 10268] Timestamp OK, checking file content…
2023-11-08 14:28:22.043 [srv 10268] [Trace] Copy is not required, the file is already exists
2023-11-08 14:28:22.043 [srv 10268] Skip installing user yml file
2023-11-08 14:28:22.043 [srv 10268] Timestamp OK, checking file content…
2023-11-08 14:28:22.044 [srv 10268] Starting upgrade(migration) process…
2023-11-08 14:28:22.044 [srv 10268] [Trace] Legacy Agent not found Upgrade is not possible
2023-11-08 14:28:22.045 [srv 10268] [Trace] trying path C:\Program Files (x86)\checkmk\service
2023-11-08 14:28:22.045 [srv 10268] Found root config on path C:\Program Files (x86)\checkmk\service\check_mk.yml
2023-11-08 14:28:22.045 [srv 10268] [Trace] Loading ‘C:\Program Files (x86)\checkmk\service\check_mk.yml’
2023-11-08 14:28:22.046 [srv 10268] [Trace] Loading ‘C:\ProgramData\checkmk\agent\bakery\check_mk.bakery.yml’
2023-11-08 14:28:22.047 [srv 10268] [Trace] Loading ‘C:\ProgramData\checkmk\agent\check_mk.user.yml’
2023-11-08 14:28:22.050 [srv 10268] [Trace] Target ‘folders’ is empty, overriding with source
2023-11-08 14:28:22.052 [srv 10268] [Trace] Target ‘folders’ is empty, overriding with source
2023-11-08 14:28:22.052 [srv 10268] Loaded Config Files by Agent [2.2.0p4,64bit,release,Jun 20 2023,17:20:33] @ ‘Win10-64 server’
root: ‘C:\Program Files (x86)\checkmk\service\check_mk.yml’ size=12750 [OK]
bakery: ‘C:\ProgramData\checkmk\agent\bakery\check_mk.bakery.yml’ size=1014 [OK]
user: ‘C:\ProgramData\checkmk\agent\check_mk.user.yml’ size=17375 [OK]
2023-11-08 14:28:22.054 [srv 10268] [Trace] Sync Plugin Entry ‘C:\ProgramData\checkmk\agent\plugins\sslcertificates.ps1’ forced to be async, due to cache_age [86400]
2023-11-08 14:28:22.055 [srv 10268] [Trace] Sync Plugin Entry ‘C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py’ forced to be async, due to cache_age [3600]
2023-11-08 14:28:22.055 [srv 10268] [Trace] Sync Plugin Entry ‘C:\ProgramData\checkmk\agent\plugins\windows_tasks.ps1’ forced to be async, due to cache_age [840]
2023-11-08 14:28:22.055 [srv 10268] [Trace] Sync Plugin Entry ‘C:\ProgramData\checkmk\agent\plugins\rds_licenses.vbs’ forced to be async, due to cache_age [240]
2023-11-08 14:28:22.055 [srv 10268] [Trace] Enabled Debug
2023-11-08 14:28:22.056 [srv 10268] Loaded start config ‘C:\Program Files (x86)\checkmk\service\check_mk.yml’,‘C:\ProgramData\checkmk\agent\bakery’,‘C:\ProgramData\checkmk\agent\check_mk.user.yml’
2023-11-08 14:28:22.056 [srv 10268] [Trace] Successful start of thread
2023-11-08 14:28:22.058 [srv 10268] The network is available
2023-11-08 14:28:22.058 [srv 10268] starting controller
2023-11-08 14:28:22.059 [srv 10268] try to kill

The Servers are both Remote Desktop Servers, did anyone else had such a Problem? It occured twice on one of the Servers within 48hrs.

Norm · November 9, 2023, 7:20am

Hi @Noah,

I think this is normal behavior. The automatic agent update installs a new msi package if there are changes that need to be installed. During the update procedure, the Checkmk Service gets restarted. Depending on your update interval and changes to the agent config, it’s quite possible that it happened twice in the last 48 hrs.
Did you see this upon looking into logs, or did this caused any “real” problems?

Looking forward to your reply!
Norm

Noah · November 9, 2023, 7:42am

Hi, I see I wasn’t clear on what exactly happens.

The following events seem to have happened:

Agent Updater detects the new Update and downloads the package
The package seems to get installed, and it restarts the service to make the changes take effect
Now “CheckmkService” restarts repeatedly within one second and doesn’t start as expected.

So the Service doesn’t come back up and fails every time.

Yesterday I installed the CheckMK Agent with “change user /install” before executing the installer, as far as I know, it wasn’t done before, so let’s see.

Noah · November 15, 2023, 12:04pm

Hi again, Today I triggered an agent update, and unfortunately the issue persists. Has anyone else had a similar issue ? I can’t restart a Server just because of an agent update, especially a terminal server.

Noah · November 16, 2023, 9:20am

Update: The Service starts normally if I set the Service user to the domain administrator. After several hours, the Service starts working again, even if I don’t touch the server at all. On one server, the service started after 4 Hours of not working, and in those 4 hours, the service showed the first described behavior of starting and stopping.

I went through the Logs from the Program Data/checkmk folder and read the Event log, I didn’t see any events except the Service restarting and stopping.

Noah · November 21, 2023, 10:18am

I’m going to close this ticket as there are no more responses. The last behavior from the server was as follows: After some time, mostly multiple hours, the server recovered itself. I don’t know what else to do, I’m going to leave the issue unsolved.

Norm · November 21, 2023, 6:05pm

Hey @Noah,

please keep in mind that this is a community forum and not a official support channel.
If you expect an answer or support you need to have an active support contract with Checkmk or a Checkmk Partner.

This is not a ticket system.

Regards
Norm