CMK Suddenly can't connect to a specific host anymore (Error Establishing TLS Connection)

Kevd · March 7, 2025, 8:45am

CMK version:
Checkmk Enterprise Edition 2.3.0p17 using Agent Bakery
OS version:
Ubuntu 22.04.5 LTS
Host OS with Problem:
Windows Server 2019 1809

So I’ve ran into a weird problem. One of our hosts suddenly has an " Error establishing TLS connection" in the WebUI, despite no changes being made. When trying nc on port 6556 from the cmk server to the host, it shows that the connection was successful.

What I’ve tried:

Restarting the Agent Service
Removing registration via delete-all AND in the WebUI and then reregistering the agent and agent updater

This didn’t help.

A cmk-agent-ctl.exe status doesn’t show any issues on the host either.

Then I tried doing a .\check_mk_agent.exe check -self. Seems like this did show an issue: Exception in IO/ip: bind: An attempt was made to access a socket in a way forbidden by its access permissions.port 6556

Here is the full output:

PS C:\Program Files (x86)\checkmk\service> .\check_mk_agent.exe check -self 10
Adhoc/Exec Mode,press any key to stop execution
Successful start of thread
starting controller
Controller has NOT started: firewall to agent
Firewall mode is set to configure, adding rule...
Removing all 'Checkmk Agent application' app: 'C:\Program Files (x86)\checkmk\service\check_mk_agent.exe'
Removed 1 old rules.
Firewall rule 'Checkmk Agent application' had been added successfully for ports [6556]
Loading module config
Processed [1] module(s)
The command 'C:\ProgramData\checkmk\agent\modules\python-3\postinstall.cmd' is successful
Module 'python-3' is prepared to work with bin 'C:\ProgramData\checkmk\agent\modules\python-3\.venv\Scripts\python.exe'
Pre Start actions
OHM file 'C:\ProgramData\checkmk\agent\bin\OpenHardwareMonitorCLI.exe' is not found
Allowed Extensions: [checkmk.py,py,exe,bat,vbs,cmd,ps1]
Left [41] files to execute
Sync Plugin Entry 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py' forced to be async, due to cache_age [1200]
Sync Plugin Entry 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py' forced to be async, due to cache_age [1200]
Plugin 'C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs'  is  async with age:90000 timeout:3600 retry:0
Plugin 'C:\ProgramData\checkmk\agent\plugins\ad_replication.bat'  is  sync with age:0 timeout:60 retry:0
Plugin 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py'  is  async with age:1200 timeout:60 retry:0
Plugin 'C:\ProgramData\checkmk\agent\plugins\mk_inventory.vbs'  is  sync with age:0 timeout:60 retry:0
Plugin 'C:\ProgramData\checkmk\agent\plugins\win_license.bat'  is  sync with age:0 timeout:60 retry:0
Left [5] files to execute in 'plugins'
Data 'C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs' is too old, age is '1080928' seconds
restarting async plugin 'C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs'
restarted thread for plugin 'C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs'
Async Thread for C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs is to be started
Data 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py' is too old, age is '1080928' seconds
Exec app 'cscript.exe //Nologo "C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs"', mode [0]
restarting async plugin 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py'
restarted thread for plugin 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py'
Async Thread for C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py is to be started
Allowed Extensions: [checkmk.py,py,exe,bat,vbs,cmd,ps1]
Plugin 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py' has updater start mode
Left [41] files to execute
Exec app 'C:\ProgramData\checkmk\agent\modules\python-3\.venv\Scripts\python.exe C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py', mode [1]
Sync Plugin Entry 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py' forced to be async, due to cache_age [1200]
Sync Plugin Entry 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py' forced to be async, due to cache_age [1200]
Plugin 'C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs'  is  async with age:90000 timeout:3600 retry:0
Plugin 'C:\ProgramData\checkmk\agent\plugins\ad_replication.bat'  is  sync with age:0 timeout:60 retry:0
Plugin 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py'  is  async with age:1200 timeout:60 retry:0
Plugin 'C:\ProgramData\checkmk\agent\plugins\mk_inventory.vbs'  is  sync with age:0 timeout:60 retry:0
Plugin 'C:\ProgramData\checkmk\agent\plugins\win_license.bat'  is  sync with age:0 timeout:60 retry:0
Left [5] files to execute in 'plugins'
Allowed Extensions: [exe,bat,vbs,cmd,ps1]
Left [0] files to execute
Left [0] files to execute in 'local'
Pre Start actions ended
Waiting for async threads [2]
Appended [386] bytes from 'C:\ProgramData\checkmk\agent\modules\python-3\.venv\Scripts\python.exe C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py', timeout is [2500ms]
perf:  In [2852] milliseconds process 'cscript.exe //Nologo "C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs"' pid:[2352] SUCCEDED - generated [86] bytes of data in [2] blocks
Thread OFF: 'C:\ProgramData\checkmk\agent\plugins\windows_updates.vbs'
Left async threads [1] after waiting 5500ms
Starting io with 6556 None
cma::world::ExternalPort::ioThreadProc:  started
main Wait Loop
Starting IO ipv6:false, used port:6556
Exception in IO/ip: bind: An attempt was made to access a socket in a way forbidden by its access permissions.port 6556
IO broken with exception bind: An attempt was made to access a socket in a way forbidden by its access permissions.
Received 2 bytes
Received 2 bytes
Plugin 'C:\ProgramData\checkmk\agent\modules\python-3\.venv\Scripts\python.exe C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py' hits timeout
Process 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py' timeout in 63 when set 60
Thread OFF: 'C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py'
Received 2 bytes
Server is going to stop
Stop Service called
Stop request is set
main Wait Loop END
Shutting down IO...
Stopping execution
Thread is stopped
Killing Agent extensions
All extensions are dead, ok.
Waiting for testing thread
Received 2 bytes
Leaving testing thread
!

Does anyone have an idea what could cause this?

Kevd · March 10, 2025, 7:42am

Just did a full Agent reinstall, that also didn’t help.

Noone able to help?

Here’s the connection test result:

Error establishing TLS connection<<<esx_vsphere_vm:cached(1741593488,90)>>>
config.datastoreUrl name VxRail-Virtual-SAN-Datastore|accessible true|capacity 122904090116096|freeSpace 94630362958556|maintenanceMode normal|type vsan|uncommitted 52916640591872|url ds:///vmfs/volumes/vsan:5263299423121df6-fe4ecf1b0d325d2f/
config.guestFullName Microsoft Windows Server 2019 (64-bit)
config.hardware.device virtualDeviceType VirtualCdrom|label CD/DVD drive 1|summary Remote device|startConnected false|allowGuestControl true|connected false|status ok@@virtualDeviceType VirtualVmxnet3|label Network adapter 1|summary DVSwitch: 50 32 5c 76 a7 d8 7e 6e-0b 2d fe 26 9f 00 33 ff|startConnected true|allowGuestControl true|connected true|status ok@@virtualDeviceType VirtualVmxnet3|label Network adapter 2|summary DVSwitch: 50 32 5c 76 a7 d8 7e 6e-0b 2d fe 26 9f 00 33 ff|startConnected true|allowGuestControl true|connected true|status ok
config.hardware.memoryMB 16384
config.hardware.numCPU 4
config.hardware.numCoresPerSocket 1
config.template false
config.uuid 423281b5-e8ef-a210-9480-6a4ddb69bee5
config.version vmx-19
guest.toolsVersion 12421
guest.toolsVersionStatus guestToolsCurrent
guestHeartbeatStatus green
name SVAD03
runtime.host vx-node-1-b.flahv.local
runtime.powerState poweredOn
snapshot.rootSnapshotList 438 1741504415 poweredOff __HYCU_Snapshot__bb8a08a2-86d4-4aac-8c8e-3bcd8fc381e0
summary.guest.hostName SVAD03.flahv.local
summary.quickStats.balloonedMemory 0
summary.quickStats.compressedMemory 0
summary.quickStats.consumedOverheadMemory 91
summary.quickStats.distributedCpuEntitlement 610
summary.quickStats.distributedMemoryEntitlement 6079
summary.quickStats.guestMemoryUsage 2293
summary.quickStats.hostMemoryUsage 16461
summary.quickStats.overallCpuDemand 610
summary.quickStats.overallCpuUsage 395
summary.quickStats.privateMemory 16370
summary.quickStats.sharedMemory 12
summary.quickStats.staticCpuEntitlement 4087
summary.quickStats.staticMemoryEntitlement 16872
summary.quickStats.swappedMemory 0
summary.quickStats.uptimeSeconds 1425667
systime 1741593488
<<<labels:sep(0)>>>
{"cmk/piggyback_source_SVVCS": "yes"}

system · March 10, 2026, 7:43am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.