"The agent updater is not registered at the deployment server" yet registering commands dont give error

Checkermaker · April 3, 2024, 8:47am

Hello again!
First of all thanks for reading this and maybe helping.
I use the commercial enterprise version. See info below.

The problem I have is that I can register my agent updater on the hosts seemingly fine but I still get this error on the CheckMK dashboard.

I have this problem on all Windows Hosts even with firewall disabled.
Here’s the register command line output:

Microsoft Windows [Version 10.0.20348.2340]
(c) Microsoft Corporation. Alle rechten voorbehouden.

C:\Windows\system32>“C:\Program Files (x86)\checkmk\service\check_mk_agent.exe” updater -r
Response without json Content-Type
Failed to connect to agent bakery: 404 Client Error: Not Found for url: https://checkmk.SITENAME.be/deploy_agent.py
Retrying with fallback URL: https://checkmk.SITENAME.be/SITENAME/check_mk
<<<cmk_update_agent_status:sep(0)>>>
{“last_check”: 1712133104.6531098, “last_update”: null, “aghash”: null, “pending_hash”: “4a6e6213d97e0673”, “update_url”: “https://checkmk.SITENAME.be”, “trusted_certs”: {“0”: {“corrupt”: false, “not_after”: “20251005071608Z”, “signature_algorithm”: “sha512WithRSAEncryption”, “common_name”: “Agent signature”}}, “error”: null}
Response without json Content-Type
Failed to connect to agent bakery: 404 Client Error: Not Found for url: https://checkmk.SITENAME.be/deploy_agent.py
Retrying with fallback URL: https://checkmk.SITENAME.be/SITENAME/check_mk

There is no further output after the retrying fallback URL, is that normal?strong text

C:\Windows\system32>“C:\Program Files (x86)\checkmk\service\check_mk_agent.exe” updater register
Using previous settings from C:\ProgramData\checkmk\agent\config\cmk-update-agent.state.
Password for user ‘cmkadmin’:

Successfully registered agent of host “HOSTNAME” for deployment.

Here’s the updater config

CMK version:
2.2.0p9
OS version:
Windows Server

Error message:
Error message on dashboard:

Version: 2.2.0p9, OS: windows, Update error: Agent Bakery: Registering/updating at remote site but found no URL to central site. Please provide it in “automatic agent updates” section at global settings.WARN , No successful connect to server yetWARN , Agent plugins: 3, Local checks: 0

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

OMD[SITENAME]:~$ cmk --debug -vvn HOSTNAME
Checkmk version 2.2.0p9

FETCHING DATA
Source: SourceInfo(hostname=‘HOSTNAME’, ipaddress=‘192.168.1.42’, ident=‘agent’, fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fab4e837990]
Read from cache: AgentFileCache(HOSTNAME, path_template=/omd/sites/SITENAME/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 36 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 192.168.1.42:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b’16’)
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b’<<')
Closing TCP connection to 192.168.1.42:6556
Write data to cache file /omd/sites/SITENAME/tmp/check_mk/cache/HOSTNAME
Trying to acquire lock on /omd/sites/SITENAME/tmp/check_mk/cache/HOSTNAME
Got lock on /omd/sites/SITENAME/tmp/check_mk/cache/HOSTNAME
Releasing lock on /omd/sites/SITENAME/tmp/check_mk/cache/HOSTNAME
Released lock on /omd/sites/SITENAME/tmp/check_mk/cache/HOSTNAME
[cpu_tracking] Stop [7fab4e837990 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=4.71000000089407))]

PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser → HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser → HostSectionParser
<<<wmi_cpuload:sep(124)>>> / Transition HostSectionParser → HostSectionParser
<<>> / Transition HostSectionParser → HostSectionParser
<<>> / Transition HostSectionParser → HostSectionParser
<<fileinfo:sep(124)>> / Transition HostSectionParser → HostSectionParser
<<<winperf_phydisk>>> / Transition HostSectionParser → HostSectionParser
<<<winperf_if>>> / Transition HostSectionParser → HostSectionParser
<<<winperf_processor>>> / Transition HostSectionParser → HostSectionParser
<<>> / Transition HostSectionParser → HostSectionParser
<<<checkmk_agent_plugins_win:sep(0)>>> / Transition HostSectionParser → HostSectionParser
<<df:sep(9)>> / Transition HostSectionParser → HostSectionParser
<<<dotnet_clrmemory:sep(124)>>> / Transition HostSectionParser → HostSectionParser
<<>> / Transition HostSectionParser → HostSectionParser
<<ps:sep(9)>> / Transition HostSectionParser → HostSectionParser
Transition HostSectionParser → NOOPParser
<<<win_firewall_status:sep(124)>>> / Transition NOOPParser → HostSectionParser
<<<cmk_update_agent_status:cached(1712131308,3600):sep(0)>>> / Transition HostSectionParser → HostSectionParser
Transition HostSectionParser → NOOPParser
<<>> / Transition NOOPParser → HostSectionParser
HostKey(hostname=‘HOSTNAME’, source_type=<SourceType.HOST: 1>) → Add sections: [‘check_mk’, ‘checkmk_agent_plugins_win’, ‘cmk_agent_ctl_status’, ‘cmk_update_agent_status’, ‘df’, ‘dotnet_clrmemory’, ‘fileinfo’, ‘logwatch’, ‘mem’, ‘ps’, ‘services’, ‘systemtime’, ‘uptime’, ‘win_firewall_status’, ‘winperf_if’, ‘winperf_phydisk’, ‘winperf_processor’, ‘wmi_cpuload’]
Received no piggyback data
[cpu_tracking] Start [7fab4e8a2ad0]
value store: synchronizing
Trying to acquire lock on /omd/sites/SITENAME/tmp/check_mk/counters/HOSTNAME
Got lock on /omd/sites/SITENAME/tmp/check_mk/counters/HOSTNAME
value store: loading from disk
Releasing lock on /omd/sites/SITENAME/tmp/check_mk/counters/HOSTNAME
Released lock on /omd/sites/SITENAME/tmp/check_mk/counters/HOSTNAME
CPU utilization Total CPU: 8.98%
Check_MK Agent Version: 2.2.0p9, OS: windows, Update error: The agent updater is not registered at the deployment server(!), No successful connect to server yet(!), Agent plugins: 3, Local checks: 0
Disk IO SUMMARY Read: 1.50 kB/s, Write: 227 kB/s, Latency: 1 millisecond
Filesystem C:/ Used: 79.56% - 71.1 GiB of 89.4 GiB (warn/crit at 92.59%/96.29% used), trend per 1 day 0 hours: +435 MiB, trend per 1 day 0 hours: +0.48%, Time left until disk full: 42 days 23 hours
Interface 1 [vmxnet3 Ethernet Adapter], (Connected), Speed: 10 GBit/s, In: 10.1 kB/s (<0.01%), Out: 90.4 kB/s (<0.01%)
Log Application No error messages
Log HardwareEvents No error messages
Log Internet Explorer No error messages
Log Key Management Service No error messages
Log OAlerts No error messages
Log PDF Writer - bioPDF No error messages
Log Security No error messages
Log System No error messages
Log Windows PowerShell No error messages
Memory RAM: 41.03% - 13.1 GiB of 32.0 GiB, Commit charge: 48.15% - 17.7 GiB of 36.7 GiB
Processor Queue 15 min load: 0.20, 15 min load per core: 0.03 (6 logical cores)
Service CSFalconService CrowdStrike Falcon Sensor Service: running (start type is auto)
Service LansweeperAgentService Lansweeper Agent Service: running (start type is auto)
Service Net2ClientSvc Net2 Client Service: running (start type is auto)
Service SessionEnv Remote Desktop Configuration: running (start type is demand)
Service Summary Autostart services: 111, Stopped services: 4
Service TermService Remote Desktop Services: running (start type is demand)
Service UmRdpService Remote Desktop Services UserMode Port Redirector: running (start type is demand)
System Time Offset: -863 milliseconds
Uptime Up since Mar 21 2024 01:52:19, Uptime: 13 days 7 hours
Windows Firewall Status Profile Domain operational state is not as expected True vs. False(!), Profile Private as expected, Profile Public as expected
[cpu_tracking] Stop [7fab4e8a2ad0 - Snapshot(process=posix.times_result(user=0.019999999999999796, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.020000003278255463))]
[agent] Success, execution time 4.7 sec | execution_time=4.730 user_time=0.030 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=4.700

Checkermaker · April 3, 2024, 9:06am

Weirdly some hosts give a different error on the dashboard:

Version: 2.2.0p9, OS: windows, Update error: Agent Bakery: Error communicating with central bakeryWARN , No successful connect to server yetWARN , Agent plugins: 3, Local checks: 0

While most others give this error

Version: 2.2.0p9, OS: windows, Update error: The agent updater is not registered at the deployment serverWARN , No successful connect to server yetWARN , Agent plugins: 3, Local checks: 0

I’ve not found why yet

Checkermaker · April 3, 2024, 9:16am

Error communicating with central bakery

I found that the hosts that give this error are looking for the /deploy_agent.py URL but it doesnt exists anywhere, not even on my central site which is where they should get the updates from

jwiederh · April 3, 2024, 9:19am

Hi,

do you have a distributed setup, because it says

Registering/updating at remote site but found no URL to central site.

If so, you need to configure a connection to the central agent bakery on the satellite via the site specific global settings (under Setup → Distributed Monitoring):

Cheers Jeronimo

aeckstein · April 3, 2024, 9:24am

Additionally to what Jeronimo wrote:

Your satellite has to be able to talk to the central checkmk server via https.
The connection is initiated by the satellite, so DNS and firewalling / routing has to be configured correctly.

Checkermaker · April 3, 2024, 9:54am

@jwiederh

Indeed I have 5 satellites.
I checked that config and it is filled in. The url to the central site is filled in for all the remote sites.
I put it in as HTTPS://domainname.domain.x Should anything else be added?
To be sure I rebaked the agent and reinstalled/reregistered but it did not help.

@aeckstein

It gives the same error for some hosts that are on the central site so it doesn’t look like a network issue.
To confirm this I tried telnet on port 443 from a satellite to the central site, telnet can connect without issue

aeckstein · April 3, 2024, 10:05am

The two error messages are completely different stories:

This means, that you have not initially registered these hosts successfully.
You will have to issue the register command as described here:

Version: 2.2.0p9, OS: windows, Update error: Agent Bakery: Error communicating with central bakeryWARN , No successful connect to server yetWARN , Agent plugins: 3, Local checks: 0

This is the issue, that the central site is either not reachable by the satellite or configured wrong on the satellite.
You can test that with the curl command on the satellite with the central checkmk server name you configured in the"URL to central site" global setting.

Checkermaker · April 3, 2024, 11:21am

Hey @aeckstein , thanks for the reply

So for the first error of “The agent updater is not registered at the deployment server”
it’s strange because when I run the register command it looks succesful.
Are there any logs i can check or do you have any troubleshoot tips?

Yet the dashboard alarm remains even after waiting an hour:

For the second error I was able to find the issue with the curl command, so that’s solved. Thanks!

aeckstein · April 3, 2024, 11:27am

In the log directory of the agent in c:\programdata\checkmk\ are the agent updater logs.
Make sure, that the server name KO-DUA is the correct case sensitive hostname in checkmk.

Checkermaker · April 3, 2024, 2:20pm

Thanks

After running

"C:\Program Files (x86)\checkmk\service\check_mk_agent.exe" updater register

I get the message that registration is succesful, but then i run

"C:\Program Files (x86)\checkmk\service\check_mk_agent.exe" updater -r

And i get these errors in the logs, looks like its unable to get the certificate somehow.
I will investigate further but if you have any tips to give, i’d love them

2024-04-03 16:15:46,461 [6476] DEBUG: Starting Checkmk Agent Updater v2.2.0p9
2024-04-03 16:15:46,461 [6476] DEBUG: Successfully read C:\ProgramData\checkmk\agent\config\cmk-update-agent.state.
2024-04-03 16:15:46,461 [6476] DEBUG: Successfully read C:\ProgramData\checkmk\agent\config\cmk-update-agent.cfg.
2024-04-03 16:15:46,461 [6476] DEBUG: Updating the certificate store “C:\ProgramData\checkmk\agent\config\cas\all_certs.pem”…
2024-04-03 16:15:46,486 [6476] INFO: Updated the certificate store “C:\ProgramData\checkmk\agent\config\cas\all_certs.pem” with 2 certificate(s)
2024-04-03 16:15:46,486 [6476] DEBUG: MSI deposit path is C:\ProgramData\checkmk\agent\update\check_mk_agent.msi
2024-04-03 16:15:46,486 [6476] DEBUG: aghash info path is C:\ProgramData\checkmk\agent\install\checkmk.dat
2024-04-03 16:15:46,486 [6476] INFO: Starting Update mode as plugin.
2024-04-03 16:15:46,486 [6476] INFO: Getting target agent configuration for host ‘K7-VEEAM-WANACC’ from deployment server
2024-04-03 16:15:46,486 [6476] DEBUG: Fetching content (using requests): https://checkmkk7.SITENAME.be/K7/check_mk/deploy_agent.py
2024-04-03 16:15:46,566 [6476] DEBUG: Response from agent bakery:
{‘result_code’: 1, ‘result’: “Error communicating with central bakery\nHTTPSConnectionPool(host=‘checkmk.SITENAME.be’, port=443): Max retries exceeded with url: /deploy_agent.py (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)’)))”, ‘severity’: ‘error’}
2024-04-03 16:15:46,582 [6476] DEBUG: Successfully read C:\ProgramData\checkmk\agent\config\cmk-update-agent.state.
2024-04-03 16:15:46,582 [6476] DEBUG: Successfully read C:\ProgramData\checkmk\agent\config\cmk-update-agent.state.
2024-04-03 16:15:46,599 [6476] DEBUG: Saved deployment status to C:\ProgramData\checkmk\agent\config\cmk-update-agent.state.
2024-04-03 16:15:46,599 [6476] DEBUG: Caught Exception:
Traceback (most recent call last):
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 1572, in _do_update_as_plugin
send_feedback = self._do_update_agent()
^^^^^^^^^^^^^^^^^^^^^^^
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 1621, in _do_update_agent
target_state = self._get_target_state()
^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 1673, in _get_target_state
response = self._fetch_agent_info(mode=“status”)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 1735, in _fetch_agent_info
return self._http.fetch_data_from_server(“deploy_agent.py”, post_args=post_vars)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 496, in fetch_data_from_server
return self._do_request(base_url, site_rel_url, post_args, auth)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 533, in _do_request
self._check_for_response_error(response)
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 580, in _check_for_response_error
raise Exception(“Agent Bakery: %s” % response.json()[“result”])
Exception: Agent Bakery: Error communicating with central bakery
HTTPSConnectionPool(host=‘checkmk.SITENAME.be’, port=443): Max retries exceeded with url: /deploy_agent.py (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)’)))
2024-04-03 16:15:46,599 [6476] DEBUG: Writing agent section to stdout:
<<<cmk_update_agent_status:sep(0)>>>
{“last_check”: null, “last_update”: null, “aghash”: null, “pending_hash”: null, “update_url”: “https://checkmkk7.SITENAME.be/K7/check_mk”, “trusted_certs”: {“0”: {“corrupt”: false, “not_after”: “20251005071608Z”, “signature_algorithm”: “sha512WithRSAEncryption”, “common_name”: “Agent signature”}}, “error”: “Agent Bakery: Error communicating with central bakery\nHTTPSConnectionPool(host=‘checkmk.SITENAME.be’, port=443): Max retries exceeded with url: /deploy_agent.py (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)’)))”}

2024-04-03 16:15:46,599 [6476] DEBUG: Sending new state data to agent bakery
2024-04-03 16:15:46,599 [6476] DEBUG: Fetching content (using requests): https://checkmkk7.SITENAME.be/K7/check_mk/deploy_agent.py
2024-04-03 16:15:46,678 [6476] DEBUG: Response from agent bakery:
{‘result_code’: 1, ‘result’: “Error communicating with central bakery\nHTTPSConnectionPool(host=‘checkmk.SITENAME.be’, port=443): Max retries exceeded with url: /deploy_agent.py (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)’)))”, ‘severity’: ‘error’}
2024-04-03 16:15:46,686 [6476] DEBUG: Caught Exception:
Traceback (most recent call last):
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 1604, in _push_status
self._fetch_agent_info(mode=“status”)
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 1735, in _fetch_agent_info
return self._http.fetch_data_from_server(“deploy_agent.py”, post_args=post_vars)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 496, in fetch_data_from_server
return self._do_request(base_url, site_rel_url, post_args, auth)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 533, in _do_request
self._check_for_response_error(response)
File “C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.py”, line 580, in _check_for_response_error
raise Exception(“Agent Bakery: %s” % response.json()[“result”])
Exception: Agent Bakery: Error communicating with central bakery
HTTPSConnectionPool(host=‘checkmk.SITENAME.be’, port=443): Max retries exceeded with url: /deploy_agent.py (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)’)))
2024-04-03 16:15:46,690 [6476] DEBUG: Done.

aeckstein · April 3, 2024, 2:28pm

The satellite does not seem to trust the root CA certificate of the central servers web server certificate. Maybe you will have to add the CA certificate to the trusted root CAs in checkmk on the satellites or import it to the certificate store of your linux distribution.

Checkermaker · April 4, 2024, 8:14am

@aeckstein I’ve noticed the servers in the same subnet as the central site server do not have this certificate problem. I’m starting to think maybe it’s a port issue so i’ll go check the firewall logs.

I thought it only needed https access towards the central site server, are any other ports needed?

Checkermaker · April 4, 2024, 8:19am

The weird thing is, in the error I get after running updater -r it mentions its local checkmk server, while I would expect the central one instead.
It’s the central one that’s configured in the updater settings

So some cert issue between the central server and the satellites maybe?
But everything else is working, I can push updates to the satellites

C:\Windows\system32>“C:\Program Files (x86)\checkmk\service\check_mk_agent.exe” updater -r
<<<cmk_update_agent_status:sep(0)>>>
{“last_check”: null, “last_update”: null, “aghash”: null, “pending_hash”: null, “update_url”: “https://localserver.sitename.be/K4/check_mk”, “trusted_certs”: {“0”: {“corrupt”: false, “not_after”: “20251005071608Z”, “signature_algorithm”: “sha512WithRSAEncryption”, “common_name”: “Agent signature”}}, “error”: “Agent Bakery: Error communicating with central bakery\nHTTPSConnectionPool(host=‘centralserver.sitename.be’, port=443): Max retries exceeded with url: /deploy_agent.py (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)’)))”}

robin.gierse · April 5, 2024, 5:41pm

This is your error right there. Remote site is talking to the central site through HTTPS, but cannot verify the central site’s certificate. As soon as you enable this trust (not a Checkmk issue), you are all set.

Depending on what certificate your central site uses, you need to install the root certificate or the certificate itself on your remote site(s). But that is TLS 101, so I am not diving into that.