Error while registering windows agent

Troubleshooting
,
1

CMK version: 2.2.0p39 Raw Edition
OS version: Ubuntu Server 20.04LTS on slave site

Error message: Error registering existing host at https://IP:8000/SITE

Hi everyone, got a weird issue on my hands here. Here’s a brief overview:

2 sites, both at version 2.2.0p39. After updating all agents on the master site to 2.2 and successfully registering the agents I went on to update the agents on our slave site. After updating the agent on the first server (windows server 2022) I ran into an issue while registering.

Here’s the command I ran:

.\cmk-agent-ctl.exe -v register --hostname HOSTNAME --server IP:8000 --site SITE --user automation --password AUTOMATION_SECRET --trust-cert

Here’s the output:
[2025-04-22 14:04:27.502348 +02:00] INFO [cmk_agent_ctl] src\main.rs:14: starting
[2025-04-22 14:04:27.511512 +02:00] INFO [cmk_agent_ctl] src\lib.rs:50: Loaded config from ‘“C:\ProgramData\checkmk\agent\cmk-agent-ctl.toml”’, connection registry from ‘“C:\ProgramData\checkmk\agent\registered_connections.json”’
[2025-04-22 14:05:05.795766 +02:00] ERROR [cmk_agent_ctl] src\main.rs:29: Error registering existing host at https://IP:8000/SITE

The file c:\programdata\check\agent\registered_connections.json has the following content.

{
  "push": {},
  "pull": {},
  "pull_imported": []
}

I double checked the assigned site for this host, this is correct. A quick “omd config SITE show |grep AGENT_RECEIVER” shows the correct port 8000. A Test-NetConnection in powershell is successful, “tnc IP -Port 8000” gives me a “TcpTestSucceeded: True”.

Here are the latest log files on the slave-site:
…/var/log/agent-receiver/access.log

::ffff:IPv4:50671 - "POST /SITE/agent-receiver/register_existing HTTP/1.1" 500

…/var/log/agent-receiver/error.log

Traceback (most recent call last):
  File "/omd/sites/SITE/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/routing.py", line 460, in handle
    await self.app(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
   File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/omd/sites/SITE/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/agent_receiver/endpoints.py", line 118, in register_existing
    _sign_agent_csr(
  File "/omd/sites/SITE/lib/python3.11/site-packages/agent_receiver/endpoints.py", line 85, in _sign_agent_csr
    lifetime = controller_certificate_settings(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/agent_receiver/checkmk_rest_api.py", line 113, in wrapper
    return endpoint_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/agent_receiver/checkmk_rest_api.py", line 131, in controller_certificate_settings
    response = _forward_get(
               ^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/agent_receiver/checkmk_rest_api.py", line 76, in _forward_get
    return requests.get(
           ^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/SITE/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /SITE/check_mk/api/1.0/agent_controller_certificates_settings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7afcb52350>: Failed to establish a new connection: [Errno 111] Connection refused'))

Side note: I don’t understand why there’s a connection on localhost port 5000, have never seen that port in the documentation.

…/var/log/agent-receiver/agent-receiver.log
empty

I’ve temporarily assigned the roles “administrator” and “agent registration user” to the user automation. I reset the automation secret without success. The file “/omd/sites/SITE/var/check_mk/web/automation/automation.secret” contains the correct set password. I’ve created a completely new user with the same permissions, sadly without change.

I’m at the end of my road here, happy for any ideas you guys can throw at me…
Thanks in advance!

2

I ran into something similar a while back while trying to use the automation user and it’s secret. I eventually determined that because that use used a secret instead of a password (htpasswd), it wouldn’t complete. Once I used the cmkadmin user, it registered fine.

Try making a new local/htpasswd user (not “Automation secret for machine accounts”) with admin permissions and then using that new account.

3

the checkmk agent receiver (that runs on port 8000 and is talking to your client) talks to the checkmk web application on the local apache, which runs on port 5000, however, only on 127.0.0.1:5000 i.e. only on IPv4

As your log contains ipv6, maybe there is some confusion on your host about what “localhost” points to? Maybe check your /etc/hosts if “localhost” only points to 127.0.0.1?

4

Thanks for your reply. sadly this didn’t work either. Created a new user with administrator and agent registration roles with a normal user login. No change in behavior.

5

thanks for the explanation. Here’s the content of my /etc/hosts on the omd site server.

127.0.1.1 HOSTNAME.DOMAIN HOSTNAME localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
~

A quick “ping localhost” returns the hostname as well as 127.0.1.1. “ping -6 localhost” returns “::1” which should be the proper return as well.

I didn’t have the chance to properly set up IPv6 on those systems as I recently took over this branch. So that might be the system default configuration.

6

Let’s check a basic, yet easily overlooked, setting. If you run omd config as the site user and go into Basic, is the agent receiver on? I suspect it is, but I tend to dig too deep too fast and often miss simple high-level things like this.

Are you trying to register TLS against the site responsible for monitoring that Windows server? If not, give that a try. Agent updates need to be against the central site, but the TLS needs to be against the site monitoring that server.

What core are you using? I’m assuming Nagios since you’re on Raw Edition.

Beyond that, I am not sure what to check next. Very few of my customers use TLS on their agents.

7

Agent_Receiver is enabled and configured to listen on port 8000. We’re indeed using nagios as the core.

I’m trying the registration on the same site as shown under “host configuration” - “monitored on site” - slave-site.

TBH I’m not hellbent on using agent registration. The connection configured in distributed monitoring already uses TLS-encryption. I just read through the documentation and interpreted this as best behavior.

I’m thinking about upgrading to 2.3.0 anyway. Since this system hasn’t been touched by my predecessor I wanted to slowly upgrade while keeping availability high. Since this involves an OS upgrade or migration (not sure yet which way to go) this also could clear up any old or wrong configuration.

8

Does it really say 127.0.1.1? The third octet should be Zero. I.e. 127.0.0.1

1 Like
9

It does say 127.0.1.1. That seemed weird to me too.

In the debian reference under 10.4 this get’s explained a bit more. Since this system has a static IP I changed the entry to 127.0.0.1. The line in /etc/hosts now looks like this:

127.0.0.1 HOSTNAME.DOMAIN HOSTNAME localhost

I then retried the registration with the following parameter.

.\cmk-agent-ctl.exe -v register --hostname HOSTNAME --server IP --site SITE --user automation --password AUTOMATION_SECRET --trust-cert

And guess what: this time it worked! What a weird issue. It always has to be DNS huh -.-

So to summarize: The agent tries to connect to the site but while redirecting the request to localhost:5000 the wrong dns-entry misdirects the request into most likely nothing. Could be an issue with the listening address of the agent registration service which only listen on 127.0.0.1 instead of 127.0.1.1 or localhost.

Anyway, thanks for all your help!

1 Like