Agent TLS registration fails with Server Error 500

Troubleshooting
1

Checkmk Enterprise Edition 2.1.0p20
Debian 11

Hi everyone,
below is the output of the “cmk-agent-ctl.exe register --trust-cert -vv” command:

[2023-02-10 12:54:18.488899 +01:00] INFO [cmk_agent_ctl] src\main.rs:14: starting [2023-02-10 12:54:18.489987 +01:00] INFO [cmk_agent_ctl] src\lib.rs:41: Loaded config from '"C:\\ProgramData\\checkmk\\agent\\cmk-agent-ctl.toml"', legacy pull 'LegacyPullMarker("C:\\ProgramData\\checkmk\\agent\\allow-legacy-pull")' exists [2023-02-10 12:54:18.491014 +01:00] DEBUG [reqwest::connect] C:\Users\sk\.cargo\registry\src\github.com-1ecc6299db9ec823\reqwest-0.11.14\src\connect.rs:429: starting new connection: https://172.xxx.xxx.xxx/ [2023-02-10 12:54:18.764387 +01:00] DEBUG [reqwest::connect] C:\Users\sk\.cargo\registry\src\github.com-1ecc6299db9ec823\reqwest-0.11.14\src\connect.rs:429: starting new connection: https://172.23.15.9:8000/ [2023-02-10 12:54:18.808115 +01:00] ERROR [cmk_agent_ctl] src\main.rs:29: Error pairing with 172.xxx.xxx.xxx:8000/rmk Caused by: Request failed with code 500 Internal Server Error: Internal Server Error

It worked flawlessy for the last 300 Hosts.
What has changed? I installed python3-pip and pypsrp to test the windows alert handler script located under “~/share/doc/check_mk/treasures/alert_handlers/windows”

Maybe this updated some python ssl packages or something similar?
Any input is welcomed - maybe someone faced something similar or can guide me to some helpful log files/commands.

Thank you and greetings
Simon

Update:
The output of var/log/agent-receiver/error.log

[2023-02-10 14:07:01 +0100] [9459] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/omd/sites/rmk/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/routing.py", line 390, in handle
    await self.app(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/omd/sites/rmk/lib/python3.9/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/omd/sites/rmk/lib/python3.9/site-packages/fastapi/routing.py", line 219, in app
    raw_response = await run_endpoint_function(
  File "/omd/sites/rmk/lib/python3.9/site-packages/fastapi/routing.py", line 152, in run_endpoint_function
    return await dependant.call(**values)
  File "/omd/sites/rmk/lib/python3.9/site-packages/agent_receiver/endpoints.py", line 50, in pairing
    uuid = uuid_from_pem_csr(pairing_body.csr)
  File "/omd/sites/rmk/lib/python3.9/site-packages/agent_receiver/utils.py", line 92, in uuid_from_pem_csr
    load_pem_x509_csr(pem_csr.encode())
  File "/omd/sites/rmk/local/lib/python3/cryptography/x509/base.py", line 569, in load_pem_x509_csr
    return rust_x509.load_pem_x509_csr(data)
cryptography.x509.base.InvalidVersion: 2 is not a valid CSR version

And this is the python3 directory of our site:

OMD[site]:~$ ls -lah local/lib/python3
total 1004K
drwxr-xr-x 11 rmk rmk 4.0K Feb 10 11:16 ./
drwxr-xr-x 7 rmk rmk 4.0K Feb 10 11:16 …/
drwxrwx— 3 rmk rmk 4.0K Feb 10 11:16 OpenSSL/
-rwxrwx–x 1 rmk rmk 959K Feb 7 13:37 _cffi_backend.cpython-39-x86_64-linux-gnu.so*
drwxrwx— 3 rmk rmk 4.0K Feb 10 11:16 cffi/
drwxrwx— 2 rmk rmk 4.0K Feb 10 11:16 cffi-1.15.1.dist-info/
drwxr-xr-x 5 rmk rmk 4.0K Feb 10 11:16 cmk/
drwxrwx— 5 rmk rmk 4.0K Feb 10 11:16 cryptography/
drwxrwx— 2 rmk rmk 4.0K Feb 10 11:16 cryptography-39.0.0.dist-info/
drwxrwx— 2 rmk rmk 4.0K Feb 10 11:16 pyOpenSSL-23.0.0.dist-info/
drwxrwx— 4 rmk rmk 4.0K Feb 10 11:16 pycparser/
drwxrwx— 2 rmk rmk 4.0K Feb 10 11:16 pycparser-2.21.dist-info/

Maybe someone can tell me which packages to delete and how to do that?

2

Is your agent reciever runnuing in the site?

3

I think so:

> root@checkmk:~# netstat -tupaln | grep 8000

tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 9455/python3

> root@checkmk:~# ps -ef | grep 9455

instance 9455 1 0 11:18 ? 00:00:01 python3 /omd/sites/instance/bin/gunicorn -D -p /omd/sites/instance/tmp/run/agent-receiver.pid --error-logfile /omd/sites/instance/var/log/agent-receiver/error.log --access-logfile /omd/sites/instance/var/log/agent-receiver/access.log --keyfile /omd/sites/instance/etc/ssl/agent_receiver_cert.pem --certfile /omd/sites/instance/etc/ssl/agent_receiver_cert.pem --ca-certs /omd/sites/instance/etc/ssl/ca.pem --cert-reqs 1 -b 0.0.0.0:8000 -k agent_receiver.worker.ClientCertWorker agent_receiver.apps:main_app()

4

OMD[beta]:~$ omd status
agent-receiver: running

Maybe try to restart the service

5

First of all - thank you for trying to help me!

OMD[instance]:~$ omd status
agent-receiver: running
mkeventd:       running
liveproxyd:     running
mknotifyd:      running
rrdcached:      running
cmc:            running
apache:         running
dcd:            running
redis:          running
crontab:        running
-----------------------
Overall state:  running

Restart, reboot all done :wink:

I still believe my mistake was to install python3-pip with all recommended packages from apt.

6

I hope you have a backup

7

I do :wink:
But maybe I can sort this out - if not I will do a clean Debian install and restore the instance

8
9

As mentioned in another thread, you actually ran into a newly implemented CSR version check that’s done by (in your case) cryptography 39.0.0: Agent-receiver pairing: "2 is not a valid CSR version" - #4 by antonio.furno

However, it’s still true that we got that version wrong and we’ll fix it before the error will occur in a released version (with our selection of python packages :wink: )

Cheers
Andi

1 Like
10

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.