Certificate error when registering agent for automatic updates in CheckMK 2.0.0

scarpatommaso · September 15, 2023, 9:29am

CMK version:
2.0.0p23

OS version:
RHEL7

Error message:

Updated the certificate store "/var/lib/check_mk_agent/cas/all_certs.pem" with 1 certificate(s)
Going to register agent at deployment server
Trying to import certificate from the server's certificate chain but found no self-signed certificate or CA certificate. Aborting import.
HTTPSConnectionPool(host='mymonitoringserver.it', port=8443): Max retries exceeded with url: /cmk/check_mk/login.py (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
See syslog or Logfile at /var/lib/check_mk_agent/cmk-update-agent.log for details.

“/var/lib/check_mk_agent/cmk-update-agent.log”:

2023-09-15 10:51:40,698 DEBUG: Authenticating at Checkmk Server (using requests): https://mymonitoringserver.it:8443/cmk/check_mk/login.py
2023-09-15 10:51:40,705 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
  File "site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
  File "site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
  File "site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages/urllib3/connectionpool.py", line 677, in urlopen
  File "site-packages/urllib3/connectionpool.py", line 381, in _make_request
  File "site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
  File "site-packages/urllib3/connection.py", line 370, in connect
  File "site-packages/urllib3/util/ssl_.py", line 377, in ssl_wrap_socket
  File "site-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages/requests/adapters.py", line 449, in send
  File "site-packages/urllib3/connectionpool.py", line 725, in urlopen
  File "site-packages/urllib3/util/retry.py", line 439, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mymonitoringserver.it', port=8443): Max retries exceeded with url: /cmk/check_mk/login.py (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cmk_update_agent.py", line 1889, in main
  File "cmk_update_agent.py", line 878, in run
  File "cmk_update_agent.py", line 1001, in _run_mode
  File "cmk_update_agent.py", line 1108, in _register_agent
  File "cmk_update_agent.py", line 323, in fetch_data_from_server
  File "cmk_update_agent.py", line 316, in fetch_data_from_server
  File "cmk_update_agent.py", line 331, in _do_request
  File "cmk_update_agent.py", line 384, in _login_site
  File "site-packages/requests/sessions.py", line 578, in post
  File "site-packages/requests/sessions.py", line 530, in request
  File "site-packages/requests/sessions.py", line 643, in send
  File "site-packages/requests/adapters.py", line 514, in send
requests.exceptions.SSLError: HTTPSConnectionPool(host='mymonitoringserver.it', port=8443): Max retries exceeded with url: /cmk/check_mk/login.py (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
2023-09-15 10:51:40,706 ERROR: HTTPSConnectionPool(host='mymonitoringserver.it', port=8443): Max retries exceeded with url: /cmk/check_mk/login.py (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Output of “cmk-update-agent register --server mymonitoringserver.it:8443 --protocol https --site cmk --hostname hostname.it --user cmkadmin --password “password” --trust-cert --verbose”:

In our checkmk site we configured CA certificate in global settings:

And we put the same CA certificate inside agent updater rule:

However, we always get the same error when we try to register and we see it on the Check_MK Agent service:

r.sander · September 15, 2023, 9:46am

The updater needs the CA certificate that signs the webserver certificate of the monitoring server.

If there is an intermediate certificate involved the webserver must also serve that together with its own certificate.

scarpatommaso · September 15, 2023, 2:25pm

In the global configuration I have only the Root CA and every other connection is working fine.
Do I need to add another certificate here?

r.sander · September 15, 2023, 2:37pm

This has nothing to do with the settings in the global configuration.
You need the root CA certificate of the certificate chain of the webserver.

scarpatommaso · September 15, 2023, 3:36pm

By webserver do you mean the server where I installed the agent?
If so, do you mean that I should install a certificate with same Root CA also on the agent?

I did this check on the CheckMK site and It does seem that the agent does not have a certificate installed:

OMD[cmk]:~$ openssl s_client -connect mymonitoredserver.it:6556
CONNECTED(00000003)
140716469011904:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:331:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 331 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
OMD[cmk]:~$

If this is the issue, how can I tell the agent to use the client server SSL certificate when connecting with the site?

r.sander · September 18, 2023, 6:53am

No. The webserver running on the monitoring server is meant here.

scarpatommaso · September 18, 2023, 7:53am

Ah ok. Thank you.
Can you tell me in which folder should I put the certificate? Do you mean following the instructions here Securing the web interface with HTTPS (checkmk.com) ?

But isn’t this the certificate that should be returned when I connect to mymonitoringserver.it:8443?