SSLError 'certificate verify failed' Version: 2.1.0p33, OS: linux with self signed certificate

BFP_IT_Sebastian · September 12, 2023, 11:59am

Hey Everyone,

we have updated CMK to 2.1 and setup TLS for communication.
Monitoring over secured connection work fine after registration.
After registering for Update over https, Windows Systems it works fine but on Ubuntu 22.04 I received folloing message and the update failed:

Version: 2.1.0p33, OS: linux, Update error: HTTPSConnectionPool(host=‘checkmk.xxx.local’, port=443): Max retries exceeded with url: /monitoring/check_mk/deploy_agent.py (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)])”)))

I use a self signed certificate which I include in the agents settings.

Based on https://forum.checkmk.com/t/sslerror-bad-handshake-certificate-verify-failed/34146 I try e.g.

openssl s_client -connect checkmk.xxx.local

…
CONNECTED(00000003)
depth=0 CN = bfp.local
verify return:1
…

wget https://checkmk.xxx.local/monitoring/check_mk/agents/plugins/mk_apt also works

manual update:

sudo cmk-update-agent -v

±------------------------------------------------------------------+
| |
| Checkmk Agent Updater v2.1.0p33 - Update |
| |
±------------------------------------------------------------------+
Getting target agent configuration for host ‘hostname’ from deployment server
HTTPSConnectionPool(host=‘checkmk.xxx.local’, port=443): Max retries exceeded wi th url: /monitoring/check_mk/deploy_agent.py (Caused by SSLError(SSLError(“bad h andshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)])”)))
See syslog or Logfile at /var/lib/check_mk_agent/cmk-update-agent.log for details.

manual update, insecure and add certificates:

sudo cmk-update-agent -v -x -t All HTTPS connections are done insecurely, as you requested. As a consequence, n o TLS verification will be done, i.e. the authenticity of the Checkmk server can not be guaranteed. However, HTTPS connections are still TLS-encrypted while usin g the “–insecure” option.

±------------------------------------------------------------------+
| |
| Checkmk Agent Updater v2.1.0p33 - Update |
| |
±------------------------------------------------------------------+
Getting target agent configuration for host ‘hostname’ from deployment server
Target state (from deployment server):

Agent Available: True*
Signatures: 1*
Target Hash: 92969f963c17270e*
Downloaded agent has size 17441636 bytes.
Signature check OK.
Invoking package manager: dpkg -i /tmp/check-mk-agent-h48122kd
Output from dpkg:
(Reading database … 135927 files and directories currently installed.)*
Preparing to unpack /tmp/check-mk-agent-h48122kd …*
Removing agent controller: /usr/bin/cmk-agent-ctl*
Removing deployed systemd units: check-mk-agent-async.service, cmk-update-agent.service, check-mk-agent@.service, check-mk-agent.socket, cmk-agent-ctl-daemon. service, cmk-update-agent.timer*
Unpacking check-mk-agent (2.1.0p33-6.92969f963c17270e) over (2.1.0p33-2.5f6473631dd1d9e)…
Setting up check-mk-agent (2.1.0p33-6.92969f963c17270e) …
Deploying agent controller: /usr/bin/cmk-agent-ctl*
Deploying systemd units: check-mk-agent-async.service cmk-update-agent.service check-mk-agent@.service check-mk-agent.socket cmk-agent-ctl-daemon.service cmk- update-agent.timer*
Deployed systemd*
Creating/updating cmk-agent user account …*
Activating systemd unit ‘check-mk-agent-async.service’…*
Activating systemd unit ‘check-mk-agent.socket’…*
Created symlink /etc/systemd/system/sockets.target.wants/check-mk-agent.socket → /lib/systemd/system/check-mk-agent.socket.*
Activating systemd unit ‘cmk-agent-ctl-daemon.service’…*
Activating systemd unit ‘cmk-update-agent.timer’…*

Successfully installed agent 92969f963c17270e.

manual update, again

sudo cmk-update-agent -v
Updated the certificate store “/var/lib/check_mk_agent/cas/all_certs.pem” with 2 certificate(s)

±------------------------------------------------------------------+
| |
| Checkmk Agent Updater v2.1.0p33 - Update |
| |
±------------------------------------------------------------------+
Getting target agent configuration for host ‘hostname’ from deployment server
HTTPSConnectionPool(host=‘checkmk.xxx.local’, port=443): Max retries exceeded with url: /monitoring/check_mk/deploy_agent.py (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)])”)))
See syslog or Logfile at /var/lib/check_mk_agent/cmk-update-agent.log for details.

Logfile:
…
2023-09-12 13:30:08,054 DEBUG: Starting Checkmk Agent Updater v2.1.0p33
2023-09-12 13:30:08,054 DEBUG: Successfully read /etc/cmk-update-agent.state.
2023-09-12 13:30:08,055 DEBUG: Successfully read /etc/check_mk/cmk-update-agent.cfg.
2023-09-12 13:30:08,055 DEBUG: Updating the certificate store “/var/lib/check_mk_agent/cas/all_certs.pem”…
2023-09-12 13:30:08,058 INFO: Updated the certificate store “/var/lib/check_mk_agent/cas/all_certs.pem” with 2 certificate(s)
2023-09-12 13:30:08,058 DEBUG: Running agent updater in InstallMode… Found no pending agent hash for installation. Nothing to do for us.
2023-09-12 13:30:08,059 DEBUG: Done.

2023-09-12 13:30:16,201 DEBUG: Starting Checkmk Agent Updater v2.1.0p33
2023-09-12 13:30:16,201 DEBUG: Successfully read /etc/cmk-update-agent.state.
2023-09-12 13:30:16,202 DEBUG: Successfully read /etc/check_mk/cmk-update-agent.cfg.
2023-09-12 13:30:16,202 DEBUG: Updating the certificate store “/var/lib/check_mk_agent/cas/all_certs.pem”…
2023-09-12 13:30:16,205 INFO: Updated the certificate store “/var/lib/check_mk_agent/cas/all_certs.pem” with 2 certificate(s)
2023-09-12 13:30:16,206 DEBUG: Starting manual update mode.
2023-09-12 13:30:16,206 INFO: Getting target agent configuration for host ‘hostname’ from deployment server
2023-09-12 13:30:16,207 DEBUG: Fetching content (using requests): https://checkmk.xxx.local/monitoring/check_mk/deploy_agent.py
2023-09-12 13:30:16,218 DEBUG: Caught Exception:
Traceback (most recent call last):

File “site-packages/urllib3/contrib/pyopenssl.py”, line 488, in wrap_socket*
File “site-packages/OpenSSL/SSL.py”, line 1934, in do_handshake*
File “site-packages/OpenSSL/SSL.py”, line 1671, in _raise_ssl_error*
File “site-packages/OpenSSL/_util.py”, line 54, in exception_from_error_queue*
OpenSSL.SSL.Error: [(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File “site-packages/urllib3/connectionpool.py”, line 677, in urlopen*
File “site-packages/urllib3/connectionpool.py”, line 381, in _make_request*
File “site-packages/urllib3/connectionpool.py”, line 976, in _validate_conn*
File “site-packages/urllib3/connection.py”, line 370, in connect*
File “site-packages/urllib3/util/ssl_.py”, line 377, in ssl_wrap_socket*
File “site-packages/urllib3/contrib/pyopenssl.py”, line 494, in wrap_socket*
ssl.SSLError: (“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)])”,)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File “site-packages/requests/adapters.py”, line 449, in send*
File “site-packages/urllib3/connectionpool.py”, line 725, in urlopen*
File “site-packages/urllib3/util/retry.py”, line 439, in increment*
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=‘checkmk.xxx.local’, port=443): Max retries exceeded with url: /monitoring/check_mk/deploy_agent.py (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)])”)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File “cmk_update_agent.py”, line 2066, in main*
File “cmk_update_agent.py”, line 936, in run*
File “cmk_update_agent.py”, line 1646, in _run_mode*
File “cmk_update_agent.py”, line 1328, in _do_update_as_command*
File “cmk_update_agent.py”, line 1379, in _do_update_agent*
File “cmk_update_agent.py”, line 1427, in _get_target_state*
File “cmk_update_agent.py”, line 1491, in _fetch_agent_info*
File “cmk_update_agent.py”, line 355, in fetch_data_from_server*
File “cmk_update_agent.py”, line 348, in fetch_data_from_server*
File “cmk_update_agent.py”, line 377, in _do_request*
File “site-packages/requests/sessions.py”, line 578, in post*
File “site-packages/requests/sessions.py”, line 530, in request*
File “site-packages/requests/sessions.py”, line 643, in send*
File “site-packages/requests/adapters.py”, line 514, in send*
requests.exceptions.SSLError: HTTPSConnectionPool(host=‘checkmk.xxx.local’, port=443): Max retries exceeded with url: /monitoring/check_mk/deploy_agent.py (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)])”)))
2023-09-12 13:30:16,219 ERROR: HTTPSConnectionPool(host=‘checkmk.xxx.local’, port=443): Max retries exceeded with url: /monitoring/check_mk/deploy_agent.py (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)])”)))

It look like that update runs fine what isn’t the truth an will noch be shown in the monitoring.
Manual update makes anything diffrent and shows the error from the monitoring.

Please help
Sebastian

andreas-doehler · September 14, 2023, 8:16am

If your certificate looks like this

and you call it with this

then the error message is perfectly fine.
The CNs and also the SANs should match the called names.

BFP_IT_Sebastian · September 14, 2023, 9:27am

Hello Andreas,

thanks for your post.
you are right, but bfp.local and xxx.local are the same. Only forget to to alienate one entry.
If this entry is realy wrong, the certification validation will fail, which is not the case.

Any other ideas?

Sebastian

andreas-doehler · September 14, 2023, 9:45am

I don’t think so - bfp.local is not the same as checkmk.bfp.local
For certificate problems it is not helpful to try to obfuscate domain names as these are most times the crucial informations.