requests.exceptions.SSLError

Troubleshooting
1

CMK version: 1.6.0 p27

Error message:

requests.exceptions.SSLError: HTTPSConnectionPool(host=‘hooks.slack.com’, port=443): Max retries exceeded with url: /services/xxxxxxxx/yyyyyyyyyyyyyyyyyyyyyy (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘ssl3_get_server_certificate’, ‘certificate verify failed’)],)”,),))

Traceback (most recent call last): – File “/omd/sites/cw/share/check_mk/notifications/slack”, line 7, in – post_request(slack_msg) – File “/omd/sites/cw/lib/python/cmk/notification_plugins/utils.py”, line 285, in post_request – r = requests.post(url=url, json=message_constructor(context), proxies=proxies) – File “/omd/sites/cw/lib/python/requests/api.py”, line 116, in post – return request(‘post’, url, data=data, json=json, **kwargs) – File “/omd/sites/cw/lib/python/requests/api.py”, line 60, in request – return session.request(method=method, url=url, **kwargs) – File “/omd/sites/cw/lib/python/requests/sessions.py”, line 533, in request – resp = self.send(prep, **send_kwargs) – File “/omd/sites/cw/lib/python/requests/sessions.py”, line 646, in send – r = adapter.send(request, **kwargs) – File “/omd/sites/cw/lib/python/requests/adapters.py”, line 514, in send – raise SSLError(e, request=request) – requests.exceptions.SSLError: HTTPSConnectionPool(host=‘hooks.slack.com’, port=443): Max retries exceeded with url: /services/xxxxxxxx/yyyyyyyyyyyyyyyyyyyyyy (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘ssl3_get_server_certificate’, ‘certificate verify failed’)],)”,),))

Hello,

Since the hooks.slack.com certificate was renewed on Tue, 14 Mar 2023 05:17:13 GMT, the slack webhook failed to work, so for any host monitored by CheckMK, I have the above error, and host warnings from checkmk, will not be posted to a Slack Channel.

How can I fix this ??

2

Hello Bogdan,

The first thing that comes to mind is that maybe your Checkmk server is missing the root certificate to which the renewed certificate on hooks.slack.com rolls up.

What happens if you run the following command from the Checkmk server command line?

curl -v https://hooks.slack.com/

Hope this helps,
Jason

3

This is the output:

  • About to connect() to hooks.slack.com port 443 (#0)
  • Trying 3.68.170.153…
  • Connected to hooks.slack.com (3.68.170.153) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  •   subject: CN=slack.com

  •   start date: Mar 14 05:17:13 2023 GMT

  •   expire date: Jun 12 05:17:12 2023 GMT

  •   common name: slack.com

  •   issuer: CN=R3,O=Let's Encrypt,C=US

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: hooks.slack.com
Accept: /

< HTTP/1.1 302 Found
< date: Tue, 21 Mar 2023 19:50:07 GMT
< server: Apache
< x-powered-by: HHVM/4.153.1
< x-frame-options: SAMEORIGIN
< referrer-policy: no-referrer
< x-slack-backend: r
< x-slack-unique-id: ZBoKb1jG313ZdWHYAQTKBwAAABs
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< vary: Accept-Encoding
< location: https://api.slack.com/
< content-length: 0
< content-type: text/html
< x-envoy-upstream-service-time: 99
< x-backend: main_normal main_canary_with_overflow main_control_with_overflow
< x-server: slack-www-hhvm-main-iad-pewl
< x-slack-shared-secret-outcome: no-match
< via: envoy-www-iad-xerk, envoy-edge-fra-pgmz
< x-edge-backend: envoy-www
< x-slack-edge-shared-secret-outcome: no-match
<

4

Hi Bogdan,

Since curl works, the server must have the correct root certificate.

Under Global Settings → Trusted certificate authorities for SSL, do you have Trust system wide configured CAs turned on?


If yes, then certificate trust is probably not the issue and someone with deeper knowledge of the inner workings of the Slack plugin will need to weigh in.

If no you can either try enabling it or, if there are reasons why it should not be enabled in your environment, you can try downloading the root certificate and manually adding it to Checkmk.

Regards,
Jason

5

Is it turned on, indeed.

The main fact is that if I will curl -v Customize your workspace | Slack (this is not the actual webhook) I will get:
< HTTP/1.1 400 Bad Request
< date: Tue, 21 Mar 2023 20:38:07 GMT
< server: Apache
< x-powered-by: HHVM/4.153.1
< x-frame-options: SAMEORIGIN
< access-control-allow-origin: *
< referrer-policy: no-referrer
< x-slack-backend: r
< x-slack-unique-id: ZBoVr99uM2AkHMR8Xp7QkwAAADk
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< vary: Accept-Encoding
< content-type: text/html
< x-envoy-upstream-service-time: 105
< x-backend: main_normal main_canary_with_overflow main_control_with_overflow
< x-server: slack-www-hhvm-main-iad-yfnp
< x-slack-shared-secret-outcome: no-match
< via: envoy-www-iad-wlub, envoy-edge-fra-wqnb
< x-edge-backend: envoy-www
< x-slack-edge-shared-secret-outcome: no-match
< transfer-encoding: chunked
<

6

Hi Bogdan,

Please use the preformatted text format for code for better readability. You can do that by highlighting the text, then pressing Ctrl+E.

Not sure if the shipped python in 1.6.0p27 is maybe coming with an older urllib or something that has issues with the new certificate, can you reproduce this issue in a 2.1.0p24 test site? (if not, maybe this is the IT gods way of giving you incentive to update from 1.6 which was EOL last september :confused: )

Gerd

7

Sadly I don’t have now a 2.1.0p24 test site, but I will contact Slack Team for this, as this is quite strange, that until they have renewed the hooks.slack.com certificate, this has worked

8

Hi Bogdan,

I wouldn’t be surprised if slack points right back towards checkmk since curl is happy with the certificate. Setting up a test site should be easy though :wink: you can install checkkm 2.1.0p24 in parallel to your existing 1.6 and just create a new site, it will not interfere with the running 1.6 production.

9

So a simple update to v2.0.0 p34 solves the issue, as I’ve tested previously, but the question which still remains, why doesn’t work on v1.6.0 p27 ?
Strange behavior, as it worked before

10

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.