Cannot make SSL connection Error for check-http Only

Troubleshooting
,
1

CMK version: Checkmk Raw Edition 2.1.0p20
OS version: Ubuntu 20.04 LTS

Error message: 140251841304384:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:ssl/record/rec_layer_s3.c:1543:SSL alert number 112

Output of “cmk --debug -vvn hostname”: N/A

We are having an issue with monitoring 1 of our HTTPS websites. The certificate age check is working properly, but the active-http check shows “CRITICAL - Cannot make SSL connection.”

1265×573 21.1 KB

On the service page, we see the above error message in the Details section.

There are 2 different DNS names for this website, that map to 2 different virtual hosts on a NetScaler appliance that then routes the traffic to the same cluster of web servers in the back end. This issue affects both websites, with the same error message being shown for each.

The NetScaler is configured via profiles, and there are other virtual hosts using the same profile on the same NetScaler that are working properly. It is only the 2 virtual hosts related to this specific website that are affected.

There are other instances of this same website hosted in other environments and the check-http monitor is working properly on those. It is just this one environment that is affected.

The checks have been like this since the day we added the host to the monitoring system, so it’s not a case of something having changed recently.

When we first added the host there was an issue with an infinite redirection loop, but that has since been resolved. The website loads fine in a browser, for both DNS names, and SSL Server Test (Powered by Qualys SSL Labs) reports no SSL configuration issues.

Does anyone have any suggestions on things we should check or how we can troubleshoot this further? SSL is clearly working, even from the Checkmk server, because otherwise the certificate age check would have the same error.

Any advice or suggestions welcome.

Thanks in advance,
Jason

2

run curl from the server (and preferably the Checkmk SITE) with -vv and look for the SSL part.

Checkmk needs to be able to read the WHOLE cert chain and if it cant you will get issues. Checkmk cannot download the intermediate cert that a browser sometimes can.

A rule of thumb dealing with SSL in Checkmk is that you actually need to be an expert more or less in HTTPS to be able to troubleshoot this.

3

Thank you for the suggestion Anders.

I tested using cURL from the site’s command line and confirmed that the connection is established as expected. I also tested another instance of the site (that is working in Checkmk) for comparison but both seem to be the same.

OMD[site]:~$ curl -vv https://uathostname.domain.com/healthcheck/healthcheck.html
*   Trying 1.2.3.4:443...
* TCP_NODELAY set
* Connected to uathostname.domain.com (1.2.3.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /omd/sites/cama/var/ssl/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.domain.com
*  start date: Nov 16 00:00:00 2022 GMT
*  expire date: Dec 17 23:59:59 2023 GMT
*  subjectAltName: host "uathostname.domain.com" matched cert's "*.domain.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
> GET /healthcheck/healthcheck.html HTTP/1.1
> Host: uathostname.domain.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/html
< Last-Modified: Fri, 03 Mar 2023 17:07:58 GMT
< Accept-Ranges: bytes
< ETag: "0abe9b3f24dd91:0"
< Server: Microsoft-IIS/8.5
< X-Powered-By: ASP.NET
< X-Frame-Options: SAMEORIGIN
< Date: Fri, 17 Mar 2023 15:37:00 GMT
< Content-Length: 3
< Strict-Transport-Security: max-age=0
<
* Connection #0 to host uathostname.domain.com left intact
OMD[site]:~$ curl -vv https://hostname.domain.com/healthcheck/healthcheck.html
*   Trying 146.20.205.156:443...
* TCP_NODELAY set
* Connected to hostname.domain.com (146.20.205.156) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /omd/sites/cama/var/ssl/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: REDACTED
*  start date: Jan 31 00:00:00 2023 GMT
*  expire date: Mar  2 23:59:59 2024 GMT
*  subjectAltName: host "hostname.domain.com" matched cert's "hostname.domain.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
> GET /healthcheck/healthcheck.html HTTP/1.1
> Host: hostname.domain.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/html
< Last-Modified: Thu, 16 Dec 2021 02:33:14 GMT
< Accept-Ranges: bytes
< ETag: "0516d4625f2d71:0"
< Server: Microsoft-IIS/8.5
< X-Powered-By: ASP.NET
< X-Frame-Options: SAMEORIGIN
< Date: Fri, 17 Mar 2023 15:37:14 GMT
< Content-Length: 3
< Strict-Transport-Security: max-age=157680000
<
* Connection #0 to host hostname.domain.com left intact
OMD[site]:~$

Any other thoughts on possible troubleshooting steps?

Regards,
Jason

4

I assume you have already set a “Virtual host” in the Host Section in the check_http rule?
If you are using SNI, you should also not disable the parameter “Advanced: Disable SSL/TLS hostname extension support (SNI)”

If that is not helping you can execute the check script directly on the checkmk site server.
The script is located here as the site user: ~/lib/nagios/plugins/check_http
The parameters for the script are visible in the “Service check command” of the affected service.
The script has an -v Parameter which may give you more information.

Best Regards
Thomas

5

Thank you for providing the path to the check script Thomas. It allowed me to do some additional troubleshooting that identified the issue.

OMD[cama]:~$ ~/lib/nagios/plugins/check_http '-u' '/healthcheck/healthcheck.html' '--ssl' '--extended-perfdata' '-j' 'HEAD' '--sni' '1.2.3.4' 'hostname.domain.com' -v
CRITICAL - Cannot make SSL connection.
140682615199552:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:ssl/record/rec_layer_s3.c:1543:SSL alert number 112
SSL initialized
OMD[cama]:~$ ~/lib/nagios/plugins/check_http '-u' '/healthcheck/healthcheck.html' '--ssl' '--extended-perfdata' '-j' 'HEAD' '--sni' '1.2.3.4' 'hostname.domain.com' -vv
CRITICAL - Cannot make SSL connection.
139730567477056:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:ssl/record/rec_layer_s3.c:1543:SSL alert number 112
SSL initialized
OMD[cama]:~$ ~/lib/nagios/plugins/check_http '-u' '/healthcheck/healthcheck.html' '--ssl' '--extended-perfdata' '-j' 'HEAD' '--sni' '1.2.3.4' 'uathostname.domain.com' -vv
SSL initialized
HEAD /healthcheck/healthcheck.html HTTP/1.1
User-Agent: check_http/v2.3.2 (monitoring-plugins 2.3.2)
Connection: close
Host: uathostname.domain.com


https://1.2.3.4:443/healthcheck/healthcheck.html is 311 characters
STATUS: HTTP/1.1 200 OK
**** HEADER ****
Content-Length: 3
Content-Type: text/html
Last-Modified: Fri, 03 Mar 2023 17:07:58 GMT
Accept-Ranges: bytes
ETag: "0abe9b3f24dd91:0"
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
X-Frame-Options: SAMEORIGIN
Date: Mon, 20 Mar 2023 14:54:06 GMT
Strict-Transport-Security: max-age=0
**** CONTENT ****

HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.202 second response time |time=0.202088s;;;0.000000;10.000000 size=311B;;;0; time_connect=0.038536s;;;;10.000000 time_ssl=0.083125s;;;;10.000000 time_headers=0.000008s;;;;10.000000 time_firstbyte=0.080267s;;;;10.000000 time_transfer=0.080283s;;;;10.000000

It seems there is an incompatibility between the way Checkmk/Nagios use SSL and our specific NetScaler implementation.

For reasons I won’t get in to, the Checkmk rule that defines this checks uses the production Virtual Host name across all environments. The production name is not a valid SNI name on the UAT NetScaler, resulting in the unrecognized name error we are seeing.

Strangely, the check works fine for our QA environment, even though the production name is not a valid SNI name there either.

I think what is happening in our UAT environment is that it is actually the NetScaler that is throwing the exception when it cannot find a certificate to serve in response to the request’s SNI value. We’re using HAProxy for the QA environment and it is probably falling back to the default certificate when there is no SNI match.

We updated the rule to use the environment-specific name for the service and now all monitors are working as intended.

Thank you again Anders and Thomas for getting me pointed in the right direction.

Regards,
Jason

1 Like
6

Just to inform that I faced the same issue, I added this into my rule to fix it:

image

Thanks @jsmyth, your troubleshooting helped me a lot.

7

Actually I had to Set it to $HOSTNAME$ to make the service check command match the CLI test.

Check HTTP service > Host settings > X Virtual host = $HOSTNAME$

Service check command check_mk_active-http!‘–ssl’ ‘–sni’ ‘$_HOSTADDRESS_4$’ ‘$HOSTNAME$’

OMD[MYSITE]:~$ ./lib/nagios/plugins/check_http --ssl -j HEAD -E -vvvvv --sni 123.123.123.123 www.mydomain.com.au

1 Like