CERTIFICATE_VERIFY_FAILED on HTTP_Check_Form_Submit

Troubleshooting
1

CMK version:
2.0.0p22.cee
OS version:
Ubuntu 20.04 TLS
Error message:
CERTIFICATE_VERIFY_FAILED
Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Hello All,
I have encountered a strange situation while attempting to monitor a web page, not on VM I have defined for monitoring (I am mentioning this while I think it is irelevant to the problem, just in case), for which problem I have started a different topic.

The problem here is as follows:

  1. When checking the web page (through CMK GUI) I get the error - Unable to open [(https://xxx.nn/login:) [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)

NOTE the “:” at the end of the URL
The settings are:
image

  1. When checking the web page (through CLI) I get the error:

sudo /opt/omd/versions/2.0.0p22.cee/lib/nagios/plugins/check_form_submit -I xxx.178.46.xxx -u /login -p 443 -e adresa -d
GET https://xxx.178.46.xxx/login
Unable to open https://xxx.178.46.xxx/login: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for ‘xxx.178.46.xxx’. (_ssl.c:1131)

NOTE the “:” at the end of the URL

BUT, when checking with curl:

curl -v --connect-to xxx.178.46.xxx:443 --max-redirs 3 --anyauth -I --user user@gmail.com:pass https://www.xxx.nn/login
*   Trying xxx.178.46.xxx:443...
* TCP_NODELAY set
* Connected to www.xxx.nn(xxx.178.46.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=NN; ST=City; L=City; O=NN a.d., City; CN=*.xxx.nn
*  start date: Feb  2 07:21:07 2022 GMT
*  expire date: Mar  6 07:21:07 2023 GMT
*  subjectAltName: host "www.xxx.nn" matched cert's "*.xxx.nn"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
> HEAD /login HTTP/1.1
> Host: www.xxx.nn
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 4148
Content-Length: 4148
< Content-Type: text/html
Content-Type: text/html
< Last-Modified: Tue, 20 Sep 2022 20:12:15 GMT
Last-Modified: Tue, 20 Sep 2022 20:12:15 GMT
< Accept-Ranges: bytes
Accept-Ranges: bytes
< ETag: "3d5727472dcdd81:0"
ETag: "3d5727472dcdd81:0"
< Server: Microsoft-IIS/10.0
Server: Microsoft-IIS/10.0
< X-Powered-By: ASP.NET
X-Powered-By: ASP.NET
< Date: Wed, 28 Sep 2022 08:21:57 GMT
Date: Wed, 28 Sep 2022 08:21:57 GMT

<
* Connection #0 to host www.xxx.nn left intact

What and where should I correct to be able to monitor this page?

2

Does the certificate contain the IP of the host as SAN entry like IP:xxx.178.46.xxx? I guess there could be a problem regarding if the URL is called FQDN or via IP.
I assume the trailing : is just for the error message like “what i do: error message”.

To support my assumption, in your call via the check you submit the IP:

and via curl you are connecting against the URL:

2 Likes
3

Thank you for your thaughts and the effort tosch.

The ending collon (“:”) is displayed in the link in the GUI report too:

image
image1295×62 38.3 KB

and via curl you are connecting against the URL:

[/quote]

If I try the check with URL - I get as follows:

image
image1280×58 36.6 KB

Note the collon again at the end of the URL.

Or via CLI:

sudo /opt/omd/versions/2.0.0p22.cee/lib/nagios/plugins/check_form_submit -I xxx.nn -u /login -p 443 -e adresa -d
GET https://xxx.nn/login
CODE: 200 RESPONSE:
Traceback (most recent call last):
  File "/opt/omd/versions/2.0.0p22.cee/lib/nagios/plugins/check_form_submit", line 380, in <module>
    exitcode, info = main()
  File "/opt/omd/versions/2.0.0p22.cee/lib/nagios/plugins/check_form_submit", line 363, in main
    raise_host_state(client, base_url, uri, timeout, form_name, params, expect_regex)
  File "/opt/omd/versions/2.0.0p22.cee/lib/nagios/plugins/check_form_submit", line 228, in raise_host_state
    real_url, body = open_url(client, base_url + uri, timeout=timeout)
  File "/opt/omd/versions/2.0.0p22.cee/lib/nagios/plugins/check_form_submit", line 150, in open_url
    ['    %02d %s' % (index + 1, line) for index, line in enumerate(content.split('\n'))]))

TypeError: a bytes-like object is required, not 'str'

The TypeError I get - is the same as in GUI on later version (2.1). Hence I assumed that I must enter IP of the host for HTTP_check_form_submit, because it expect “bytes-like” data there and used the IP in all checks.

4

Hmmm…
No further ideas?

5

There are some points you need to do.

Please don’t execute check plugins on the command line as root.
Do this every time as the site user. Also your curl test should be done as site user to have the same environment as the check plugin.

What happens if you try the “check_form_submit” on the CLI with the following options?

~/lib/nagios/plugins/check_form_submit -I www.xxx.nn -u /login -p 443 -s -e adresa -d

Also you should check what this server really gives as an response with

openssl s_client -connect www.xxx.nn:443
2 Likes
6

Thanx Andreas,
I will keep in mind that and try it.

  /opt/omd/versions/2.0.0p22.cee/lib/nagios/plugins/check_form_submit -I 'xxx.rs' -p 443 -u '/login' -e 'adresa' -f 'Login' -q '_username=nnn@gmail.com&password=nnn.isp' 
 Unable to open https://xxx.rs/login: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)

The server response is as follows:

openssl s_client -connect www.xxx.rs:443
CONNECTED(00000003)
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = RS, ST = Beograd, L = Beograd, O = "Preduzece za telekomunikacije", CN = *.xxx.rs
verify return:1
---
Certificate chain
 0 s:C = RS, ST = Beograd, L = Beograd, O = "Preduzece za telekomunikacije", CN = *.xxx.rs
   i:C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018
 1 s:C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018
   i:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGtDCCBZygAwIBAgIMELMvZuP0IR3pnZUNMA0GCSqGSIb3DQEBCwUAMFAxCzAJ
BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSYwJAYDVQQDEx1H
czAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHwYDVR0jBBgwFoAU+O9/
8s14Z6jeb48kjYjxhwMCs+swHQYDVR0OBBYEFHMyX+BTZWsneA6RvOZeew77foau
w/zG6feYhm8IbogIps046gIhAIIvgJ46Bo5Cuqs8+fMzdw2HDwTDyqlMDLN7w0BU

BAMASDBGAiEAjliMUwkBX0dRjtZUPlBbpqMZ1Uw/IAjEM9irpF06iVYCIQCc2nn6
7JBZNFRR9pAq1eIP33Bv52TNDUY0UB+0hFqoZgB2AFWB1MIWkDYBSuoLm1c8U/DA

wWCbfjvYqGBjD10cEGsxfvvio+XOzKmpdPsZV1kmUTlrwDUoMcFYTw==
-----END CERTIFICATE-----
subject=C = RS, ST = Beograd, L = Beograd, O = "Preduzece za telekomunikacije", CN = *.xxx.rs

issuer=C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3391 bytes and written 395 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 2533F5D1D4EB51AB822DDA7B68EEF125181E3C0A70EB4F0426650C4000488A11
    Session-ID-ctx:
    Resumption PSK: 2642D51BD7080381DC9AFB8BD301E664130C96657CDF114697DE3CCA81DA516A04060A0E7FC48C7E13294E5D21D6F1C1
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 07 bc b0 bf ab bc c5 fd-15 ea ff 5a ce ba 94 60   ...........Z...`
    0010 - 73 c0 fe 48 33 5d e1 64-43 c9 4b 18 30 7a 61 04   s..H3].dC.K.0za.
    0020 - 09 32 6a 68 40 e2 0e c7-72 8f ce e3 b4 69 ca d1   .2jh@...r....i..
    0030 - 97 53 69 e2 62 db 9e e6-8a 75 9b 82 9a 69 11 92   .Si.b....u...i..
    0040 - f6 a6 7c b8 5d 9c 14 97-c7 ff b0 32 30 e2 32 54   ..|.]......20.2T
    0050 - 21 5d 76 89 07 0c 95 8c-2d 9d 55 f0 b1 0c 82 d7   !]v.....-.U.....
    0060 - 6c 73 45 dd 48 4e d4 dd -6f 55 de 80 34 82 b1 7a   lsE.HN..oU..4..z
    0070 - c0 7d 69 64 00 8f 5d 05-cf 34 a5 73 54 33 33 01   .}id..]..4.sT33.
    0080 - a9 95 02 91 4f c9 8f 99-e1 8c 8d ce 08 4e 29 6c   ....O........N)l
    0090 - 0e 34 2c 0f 00 c8 fc 35-f0 aa 5f 5a b4 e7 c8 e4   .4,....5.._Z....
    00a0 - 5f b0 9f 49 99 0c 56 11-1d e7 c3 69 fc a6 a8 dc   _..I..V....i....
    00b0 - f6 dd 2b 94 c5 67 14 9f-09 41 c2 b4 06 2a 36 61   ..+..g...A...*6a
    00c0 - 49 37 37 20 98 20 98 c2-07 38 89 a9 c6 03 8d 9e   I77 . ...8......
    00d0 - 4f 6d d0 83 77 05 c2 5f-b7 f8 c4 98 cd 38 35 40   Om..w.._.....85@

    Start Time: 1665055466
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: B0C2EDE3F3E7F90939F4AFD62A313BA27831EB9A1C9933346B50FB5E432BBCE8
    Session-ID-ctx:
    Resumption PSK: 072D90FEAF78116FA2C4A030ECC7AB8BF75E9ABF70A5FD7D98011FE42D7476D61901B3E8F0265F964E20CC6CD7C274CC
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 07 bc b0 bf ab bc c5 fd-15 ea ff 5a ce ba 94 60   ...........Z...`
    0010 - a0 ec fc 34 75 b7 a4 50-8a 60 72 57 55 50 ee b2   ...4u..P.`rWUP..
    0020 - a4 ea c3 e1 b6 20 36 ca-3d 14 65 54 4b f0 ba 3d   ..... 6.=.eTK..=
    0030 - cc e8 3b 19 0a ff 3c f1-b9 54 4c 92 4f 18 a2 bb   ..;...<..TL.O...
    0040 - 15 50 1b 8a ec 8f ac 43-10 4d db e9 44 45 11 42   .P.....C.M..DE.B
    0050 - 77 fa 6a bf 44 4d 01 4b-be b1 aa be 4b 7b 12 be   w.j.DM.K..1.K{..
    0060 - 0d e5 32 0b 42 ea f3 5a-73 03 19 94 ea 4f d3 af   ..2.B..Zs....O..
    0070 - d4 43 d8 35 42 95 9f fa-9d 17 85 01 4b c8 5a 5c   .C.5B.......K.Z\
    0080 - 6a b3 88 fe b4 14 e0 43-48 35 fa d5 f3 ae d5 3d   j......CH5.....=
    0090 - 5f 19 31 15 9a 8f b1 3f-e0 22 6c c2 0d be d2 eb   _.1....?."l.....
    00a0 - cb 40 ce 15 5c 82 d5 e8-dc ad 86 c8 58 77 00 93   .@..\.......Xw..
    00b0 - 38 82 35 a5 63 92 24 52-b1 7c 9d 45 90 2b 26 d3   8.5.c.$R.|.E.+&.
    00c0 - 32 2c c3 da 05 ec 8e 86-c2 4a a8 98 bf 62 8e 82   2,.......J...b..
    00d0 - e9 27 8f 17 70 3d 54 5a-57 86 02 b4 da 64 7d 58   .'..p=TZW....d}X

    Start Time: 1665055466
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

Might be that this:
verify error:num=20:unable to get local issuer certificate
is the problem?
However - this does not affect browsers and the connection is supposed to be secure…

curl executed as site user confirms that:

*   Trying xxx.178.46.xxx:443...
* TCP_NODELAY set
* Connected to www.xxx.rs (xxx.178.46.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=RS; ST=Beograd; L=Beograd; O=Preduzece za telekomunikacije; CN=*.xxx.rs
*  start date: Feb  2 07:21:07 2022 GMT
*  expire date: Mar  6 07:21:07 2023 GMT
*  subjectAltName: host "www.xxx.rs" matched cert's "*.xxx.rs"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
> HEAD /login HTTP/1.1
> Host: www.xxx.rs
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 4148
Content-Length: 4148
< Content-Type: text/html
Content-Type: text/html
< Last-Modified: Tue, 20 Sep 2022 20:12:15 GMT
Last-Modified: Tue, 20 Sep 2022 20:12:15 GMT
< Accept-Ranges: bytes
Accept-Ranges: bytes
< ETag: "3d5727472dcdd81:0"
ETag: "3d5727472dcdd81:0"
< Server: Microsoft-IIS/10.0
Server: Microsoft-IIS/10.0
< X-Powered-By: ASP.NET
X-Powered-By: ASP.NET
< Date: Thu, 06 Oct 2022 11:58:58 GMT
Date: Thu, 06 Oct 2022 11:58:58 GMT

<
* Connection #0 to host www.xxx.rs left intact
7

curl only checks the first certificate and not the complete chain.

Yes that’s the real problem.
I had a look at the openssl inside the site directory. It looks like there is a problem/miss-configuration with openssl.
If you do a “openssl version -a” you will get a strange “OPENSSLDIR”.
That’s the reason why it cannot find any certificates inside your site.

Quick solution for this problem

export SSL_CERT_FILE=/omd/sites/<sitename>/var/ssl/ca-certificates.crt
openssl s_client -connect www.myserver.com:443

also

~/lib/nagios/plugins/check_form_submit --port 443 --uri /login  www.myserver.com

should not show any problems now.
I would put this export inside the “.profile” of the site.

This is again something for @robin.gierse as it is a bug inside the OMD distribution and would explain some strange certificate problems inside a CMK site.

3 Likes
8

This should be fixed with this Werk: Add path truststore-path to environment

2 Likes