SSH broken as Check_MK site user

Troubleshooting
1

Hi,

I’m running in to the following issue. We’re running Check_MK 2.0.0p12 raw on SLES 15.2. Now I want to setup monitoring of a client via ssh. However, as the site user ssh is broken:

OMD[PWO]:~$ ssh <user>@<server>
ssh: relocation error: ssh: symbol EVP_KDF_CTX_free, version OPENSSL_1_1_1d not defined in file libcrypto.so.1.1 with link time reference

As a normal user this works as it should:

ssh <user>@<server>
Password:

Now, I have a very big suspicion that this is caused by the fact that CheckMK uses it’s own version of libcrypto that is incompatible with the installed version:

ldd /usr/bin/ssh
        linux-vdso.so.1 (0x00007ffd316f2000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f74b6f6f000)
        libcrypto.so.1.1 => /omd/sites/PWO/lib/libcrypto.so.1.1 (0x00007f74b6c85000)

Can I safely replace the Check_MK version with the system one? Or will I run into problems running Check_MK? It’s a production site, so I don’t want to exeriment too much.

2

An update, I realized, I also have a DEV server running. Replacing the Check_MK version of libcrypto.so.1.1 with the system version is not an option. Tried it, and when, for example, trying to check the services of a server it results in this:

Error running automation call try-inventory (exit code 1), error:

Traceback (most recent call last):
  File "/omd/sites/TWO/bin/check_mk", line 36, in 
    import cmk.base.config as config  # pylint: disable=cmk-module-layer-violation
  File "/omd/sites/TWO/lib/python3/cmk/base/config.py", line 59, in 
    import cmk.utils.version as cmk_version
  File "/omd/sites/TWO/lib/python3/cmk/utils/version.py", line 25, in 
    import livestatus
  File "/omd/sites/TWO/lib/python3/livestatus.py", line 13, in 
    import ssl
  File "/omd/sites/TWO/lib/python3.8/ssl.py", line 98, in 
    import _ssl             # if we can't import it, let the error propagate
ImportError: /omd/versions/2.0.0p12.cre/lib/libssl.so.1.1: symbol EVP_idea_cbc, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference
3

Not sure what you want to achieve, normally you use active check “Check via SSH service”. You need a key file for authentication without PW.

In case you want to do some scripting with site user:
In the omd user the LD_LIBRARY_PATH is set. Maybe try to unset before using ssh or point it to the right libraries.

I hope that helps

Michael

4

I want to monitor an IBM V7000 storage system. This type of check uses native ssh to connect to the device. The problem is that this will cause the discovery to fail:

image
image1552×152 5.49 KB

5

I use this check on CentOS 7 without any issues in 1.6.
Allow me some time to figure out what happened in 2.0 with the DS agent.

Anyway make sure that LD_LIBRARY_PATH in site user profile is set to:

LD_LIBRARY_PATH=/omd/sites//local/lib:/omd/sites//lib

BR

MF

6

The error message shows an incompatible OpenSSL version between system and CMK site. Inside the site you have OpenSSL 1.1.1k and outside in the system 1.1.1d.
Theres was in the spring a thread here in German with exactly the same problem and also with SLES 15 SP2.

7

Thanks for pointing that one out for me. Now from what I understood from the German, the original poster “solved” his issue by switching to Ubuntu. For us that’s not an option.

But in the thread something interesting was pointed out. And that is to use LD_PRELOAD=/usr/lib64/libcrypto.so.1.1 when running ssh as the site user.

So in /opt/omd/versions/2.0.0p12.cre/lib/python3/cmk/special_agents/agent_ibmsvc.py I made this change:

    #############################################################################
    # fetch information by ssh
    #############################################################################

    cmd = "LD_PRELOAD=/usr/lib64/libcrypto.so.1.1 ssh -o ConnectTimeout=%s %s %s@%s '" % (opt_timeout, opt_any_hostkey, user, host_address)

And that gets the IBM V7000 plugin working. It discovers all services and monitors them.

Normally I’d say case closed, but I believe this is something that should be fixed. Now I’ll have to manually edit this after every upgrade just to keep it working. Maybe as a workaround it’s possible to put an OS detection in the plugin that uses the LD_PRELOAD when running on SLES 15?

8

Hello,

Good that you found a solution for this issue and I agree that it should be fixed if SLES 15SP2 is supported.
Until its fixed you should be able to copy your changed code to local structure which prevents it from overwritten by an upgrade.

BR

Michael

9

Thanks, I’ll do that then. At least for now it’s working so that’s good.

10

Problem also in SLES 15 SP3 as of now, can’t ssh from site user. This also means that the integrated “check_by_ssh” plugin doesn’t work.

11

Problem also on SLES 15 SP4 as of now. Same issue, but different problem: can’t use the telegram notification as curl doesn’t work as site user.

12

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.