Check-mk-enterprise-2.4.0p34-el9-38.x86_64: /omd/sites/mysite/lib/libcrypto.so.3: version `OPENSSL_3.4.0' not found

Troubleshooting
, , , ,
1

CMK version: check-mk-enterprise-2.4.0p34-el9-38.x86_64
OS version: Rocky Linux release 9.8 (Blue Onyx)**
Error message:** /omd/sites/mysite/lib/libcrypto.so.3: version `OPENSSL_3.4.0’ not found

After upgrading to check-mk-enterprise-2.4.0p34-el9-38.x86_64 a number of commands stopped working in the checkmk user shell, examples:
$ rpm -q check-mk-enterprise
rpm: /omd/sites/mysite/lib/libcrypto.so.3: version `OPENSSL_3.4.0’ not found (required by /lib64/librpmio.so.9)
$ journalctl --user | head
journalctl: /omd/sites/mysite/lib/libcrypto.so.3: version `OPENSSL_3.4.0’ not found (required by /usr/lib64/systemd/libsystemd-shared-252.so)

Reason:
$ strings /omd/sites/mysite/lib/libcrypto.so.3 | egrep ‘^OPENSSL_3’
OPENSSL_3.0.0
OPENSSL_3.0.3
OPENSSL_3.0.8
OPENSSL_3.0.9
as opposed to:
$ strings /lib64/libcrypto.so.3 | egrep ^OPENSSL_3
OPENSSL_3.0.0
OPENSSL_3.0.1
OPENSSL_3.0.3
OPENSSL_3.0.8
OPENSSL_3.0.9
OPENSSL_3.1.0
OPENSSL_3.2.0
OPENSSL_3.3.0
OPENSSL_3.4.0
OPENSSL_3.5.0

It appears check-mk-enterprise-2.4.0p34-el9-38.x86_64 for EL9 packages at least a library that’s not compatible with up to date EL9.8. As a workaround one can use LD_PRELOAD:
$ LD_PRELOAD=/lib64/libcrypto.so.3 rpm -qa check-mk-enterprise*
check-mk-enterprise-2.1.0p49-el9-38.x86_64
check-mk-enterprise-2.4.0p34-el9-38.x86_64
check-mk-enterprise-2.3.0p48-el9-38.x86_64
check-mk-enterprise-2.2.0p47-el9-38.x86_64

2

Thanks for reporting. Forwarding to Development/CI.

3

Hello,

Thanks for the detailed report — you’ve correctly diagnosed the root cause yourself.

Each Checkmk site bundles its own OpenSSL build under /omd/sites/<site>/lib, and the site environment adds that directory to LD_LIBRARY_PATH so Checkmk’s own binaries link against a fixed, version-pinned OpenSSL rather than whatever the OS ships.

On most distros this is harmless, but on a handful (RHEL/Rocky 9 and 10, SLES 15, Debian trixie among them) the OS’s own OpenSSL moves ahead faster than what is shipped with Checkmk. Some system tools invoked from within the site user’s shell, where LD_LIBRARY_PATH is set end up loading the site’s older bundled libcrypto.so.3 instead of the system one. They fail if they need symbols (like OPENSSL_3.4.0) that only exist in the newer system library.

We already work around this for a fixed list of commonly-used system executables (curl, ssh, scp, dig, nslookup, host, nc, nmap, systemctl, resolvectl, zypper, rpmbuild, php, php-cgi, pdftoppm) on the affected distros: we install a small wrapper under that name in the site’s bin/, which finds the real system binary further down $PATH, unsets LD_LIBRARY_PATH, and re-execs it — so it’s guaranteed to use the OS’s OpenSSL.

rpm and journalctl aren’t on that wrapped list yet. We’ll add both in an upcoming release 2.4.0p36.

In the meantime, your LD_PRELOAD=/lib64/libcrypto.so.3 <command> workaround is correct for ad-hoc admin commands. If you want more permanent fix inside your installation you can use the same wrapper trick we use for the executables above:

ln -s /omd/sites/mysite/bin/use_system_openssl /omd/sites/mysite/local/bin/rpm
ln -s /omd/sites/mysite/bin/use_system_openssl /omd/sites/mysite/local/bin/journalctl

Note that it needs to go into /omd/sites/mysite/local/bin to be site specific and to survive updates, not /omd/sites/mysite/bin.

Could you let us know if there are any other commands you’ve hit this with, or expect to hit, beyond rpm and journalctl? We’d like to fold the full list into the fix rather than doing this one command at a time.

Thanks again for the clear write-up!

Nigel

4

Thanks @mschlenker and @nigel.banks for the very quick turnaround, good stuff!

Hard to tell at the moment what other commands will require wrappers installed without, say, scripting some sort of check that goes through the usual /usr/bin & similar directories to check the OpenSSL compatibility with CheckMK’s own version.

But your approach to provide own OpenSSL instead of relying on the OS one got me intrigued a bit :slight_smile:, I’m trying to balance the pros / cons of this approach and, if the wrapper approach turns out to be an uphill battle from an ops perspective, maybe there’s some merit in revisiting it.
Thanks again, looking forward to testing 2.4.0p36 once released.