Hello,
Thanks for the detailed report — you’ve correctly diagnosed the root cause yourself.
Each Checkmk site bundles its own OpenSSL build under /omd/sites/<site>/lib, and the site environment adds that directory to LD_LIBRARY_PATH so Checkmk’s own binaries link against a fixed, version-pinned OpenSSL rather than whatever the OS ships.
On most distros this is harmless, but on a handful (RHEL/Rocky 9 and 10, SLES 15, Debian trixie among them) the OS’s own OpenSSL moves ahead faster than what is shipped with Checkmk. Some system tools invoked from within the site user’s shell, where LD_LIBRARY_PATH is set end up loading the site’s older bundled libcrypto.so.3 instead of the system one. They fail if they need symbols (like OPENSSL_3.4.0) that only exist in the newer system library.
We already work around this for a fixed list of commonly-used system executables (curl, ssh, scp, dig, nslookup, host, nc, nmap, systemctl, resolvectl, zypper, rpmbuild, php, php-cgi, pdftoppm) on the affected distros: we install a small wrapper under that name in the site’s bin/, which finds the real system binary further down $PATH, unsets LD_LIBRARY_PATH, and re-execs it — so it’s guaranteed to use the OS’s OpenSSL.
rpm and journalctl aren’t on that wrapped list yet. We’ll add both in an upcoming release 2.4.0p36.
In the meantime, your LD_PRELOAD=/lib64/libcrypto.so.3 <command> workaround is correct for ad-hoc admin commands. If you want more permanent fix inside your installation you can use the same wrapper trick we use for the executables above:
ln -s /omd/sites/mysite/bin/use_system_openssl /omd/sites/mysite/local/bin/rpm
ln -s /omd/sites/mysite/bin/use_system_openssl /omd/sites/mysite/local/bin/journalctl
Note that it needs to go into /omd/sites/mysite/local/bin to be site specific and to survive updates, not /omd/sites/mysite/bin.
Could you let us know if there are any other commands you’ve hit this with, or expect to hit, beyond rpm and journalctl? We’d like to fold the full list into the fix rather than doing this one command at a time.
Thanks again for the clear write-up!
Nigel