Webgui stopped working - Apache proxy error - undefined symbol: EVP_idea_cbc, version OPENSSL_3.0.0

Troubleshooting
1

CMK version: 2.5.0.p4 Community
OS version: Debian 13

Error message: Apache Proxy Error trying to access cmk site.

Hi,

Upgraded to 2.5.0 sometime last week including all remote sites and mkps. Worked perfectly, smooth operation since then.

Today master site suddenly cant be accessed anymore. OMD status is green and checks are running.
Tried restoring Backups from known working states, tried several Backups covering the last days, all experience the same error. Creating a new omd site on the host works.

The site includes a kerberos configuration for apache if relevant.

Single sign-on with Kerberos

Any ideas / suggestions would be highly appreciated, a bit lost atm.

From /var/log/apache/error.log

Wed Jun 10 10:57:12.882468 2026] [proxy_http:error] [pid 1436:tid 1436] [client 172.23.241.15:61926] AH01114: HTTP: failed to make connection to backend: 127.0.0.1, referer:

[Wed Jun 10 12:14:58.975315 2026] [proxy_http:error] [pid 5110:tid 5110] (70007)The timeout specified has expired: [client 172.23.241.15:58902] AH01102: error reading status line from remote server 127.0.0.1:5001,

Thanks!

edit:

took out the kerberos config, no effect.
tried upgrading 2.5.0p5, no change.
stopped, backuped, removed and restored site, no change.

edit2:

upgrading to 2.5.0p6 changes it from timeout to internal server error.
Apache error log

[Thu Jun 11 09:13:34.825390 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:50026] ImportError: /omd/sites/<replaced>/lib/python3.13/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: undefined symbol: EVP_idea_cbc, version OPENSSL_3.0.0
[Thu Jun 11 09:18:49.708767 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204] mod_wsgi (pid=46633): Failed to exec Python script file '/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/index.wsgi'.
[Thu Jun 11 09:18:49.708815 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204] mod_wsgi (pid=46633): Exception occurred processing WSGI script '/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/index.wsgi'.
[Thu Jun 11 09:18:49.709095 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204] Traceback (most recent call last):
[Thu Jun 11 09:18:49.709715 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/index.wsgi", line 13, in <module>
[Thu Jun 11 09:18:49.709726 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     from cmk.gui.wsgi.applications.profile_switcher import (
[Thu Jun 11 09:18:49.709730 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     ...<3 lines>...
[Thu Jun 11 09:18:49.709735 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     )
[Thu Jun 11 09:18:49.709744 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/__init__.py", line 6, in <module>
[Thu Jun 11 09:18:49.709749 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     from cmk.gui.wsgi.applications.checkmk import CheckmkApp
[Thu Jun 11 09:18:49.709758 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 28, in <module>
[Thu Jun 11 09:18:49.709763 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     from cmk.gui import pages, sites
[Thu Jun 11 09:18:49.709771 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/pages.py", line 16, in <module>
[Thu Jun 11 09:18:49.709776 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     from cmk.gui.config import Config
[Thu Jun 11 09:18:49.709784 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/config.py", line 28, in <module>
[Thu Jun 11 09:18:49.709790 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     from cmk.gui.keypair_store import KeypairStore
[Thu Jun 11 09:18:49.709800 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/keypair_store.py", line 12, in <module>
[Thu Jun 11 09:18:49.709806 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     from cmk.crypto.hash import HashAlgorithm
[Thu Jun 11 09:18:49.709815 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]   File "/omd/sites/<replaced>/lib/python3.13/site-packages/cmk/crypto/hash.py", line 12, in <module>
[Thu Jun 11 09:18:49.709833 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     from cryptography.hazmat.primitives import hashes as crypto_hashes
[Thu Jun 11 09:18:49.709844 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]   File "/omd/sites/<replaced>/lib/python3.13/site-packages/cryptography/hazmat/primitives/hashes.py", line 9, in <module>
[Thu Jun 11 09:18:49.709851 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204]     from cryptography.hazmat.bindings._rust import openssl as rust_openssl
[Thu Jun 11 09:18:49.709869 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58204] ImportError: /omd/sites/<replaced>/lib/python3.13/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: undefined symbol: EVP_idea_cbc, version OPENSSL_3.0.0
[Thu Jun 11 09:18:50.438348 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234] mod_wsgi (pid=46633): Failed to exec Python script file '/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/index.wsgi'.
[Thu Jun 11 09:18:50.438388 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234] mod_wsgi (pid=46633): Exception occurred processing WSGI script '/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/index.wsgi'.
[Thu Jun 11 09:18:50.438564 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234] Traceback (most recent call last):
[Thu Jun 11 09:18:50.439109 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/index.wsgi", line 13, in <module>
[Thu Jun 11 09:18:50.439119 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     from cmk.gui.wsgi.applications.profile_switcher import (
[Thu Jun 11 09:18:50.439122 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     ...<3 lines>...
[Thu Jun 11 09:18:50.439125 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     )
[Thu Jun 11 09:18:50.439132 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/__init__.py", line 6, in <module>
[Thu Jun 11 09:18:50.439135 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     from cmk.gui.wsgi.applications.checkmk import CheckmkApp
[Thu Jun 11 09:18:50.439141 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 28, in <module>
[Thu Jun 11 09:18:50.439144 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     from cmk.gui import pages, sites
[Thu Jun 11 09:18:50.439149 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/pages.py", line 16, in <module>
[Thu Jun 11 09:18:50.439152 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     from cmk.gui.config import Config
[Thu Jun 11 09:18:50.439157 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/config.py", line 28, in <module>
[Thu Jun 11 09:18:50.439161 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     from cmk.gui.keypair_store import KeypairStore
[Thu Jun 11 09:18:50.439166 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]   File "/omd/sites/<replaced>/lib/python3/cmk/gui/keypair_store.py", line 12, in <module>
[Thu Jun 11 09:18:50.439169 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     from cmk.crypto.hash import HashAlgorithm
[Thu Jun 11 09:18:50.439174 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]   File "/omd/sites/<replaced>/lib/python3.13/site-packages/cmk/crypto/hash.py", line 12, in <module>
[Thu Jun 11 09:18:50.439177 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     from cryptography.hazmat.primitives import hashes as crypto_hashes
[Thu Jun 11 09:18:50.439182 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]   File "/omd/sites/<replaced>/lib/python3.13/site-packages/cryptography/hazmat/primitives/hashes.py", line 9, in <module>
[Thu Jun 11 09:18:50.439196 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234]     from cryptography.hazmat.bindings._rust import openssl as rust_openssl
[Thu Jun 11 09:18:50.439208 2026] [wsgi:error] [pid 46633:tid 46633] [client 127.0.0.1:58234] ImportError: /omd/sites/<replaced>/lib/python3.13/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: undefined symbol: EVP_idea_cbc, version OPENSSL_3.0.0
1 Like
2

We’re getting the same error on both RHEL 9.8 and RHEL 10.2. My current suspicion is Werk #19573. We’re using mod_auth_openidc for single sign-on via OpenID Connect (OIDC) authentication.

3

got my deployment back working, stable for a couple of days now, so kinda optimistic.
Also noticeably better performance.

ofc i did way too many things in parallel to be sure what exactly it was.

  • removed kerberos config
  • checked, installed, fixed, rechecked pip3 libraries in site user (like pip3 check and so on) because i got some warnings running update-config
  • upgraded everything 2.5.0p4 → p6
  • threw a ton of hardware at the vms, brought every site (even those with 2 hosts checked) up to 8gb/4 core at the minimum. master at 64gb, which is overkill i think, but actually saw some spikes 30gb+ usage.
  • removed all persistent connections to remote sites, reduced all timeouts to 1s
  • tried modifying apache process count. Right now im back where i started at 128, not sure if thats usefull though.
  • removed one particular unstable remote site, that one kept crashing the entire ui if enabled. looking into that one in more detail today.
1 Like
4

We’re working on a hot-fix release where we revert #19573 to address this.

Though in the meanwhile this script will help unblock your site if you’ve already upgraded to 2.5.0p6. It patches the init.d script for Apache to use LD_PRELOAD to make sure that the OpenSSL packaged with Checkmk is used by Apache rather than the system OpenSSL.
patch-werk-19575-apache.sh (6.0 KB)

Cheers,
Nigel

1 Like