Error when updating from 2.0.0p25.cee to 2.1.0p1.cee

[mimimi (Y0d0g)]

Hello,

I am not sure if this is the right place to post this, please let me know if not.

I tried to update a prod clone from 2.0.0p25.cee to 2.1.0p1.cee but get an error:

2022-06-07 21:17:48 - Updating site 'prod' from version 2.0.0p25.cee to 2.1.0p1.cee...

 * Installed dir  var/log/agent-receiver
 * Unwanted       var/nagios/archive (unchanged, deleted by you)
 * Unwanted       local/share/check_mk/web/htdocs/themes (unchanged, deleted by you)
 * Identical new  etc/omd
 * Updated        etc/nagvis/apache.conf
 * Installed file etc/omd/allocated_ports
 * Updated        etc/apache/conf.d/omd.conf
 * Updated        etc/apache/conf.d/02_fcgid.conf
 * Updated        etc/apache/conf.d/01_wsgi.conf
 * Updated        etc/apache/conf.d/security.conf
 * Installed dir  etc/stunnel/conf.d
 * Updated        etc/stunnel/server.conf
 * Installed file etc/stunnel/conf.d/01-livestatus.conf
 * Updated        etc/nagios/nagios.d/misc.cfg
 * Updated        etc/nagios/conf.d/check_mk_templates.cfg
 * Installed file etc/mk-livestatus/livestatus.socket
 * Updated        etc/mk-livestatus/nagios.cfg
 * Installed file etc/mk-livestatus/livestatus@.service
 * Updated        etc/init.d/redis
 * Updated        etc/init.d/stunnel
 * Permissions    0755 -> 0775 etc/init.d/mkeventd
 * Installed file etc/init.d/agent-receiver
 * Updated        etc/init.d/cmc
 * Updated        etc/init.d/xinetd
 * Updated        etc/init.d/pnp_gearman_worker
 * Installed link etc/rc.d/10-agent-receiver
 * Installed file etc/logrotate.d/agent-receiver
 * Installed file etc/logrotate.d/agent-registration
 * Updated        etc/check_mk/apache.conf
 * Vanished       etc/jmx4perl/config/websphere/jms.cfg
 * Vanished       etc/jmx4perl/config/websphere/appstate.cfg
 * Vanished       etc/jmx4perl/config/websphere/http.cfg
 * Vanished       etc/jmx4perl/config/websphere/jdbc.cfg
 * Vanished       etc/jmx4perl/config/websphere/jca.cfg
 * Vanished       etc/jmx4perl/config/websphere/threads.cfg
 * Vanished       etc/jmx4perl/config/jboss.cfg
 * Vanished       etc/jmx4perl/config/glassfish.cfg
 * Vanished       etc/jmx4perl/config/common.cfg
 * Vanished       etc/jmx4perl/config/jetty.cfg
 * Vanished       etc/jmx4perl/config/tomcat.cfg
 * Vanished       etc/jmx4perl/config/weblogic.cfg
 * Vanished       etc/jmx4perl/config/metrics.cfg
 * Vanished       etc/jmx4perl/config/memory.cfg
 * Vanished       etc/jmx4perl/config/websphere.cfg
 * Vanished       etc/jmx4perl/config/jboss7.cfg
 * Vanished       etc/jmx4perl/config/threads.cfg
 * Vanished       etc/jmx4perl/config/websphere
 * Vanished       etc/jmx4perl/config
 * Vanished       etc/nagios/conf.d/jmx4perl_nagios.cfg
 * Vanished       etc/jmx4perl
 * Vanished       .j4p
Traceback (most recent call last):
  File "/omd/versions/2.1.0p1.cee/bin/omd", line 63, in <module>
    omdlib.main.main()
  File "/omd/versions/2.1.0p1.cee/lib/python3/omdlib/main.py", line 4666, in main
    command.handler(version_info, site, global_opts, args, command_options)
  File "/omd/versions/2.1.0p1.cee/lib/python3/omdlib/main.py", line 2816, in main_update
    initialize_site_ca(site)
  File "/omd/versions/2.1.0p1.cee/lib/python3/omdlib/main.py", line 1304, in initialize_site_ca
    root_ca=RootCA.load_or_create(root_cert_path(ca_path), f"Site '{site.name}' local CA"),
  File "/omd/versions/2.1.0p1.cee/lib/python3/cmk/utils/certs.py", line 52, in load_or_create
    return cls.load(path)
  File "/omd/versions/2.1.0p1.cee/lib/python3/cmk/utils/certs.py", line 47, in load
    return cls(*load_cert_and_private_key(path))
  File "/omd/versions/2.1.0p1.cee/lib/python3/cmk/utils/certs.py", line 101, in load_cert_and_private_key
    load_pem_private_key(
  File "/omd/sites/prod/local/lib/python3/cryptography/hazmat/primitives/serialization/base.py", line 20, in load_pem_private_key
    return backend.load_pem_private_key(data, password)
  File "/omd/sites/prod/local/lib/python3/cryptography/hazmat/backends/openssl/backend.py", line 1217, in load_pem_private_key
    return self._load_key(
  File "/omd/sites/prod/local/lib/python3/cryptography/hazmat/backends/openssl/backend.py", line 1448, in _load_key
    self._handle_key_loading_error()
  File "/omd/sites/prod/local/lib/python3/cryptography/hazmat/backends/openssl/backend.py", line 1490, in _handle_key_loading_error
    raise ValueError(
ValueError: Could not deserialize key data. The data may be in an incorrect format or it may be encrypted with an unsupported algorithm.

[fayepal (Faye)]

Hey @mimimi,
It would be better to create a separate thread. I’ll move your reply to a new topic.

[robin.gierse]

You have custom python modules installed in your local structure.
Removing them should fix your problem.

[mimimi (Y0d0g)]

Thanks @robin.gierse for pointing me into the right direction.

For some third party special agents we installed some python modules and their dependencies using pip3.
E.g. for andreas-doehlers redfish ilo monitoring and others.

I am now wondering how I can distinguish which python modules are shipped with vanilla checkmk and which are installed by us doe to third party checkmk packages.

There 130 python modules, which is a lot I think:

OMD[prod]:~$ pip3 list | wc -l
130

Perhaps I have to set up a vanilla, empty checkmk site to compare “pip3 list” with my production checkmk site ?

Or is it possible to limit pip3 to the ~/local structure somehow ?

[andreas-doehler (Andreas)]

If you have a problem with Python modules in the local folder. Remove all from there, do the upgrade and reinstalled after this. Only leave the folder “cmk”. This should not be removed.
I must say i had no problems with updates until now, also from 2.0 to 2.1 with installed local modules.

The error messages suggests also some other issues beside the local Python module.
A defect local CA certificate.