Can't update to 2.3.0b3

Trying to update from 2.2.0p23cre tp 2.3.0b3.cre, getting the following error:

2024-03-20 16:12:37 - Updating site 'home' from version 2.2.0p23.cre to 2.3.0b3.cre...

 * Updated        etc/check_mk/apache.conf
 * Updated        etc/init.d/agent-receiver
 * Installed file etc/logrotate.d/audit
 * Installed file etc/logrotate.d/security
 * Updated        etc/mk-livestatus/nagios.cfg
 * Updated        etc/nagios/conf.d/check_mk_templates.cfg
 * Updated        etc/nagvis/apache.conf
 * Updated        etc/ssl/openssl.cnf
 * Updated        etc/ssl/openssl.cnf.dist
 * Updated        etc/ssl/misc/CA.pl
 * Identical      etc/ssl/misc/tsget
 * Updated        etc/ssl/misc/tsget.pl
 * Installed dir  local/lib/python3/cmk_addons
 * Installed dir  local/lib/python3/cmk/plugins
 * Installed dir  local/lib/python3/cmk_addons/plugins
 * Vanished       etc/cron.d/cmk_discovery
Creating temporary filesystem /omd/sites/home/tmp...OK
Traceback (most recent call last):
  File "/omd/sites/home/bin/cmk-update-config", line 8, in <module>
    from cmk.update_config.main import ensure_site_is_stopped, main
  File "/omd/sites/home/lib/python3/cmk/update_config/main.py", line 36, in <module>
    from cmk.gui import main_modules
  File "/omd/sites/home/lib/python3/cmk/gui/main_modules.py", line 44, in <module>
    import cmk.gui.cee.registration  # pylint: disable=no-name-in-module,cmk-module-layer-violation
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'cmk.gui.cee.registration'

I am not able to reproduce it.
Update from 2.2.0p23cre tp 2.3.0b3.cre or 2.2.0p24cre to 2.3.0b3.cre with one host registered via TLS works out of the box. No errors.

Can you tell if there are any local custom extensions or if this existing 2.2.0p23.cre was upgraded/downgraded ever to/from cee?

1 Like

I think it is possible that this error is triggered if you have an mkp installed that also has parts that are only cee usable.
Like deployment rules for agent plugins or so.

2 Likes

I just disabled all plugins to be sure, same error.

OMD[home]:~$ mkp list
Name                    Version         Title                        Author                                           Req. Version Until Version Files State
----------------------- --------------- ---------------------------- ------------------------------------------------ ------------ ------------- ----- --------
fail2ban                1.9.1           Monitor fail2ban jail status Jens Kuehnel                                     2.0.0        None          7     Disabled
snmp_uptime             20220306.v0.0.1 SNMP Uptime                  Th.L. (thl-cmk[at]outlook[dot]com)               2021.09.20   None          1     Disabled
active_check_traceroute 0.0.2-20230607  Check Traceroute (extended)  Th.L. (thl-cmk[at]outlook[dot]com)               2.1.0b1      None          4     Disabled
checkmk_update          0.1.1-20230519  Checkmk Update               Th.L. (thl-cmk[at]outlook[dot]com)               2.0.0b1      None          3     Disabled
sslcertificates         8.7.3           SSL-Certificates             Robert Sander <r.sander@heinlein-support.de>     2.0.0        None          6     Disabled
qemu                    2.0.2           Qemu / KVM                   Kuhn & Rueß GmbH                                 2.0.0        None          4     Disabled
access_logs             1.2.1           HTTP Access Logs             Notausstieg0309 <Notausstieg0309@googlemail.com> 2.0.0        None          8     Disabled

Don’t think there is anything else that specific about my installation. Never had CEE, always CRE. Is there a way to get more verbosity/debug for the update script?

Didn’t find anything on my server.

OMD[home]:~$ grep -sR "cmk.gui.cee.registration" .
OMD[home]:~$ grep -sR "cmk.gui.cee.registration" .

Is this still 2.2 ?

Also, what’s under ~/local/lib/python3/cmk/gui/ ?

1 Like

I figured out. Its this one that breaks your site: Checkmk Exchange

2024-03-21 14:34:14 - Updating site 'cre' from version 2.2.0p23.cre to 2.3.0b3.cre...

 * Installed dir  local/lib/python3/cmk_addons
 * Installed dir  local/lib/python3/cmk_addons/plugins
 * Installed dir  local/lib/python3/cmk/plugins
 * Updated        etc/mk-livestatus/nagios.cfg
 * Updated        etc/nagios/conf.d/check_mk_templates.cfg
 * Updated        etc/init.d/agent-receiver
 * Updated        etc/ssl/openssl.cnf
 * Updated        etc/ssl/openssl.cnf.dist
 * Updated        etc/ssl/misc/tsget.pl
 * Updated        etc/ssl/misc/CA.pl
 * Updated        etc/ssl/misc/tsget
 * Updated        etc/nagvis/apache.conf
 * Updated        etc/check_mk/apache.conf
 * Installed file etc/logrotate.d/audit
 * Installed file etc/logrotate.d/security
 * Vanished       etc/cron.d/cmk_discovery
Creating temporary filesystem /omd/sites/cre/tmp...OK
Traceback (most recent call last):
  File "/omd/sites/cre/bin/cmk-update-config", line 8, in <module>
    from cmk.update_config.main import ensure_site_is_stopped, main
  File "/omd/sites/cre/lib/python3/cmk/update_config/main.py", line 36, in <module>
    from cmk.gui import main_modules
  File "/omd/sites/cre/lib/python3/cmk/gui/main_modules.py", line 44, in <module>
    import cmk.gui.cee.registration  # pylint: disable=no-name-in-module,cmk-module-layer-violation
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'cmk.gui.cee.registration'
OMD[cre]:~/local/lib/python3/cmk/gui$ find
.
./cee
./cee/plugins
./cee/plugins/wato
./cee/plugins/wato/agent_bakery
./cee/plugins/wato/agent_bakery/access_logs.py
./plugins
./plugins/views
./plugins/dashboard
OMD[cre]:~/local/lib/python3/cmk/gui$
1 Like

I removed it and still got the error, so I removed every mkp one by one.

OMD[home]:~$ mkp list
Name Version Title Author Req. Version Until Version Files State
---- ------- ----- ------ ------------ ------------- ----- -----

OMD[home]:~$ omd update
Cleaning up tmp directory...OK
Cleaning up tmp directory...OK

2024-03-21 09:52:33 - Updating site 'home' from version 2.2.0p23.cre to 2.3.0b3.cre...

 * Updated        etc/check_mk/apache.conf
 * Updated        etc/init.d/agent-receiver
 * Installed file etc/logrotate.d/audit
 * Installed file etc/logrotate.d/security
 * Updated        etc/mk-livestatus/nagios.cfg
 * Updated        etc/nagios/conf.d/check_mk_templates.cfg
 * Updated        etc/nagvis/apache.conf
 * Updated        etc/ssl/openssl.cnf
 * Updated        etc/ssl/openssl.cnf.dist
 * Updated        etc/ssl/misc/CA.pl
 * Identical      etc/ssl/misc/tsget
 * Updated        etc/ssl/misc/tsget.pl
 * Installed dir  local/lib/python3/cmk_addons
 * Installed dir  local/lib/python3/cmk/plugins
 * Installed dir  local/lib/python3/cmk_addons/plugins
 * Vanished       etc/cron.d/cmk_discovery
Creating temporary filesystem /omd/sites/home/tmp...OK
Traceback (most recent call last):
  File "/omd/sites/home/bin/cmk-update-config", line 8, in <module>
    from cmk.update_config.main import ensure_site_is_stopped, main
  File "/omd/sites/home/lib/python3/cmk/update_config/main.py", line 36, in <module>
    from cmk.gui import main_modules
  File "/omd/sites/home/lib/python3/cmk/gui/main_modules.py", line 44, in <module>
    import cmk.gui.cee.registration  # pylint: disable=no-name-in-module,cmk-module-layer-violation
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'cmk.gui.cee.registration'

Same error, so not mkp related?

As for your other question:

OMD[home]:~$ find ~/local/lib/python3/cmk/gui/
/omd/sites/home/local/lib/python3/cmk/gui/
/omd/sites/home/local/lib/python3/cmk/gui/plugins
/omd/sites/home/local/lib/python3/cmk/gui/plugins/dashboard
/omd/sites/home/local/lib/python3/cmk/gui/plugins/views
/omd/sites/home/local/lib/python3/cmk/gui/plugins/metrics
/omd/sites/home/local/lib/python3/cmk/gui/plugins/metrics/__pycache__
/omd/sites/home/local/lib/python3/cmk/gui/plugins/metrics/__pycache__/traceroute.cpython-311.pyc
/omd/sites/home/local/lib/python3/cmk/gui/plugins/wato
/omd/sites/home/local/lib/python3/cmk/gui/plugins/wato/__pycache__
/omd/sites/home/local/lib/python3/cmk/gui/plugins/wato/__pycache__/active_checks_routing.cpython-311.pyc
/omd/sites/home/local/lib/python3/cmk/gui/cee
/omd/sites/home/local/lib/python3/cmk/gui/cee/plugins
/omd/sites/home/local/lib/python3/cmk/gui/cee/plugins/wato
/omd/sites/home/local/lib/python3/cmk/gui/cee/plugins/wato/agent_bakery

Can you keep the structure like this?

OMD[cre]:~/local/lib/python3$ find
.
./cmk
./cmk/base
./cmk/base/plugins
./cmk/base/plugins/agent_based
./cmk/special_agents
./cmk/gui
./cmk/gui/plugins
./cmk/gui/plugins/views
./cmk/gui/plugins/dashboard

Looks like enabling the MKP create a folder structure but removing the MKP doesn’t remove the folders it creates.

That worked. Removing the different cee directories worked.

With that being said, it’s not the first time I’m having issues updating because of MKPs. Do you think the updating process could be a little bit more resilient to this in the future?

I think the new agent api was developed to avoid exactly these problems:

With checkmk 2.4 only MKPs using the new APIs will be supported and hopefully all pain related to MKPs and updates will be history then :slight_smile:

1 Like

I do have a new issue starting the site however, will need to investigate further later today.

Starting nagios.../omd/sites/home/bin/nagios: error while loading shared libraries: libperl.so: cannot open shared object file: No such file or directory

Might be a missing dependancy somewhere or just permissions.

vm-admin:~ # locate libperl.so
/usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/CORE/libperl.so

Looked like it happened here as well: Error while loading shared libraries: libperl.so.5.26

What OS it is?

With that being said, it’s not the first time I’m having issues updating because of MKPs. Do you think the updating process could be a little bit more resilient to this in the future?

We will investigate here and let you know.

OpenSUSE 15.4 (binary-compatible with SLES15 SP4).

The libperl error is a typical problem with wrong package for your distribution.
Also if OpenSuse should be binary compatible, i would not use such a setup for any tests.

I don’t understand what you are saying here. It’s the exact same OS, except for the fact that SLES requires a paid registration that comes with support.

With the release of SLES 15 SP3 we now have 100% binary compatibility with openSUSE Leap 15.3 (our developer platform). That means that you can smoothly move workloads from development to production environments that run SLE 15 SP3 – and back again – with assured application compatibility.

Thanks, a similar solution actually works.

vm-admin:~ # ln -s /usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/CORE/libperl.so /usr/lib64/libperl.so
vm-admin:~ # omd start home nagios
Temporary filesystem already mounted
Starting nagios...OK
2 Likes

Thank you, very appreciated. I will happily test again with my broken plugins on 2.3.0b4. I noticed it does not only crash during the update process but also cause the same errors after the update if the packages are reinstalled/reenabled.

All in all it’s never too easy to update to a major or minor version (but that’s expected with betas). Patches are easy. But I got a lot of other problems and overcame most. I see some of them are already fixed on GitHub (such as the postfix plugin). Others I’m not sure if normal (for example SNMP uptimes had to be deleted and rediscovered to work for me, I see the plugin has been modified). Also got some other issues which I reported to the feedback email. I haven’t been able to make the new http plugin work but it looks promising, looking forward to make it work.

Now. This one got my head scratching, I checked the SUSE repos and no package provides libperl.so in the path Checkmk seems to be expecting it.

From our experience, custom extensions (via MKPs from the exchange, GitHub etc.) are the main reason for issues in the update process.
We have added many developer APIs in Checkmk 2.3, so that developers of such extensions can built their extensions on these APIs and users can then benefit from much higher stability during major updates as well.
We will also start cleaning up the exchange sooner or later as many plug-ins there are broken and not maintained anymore.

2 Likes