GUI crash after update to 2.4.0p1

CMK version: CRE 2.4.0p1
OS version: Ubuntu 24.04 LTS

Error message: [Errno 2] No such file or directory: ‘/omd/sites/monitoring/local/lib/check_mk/gui/plugins/views’

I updated my site as usual, but it had two issues:
First, something dashboard related:

-| Updating Checkmk configuration...
-|  01/30 User attributes...
-|  02/30 Create precompiled host and folder files...
-|  03/30 Update dashboards...
-|  + "Update dashboards" failed
-| Traceback (most recent call last):
-|   File "/omd/sites/monitoring/lib/python3/cmk/update_config/main.py", line 266, in update_config
-|     action(logger)
-|   File "/omd/sites/monitoring/lib/python3/cmk/update_config/plugins/actions/dashboards.py", line 27, in __call__
-|     dashboards = get_all_dashboards()
-|                  ^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/dashboard/store.py", line 95, in get_all_dashboards
-|     return DashboardStore.get_instance().all
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/hooks.py", line 135, in wrapper
-|     return cached_func(*args, **kwargs)  # type: ignore[arg-type]
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/dashboard/store.py", line 37, in get_instance
-|     return cls()
-|            ^^^^^
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/dashboard/store.py", line 40, in __init__
-|     self.all = self._load_all()
-|                ^^^^^^^^^^^^^^^^
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/dashboard/store.py", line 46, in _load_all
-|     return visuals.load(
-|            ^^^^^^^^^^^^^
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py", line 90, in load
-|     _CombinedVisualsCache[TVisual](what).load(
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py", line 152, in load
-|     return self._compute_and_write_cache(internal_to_runtime_transformer)
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py", line 182, in _compute_and_write_cache
-|     visuals = _load_custom_user_visuals(self._visual_type, internal_to_runtime_transformer)
-|               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py", line 243, in _load_custom_user_visuals
-|     _get_packaged_visuals(
-|   File "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py", line 290, in _get_packaged_visuals
-|     for dirpath in local_path.iterdir():
-|   File "/omd/sites/monitoring/lib/python3.12/pathlib.py", line 1056, in iterdir
-|     for name in os.listdir(self):
-|                 ^^^^^^^^^^^^^^^^
-| FileNotFoundError: [Errno 2] No such file or directory: '/omd/sites/monitoring/local/lib/check_mk/gui/plugins/dashboard'

Then an issue with (probably old) notification rules of a telegram plugin, but this is probably unrelated:

-|  29/30 Validating configuration files...
-|
-|        We have identified an issue with the configuration of your site.
-|
-|        Currently, this is a warning to make you aware of a potential problem.
-|        Our validation process checks your configuration files against a work-in-progress internal representation.
-|        In this case, we found at least one mismatch between the two.
-|
-|        For now you can proceed with the update of your site.
-|        However, in the future we will treat this as an error and stop the update procedure.
-|        To prevent any interruptions, we kindly ask you to notify us about this issue.
-|
-|        Please send us a support ticket if you believe there are no issues with your relevant configuration mk files.
-|        Be sure to include the name of the configuration file, the displayed error message and
-|        if possible the mk file itself.
-|        This information will help us investigate further and determine whether improvements are needed.
-|
-|        The following mk files had issues during the validation:
-|
-|          etc/check_mk/conf.d/wato/notifications.mk................... Failed
-|      2 config errors
-|      {'input': {'bot_token': '12345678:ABCDEFabcdef',
-|                 'chat_id': '123456789'},
-|       'loc': (6, 'notify_plugin', 1),
-|       'msg': 'Input should be a valid string',
-|       'type': 'string_type',
-|       'url': 'https://errors.pydantic.dev/2.10/v/string_type'}
-|
-|      {'input': {},
-|       'loc': (7, 'notify_plugin', 1),
-|       'msg': 'Input should be a valid string',
-|       'type': 'string_type',
-|       'url': 'https://errors.pydantic.dev/2.10/v/string_type'}
-|
-|
-|  30/30 Update core config...
-| Generating configuration for core (type nagios)...
-| Precompiling host checks...OK
-| Done (with errors)
ERROR (exit code: 1)

I tried starting nonetheless, and it looked good at first:

OMD[monitoring]:~$ cmk -R
Generating configuration for core (type nagios)...
Precompiling host checks...OK
Restarting monitoring core...OK
OMD[monitoring]:~$ omd start
Starting agent-receiver...OK
Starting mkeventd (builtin: syslog-udp)...OK
Starting rrdcached...OK
Starting redis...OK
Starting npcd...OK
Starting automation-helper...OK
Starting ui-job-scheduler...OK
Starting nagios...already running.
Starting apache...OK
Starting crontab...OK

I then refreshed the GUI, logged in but then got an error and the dashboard couldn’t be displayed:
image

The error is [Errno 2] No such file or directory: '/omd/sites/monitoring/local/lib/check_mk/gui/plugins/views'. I’ve checked and the plugins folder is empty. However, it was also empty in the backup of last night. I don’t quite know what could cause this.

I’m not sure if it’s a general issue or somehow related to my setup, so I decided to post this instead of restoring the backup and hoping next try it’ll be better…
Any ideas what I could try?

Here’s the crash.info if it helps anyone:

{
  "core": "nagios",
  "crash_type": "gui",
  "details": {
    "is_mobile": false,
    "is_ssl_request": false,
    "language": "en",
    "page": "index.py",
    "referer": "https://my-checkmk-domain/monitoring/check_mk/login.py?_origtarget=index.py",
    "request_method": "GET",
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
    "username": "myuser",
    "vars": {}
  },
  "edition": "cre",
  "exc_traceback": [
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/wsgi/applications/checkmk.py",
      195,
      "_process_request",
      "resp = page_handler()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/wsgi/applications/utils.py",
      135,
      "_call_auth",
      "func()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/pages.py",
      169,
      "wrapper",
      "return hc().handle_page()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/pages.py",
      53,
      "handle_page",
      "self.page()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/pages.py",
      147,
      "<lambda>",
      "\"page\": lambda self: self._wrapped_callable[0](),"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/main.py",
      34,
      "page_index",
      "SidebarRenderer().show(title, content)"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/sidebar/__init__.py",
      388,
      "show",
      "self._show_sidebar()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/sidebar/__init__.py",
      422,
      "_show_sidebar",
      "self._show_sidebar_head()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/sidebar/__init__.py",
      625,
      "_show_sidebar_head",
      "MainMenuRenderer().show()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/sidebar/main_menu.py",
      41,
      "show",
      "self._show_main_menu_content()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/sidebar/main_menu.py",
      45,
      "_show_main_menu_content",
      "for menu_item in self._get_main_menu_items():"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/sidebar/main_menu.py",
      85,
      "_get_main_menu_items",
      "if not menu.topics():"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/hooks.py",
      135,
      "wrapper",
      "return cached_func(*args, **kwargs)  # type: ignore[arg-type]"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/sidebar/_snapin/_views.py",
      86,
      "default_view_menu_topics",
      "return make_topic_menu(view_menu_items())"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/sidebar/_snapin/_views.py",
      110,
      "view_menu_items",
      "for name, view in get_permitted_views().items()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/views/store.py",
      87,
      "get_permitted_views",
      "return ViewStore.get_instance().permitted"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/hooks.py",
      135,
      "wrapper",
      "return cached_func(*args, **kwargs)  # type: ignore[arg-type]"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/views/store.py",
      57,
      "get_instance",
      "return cls()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/views/store.py",
      60,
      "__init__",
      "self.all: Final = ViewStore._load_all_views()"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/views/store.py",
      68,
      "_load_all_views",
      "for k, v in visuals.load("
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py",
      90,
      "load",
      "_CombinedVisualsCache[TVisual](what).load("
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py",
      152,
      "load",
      "return self._compute_and_write_cache(internal_to_runtime_transformer)"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py",
      182,
      "_compute_and_write_cache",
      "visuals = _load_custom_user_visuals(self._visual_type, internal_to_runtime_transformer)"
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py",
      243,
      "_load_custom_user_visuals",
      "_get_packaged_visuals("
    ],
    [
      "/omd/sites/monitoring/lib/python3/cmk/gui/visuals/_store.py",
      290,
      "_get_packaged_visuals",
      "for dirpath in local_path.iterdir():"
    ],
    [
      "/omd/sites/monitoring/lib/python3.12/pathlib.py",
      1056,
      "iterdir",
      "for name in os.listdir(self):"
    ]
  ],
  "exc_type": "FileNotFoundError",
  "exc_value": "[Errno 2] No such file or directory: '/omd/sites/monitoring/local/lib/check_mk/gui/plugins/views'",
  "id": "0ab2a08e-31be-11f0-bb71-bc2411e0664e",
  "local_vars": "redacted - not sure if sensitive things in there",
  "os": "Ubuntu 24.04.2 LTS",
  "python_paths": [
    "/omd/sites/monitoring/local/lib/python3",
    "/omd/sites/monitoring/lib/python3/cloud",
    "/omd/sites/monitoring/lib/python312.zip",
    "/omd/sites/monitoring/lib/python3.12",
    "/omd/sites/monitoring/lib/python3.12/lib-dynload",
    "/omd/sites/monitoring/lib/python3.12/site-packages",
    "/omd/sites/monitoring/lib/python3"
  ],
  "python_version": "3.12.9 (main, May  3 2025, 01:23:56) [GCC 13.2.0]",
  "time": 1747335274.1492572,
  "version": "2.4.0p1"
}

I restored from a snapshot and tried again, trying to find what causes the failure, and I found out that it has to do with an incompatible mkp (fritzbox_smarthome). Here’s the updater output:

2025-05-16 03:01:42 - Updating site 'monitoring' from version 2.4.0.cre to 2.4.0p1.cre...

Creating temporary filesystem /omd/sites/monitoring/tmp...OK
Executing 'cmk-update-config --conflict ask --dry-run'
-| ATTENTION
-|   Some steps may take a long time depending on your installation.
-|   Please be patient.
-|
-| Cleanup precompiled host and folder files
-| Verifying Checkmk configuration...
-|   Error in metrics plug-in 'fritzbox_smarthome'
-| Traceback (most recent call last):
-|   File "/omd/sites/monitoring/lib/python3/cmk/utils/plugin_loader.py", line 51, in load_plugins
-|     yield name, importlib.import_module(full_name)
-|                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/monitoring/lib/python3.12/importlib/__init__.py", line 90, in import_module
-|     return _bootstrap._gcd_import(name[level:], package, level)
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
-|   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
-|   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
-|   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
-|   File "<frozen importlib._bootstrap_external>", line 999, in exec_module
-|   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
-|   File "/omd/sites/monitoring/local/lib/python3/cmk/gui/plugins/metrics/fritzbox_smarthome.py", line 12, in <module>
-|     from cmk.gui.plugins.metrics.utils import (
-| ImportError: cannot import name 'metric_info' from 'cmk.gui.plugins.metrics.utils' (/omd/sites/monitoring/lib/python3/cmk/gui/plugins/metrics/utils.py)
-|  01/08 Legacy check plug-ins...
-|  02/08 Rulesets...
-|  03/08 UI extensions...
-| [fritzbox_smarthome 0.9.1-20240617]: Ignoring problems (MKP will be disabled on target version)
-|  04/08 Migrate Azure Databases...
-|  05/08 Agent based plugins...
-| [fritzbox_smarthome 0.9.1-20240617]: Ignoring problems (MKP will be disabled on target version)
-|  06/08 Autochecks...
-|  07/08 Invalid hosts labels...
-|  08/08 Deprecated .mk configuration of plugins...
-| Done (success)
-|

Completed verifying site configuration. Your site now has version 2.4.0p1.cre.
Executing update-pre-hooks script "01_mkp-disable-outdated"...
-| [proxmox_provisioned 1.3.1]: enabled
-| [sas2ircu 1.1]: enabled
-| [check_calculate 0.2.2]: disabled
-| [fritzbox_smarthome 0.9.1-20240617]: disabled
OK

Shortly after the Completed verifying site configuration. Your site now has version 2.4.0p1.cre. line appeared, but before the Executing update-pre-hooks script line appeared, the folder /opt/omd/sites/monitoring/local/lib/python3/cmk/gui/plugins was emptied. This is the folder that contains the missing views and dashboards folders.

This does not happen when I disable that mkp beforehand (which solves my problem).

The instance did complain that this plugin was incompatible before, but I had enabled it anyway - my fault, I guess. Still, I feel like this shouldn’t lead to an update rendering my instance inoperable, so I think there’s room for improvement :slight_smile:

I can provide a screen recording of the update process next to a watch -n 0.1 ls -la /opt/omd/sites/monitoring/local/lib/python3/cmk/gui/plugins where you can see when exactly it happened if necessary.

Hi @Jay2k1,

good that you were able to narrow it down.
I am with you there - even an incompatible MPK should not result in a hanging/crashing site.

Would you be able to create a crash report?

Sunny Greetings and thank you
Hartmut

Hi,

the very end of the first posting is the contents of the crash report file.

Because I was unable to get this back to a working state, after the bug report, I rolled back and hence no longer have the crash report in my instance. If you absolutely need me to, I can roll back again, enable the mkp in question and try the update, but maybe the contents of the crash.info will suffice? :slight_smile:

If you guys need to repro this, install this mkp and update the instance (actually, I used a more recent version than what is available in the exchange, you can get it here in case it makes a difference). I hope this will be enough to reproduce the issue.

Good Evening @Jay2k1,

thank you.

I think the crash information in the thread start and the reproduction steps should suffice for now.

Sunny Greetings
Hartmut

Hi @Jay2k1,

thanks for reporting this problem. It turned out, that simply disabling an MKP (without running an update) reproduced the problem. Fixes will become available with upcoming patch releases.

For details see: Werk #18211: Deinstallation of MKP causes GUI to crash

Have a nice day, Andi

3 Likes