Upgrade error from 2.1 to 2.2

CMK version: 2.1.0p36 cee → 2.2.0p17 cee
OS version: Debian 11

Hello everyone,

I have just tried to upgrade from checkmk cee 2.1.0p36 to checkmk cee 2.2.0p17 on a test copy and I get this error output, which I can’t really do anything with.
I am not aware of having done anything with the “agent_bakery_docker”.
What must/can I do to fix the error.

-|ATTENTION
-|   Some steps may take a long time depending on your installation.
-|   Please be patient.
-|
-| Verifying Checkmk configuration...
-|01/04 Rulesets...
-| Failed to load plugin /omd/sites/updatetest/local/share/check_mk/web/plugins/wato/agent_bakery_docker.py: name 'DropdownChoice' is not defined
-| Traceback (most recent call last):
-|   File "/omd/sites/updatetest/lib/python3/cmk/gui/utils/__init__.py", line 66, in load_web_plugins
-|     exec(f.read(), globalvars)
-|   File "<string>", line 32, in <module>
-| NameError: name 'DropdownChoice' is not defined
-| Failed to load plugin /omd/sites/updatetest/local/share/check_mk/web/plugins/wato/agent_bakery_docker.py: name 'DropdownChoice' is not defined
-| Traceback (most recent call last):
-|   File "/omd/sites/updatetest/lib/python3/cmk/gui/utils/__init__.py", line 66, in load_web_plugins
-|     exec(f.read(), globalvars)
-|   File "<string>", line 32, in <module>
-| NameError: name 'DropdownChoice' is not defined
-|02/04 UI extensions...
-| Failed to load plugin /omd/sites/updatetest/local/share/check_mk/web/plugins/wato/agent_bakery_docker.py: name 'DropdownChoice' is not defined
-| Traceback (most recent call last):
-|   File "/omd/sites/updatetest/lib/python3/cmk/gui/utils/__init__.py", line 66, in load_web_plugins
-|     exec(f.read(), globalvars)
-|   File "<string>", line 32, in <module>
-| NameError: name 'DropdownChoice' is not defined
-| Incompatible local file 'agent_bakery_docker'.
-| Error: name 'DropdownChoice' is not defined
-|
-| You can abort the update process (A) and try to fix the incompatibilities or continue the update (c).
-|
-| Abort the update process? [A/c]

Then somebody else has as there is a copy in the local hierarchy.
You may remove that file if it has no use any more.

1 Like

Ok, i check this.
Thanks

I have now fixed the 2-3 errors that occurred during the test upgrade run and it runs very well, but at the end when writing the config, an error occurs that cancels the upgrade.

Updating core configuration...
Generating configuration for core (type cmc)...
Configuration Error: Error creating configuration: 'cmk.influxdb'
Could not update core configuration. Aborting.

I have not yet done or configured anything with the influxdb integration.
What can I check/do to eliminate the problem?

Your upgrade is finished only the core config generation fails.
After this error you can do a “cmk --debug -vvU” to generate the core config with debug mode on.

What must i do, if i get this?

OMD[test3]:~$ cmk --debug -vvU
Trying to acquire lock on /omd/sites/test3/etc/check_mk/main.mk
Got lock on /omd/sites/test3/etc/check_mk/main.mk
Generating configuration for core (type cmc)...
Trying to acquire lock on /omd/sites/test3/var/check_mk/core/helper_config/serial.mk
Got lock on /omd/sites/test3/var/check_mk/core/helper_config/serial.mk
Releasing lock on /omd/sites/test3/var/check_mk/core/helper_config/serial.mk
Released lock on /omd/sites/test3/var/check_mk/core/helper_config/serial.mk
Trying to acquire lock on /omd/sites/test3/var/check_mk/licensing/licensed_state
Got lock on /omd/sites/test3/var/check_mk/licensing/licensed_state
Releasing lock on /omd/sites/test3/var/check_mk/licensing/licensed_state
Released lock on /omd/sites/test3/var/check_mk/licensing/licensed_state

Releasing lock on /omd/sites/test3/etc/check_mk/main.mk
Released lock on /omd/sites/test3/etc/check_mk/main.mk
Configuration Error: 'cmk.influxdb'
Trying to acquire lock on /omd/sites/test3/var/check_mk/crashes/base/30b2b2c4-accc-11ee-9c2d-2c768aacbc0c/crash.info
Got lock on /omd/sites/test3/var/check_mk/crashes/base/30b2b2c4-accc-11ee-9c2d-2c768aacbc0c/crash.info
Releasing lock on /omd/sites/test3/var/check_mk/crashes/base/30b2b2c4-accc-11ee-9c2d-2c768aacbc0c/crash.info
Released lock on /omd/sites/test3/var/check_mk/crashes/base/30b2b2c4-accc-11ee-9c2d-2c768aacbc0c/crash.info
Traceback (most recent call last):
  File "/omd/sites/test3/bin/cmk", line 112, in <module>
    exit_status = modes.call(mode_name, mode_args, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test3/lib/python3/cmk/base/modes/__init__.py", line 68, in call
    return handler(*handler_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test3/lib/python3/cmk/base/modes/check_mk.py", line 1223, in mode_update
    do_create_config(
  File "/omd/sites/test3/lib/python3/cmk/base/core_config.py", line 271, in do_create_config
    _create_core_config(
  File "/omd/sites/test3/lib/python3/cmk/base/core_config.py", line 359, in _create_core_config
    core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)
  File "/omd/sites/test3/lib/python3/cmk/base/core_config.py", line 79, in create_config
    self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)
  File "/omd/sites/test3/lib/python3/cmk/base/cee/microcore_config.py", line 3216, in _create_config
    data = create_core_config(
           ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test3/lib/python3/cmk/base/cee/microcore_config.py", line 3135, in create_core_config
    data["globals"] = _serialize_config_globals(string_pool, serial)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test3/lib/python3/cmk/base/cee/microcore_config.py", line 546, in _serialize_config_globals
    _fill_influx_db(gl, string_pool)
  File "/omd/sites/test3/lib/python3/cmk/base/cee/microcore_config.py", line 420, in _fill_influx_db
    influx_db.log_level = _to_log_level(base_config.cmc_log_levels["cmk.influxdb"])
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'cmk.influxdb'

and

cat /omd/sites/test3/var/check_mk/crashes/base/30b2b2c4-accc-11ee-9c2d-2c768aacbc0c/crash.info
{"time": 1704570348.0317671, "os": "Debian GNU/Linux 11 (bullseye)", "version": "2.2.0p17", "edition": "cee", "core": "cmc", "python_version": "3.11.5 (main, Dec  1 2023, 14:44:52) [GCC 13.2.0]", "python_paths": ["/opt/omd/versions/2.2.0p17.cee/bin", "/omd/sites/test3/local/lib/python3", "/omd/sites/test3/lib/python3/cloud", "/omd/sites/test3/lib/python311.zip", "/omd/sites/test3/lib/python3.11", "/omd/sites/test3/lib/python3.11/lib-dynload", "/omd/sites/test3/lib/python3.11/site-packages", "/omd/sites/test3/lib/python3"], "id": "30b2b2c4-accc-11ee-9c2d-2c768aacbc0c", "crash_type": "base", "exc_type": "KeyError", "exc_value": "cmk.influxdb", "exc_traceback": [["/omd/sites/test3/bin/cmk", 112, "<module>", "exit_status = modes.call(mode_name, mode_args, opts, args)"], ["/omd/sites/test3/lib/python3/cmk/base/modes/__init__.py", 68, "call", "return handler(*handler_args)"], ["/omd/sites/test3/lib/python3/cmk/base/modes/check_mk.py", 1223, "mode_update", "do_create_config("], ["/omd/sites/test3/lib/python3/cmk/base/core_config.py", 271, "do_create_config", "_create_core_config("], ["/omd/sites/test3/lib/python3/cmk/base/core_config.py", 359, "_create_core_config", "core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)"], ["/omd/sites/test3/lib/python3/cmk/base/core_config.py", 79, "create_config", "self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)"], ["/omd/sites/test3/lib/python3/cmk/base/cee/microcore_config.py", 3216, "_create_config", "data = create_core_config("], ["/omd/sites/test3/lib/python3/cmk/base/cee/microcore_config.py", 3135, "create_core_config", "data[\"globals\"] = _serialize_config_globals(string_pool, serial)"], ["/omd/sites/test3/lib/python3/cmk/base/cee/microcore_config.py", 546, "_serialize_config_globals", "_fill_influx_db(gl, string_pool)"], ["/omd/sites/test3/lib/python3/cmk/base/cee/microcore_config.py", 420, "_fill_influx_db", "influx_db.log_level = _to_log_level(base_config.cmc_log_levels[\"cmk.influxdb\"])"]], "local_vars": "eydnbCc6IGdlbmVyYWwgewogIHNlcmlhbDogMTIwNQogIGNvcmVfZGlyZWN0b3J5IHsKICB9CiAgbG9nX2ZpbGUgewogICAgc3RyX3JlZjogMQogIH0KICBjbWtfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAyCiAgfQogIGNyYXNoX3JlcG9ydHNfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDMKICB9CiAgbGljZW5zZV91c2FnZV9oaXN0b3J5X2ZpbGUgewogICAgc3RyX3JlZjogNAogIH0KICBpbnZlbnRvcnlfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDUKICB9CiAgc3RydWN0dXJlZF9zdGF0dXNfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDYKICB9CiAgcm9ib3Rta19odG1sX2xvZ19kaXJlY3RvcnkgewogICAgc3RyX3JlZjogNwogIH0KICBsb2d3YXRjaF9kaXJlY3RvcnkgewogICAgc3RyX3JlZjogOAogIH0KICBldmVudF9jb25zb2xlX3N0YXR1c19zb2NrZXQgewogICAgc3RyX3JlZjogOQogIH0KICBzdGF0ZV9maWxlX2NyZWF0ZWRfZmlsZSB7CiAgICBzdHJfcmVmOiAxMAogIH0KICBsaWNlbnNlZF9zdGF0ZV9maWxlIHsKICAgIHN0cl9yZWY6IDExCiAgfQogIGxvZ19sZXZlbF9jb3JlOiBMT0dfTEVWRUxfTk9USUNFCiAgbG9nX2xldmVsX2Rvd250aW1lOiBMT0dfTEVWRUxfTk9USUNFCiAgaG91c2VrZWVwaW5nX2ludGVydmFsIHsKICAgIHNlY29uZHM6IDEKICB9CiAgc3RhdGVfcmV0ZW50aW9uX2ludGVydmFsIHsKICAgIHNlY29uZHM6IDYwMAogIH0KICBldmVudF9jb25zb2xlX2VuYWJsZWQ6IHRydWUKICBlZGl0aW9uX3JlZjogMTIKICBtYXhfbG9uZ19vdXRwdXRfc2l6ZTogMjAwMAp9CnNtYXJ0X3BpbmcgewogIGNoZWNrX2ludGVydmFsIHsKICAgIG5hbm9zOiA1MDAwMDAwMDAKICB9CiAgbnVtX3NvY2tldHM6IDgKICB3YWl0X2R1cmF0aW9uIHsKICB9CiAgYnVsa19zaXplOiAxMDAwCiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfTk9USUNFCiAgaWNtcF9zZW5kZXJfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAxMwogIH0KICBpY21wX3JlY2VpdmVyX2V4ZWN1dGFibGUgewogICAgc3RyX3JlZjogMTQKICB9Cn0KY2hlY2tpbmcgewogIGdlbmVyaWNfaGVscGVyX2V4ZWN1dGFibGUgewogICAgc3RyX3JlZjogMTUKICB9CiAgbnVtX2dlbmVyaWNfaGVscGVyczogMTUKICBudW1fcmVhbF90aW1lX2hlbHBlcnM6IDYKICByZWFsX3RpbWVfcG9ydCB7CiAgfQogIGZldGNoZXJfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAxNgogIH0KICBudW1fZmV0Y2hlcnM6IDEzCiAgZmV0Y2hlcnNfZW5hYmxlZDogdHJ1ZQogIG51bV9jaGVja2VyczogMTAKICBsb2dfbGV2ZWw6IExPR19MRVZFTF9OT1RJQ0UKICBpbml0aWFsX3NjaGVkdWxpbmdfYnVyc3Q6IDEwCiAgaW5pdGlhbF9nZW5lcmljX3NjaGVkdWxpbmdfaW50ZXJ2YWwgewogICAgbmFub3M6IDE1MDAwMDAwMAogIH0KICBpbml0aWFsX2NoZWNrbWtfc2NoZWR1bGluZ19pbnRlcnZhbCB7CiAgICBzZWNvbmRzOiAxCiAgICBuYW5vczogMjAwMDAwMDAwCiAgfQp9CmxpdmVzdGF0dXMgewogIGRlbGF5IHsKICB9CiAgZGVsYXlfcmFuZG9taXplIHsKICB9CiAgc29ja2V0IHsKICAgIHN0cl9yZWY6IDE3CiAgfQogIGNvbW1hbmRfcGlwZSB7CiAgICBzdHJfcmVmOiAxOAogIH0KICBzdGFja19zaXplOiA0MTk0MzA0CiAgbnVtX3RocmVhZHM6IDIwCiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfREVCVUcKICBzZXJ2aWNlX2F1dGhvcml6YXRpb246IFNFUlZJQ0VfQVVUSE9SSVpBVElPTl9MT09TRQogIGdyb3VwX2F1dGhvcml6YXRpb246IEdST1VQX0FVVEhPUklaQVRJT05fTE9PU0UKfQpub3RpZmljYXRpb25faGFuZGxlciB7CiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfTk9USUNFCn0KYWxlcnRfaGFuZGxlciB7CiAgZXZlbnRfdHlwZXM6IDEKICBsb2dfbGV2ZWw6IExPR19MRVZFTF9OT1RJQ0UKfQpjYXJib24gewogIGxvZ19sZXZlbDogTE9HX0xFVkVMX05PVElDRQp9CiwKICdpbmZsdXhfZGInOiAsCiAnc3RyaW5nX3Bvb2wnOiA8Y21rLmJhc2UuY2VlLm1pY3JvY29yZV9jb25maWcuX1N0cmluZ1Bvb2wgb2JqZWN0IGF0IDB4N2ZkYTdmZGU3OGUwPn0=", "details": {"argv": ["/omd/sites/test3/bin/cmk", "--debug", "-vvU"], "env": {"SHELL": "/bin/bash", "OMD_ROOT": "/omd/sites/test3", "NAGIOS_PLUGIN_STATE_DIRECTORY": "/omd/sites/test3/var/monitoring-plugins", "PWD": "/omd/sites/test3", "LOGNAME": "test3", "MANPATH": "/omd/sites/test3/share/man:", "MODULEBUILDRC": "/omd/sites/test3/.modulebuildrc", "HOME": "/omd/sites/test3", "LANG": "C.UTF-8", "PERL5LIB": "/omd/sites/test3/local/lib/perl5/lib/perl5:/omd/sites/test3/lib/perl5/lib/perl5:", "OMD_SITE": "test3", "TERM": "xterm", "USER": "test3", "PERL_MM_OPT": "INSTALL_BASE=/omd/sites/test3/local/lib/perl5/", "SHLVL": "1", "LD_LIBRARY_PATH": "/omd/sites/test3/local/lib:/omd/sites/test3/lib", "SSL_CERT_FILE": "/omd/sites/test3/var/ssl/ca-certificates.crt", "REQUESTS_CA_BUNDLE": "/omd/sites/test3/var/ssl/ca-certificates.crt", "LC_ALL": "C.UTF-8", "PATH": "/omd/sites/test3/lib/perl5/bin:/omd/sites/test3/local/bin:/omd/sites/test3/bin:/omd/sites/test3/local/lib/perl5/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games", "MP_STATE_DIRECTORY": "/omd/sites/test3/var/monitoring-plugins", "MAIL": "/var/mail/test3", "MAILRC": "/omd/sites/test3/etc/mail.rc", "_": "/omd/sites/test3/bin/cmk"}}}

It looks like there is some old “influxdb” connector configuration existing inside your “~/etc/check_mk/conf.d/” folder.
You can grep over the config folder and try to find the configuration.

The only thing i found was this ~/etc/check_mk/conf.d/wato/influxdb_connections.mk

I tested an update attempt after deleting the file, but the result looked pretty much the same as above and it still gave the “KeyError: ‘cmk.influxdb’” message. :frowning:

I have make a new try from my backup.

OMD[test5]:~$ cmk --debug -vvU
Trying to acquire lock on /omd/sites/test5/etc/check_mk/main.mk
Got lock on /omd/sites/test5/etc/check_mk/main.mk
Generating configuration for core (type cmc)...
Trying to acquire lock on /omd/sites/test5/var/check_mk/core/helper_config/serial.mk
Got lock on /omd/sites/test5/var/check_mk/core/helper_config/serial.mk
Releasing lock on /omd/sites/test5/var/check_mk/core/helper_config/serial.mk
Released lock on /omd/sites/test5/var/check_mk/core/helper_config/serial.mk
Trying to acquire lock on /omd/sites/test5/var/check_mk/licensing/licensed_state
Got lock on /omd/sites/test5/var/check_mk/licensing/licensed_state
Releasing lock on /omd/sites/test5/var/check_mk/licensing/licensed_state
Released lock on /omd/sites/test5/var/check_mk/licensing/licensed_state

Releasing lock on /omd/sites/test5/etc/check_mk/main.mk
Released lock on /omd/sites/test5/etc/check_mk/main.mk
Configuration Error: 'cmk.influxdb'
Trying to acquire lock on /omd/sites/test5/var/check_mk/crashes/base/0549194e-ae38-11ee-a8ed-2c768aacbc0c/crash.info
Got lock on /omd/sites/test5/var/check_mk/crashes/base/0549194e-ae38-11ee-a8ed-2c768aacbc0c/crash.info
Releasing lock on /omd/sites/test5/var/check_mk/crashes/base/0549194e-ae38-11ee-a8ed-2c768aacbc0c/crash.info
Released lock on /omd/sites/test5/var/check_mk/crashes/base/0549194e-ae38-11ee-a8ed-2c768aacbc0c/crash.info
Traceback (most recent call last):
  File "/omd/sites/test5/bin/cmk", line 112, in <module>
    exit_status = modes.call(mode_name, mode_args, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test5/lib/python3/cmk/base/modes/__init__.py", line 68, in call
    return handler(*handler_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test5/lib/python3/cmk/base/modes/check_mk.py", line 1223, in mode_update
    do_create_config(
  File "/omd/sites/test5/lib/python3/cmk/base/core_config.py", line 271, in do_create_config
    _create_core_config(
  File "/omd/sites/test5/lib/python3/cmk/base/core_config.py", line 359, in _create_core_config
    core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)
  File "/omd/sites/test5/lib/python3/cmk/base/core_config.py", line 79, in create_config
    self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)
  File "/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", line 3216, in _create_config
    data = create_core_config(
           ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", line 3135, in create_core_config
    data["globals"] = _serialize_config_globals(string_pool, serial)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", line 546, in _serialize_config_globals
    _fill_influx_db(gl, string_pool)
  File "/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", line 420, in _fill_influx_db
    influx_db.log_level = _to_log_level(base_config.cmc_log_levels["cmk.influxdb"])
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'cmk.influxdb'

OMD[test5]:~$ cat /omd/sites/test5/var/check_mk/crashes/base/0549194e-ae38-11ee-a8ed-2c768aacbc0c/crash.info
{"time": 1704726612.0430887, "os": "Debian GNU/Linux 11 (bullseye)", "version": "2.2.0p17", "edition": "cee", "core": "cmc", "python_version": "3.11.5 (main, Dec  1 2023, 14:44:52) [GCC 13.2.0]", "python_paths": ["/opt/omd/versions/2.2.0p17.cee/bin", "/omd/sites/test5/local/lib/python3", "/omd/sites/test5/lib/python3/cloud", "/omd/sites/test5/lib/python311.zip", "/omd/sites/test5/lib/python3.11", "/omd/sites/test5/lib/python3.11/lib-dynload", "/omd/sites/test5/lib/python3.11/site-packages", "/omd/sites/test5/lib/python3"], "id": "0549194e-ae38-11ee-a8ed-2c768aacbc0c", "crash_type": "base", "exc_type": "KeyError", "exc_value": "cmk.influxdb", "exc_traceback": [["/omd/sites/test5/bin/cmk", 112, "<module>", "exit_status = modes.call(mode_name, mode_args, opts, args)"], ["/omd/sites/test5/lib/python3/cmk/base/modes/__init__.py", 68, "call", "return handler(*handler_args)"], ["/omd/sites/test5/lib/python3/cmk/base/modes/check_mk.py", 1223, "mode_update", "do_create_config("], ["/omd/sites/test5/lib/python3/cmk/base/core_config.py", 271, "do_create_config", "_create_core_config("], ["/omd/sites/test5/lib/python3/cmk/base/core_config.py", 359, "_create_core_config", "core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)"], ["/omd/sites/test5/lib/python3/cmk/base/core_config.py", 79, "create_config", "self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)"], ["/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", 3216, "_create_config", "data = create_core_config("], ["/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", 3135, "create_core_config", "data[\"globals\"] = _serialize_config_globals(string_pool, serial)"], ["/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", 546, "_serialize_config_globals", "_fill_influx_db(gl, string_pool)"], ["/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", 420, "_fill_influx_db", "influx_db.log_level = _to_log_level(base_config.cmc_log_levels[\"cmk.influxdb\"])"]], "local_vars": "eydnbCc6IGdlbmVyYWwgewogIHNlcmlhbDogMTIwNwogIGNvcmVfZGlyZWN0b3J5IHsKICB9CiAgbG9nX2ZpbGUgewogICAgc3RyX3JlZjogMQogIH0KICBjbWtfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAyCiAgfQogIGNyYXNoX3JlcG9ydHNfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDMKICB9CiAgbGljZW5zZV91c2FnZV9oaXN0b3J5X2ZpbGUgewogICAgc3RyX3JlZjogNAogIH0KICBpbnZlbnRvcnlfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDUKICB9CiAgc3RydWN0dXJlZF9zdGF0dXNfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDYKICB9CiAgcm9ib3Rta19odG1sX2xvZ19kaXJlY3RvcnkgewogICAgc3RyX3JlZjogNwogIH0KICBsb2d3YXRjaF9kaXJlY3RvcnkgewogICAgc3RyX3JlZjogOAogIH0KICBldmVudF9jb25zb2xlX3N0YXR1c19zb2NrZXQgewogICAgc3RyX3JlZjogOQogIH0KICBzdGF0ZV9maWxlX2NyZWF0ZWRfZmlsZSB7CiAgICBzdHJfcmVmOiAxMAogIH0KICBsaWNlbnNlZF9zdGF0ZV9maWxlIHsKICAgIHN0cl9yZWY6IDExCiAgfQogIGxvZ19sZXZlbF9jb3JlOiBMT0dfTEVWRUxfTk9USUNFCiAgbG9nX2xldmVsX2Rvd250aW1lOiBMT0dfTEVWRUxfTk9USUNFCiAgaG91c2VrZWVwaW5nX2ludGVydmFsIHsKICAgIHNlY29uZHM6IDEKICB9CiAgc3RhdGVfcmV0ZW50aW9uX2ludGVydmFsIHsKICAgIHNlY29uZHM6IDYwMAogIH0KICBldmVudF9jb25zb2xlX2VuYWJsZWQ6IHRydWUKICBlZGl0aW9uX3JlZjogMTIKICBtYXhfbG9uZ19vdXRwdXRfc2l6ZTogMjAwMAp9CnNtYXJ0X3BpbmcgewogIGNoZWNrX2ludGVydmFsIHsKICAgIG5hbm9zOiA1MDAwMDAwMDAKICB9CiAgbnVtX3NvY2tldHM6IDgKICB3YWl0X2R1cmF0aW9uIHsKICB9CiAgYnVsa19zaXplOiAxMDAwCiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfTk9USUNFCiAgaWNtcF9zZW5kZXJfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAxMwogIH0KICBpY21wX3JlY2VpdmVyX2V4ZWN1dGFibGUgewogICAgc3RyX3JlZjogMTQKICB9Cn0KY2hlY2tpbmcgewogIGdlbmVyaWNfaGVscGVyX2V4ZWN1dGFibGUgewogICAgc3RyX3JlZjogMTUKICB9CiAgbnVtX2dlbmVyaWNfaGVscGVyczogMTUKICBudW1fcmVhbF90aW1lX2hlbHBlcnM6IDYKICByZWFsX3RpbWVfcG9ydCB7CiAgfQogIGZldGNoZXJfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAxNgogIH0KICBudW1fZmV0Y2hlcnM6IDEzCiAgZmV0Y2hlcnNfZW5hYmxlZDogdHJ1ZQogIG51bV9jaGVja2VyczogMTAKICBsb2dfbGV2ZWw6IExPR19MRVZFTF9OT1RJQ0UKICBpbml0aWFsX3NjaGVkdWxpbmdfYnVyc3Q6IDEwCiAgaW5pdGlhbF9nZW5lcmljX3NjaGVkdWxpbmdfaW50ZXJ2YWwgewogICAgbmFub3M6IDE1MDAwMDAwMAogIH0KICBpbml0aWFsX2NoZWNrbWtfc2NoZWR1bGluZ19pbnRlcnZhbCB7CiAgICBzZWNvbmRzOiAxCiAgICBuYW5vczogMjAwMDAwMDAwCiAgfQp9CmxpdmVzdGF0dXMgewogIGRlbGF5IHsKICB9CiAgZGVsYXlfcmFuZG9taXplIHsKICB9CiAgc29ja2V0IHsKICAgIHN0cl9yZWY6IDE3CiAgfQogIGNvbW1hbmRfcGlwZSB7CiAgICBzdHJfcmVmOiAxOAogIH0KICBzdGFja19zaXplOiA0MTk0MzA0CiAgbnVtX3RocmVhZHM6IDIwCiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfREVCVUcKICBzZXJ2aWNlX2F1dGhvcml6YXRpb246IFNFUlZJQ0VfQVVUSE9SSVpBVElPTl9MT09TRQogIGdyb3VwX2F1dGhvcml6YXRpb246IEdST1VQX0FVVEhPUklaQVRJT05fTE9PU0UKfQpub3RpZmljYXRpb25faGFuZGxlciB7CiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfTk9USUNFCn0KYWxlcnRfaGFuZGxlciB7CiAgZXZlbnRfdHlwZXM6IDEKICBsb2dfbGV2ZWw6IExPR19MRVZFTF9OT1RJQ0UKfQpjYXJib24gewogIGxvZ19sZXZlbDogTE9HX0xFVkVMX05PVElDRQp9CiwKICdpbmZsdXhfZGInOiAsCiAnc3RyaW5nX3Bvb2wnOiA8Y21rLmJhc2UuY2VlLm1pY3JvY29yZV9jb25maWcuX1N0cmluZ1Bvb2wgb2JqZWN0IGF0IDB4N2YyMzhkZDQ1YWIwPn0=", "details": {"argv": ["/omd/sites/test5/bin/cmk", "--debug", "-vvU"], "env": {"SHELL": "/bin/bash", "OMD_ROOT": "/omd/sites/test5", "NAGIOS_PLUGIN_STATE_DIRECTORY": "/omd/sites/test5/var/monitoring-plugins", "PWD": "/omd/sites/test5", "LOGNAME": "test5", "MANPATH": "/omd/sites/test5/share/man:", "MODULEBUILDRC": "/omd/sites/test5/.modulebuildrc", "HOME": "/omd/sites/test5", "LANG": "C.UTF-8", "PERL5LIB": "/omd/sites/test5/local/lib/perl5/lib/perl5:/omd/sites/test5/lib/perl5/lib/perl5:", "OMD_SITE": "test5", "TERM": "xterm-256color", "USER": "test5", "PERL_MM_OPT": "INSTALL_BASE=/omd/sites/test5/local/lib/perl5/", "SHLVL": "1", "LD_LIBRARY_PATH": "/omd/sites/test5/local/lib:/omd/sites/test5/lib", "SSL_CERT_FILE": "/omd/sites/test5/var/ssl/ca-certificates.crt", "REQUESTS_CA_BUNDLE": "/omd/sites/test5/var/ssl/ca-certificates.crt", "LC_ALL": "C.UTF-8", "PATH": "/omd/sites/test5/lib/perl5/bin:/omd/sites/test5/local/bin:/omd/sites/test5/bin:/omd/sites/test5/local/lib/perl5/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games", "MP_STATE_DIRECTORY": "/omd/sites/test5/var/monitoring-plugins", "MAIL": "/var/mail/test5", "MAILRC": "/omd/sites/test5/etc/mail.rc", "_": "/omd/sites/test5/bin/cmk"}}}

The emcypted local_vars from the crash file are

{'gl': general {
  serial: 1207
  core_directory {
  }
  log_file {
    str_ref: 1
  }
  cmk_executable {
    str_ref: 2
  }
  crash_reports_directory {
    str_ref: 3
  }
  license_usage_history_file {
    str_ref: 4
  }
  inventory_directory {
    str_ref: 5
  }
  structured_status_directory {
    str_ref: 6
  }
  robotmk_html_log_directory {
    str_ref: 7
  }
  logwatch_directory {
    str_ref: 8
  }
  event_console_status_socket {
    str_ref: 9
  }
  state_file_created_file {
    str_ref: 10
  }
  licensed_state_file {
    str_ref: 11
  }
  log_level_core: LOG_LEVEL_NOTICE
  log_level_downtime: LOG_LEVEL_NOTICE
  housekeeping_interval {
    seconds: 1
  }
  state_retention_interval {
    seconds: 600
  }
  event_console_enabled: true
  edition_ref: 12
  max_long_output_size: 2000
}
smart_ping {
  check_interval {
    nanos: 500000000
  }
  num_sockets: 8
  wait_duration {
  }
  bulk_size: 1000
  log_level: LOG_LEVEL_NOTICE
  icmp_sender_executable {
    str_ref: 13
  }
  icmp_receiver_executable {
    str_ref: 14
  }
}
checking {
  generic_helper_executable {
    str_ref: 15
  }
  num_generic_helpers: 15
  num_real_time_helpers: 6
  real_time_port {
  }
  fetcher_executable {
    str_ref: 16
  }
  num_fetchers: 13
  fetchers_enabled: true
  num_checkers: 10
  log_level: LOG_LEVEL_NOTICE
  initial_scheduling_burst: 10
  initial_generic_scheduling_interval {
    nanos: 150000000
  }
  initial_checkmk_scheduling_interval {
    seconds: 1
    nanos: 200000000
  }
}
livestatus {
  delay {
  }
  delay_randomize {
  }
  socket {
    str_ref: 17
  }
  command_pipe {
    str_ref: 18
  }
  stack_size: 4194304
  num_threads: 20
  log_level: LOG_LEVEL_DEBUG
  service_authorization: SERVICE_AUTHORIZATION_LOOSE
  group_authorization: GROUP_AUTHORIZATION_LOOSE
}
notification_handler {
  log_level: LOG_LEVEL_NOTICE
}
alert_handler {
  event_types: 1
  log_level: LOG_LEVEL_NOTICE
}
carbon {
  log_level: LOG_LEVEL_NOTICE
}
,
 'influx_db': ,
 'string_pool': <cmk.base.cee.microcore_config._StringPool object at 0x7f238dd45ab0>}

I have search for influxdb at ~/etc/check_mk/conf.d. and it found two points.

OMD[test5]:~$ grep -R 'influxdb' ~/etc/check_mk/conf.d
/omd/sites/test5/etc/check_mk/conf.d/wato/influxdb_connections.mk:influxdb_connections.update({})
/omd/sites/test5/etc/check_mk/conf.d/wato/global.mk: 'cmk.influxdb': 5,

What are now the best steps to fix this upgrade issue?

I don’t know from where this setting comes. It should not be there.
Comment this line and try to generate the config with “cmk --debug -vvU”

I also don’t know from where this came.

I have comment out this line and run “cmk --debug -vvU”. And the result are the same fail.

OMD[test5]:~$ cmk --debug -vvU
Trying to acquire lock on /omd/sites/test5/etc/check_mk/main.mk
Got lock on /omd/sites/test5/etc/check_mk/main.mk
Generating configuration for core (type cmc)...
Trying to acquire lock on /omd/sites/test5/var/check_mk/core/helper_config/serial.mk
Got lock on /omd/sites/test5/var/check_mk/core/helper_config/serial.mk
Releasing lock on /omd/sites/test5/var/check_mk/core/helper_config/serial.mk
Released lock on /omd/sites/test5/var/check_mk/core/helper_config/serial.mk
Trying to acquire lock on /omd/sites/test5/var/check_mk/licensing/licensed_state
Got lock on /omd/sites/test5/var/check_mk/licensing/licensed_state
Releasing lock on /omd/sites/test5/var/check_mk/licensing/licensed_state
Released lock on /omd/sites/test5/var/check_mk/licensing/licensed_state

Releasing lock on /omd/sites/test5/etc/check_mk/main.mk
Released lock on /omd/sites/test5/etc/check_mk/main.mk
Configuration Error: 'cmk.influxdb'
Trying to acquire lock on /omd/sites/test5/var/check_mk/crashes/base/cafcc59e-ae49-11ee-8528-2c768aacbc0c/crash.info
Got lock on /omd/sites/test5/var/check_mk/crashes/base/cafcc59e-ae49-11ee-8528-2c768aacbc0c/crash.info
Releasing lock on /omd/sites/test5/var/check_mk/crashes/base/cafcc59e-ae49-11ee-8528-2c768aacbc0c/crash.info
Released lock on /omd/sites/test5/var/check_mk/crashes/base/cafcc59e-ae49-11ee-8528-2c768aacbc0c/crash.info
Traceback (most recent call last):
  File "/omd/sites/test5/bin/cmk", line 112, in <module>
    exit_status = modes.call(mode_name, mode_args, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test5/lib/python3/cmk/base/modes/__init__.py", line 68, in call
    return handler(*handler_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test5/lib/python3/cmk/base/modes/check_mk.py", line 1223, in mode_update
    do_create_config(
  File "/omd/sites/test5/lib/python3/cmk/base/core_config.py", line 271, in do_create_config
    _create_core_config(
  File "/omd/sites/test5/lib/python3/cmk/base/core_config.py", line 359, in _create_core_config
    core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)
  File "/omd/sites/test5/lib/python3/cmk/base/core_config.py", line 79, in create_config
    self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)
  File "/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", line 3216, in _create_config
    data = create_core_config(
           ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", line 3135, in create_core_config
    data["globals"] = _serialize_config_globals(string_pool, serial)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", line 546, in _serialize_config_globals
    _fill_influx_db(gl, string_pool)
  File "/omd/sites/test5/lib/python3/cmk/base/cee/microcore_config.py", line 420, in _fill_influx_db
    influx_db.log_level = _to_log_level(base_config.cmc_log_levels["cmk.influxdb"])
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'cmk.influxdb'

Before you update have a look at this setting here.


Set something for InfluxDB and then after the setting reset this to factory default.
After this it should be possible to upgrade.

I did this with a new copy of the original page.
A “grep -R ‘influxdb’ ~/etc/check_mk/conf.d” no longer finds anything in the files.

But the error is still the same. :frowning:

OMD[test7]:~$ cmk --debug -vvU
Trying to acquire lock on /omd/sites/test7/etc/check_mk/main.mk
Got lock on /omd/sites/test7/etc/check_mk/main.mk
Generating configuration for core (type cmc)...
Trying to acquire lock on /omd/sites/test7/var/check_mk/core/helper_config/serial.mk
Got lock on /omd/sites/test7/var/check_mk/core/helper_config/serial.mk
Releasing lock on /omd/sites/test7/var/check_mk/core/helper_config/serial.mk
Released lock on /omd/sites/test7/var/check_mk/core/helper_config/serial.mk
Trying to acquire lock on /omd/sites/test7/var/check_mk/licensing/licensed_state
Got lock on /omd/sites/test7/var/check_mk/licensing/licensed_state
Releasing lock on /omd/sites/test7/var/check_mk/licensing/licensed_state
Released lock on /omd/sites/test7/var/check_mk/licensing/licensed_state

Releasing lock on /omd/sites/test7/etc/check_mk/main.mk
Released lock on /omd/sites/test7/etc/check_mk/main.mk
Configuration Error: 'cmk.influxdb'
Trying to acquire lock on /omd/sites/test7/var/check_mk/crashes/base/790bc8fa-aebe-11ee-8d5f-2c768aacbc0c/crash.info
Got lock on /omd/sites/test7/var/check_mk/crashes/base/790bc8fa-aebe-11ee-8d5f-2c768aacbc0c/crash.info
Releasing lock on /omd/sites/test7/var/check_mk/crashes/base/790bc8fa-aebe-11ee-8d5f-2c768aacbc0c/crash.info
Released lock on /omd/sites/test7/var/check_mk/crashes/base/790bc8fa-aebe-11ee-8d5f-2c768aacbc0c/crash.info
Traceback (most recent call last):
  File "/omd/sites/test7/bin/cmk", line 112, in <module>
    exit_status = modes.call(mode_name, mode_args, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test7/lib/python3/cmk/base/modes/__init__.py", line 68, in call
    return handler(*handler_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test7/lib/python3/cmk/base/modes/check_mk.py", line 1223, in mode_update
    do_create_config(
  File "/omd/sites/test7/lib/python3/cmk/base/core_config.py", line 271, in do_create_config
    _create_core_config(
  File "/omd/sites/test7/lib/python3/cmk/base/core_config.py", line 359, in _create_core_config
    core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)
  File "/omd/sites/test7/lib/python3/cmk/base/core_config.py", line 79, in create_config
    self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)
  File "/omd/sites/test7/lib/python3/cmk/base/cee/microcore_config.py", line 3216, in _create_config
    data = create_core_config(
           ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test7/lib/python3/cmk/base/cee/microcore_config.py", line 3135, in create_core_config
    data["globals"] = _serialize_config_globals(string_pool, serial)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test7/lib/python3/cmk/base/cee/microcore_config.py", line 546, in _serialize_config_globals
    _fill_influx_db(gl, string_pool)
  File "/omd/sites/test7/lib/python3/cmk/base/cee/microcore_config.py", line 420, in _fill_influx_db
    influx_db.log_level = _to_log_level(base_config.cmc_log_levels["cmk.influxdb"])
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'cmk.influxdb'
OMD[test7]:~$ cat /omd/sites/test7/var/check_mk/crashes/base/790bc8fa-aebe-11ee-8d5f-2c768aacbc0c/crash.info
{"time": 1704784358.84961, "os": "Debian GNU/Linux 11 (bullseye)", "version": "2.2.0p17", "edition": "cee", "core": "cmc", "python_version": "3.11.5 (main, Dec  1 2023, 14:44:52) [GCC 13.2.0]", "python_paths": ["/opt/omd/versions/2.2.0p17.cee/bin", "/omd/sites/test7/local/lib/python3", "/omd/sites/test7/lib/python3/cloud", "/omd/sites/test7/lib/python311.zip", "/omd/sites/test7/lib/python3.11", "/omd/sites/test7/lib/python3.11/lib-dynload", "/omd/sites/test7/lib/python3.11/site-packages", "/omd/sites/test7/lib/python3"], "id": "790bc8fa-aebe-11ee-8d5f-2c768aacbc0c", "crash_type": "base", "exc_type": "KeyError", "exc_value": "cmk.influxdb", "exc_traceback": [["/omd/sites/test7/bin/cmk", 112, "<module>", "exit_status = modes.call(mode_name, mode_args, opts, args)"], ["/omd/sites/test7/lib/python3/cmk/base/modes/__init__.py", 68, "call", "return handler(*handler_args)"], ["/omd/sites/test7/lib/python3/cmk/base/modes/check_mk.py", 1223, "mode_update", "do_create_config("], ["/omd/sites/test7/lib/python3/cmk/base/core_config.py", 271, "do_create_config", "_create_core_config("], ["/omd/sites/test7/lib/python3/cmk/base/core_config.py", 359, "_create_core_config", "core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)"], ["/omd/sites/test7/lib/python3/cmk/base/core_config.py", 79, "create_config", "self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)"], ["/omd/sites/test7/lib/python3/cmk/base/cee/microcore_config.py", 3216, "_create_config", "data = create_core_config("], ["/omd/sites/test7/lib/python3/cmk/base/cee/microcore_config.py", 3135, "create_core_config", "data[\"globals\"] = _serialize_config_globals(string_pool, serial)"], ["/omd/sites/test7/lib/python3/cmk/base/cee/microcore_config.py", 546, "_serialize_config_globals", "_fill_influx_db(gl, string_pool)"], ["/omd/sites/test7/lib/python3/cmk/base/cee/microcore_config.py", 420, "_fill_influx_db", "influx_db.log_level = _to_log_level(base_config.cmc_log_levels[\"cmk.influxdb\"])"]], "local_vars": "eydnbCc6IGdlbmVyYWwgewogIHNlcmlhbDogMTIxMQogIGNvcmVfZGlyZWN0b3J5IHsKICB9CiAgbG9nX2ZpbGUgewogICAgc3RyX3JlZjogMQogIH0KICBjbWtfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAyCiAgfQogIGNyYXNoX3JlcG9ydHNfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDMKICB9CiAgbGljZW5zZV91c2FnZV9oaXN0b3J5X2ZpbGUgewogICAgc3RyX3JlZjogNAogIH0KICBpbnZlbnRvcnlfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDUKICB9CiAgc3RydWN0dXJlZF9zdGF0dXNfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDYKICB9CiAgcm9ib3Rta19odG1sX2xvZ19kaXJlY3RvcnkgewogICAgc3RyX3JlZjogNwogIH0KICBsb2d3YXRjaF9kaXJlY3RvcnkgewogICAgc3RyX3JlZjogOAogIH0KICBldmVudF9jb25zb2xlX3N0YXR1c19zb2NrZXQgewogICAgc3RyX3JlZjogOQogIH0KICBzdGF0ZV9maWxlX2NyZWF0ZWRfZmlsZSB7CiAgICBzdHJfcmVmOiAxMAogIH0KICBsaWNlbnNlZF9zdGF0ZV9maWxlIHsKICAgIHN0cl9yZWY6IDExCiAgfQogIGxvZ19sZXZlbF9jb3JlOiBMT0dfTEVWRUxfTk9USUNFCiAgbG9nX2xldmVsX2Rvd250aW1lOiBMT0dfTEVWRUxfTk9USUNFCiAgaG91c2VrZWVwaW5nX2ludGVydmFsIHsKICAgIHNlY29uZHM6IDEKICB9CiAgc3RhdGVfcmV0ZW50aW9uX2ludGVydmFsIHsKICAgIHNlY29uZHM6IDYwMAogIH0KICBldmVudF9jb25zb2xlX2VuYWJsZWQ6IHRydWUKICBlZGl0aW9uX3JlZjogMTIKICBtYXhfbG9uZ19vdXRwdXRfc2l6ZTogMjAwMAp9CnNtYXJ0X3BpbmcgewogIGNoZWNrX2ludGVydmFsIHsKICAgIG5hbm9zOiA1MDAwMDAwMDAKICB9CiAgbnVtX3NvY2tldHM6IDgKICB3YWl0X2R1cmF0aW9uIHsKICB9CiAgYnVsa19zaXplOiAxMDAwCiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfTk9USUNFCiAgaWNtcF9zZW5kZXJfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAxMwogIH0KICBpY21wX3JlY2VpdmVyX2V4ZWN1dGFibGUgewogICAgc3RyX3JlZjogMTQKICB9Cn0KY2hlY2tpbmcgewogIGdlbmVyaWNfaGVscGVyX2V4ZWN1dGFibGUgewogICAgc3RyX3JlZjogMTUKICB9CiAgbnVtX2dlbmVyaWNfaGVscGVyczogMTUKICBudW1fcmVhbF90aW1lX2hlbHBlcnM6IDYKICByZWFsX3RpbWVfcG9ydCB7CiAgfQogIGZldGNoZXJfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAxNgogIH0KICBudW1fZmV0Y2hlcnM6IDEzCiAgZmV0Y2hlcnNfZW5hYmxlZDogdHJ1ZQogIG51bV9jaGVja2VyczogMTAKICBsb2dfbGV2ZWw6IExPR19MRVZFTF9OT1RJQ0UKICBpbml0aWFsX3NjaGVkdWxpbmdfYnVyc3Q6IDEwCiAgaW5pdGlhbF9nZW5lcmljX3NjaGVkdWxpbmdfaW50ZXJ2YWwgewogICAgbmFub3M6IDE1MDAwMDAwMAogIH0KICBpbml0aWFsX2NoZWNrbWtfc2NoZWR1bGluZ19pbnRlcnZhbCB7CiAgICBzZWNvbmRzOiAxCiAgICBuYW5vczogMjAwMDAwMDAwCiAgfQp9CmxpdmVzdGF0dXMgewogIGRlbGF5IHsKICB9CiAgZGVsYXlfcmFuZG9taXplIHsKICB9CiAgc29ja2V0IHsKICAgIHN0cl9yZWY6IDE3CiAgfQogIGNvbW1hbmRfcGlwZSB7CiAgICBzdHJfcmVmOiAxOAogIH0KICBzdGFja19zaXplOiA0MTk0MzA0CiAgbnVtX3RocmVhZHM6IDIwCiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfREVCVUcKICBzZXJ2aWNlX2F1dGhvcml6YXRpb246IFNFUlZJQ0VfQVVUSE9SSVpBVElPTl9MT09TRQogIGdyb3VwX2F1dGhvcml6YXRpb246IEdST1VQX0FVVEhPUklaQVRJT05fTE9PU0UKfQpub3RpZmljYXRpb25faGFuZGxlciB7CiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfTk9USUNFCn0KYWxlcnRfaGFuZGxlciB7CiAgZXZlbnRfdHlwZXM6IDEKICBsb2dfbGV2ZWw6IExPR19MRVZFTF9OT1RJQ0UKfQpjYXJib24gewogIGxvZ19sZXZlbDogTE9HX0xFVkVMX05PVElDRQp9CiwKICdpbmZsdXhfZGInOiAsCiAnc3RyaW5nX3Bvb2wnOiA8Y21rLmJhc2UuY2VlLm1pY3JvY29yZV9jb25maWcuX1N0cmluZ1Bvb2wgb2JqZWN0IGF0IDB4N2ZmMmQwYzU0MmIwPn0=", "details": {"argv": ["/omd/sites/test7/bin/cmk", "--debug", "-vvU"], "env": {"SHELL": "/bin/bash", "OMD_ROOT": "/omd/sites/test7", "NAGIOS_PLUGIN_STATE_DIRECTORY": "/omd/sites/test7/var/monitoring-plugins", "PWD": "/omd/sites/test7", "LOGNAME": "test7", "MANPATH": "/omd/sites/test7/share/man:", "MODULEBUILDRC": "/omd/sites/test7/.modulebuildrc", "HOME": "/omd/sites/test7", "LANG": "C.UTF-8", "PERL5LIB": "/omd/sites/test7/local/lib/perl5/lib/perl5:/omd/sites/test7/lib/perl5/lib/perl5:", "OMD_SITE": "test7", "TERM": "xterm-256color", "USER": "test7", "PERL_MM_OPT": "INSTALL_BASE=/omd/sites/test7/local/lib/perl5/", "SHLVL": "1", "LD_LIBRARY_PATH": "/omd/sites/test7/local/lib:/omd/sites/test7/lib", "SSL_CERT_FILE": "/omd/sites/test7/var/ssl/ca-certificates.crt", "REQUESTS_CA_BUNDLE": "/omd/sites/test7/var/ssl/ca-certificates.crt", "LC_ALL": "C.UTF-8", "PATH": "/omd/sites/test7/lib/perl5/bin:/omd/sites/test7/local/bin:/omd/sites/test7/bin:/omd/sites/test7/local/lib/perl5/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games", "MP_STATE_DIRECTORY": "/omd/sites/test7/var/monitoring-plugins", "MAIL": "/var/mail/test7", "MAILRC": "/omd/sites/test7/etc/mail.rc", "_": "/omd/sites/test7/bin/cmk"}}}

@andreas-doehler do you have any more ideas about what else could be done?

I have now also tried the whole thing to the effect that I have set something for everything that has to do with Influxdb.
Influxdb Connection and Send metrics to InfluxDB created but rule set to inactive.

But even then the upgrade does not work and it is canceled with the error message:

Configuration Error: Error creating configuration: 'cmk.influxdb'
Could not update core configuration. Aborting.

Are you sure you commented out cmk.influxdb in global.mkproperly?
Maybe try to delete it, as the configuration generator apparently still sees it.

I have now remove all influxdb settings over wato, the loglevel all set to default
A “grep -R ‘influxdb’ ~/etc/check_mk/conf.d” no longer finds anything in the files.
The global.mk are clean and minimal.

But the upgrade still fail with the cmk.influxdb creation error.

cmk --debug -vvU result are this:

OMD[test15]:~$ cmk --debug -vvU
Trying to acquire lock on /omd/sites/test15/etc/check_mk/main.mk
Got lock on /omd/sites/test15/etc/check_mk/main.mk
Generating configuration for core (type cmc)...
Trying to acquire lock on /omd/sites/test15/var/check_mk/core/helper_config/serial.mk
Got lock on /omd/sites/test15/var/check_mk/core/helper_config/serial.mk
Releasing lock on /omd/sites/test15/var/check_mk/core/helper_config/serial.mk
Released lock on /omd/sites/test15/var/check_mk/core/helper_config/serial.mk
Trying to acquire lock on /omd/sites/test15/var/check_mk/licensing/licensed_state
Got lock on /omd/sites/test15/var/check_mk/licensing/licensed_state
Releasing lock on /omd/sites/test15/var/check_mk/licensing/licensed_state
Released lock on /omd/sites/test15/var/check_mk/licensing/licensed_state

Releasing lock on /omd/sites/test15/etc/check_mk/main.mk
Released lock on /omd/sites/test15/etc/check_mk/main.mk
Configuration Error: 'cmk.influxdb'
Trying to acquire lock on /omd/sites/test15/var/check_mk/crashes/base/542981de-b0b3-11ee-b417-2c768aacbc0c/crash.info
Got lock on /omd/sites/test15/var/check_mk/crashes/base/542981de-b0b3-11ee-b417-2c768aacbc0c/crash.info
Releasing lock on /omd/sites/test15/var/check_mk/crashes/base/542981de-b0b3-11ee-b417-2c768aacbc0c/crash.info
Released lock on /omd/sites/test15/var/check_mk/crashes/base/542981de-b0b3-11ee-b417-2c768aacbc0c/crash.info
Traceback (most recent call last):
  File "/omd/sites/test15/bin/cmk", line 112, in <module>
    exit_status = modes.call(mode_name, mode_args, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test15/lib/python3/cmk/base/modes/__init__.py", line 68, in call
    return handler(*handler_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test15/lib/python3/cmk/base/modes/check_mk.py", line 1223, in mode_update
    do_create_config(
  File "/omd/sites/test15/lib/python3/cmk/base/core_config.py", line 271, in do_create_config
    _create_core_config(
  File "/omd/sites/test15/lib/python3/cmk/base/core_config.py", line 359, in _create_core_config
    core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)
  File "/omd/sites/test15/lib/python3/cmk/base/core_config.py", line 79, in create_config
    self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)
  File "/omd/sites/test15/lib/python3/cmk/base/cee/microcore_config.py", line 3216, in _create_config
    data = create_core_config(
           ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test15/lib/python3/cmk/base/cee/microcore_config.py", line 3135, in create_core_config
    data["globals"] = _serialize_config_globals(string_pool, serial)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/test15/lib/python3/cmk/base/cee/microcore_config.py", line 546, in _serialize_config_globals
    _fill_influx_db(gl, string_pool)
  File "/omd/sites/test15/lib/python3/cmk/base/cee/microcore_config.py", line 420, in _fill_influx_db
    influx_db.log_level = _to_log_level(base_config.cmc_log_levels["cmk.influxdb"])
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'cmk.influxdb'
OMD[test15]:~$ ^C
OMD[test15]:~$ cat /omd/sites/test15/var/check_mk/crashes/base/542981de-b0b3-11ee-b417-2c768aacbc0c/crash.info
{"time": 1704999474.814274, "os": "Debian GNU/Linux 11 (bullseye)", "version": "2.2.0p17", "edition": "cee", "core": "cmc", "python_version": "3.11.5 (main, Dec  1 2023, 14:44:52) [GCC 13.2.0]", "python_paths": ["/opt/omd/versions/2.2.0p17.cee/bin", "/omd/sites/test15/local/lib/python3", "/omd/sites/test15/lib/python3/cloud", "/omd/sites/test15/lib/python311.zip", "/omd/sites/test15/lib/python3.11", "/omd/sites/test15/lib/python3.11/lib-dynload", "/omd/sites/test15/lib/python3.11/site-packages", "/omd/sites/test15/lib/python3"], "id": "542981de-b0b3-11ee-b417-2c768aacbc0c", "crash_type": "base", "exc_type": "KeyError", "exc_value": "cmk.influxdb", "exc_traceback": [["/omd/sites/test15/bin/cmk", 112, "<module>", "exit_status = modes.call(mode_name, mode_args, opts, args)"], ["/omd/sites/test15/lib/python3/cmk/base/modes/__init__.py", 68, "call", "return handler(*handler_args)"], ["/omd/sites/test15/lib/python3/cmk/base/modes/check_mk.py", 1223, "mode_update", "do_create_config("], ["/omd/sites/test15/lib/python3/cmk/base/core_config.py", 271, "do_create_config", "_create_core_config("], ["/omd/sites/test15/lib/python3/cmk/base/core_config.py", 359, "_create_core_config", "core.create_config(config_path, config_cache, hosts_to_update=hosts_to_update)"], ["/omd/sites/test15/lib/python3/cmk/base/core_config.py", 79, "create_config", "self._create_config(config_path, config_cache, licensing_handler, hosts_to_update)"], ["/omd/sites/test15/lib/python3/cmk/base/cee/microcore_config.py", 3216, "_create_config", "data = create_core_config("], ["/omd/sites/test15/lib/python3/cmk/base/cee/microcore_config.py", 3135, "create_core_config", "data[\"globals\"] = _serialize_config_globals(string_pool, serial)"], ["/omd/sites/test15/lib/python3/cmk/base/cee/microcore_config.py", 546, "_serialize_config_globals", "_fill_influx_db(gl, string_pool)"], ["/omd/sites/test15/lib/python3/cmk/base/cee/microcore_config.py", 420, "_fill_influx_db", "influx_db.log_level = _to_log_level(base_config.cmc_log_levels[\"cmk.influxdb\"])"]], "local_vars": "eydnbCc6IGdlbmVyYWwgewogIHNlcmlhbDogMTI1MgogIGNvcmVfZGlyZWN0b3J5IHsKICB9CiAgbG9nX2ZpbGUgewogICAgc3RyX3JlZjogMQogIH0KICBjbWtfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAyCiAgfQogIGNyYXNoX3JlcG9ydHNfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDMKICB9CiAgbGljZW5zZV91c2FnZV9oaXN0b3J5X2ZpbGUgewogICAgc3RyX3JlZjogNAogIH0KICBpbnZlbnRvcnlfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDUKICB9CiAgc3RydWN0dXJlZF9zdGF0dXNfZGlyZWN0b3J5IHsKICAgIHN0cl9yZWY6IDYKICB9CiAgcm9ib3Rta19odG1sX2xvZ19kaXJlY3RvcnkgewogICAgc3RyX3JlZjogNwogIH0KICBsb2d3YXRjaF9kaXJlY3RvcnkgewogICAgc3RyX3JlZjogOAogIH0KICBldmVudF9jb25zb2xlX3N0YXR1c19zb2NrZXQgewogICAgc3RyX3JlZjogOQogIH0KICBzdGF0ZV9maWxlX2NyZWF0ZWRfZmlsZSB7CiAgICBzdHJfcmVmOiAxMAogIH0KICBsaWNlbnNlZF9zdGF0ZV9maWxlIHsKICAgIHN0cl9yZWY6IDExCiAgfQogIGxvZ19sZXZlbF9jb3JlOiBMT0dfTEVWRUxfTk9USUNFCiAgbG9nX2xldmVsX2Rvd250aW1lOiBMT0dfTEVWRUxfTk9USUNFCiAgaG91c2VrZWVwaW5nX2ludGVydmFsIHsKICAgIHNlY29uZHM6IDEKICB9CiAgc3RhdGVfcmV0ZW50aW9uX2ludGVydmFsIHsKICAgIHNlY29uZHM6IDYwMAogIH0KICBldmVudF9jb25zb2xlX2VuYWJsZWQ6IHRydWUKICBlZGl0aW9uX3JlZjogMTIKICBtYXhfbG9uZ19vdXRwdXRfc2l6ZTogMjAwMAp9CnNtYXJ0X3BpbmcgewogIGNoZWNrX2ludGVydmFsIHsKICAgIG5hbm9zOiA1MDAwMDAwMDAKICB9CiAgbnVtX3NvY2tldHM6IDgKICB3YWl0X2R1cmF0aW9uIHsKICB9CiAgYnVsa19zaXplOiAxMDAwCiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfTk9USUNFCiAgaWNtcF9zZW5kZXJfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAxMwogIH0KICBpY21wX3JlY2VpdmVyX2V4ZWN1dGFibGUgewogICAgc3RyX3JlZjogMTQKICB9Cn0KY2hlY2tpbmcgewogIGdlbmVyaWNfaGVscGVyX2V4ZWN1dGFibGUgewogICAgc3RyX3JlZjogMTUKICB9CiAgbnVtX2dlbmVyaWNfaGVscGVyczogMTUKICBudW1fcmVhbF90aW1lX2hlbHBlcnM6IDYKICByZWFsX3RpbWVfcG9ydCB7CiAgfQogIGZldGNoZXJfZXhlY3V0YWJsZSB7CiAgICBzdHJfcmVmOiAxNgogIH0KICBudW1fZmV0Y2hlcnM6IDEzCiAgZmV0Y2hlcnNfZW5hYmxlZDogdHJ1ZQogIG51bV9jaGVja2VyczogMTAKICBsb2dfbGV2ZWw6IExPR19MRVZFTF9OT1RJQ0UKICBpbml0aWFsX3NjaGVkdWxpbmdfYnVyc3Q6IDEwCiAgaW5pdGlhbF9nZW5lcmljX3NjaGVkdWxpbmdfaW50ZXJ2YWwgewogICAgbmFub3M6IDE1MDAwMDAwMAogIH0KICBpbml0aWFsX2NoZWNrbWtfc2NoZWR1bGluZ19pbnRlcnZhbCB7CiAgICBzZWNvbmRzOiAxCiAgICBuYW5vczogMjAwMDAwMDAwCiAgfQp9CmxpdmVzdGF0dXMgewogIGRlbGF5IHsKICB9CiAgZGVsYXlfcmFuZG9taXplIHsKICB9CiAgc29ja2V0IHsKICAgIHN0cl9yZWY6IDE3CiAgfQogIGNvbW1hbmRfcGlwZSB7CiAgICBzdHJfcmVmOiAxOAogIH0KICBzdGFja19zaXplOiA0MTk0MzA0CiAgbnVtX3RocmVhZHM6IDIwCiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfREVCVUcKICBzZXJ2aWNlX2F1dGhvcml6YXRpb246IFNFUlZJQ0VfQVVUSE9SSVpBVElPTl9MT09TRQogIGdyb3VwX2F1dGhvcml6YXRpb246IEdST1VQX0FVVEhPUklaQVRJT05fTE9PU0UKfQpub3RpZmljYXRpb25faGFuZGxlciB7CiAgbG9nX2xldmVsOiBMT0dfTEVWRUxfTk9USUNFCn0KYWxlcnRfaGFuZGxlciB7CiAgZXZlbnRfdHlwZXM6IDEKICBsb2dfbGV2ZWw6IExPR19MRVZFTF9OT1RJQ0UKfQpjYXJib24gewogIGxvZ19sZXZlbDogTE9HX0xFVkVMX05PVElDRQp9CiwKICdpbmZsdXhfZGInOiAsCiAnc3RyaW5nX3Bvb2wnOiA8Y21rLmJhc2UuY2VlLm1pY3JvY29yZV9jb25maWcuX1N0cmluZ1Bvb2wgb2JqZWN0IGF0IDB4N2YyMmQxODY3OGUwPn0=", "details": {"argv": ["/omd/sites/test15/bin/cmk", "--debug", "-vvU"], "env": {"SHELL": "/bin/bash", "OMD_ROOT": "/omd/sites/test15", "NAGIOS_PLUGIN_STATE_DIRECTORY": "/omd/sites/test15/var/monitoring-plugins", "PWD": "/omd/sites/test15", "LOGNAME": "test15", "MANPATH": "/omd/sites/test15/share/man:", "MODULEBUILDRC": "/omd/sites/test15/.modulebuildrc", "HOME": "/omd/sites/test15", "LANG": "C.UTF-8", "PERL5LIB": "/omd/sites/test15/local/lib/perl5/lib/perl5:/omd/sites/test15/lib/perl5/lib/perl5:", "OMD_SITE": "test15", "TERM": "xterm", "USER": "test15", "PERL_MM_OPT": "INSTALL_BASE=/omd/sites/test15/local/lib/perl5/", "SHLVL": "1", "LD_LIBRARY_PATH": "/omd/sites/test15/local/lib:/omd/sites/test15/lib", "SSL_CERT_FILE": "/omd/sites/test15/var/ssl/ca-certificates.crt", "REQUESTS_CA_BUNDLE": "/omd/sites/test15/var/ssl/ca-certificates.crt", "LC_ALL": "C.UTF-8", "PATH": "/omd/sites/test15/lib/perl5/bin:/omd/sites/test15/local/bin:/omd/sites/test15/bin:/omd/sites/test15/local/lib/perl5/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games", "MP_STATE_DIRECTORY": "/omd/sites/test15/var/monitoring-plugins", "MAIL": "/var/mail/test15", "MAILRC": "/omd/sites/test15/etc/mail.rc", "_": "/omd/sites/test15/bin/cmk"}}}

The BASE64 Decode from the local_vars are this:

{'gl': general {
  serial: 1252
  core_directory {
  }
  log_file {
    str_ref: 1
  }
  cmk_executable {
    str_ref: 2
  }
  crash_reports_directory {
    str_ref: 3
  }
  license_usage_history_file {
    str_ref: 4
  }
  inventory_directory {
    str_ref: 5
  }
  structured_status_directory {
    str_ref: 6
  }
  robotmk_html_log_directory {
    str_ref: 7
  }
  logwatch_directory {
    str_ref: 8
  }
  event_console_status_socket {
    str_ref: 9
  }
  state_file_created_file {
    str_ref: 10
  }
  licensed_state_file {
    str_ref: 11
  }
  log_level_core: LOG_LEVEL_NOTICE
  log_level_downtime: LOG_LEVEL_NOTICE
  housekeeping_interval {
    seconds: 1
  }
  state_retention_interval {
    seconds: 600
  }
  event_console_enabled: true
  edition_ref: 12
  max_long_output_size: 2000
}
smart_ping {
  check_interval {
    nanos: 500000000
  }
  num_sockets: 8
  wait_duration {
  }
  bulk_size: 1000
  log_level: LOG_LEVEL_NOTICE
  icmp_sender_executable {
    str_ref: 13
  }
  icmp_receiver_executable {
    str_ref: 14
  }
}
checking {
  generic_helper_executable {
    str_ref: 15
  }
  num_generic_helpers: 15
  num_real_time_helpers: 6
  real_time_port {
  }
  fetcher_executable {
    str_ref: 16
  }
  num_fetchers: 13
  fetchers_enabled: true
  num_checkers: 10
  log_level: LOG_LEVEL_NOTICE
  initial_scheduling_burst: 10
  initial_generic_scheduling_interval {
    nanos: 150000000
  }
  initial_checkmk_scheduling_interval {
    seconds: 1
    nanos: 200000000
  }
}
livestatus {
  delay {
  }
  delay_randomize {
  }
  socket {
    str_ref: 17
  }
  command_pipe {
    str_ref: 18
  }
  stack_size: 4194304
  num_threads: 20
  log_level: LOG_LEVEL_DEBUG
  service_authorization: SERVICE_AUTHORIZATION_LOOSE
  group_authorization: GROUP_AUTHORIZATION_LOOSE
}
notification_handler {
  log_level: LOG_LEVEL_NOTICE
}
alert_handler {
  event_types: 1
  log_level: LOG_LEVEL_NOTICE
}
carbon {
  log_level: LOG_LEVEL_NOTICE
}
,
 'influx_db': ,
 'string_pool': <cmk.base.cee.microcore_config._StringPool object at 0x7f22d18678e0>}

I don’t understand where the influx_db at the end comes from.

Good morning @robin.gierse, @andreas-doehler und @r.sander , do you have any other ideas on how I might be able to solve the problem? :pray:

Honestly, I can only point towards paid support. This is way too in depth and complex to do it on the side. Also, one might need to inspect your issue in more depth, which you certainly do not want to do in a public forum.

Maybe the other wizards here can do something, but I am afraid, this is as far as I can go.

1 Like

I can only write the following.

There must be somewhere in the system a config problem.
This problem is not directly InfluxDB related.
You have a problem with your “cmc_log_levels” definition.
That’s why i said in one of my posts before to define some log levels for your system.
If you grep your “~/etc/check_mk/” folder for “cmc_log_levels” or some other variables starting with “cmc_…” what do you find?

The default value for this config variable is

cmc_log_levels: dict[str, int] = {
    "cmk.alert": 5,
    "cmk.carbon": 5,
    "cmk.core": 5,
    "cmk.downtime": 5,
    "cmk.helper": 5,
    "cmk.livestatus": 5,
    "cmk.notification": 5,
    "cmk.rrd": 5,
    "cmk.influxdb": 5,
    "cmk.smartping": 5,
}

If a key is missing inside this setting you will get errors like your one.

1 Like