Update master Checkmk from 2.0.0p8 to 2.2.0b6

PablooS · June 16, 2023, 10:05am

CMK version: 2.2.0b6
OS version: Ubuntu 22.04.2 LTS

Description of the problem:
Hello,

I had CheckMK RAW version 2.0.0p8 on Linux 20.04. I updated Linux to version 22.04 and CheckMK RAW to the (then latest) version 2.2.0b6, which I currently have.
Since then, I have been experiencing errors in various parts of CheckMK related to distributed monitoring.
Whenever I try to do something with them (such as updating through “activate pending changes” or doing “run service discovery”), I receive errors:

Activate pending changes:

Got invalid data:

Internal automation error: unhashable type: &#x27;dict&#x27;
Traceback (most recent call last):
  File &quot;/omd/sites/remote_site/lib/python3/cmk/gui/wato/pages/automation.py&quot;, line 167, in _execute_automation_command
    html.write(repr(automation.execute(automation.get_request())))
  File &quot;/omd/sites/remote_site/lib/python3/cmk/gui/wato/pages/activate_changes.py&quot;, line 663, in execute
    return cmk.gui.watolib.activate_changes.execute_activate_changes(request.domains)
  File &quot;/omd/sites/remote_site/lib/python3/cmk/gui/watolib/activate_changes.py&quot;, line 1776, in execute_activate_changes
    activation_domains = set(domains).union(ABCConfigDomain.get_always_activate_domain_idents())
TypeError: unhashable type: &#x27;dict&#x27;

Run service discovery

Got invalid data:

Internal automation error: Cannot read configuration file "/omd/sites/remote_site/etc/check_mk/conf.d/wato/it_consulting/network/hosts.mk": name 'folder_attributes' is not defined
Traceback (most recent call last):
  File "/omd/sites/remote_site/lib/python3/cmk/utils/store.py", line 139, in load_mk_file
    exec(f.read(), globals(), default)
  File "<string>", line 17, in <module>
NameError: name 'folder_attributes' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1427, in find_host_by_lookup_cache
    folder_lookup_cache = Folder.get_folder_lookup_cache()
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1458, in get_folder_lookup_cache
    Folder.build_host_lookup_cache(cache_path)
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1470, in build_host_lookup_cache
    for host_name, host in Folder.root_folder().all_hosts_recursively().items():
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1154, in all_hosts_recursively
    hosts.update(subfolder.all_hosts_recursively())
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1152, in all_hosts_recursively
    hosts.update(self.hosts())
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1137, in hosts
    self._load_hosts_on_demand()
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 647, in _load_hosts_on_demand
    self._load_hosts()
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 656, in _load_hosts
    variables = self._load_hosts_file()
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 833, in _load_hosts_file
    return store.load_mk_file(self.hosts_file_path(), variables)
  File "/omd/sites/remote_site/lib/python3/cmk/utils/store.py", line 149, in load_mk_file
    raise MKGeneralException(_("Cannot read configuration file \"%s\": %s") % (path, e))
cmk.utils.exceptions.MKGeneralException: Cannot read configuration file "/omd/sites/remote_site/etc/check_mk/conf.d/wato/it_consulting/hosts.mk": name 'folder_attributes' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/omd/sites/remote_site/lib/python3/cmk/utils/store.py", line 139, in load_mk_file
    exec(f.read(), globals(), default)
  File "<string>", line 24, in <module>
NameError: name 'folder_attributes' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/omd/sites/remote_site/lib/python3/cmk/gui/wato/pages/automation.py", line 167, in _execute_automation_command
    html.write(repr(automation.execute(automation.get_request())))
  File "/omd/sites/remote_site/lib/python3/cmk/gui/wato/pages/services.py", line 170, in get_request
    host = watolib.Host.host(host_name)
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 2321, in host
    return Folder.find_host_by_lookup_cache(host_name)
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1451, in find_host_by_lookup_cache
    return Folder.root_folder().find_host_recursively(host_name)
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1411, in find_host_recursively
    host = subfolder.find_host_recursively(host_name)
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1411, in find_host_recursively
    host = subfolder.find_host_recursively(host_name)
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1406, in find_host_recursively
    host: Optional[CREHost] = self.host(host_name)
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 290, in host
    return self.hosts().get(host_name)
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 1137, in hosts
    self._load_hosts_on_demand()
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 647, in _load_hosts_on_demand
    self._load_hosts()
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 656, in _load_hosts
    variables = self._load_hosts_file()
  File "/omd/sites/remote_site/lib/python3/cmk/gui/watolib/hosts_and_folders.py", line 833, in _load_hosts_file
    return store.load_mk_file(self.hosts_file_path(), variables)
  File "/omd/sites/remote_site/lib/python3/cmk/utils/store.py", line 149, in load_mk_file
    raise MKGeneralException(_("Cannot read configuration file \"%s\": %s") % (path, e))
cmk.utils.exceptions.MKGeneralException: Cannot read configuration file "/omd/sites/remote_site/etc/check_mk/conf.d/wato/it_consulting/network/hosts.mk": name 'folder_attributes' is not defined

My remote servers are running CheckMK RAW 2.0.0p8 on Raspbian GNU/Linux 10 (buster).

Output of “cmk --debug -vvn hostname”: I made this from omd su master (hostname is cmk_remote site)

OMD[master]:~$ cmk --debug -vvn cmk_remote
Checkmk version 2.2.0b6
+ FETCHING DATA
  Source: SourceInfo(hostname='cmk_remote', ipaddress='192.168.178.170', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f4d6ea85c10]
Read from cache: AgentFileCache(cmk_remote, path_template=/omd/sites/master/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (does not exist)
[TCPFetcher] Execute data source
Connecting via TCP to 192.168.178.170:6556 (5.0s timeout)
Closing TCP connection to 192.168.178.170:6556
[cpu_tracking] Stop [7f4d6ea85c10 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=5.010000001639128))]
  Source: SourceInfo(hostname='cmk_remote', ipaddress='192.168.178.170', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f4d6ea87010]
Read from cache: NoCache(cmk_remote, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for 'cmk_remote'. Skip processing.
No piggyback files for '192.168.178.170'. Skip processing.
[cpu_tracking] Stop [7f4d6ea87010 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
No persisted sections
  HostKey(hostname='cmk_remote', source_type=<SourceType.HOST: 1>)  -> Not adding sections: TimeoutError('timed out')
  HostKey(hostname='cmk_remote', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
[cpu_tracking] Start [7f4d6ea98950]
value store: synchronizing
Trying to acquire lock on /omd/sites/master/tmp/check_mk/counters/cmk_remote
Got lock on /omd/sites/master/tmp/check_mk/counters/cmk_remote
value store: loading from disk
Releasing lock on /omd/sites/master/tmp/check_mk/counters/cmk_remote
Released lock on /omd/sites/master/tmp/check_mk/counters/cmk_remote
No piggyback files for 'cmk_remote'. Skip processing.
No piggyback files for '192.168.178.170'. Skip processing.
[cpu_tracking] Stop [7f4d6ea98950 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
[agent] TimeoutError('timed out')(!!), [piggyback] Success (but no data found for this host), execution time 5.0 sec | execution_time=5.020 user_time=0.000 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=5.010

I also updated one of the Raspberry Pi devices to Raspbian GNU/Linux 11 (bullseye) and CheckMK RAW version 2.1.0p28, and I’m encountering the following error in “distributed monitoring”:
Missing secret for automation command.

I read in an forum old thread that logging out and logging back in might help, but then I get this error (user and password are still correct):

Cannot connect to remote site: Authentication to web service failed.
Message:
Checkmk Local site MYSITE Username:Password:Version: 2.1.0p28 - &copy; Checkmk GmbHYou can use, modify and distribute Check_MK under the terms of the GNU GPL Version 2.

Output of “cmk --debug -vvn hostname”: (here is output from updated Raspberry Pi which monitored my hosts in office)

OMD[master]:~$ cmk --debug -vvn cmk_monitoring
Checkmk version 2.2.0b6
+ FETCHING DATA
  Source: SourceInfo(hostname='cmk_monitoring', ipaddress='192.168.222.13', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f970a83dbd0]
Read from cache: AgentFileCache(cmk_monitoring, path_template=/omd/sites/master/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 1580 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 192.168.222.13:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Reading data from agent
Closing TCP connection to 192.168.222.13:6556
Write data to cache file /omd/sites/master/tmp/check_mk/cache/cmk_monitoring
Trying to acquire lock on /omd/sites/master/tmp/check_mk/cache/cmk_monitoring
Got lock on /omd/sites/master/tmp/check_mk/cache/cmk_monitoring
Releasing lock on /omd/sites/master/tmp/check_mk/cache/cmk_monitoring
Released lock on /omd/sites/master/tmp/check_mk/cache/cmk_monitoring
[cpu_tracking] Stop [7f970a83dbd0 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=7.6900000013411045))]
  Source: SourceInfo(hostname='cmk_monitoring', ipaddress='192.168.222.13', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f970a83c9d0]
Read from cache: NoCache(cmk_monitoring, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for 'cmk_monitoring'. Skip processing.
No piggyback files for '192.168.222.13'. Skip processing.
[cpu_tracking] Stop [7f970a83c9d0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_status:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_ssl_certs:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<mkeventd_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cmk_site_statistics:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_thermal:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_status:cached(1686909503,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<mknotifyd:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_apache:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_diskusage:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_info:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<timesyncd>>> / Transition HostSectionParser -> HostSectionParser
<<<timesyncd_ntpmessage:sep(10)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
No persisted sections
  HostKey(hostname='cmk_monitoring', source_type=<SourceType.HOST: 1>)  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'cifsmounts', 'cmk_site_statistics', 'cpu', 'df_v2', 'diskstat', 'job', 'kernel', 'labels', 'livestatus_ssl_certs', 'livestatus_status', 'lnx_if', 'lnx_thermal', 'local', 'md', 'mem', 'mkeventd_status', 'mknotifyd', 'mounts', 'nfsmounts_v2', 'omd_apache', 'omd_diskusage', 'omd_info', 'omd_status', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'timesyncd', 'timesyncd_ntpmessage', 'uptime', 'vbox_guest']
  HostKey(hostname='cmk_monitoring', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
[cpu_tracking] Start [7f970a7b8850]
value store: synchronizing
Trying to acquire lock on /omd/sites/master/tmp/check_mk/counters/cmk_monitoring
Got lock on /omd/sites/master/tmp/check_mk/counters/cmk_monitoring
value store: loading from disk
Releasing lock on /omd/sites/master/tmp/check_mk/counters/cmk_monitoring
Released lock on /omd/sites/master/tmp/check_mk/counters/cmk_monitoring
+ EXECUTING INVENTORY PLUGINS
 allnet_ip_sensoric: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 check_mk: ok
 check_mk: skipped (no data)
 checkmk_agent_plugins: ok
 checkmk_agent_plugins: skipped (no data)
 cisco_meraki_org_device_info: skipped (no data)
 cisco_meraki_org_device_info: skipped (no data)
 dell_hw_info: skipped (no data)
 dell_hw_info: skipped (no data)
 docker_node_info: skipped (no data)
 docker_node_info: skipped (no data)
 esx_systeminfo: skipped (no data)
 esx_systeminfo: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fritz: skipped (no data)
 fritz: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ibm_mq_queues: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_if: ok
 inv_if: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_packages: skipped (no data)
 aix_packages: skipped (no data)
 aix_service_packs: skipped (no data)
 aix_service_packs: skipped (no data)
 azure_load_balancer: skipped (no data)
 azure_load_balancer: skipped (no data)
 inventory_checkmk: ok
 inventory_checkmk: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_state: skipped (no data)
 citrix_state: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 dmidecode: skipped (no data)
 dmidecode: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_network: skipped (no data)
 inventory_docker_container_node_name: skipped (no data)
 inventory_docker_container_node_name: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_network: skipped (no data)
 docker_node_network: skipped (no data)
 enviromux_micro_information: skipped (no data)
 enviromux_micro_information: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortigate_ha: skipped (no data)
 fortigate_ha: skipped (no data)
 fortimail_system: skipped (no data)
 fortimail_system: skipped (no data)
 fortinet_firewall_network_interfaces: skipped (no data)
 fortinet_firewall_network_interfaces: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_system: skipped (no data)
 fortisandbox_system: skipped (no data)
 ipmi_firmware: skipped (no data)
 ipmi_firmware: skipped (no data)
 k8s_service_info: skipped (no data)
 k8s_service_info: skipped (no data)
 kube_cluster: skipped (no data)
 kube_cluster: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_deployment: skipped (no data)
 kube_deployment: skipped (no data)
 kube_namespace: skipped (no data)
 kube_namespace: skipped (no data)
 kube_node: skipped (no data)
 kube_node: skipped (no data)
 kube_pod: skipped (no data)
 kube_pod: skipped (no data)
 kube_statefulset: skipped (no data)
 kube_statefulset: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_video: skipped (no data)
 lnx_video: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_clusters: skipped (no data)
 oracle_systemparameter: skipped (no data)
 oracle_systemparameter: skipped (no data)
 primekey: skipped (no data)
 primekey: skipped (no data)
 snmp_extended_info: skipped (no data)
 snmp_extended_info: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_routes: skipped (no data)
 statgrab_net: skipped (no data)
 statgrab_net: skipped (no data)
 win_bios: skipped (no data)
 win_bios: skipped (no data)
 win_disks: skipped (no data)
 win_disks: skipped (no data)
 win_exefiles: skipped (no data)
 win_exefiles: skipped (no data)
 win_ip_r: skipped (no data)
 win_ip_r: skipped (no data)
 win_networkadapter: skipped (no data)
 win_networkadapter: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_video: skipped (no data)
 win_video: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_updates: skipped (no data)
 win_wmi_updates: skipped (no data)
 juniper_info: skipped (no data)
 juniper_info: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_pod_container: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_if: ok
 lnx_if: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_uname: skipped (no data)
 lparstat_aix: skipped (no data)
 lparstat_aix: skipped (no data)
 mem: ok
 mem: skipped (no data)
 mem_used: skipped (no data)
 mem_used: skipped (no data)
 mssql_instance: skipped (no data)
 mssql_instance: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_info: skipped (no data)
 netapp_api_info: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_tablespaces: skipped (no data)
 oracle_tablespaces: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_psmu: skipped (no data)
 perle_psmu: skipped (no data)
 prtconf: skipped (no data)
 prtconf: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_info: skipped (no data)
 snmp_info: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_uname: skipped (no data)
 solaris_uname: skipped (no data)
 suseconnect: skipped (no data)
 suseconnect: skipped (no data)
 win_computersystem: skipped (no data)
 win_computersystem: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_os: skipped (no data)
 win_os: skipped (no data)
 win_system: skipped (no data)
 win_system: skipped (no data)
 winperf_if: skipped (no data)
 winperf_if: skipped (no data)
Trying to acquire lock on /omd/sites/master/tmp/check_mk/status_data/cmk_monitoring
Got lock on /omd/sites/master/tmp/check_mk/status_data/cmk_monitoring
Releasing lock on /omd/sites/master/tmp/check_mk/status_data/cmk_monitoring
Released lock on /omd/sites/master/tmp/check_mk/status_data/cmk_monitoring
Trying to acquire lock on /omd/sites/master/tmp/check_mk/status_data/cmk_monitoring.gz
Got lock on /omd/sites/master/tmp/check_mk/status_data/cmk_monitoring.gz
Releasing lock on /omd/sites/master/tmp/check_mk/status_data/cmk_monitoring.gz
Released lock on /omd/sites/master/tmp/check_mk/status_data/cmk_monitoring.gz
No piggyback files for 'cmk_monitoring'. Skip processing.
No piggyback files for '192.168.222.13'. Skip processing.
[cpu_tracking] Stop [7f970a7b8850 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.010000001639127731))]
[agent] Success, [piggyback] Success (but no data found for this host), execution time 7.7 sec | execution_time=7.700 user_time=0.020 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=7.680

cmk --debug -vvI hostname: on Raspberry

OMD[MYSITE]:~$ cmk --debug -vvI cmk_monitoring
Discovering services and host labels on: cmk_monitoring
cmk_monitoring:
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [f4a53118]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(cmk_monitoring, base_path=/omd/sites/MYSITE/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Using data from cache file /omd/sites/MYSITE/tmp/check_mk/cache/cmk_monitoring
Got 263115 bytes data from cache
[TCPFetcher] Use cached data
[cpu_tracking] Stop [f4a53118 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.010000001639127731))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [f4a535f8]
[PiggybackFetcher] Fetch with cache settings: NoCache(cmk_monitoring, base_path=/omd/sites/MYSITE/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'cmk_monitoring'. Skip processing.
No piggyback files for '192.168.222.13'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [f4a535f8 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_status:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_ssl_certs:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<mkeventd_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cmk_site_statistics:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_thermal:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_status:cached(1686910228,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<mknotifyd:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_apache:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_diskusage:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_info:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<timesyncd>>> / Transition HostSectionParser -> HostSectionParser
<<<timesyncd_ntpmessage:sep(10)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'cifsmounts', 'cmk_site_statistics', 'cpu', 'df_v2', 'diskstat', 'job', 'kernel', 'labels', 'livestatus_ssl_certs', 'livestatus_status', 'lnx_if', 'lnx_thermal', 'local', 'md', 'mem', 'mkeventd_status', 'mknotifyd', 'mounts', 'nfsmounts_v2', 'omd_apache', 'omd_diskusage', 'omd_info', 'omd_status', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'timesyncd', 'timesyncd_ntpmessage', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
+ ANALYSE DISCOVERED HOST LABELS
Trying host label discovery with: check_mk, checkmk_agent_plugins_lnx, cifsmounts, cmk_site_statistics, cpu, df_v2, diskstat, job, kernel, labels, livestatus_ssl_certs, livestatus_status, lnx_if, lnx_thermal, local, md, mem, mkeventd_status, mknotifyd, mounts, nfsmounts_v2, omd_apache, omd_diskusage, omd_info, omd_status, ps_lnx, systemd_units, tcp_conn_stats, timesyncd, timesyncd_ntpmessage, uptime, vbox_guest
  cmk/os_family: linux (check_mk)
  cmk/check_mk_server: yes (omd_info)
Trying host label discovery with:
Trying to acquire lock on /omd/sites/MYSITE/var/check_mk/discovered_host_labels/cmk_monitoring.mk
Got lock on /omd/sites/MYSITE/var/check_mk/discovered_host_labels/cmk_monitoring.mk
Releasing lock on /omd/sites/MYSITE/var/check_mk/discovered_host_labels/cmk_monitoring.mk
Released lock on /omd/sites/MYSITE/var/check_mk/discovered_host_labels/cmk_monitoring.mk
SUCCESS - Found no new host labels
+ ANALYSE DISCOVERED SERVICES
+ EXECUTING DISCOVERY PLUGINS (38)
  Trying discovery with: tcp_conn_stats, lnx_thermal, ps, mknotifyd, local, mem_linux, md, df, docker_container_status_uptime, mkeventd_status, kernel_util, domino_tasks, kernel, mknotifyd_connection, omd_status, mem_win, mssql_datafiles, diskstat, systemd_units_services, mssql_transactionlogs, kernel_performance, mem_vmalloc, systemd_units_services_summary, checkmk_agent, vbox_guest, livestatus_status, cpu_loads, job, uptime, cpu_threads, cifsmounts, k8s_stats_network, check_mk_only_from, mounts, lnx_if, omd_apache, cmk_site_statistics, timesyncd
  1 cmk_site_statistics
  1 df
  1 livestatus_status
  1 mkeventd_status
  1 omd_apache
  1 omd_status
SUCCESS - Found 6 services

scotsie · June 16, 2023, 4:12pm

In the updating documentation it explicitly calls out a requirement nowadays to update in steps.
You likely are missing some intermediary changes between 2.0 > 2.1 > 2.2

Update the Checkmk version

Before updating to version 2.2.0, the Checkmk site must have version 2.1.0 installed.

We have previously advised against omitting an intermediate major version when performing a major update, as there are simply too many changes in between that will hinder a smooth update and almost certainly cause problems. With version 2.1.0, this recommendation was turned into a requirement — and a lock was introduced that prevents, for example, a direct update from version 2.0.0 to 2.2.0.

The update to 2.2.0 currently requires at least 2.1.0p23. However, in the future, a specific 2.2.0 patch version may require a higher 2.1.0 patch version for the update. In general, we recommend updating Checkmk to the latest 2.1.0 patch version first, and only then updating to 2.2.0.

There are also some best practices mentioned with distributed monitoring about updating the remote nodes before updating your central node or doing them all at once.

If you have a backup or copy of your old instance, perhaps you can rollback and then perform a 2.1 update before migration to 2.2.

Hope this helps.

Sincerely,
Scotsie

PablooS · June 20, 2023, 8:33am

Thank you for your response.

I’m not 100% sure about the exact process of the upgrade because I was involved in it some time ago, then had other tasks, and returned to CheckMK. However, I definitely performed multiple updates, “jumping” through several versions. I have backups of the master instance starting from version 2.0.0p8, then 2.0.0p35, then 2.1.0p28, and now it’s on version 2.2.0b6.

I have backups only for the master instance and for the mentioned Raspberry Pi instance, which I upgraded to version 2.1.0p28 and have local access to. I didn’t have access to other remote instances at that time (now I have access to all instances, but another instances haven’t backup)

When I log in to the remote instances, I immediately encounter the following error after logging in:

Internal error: **new**() takes 6 positional arguments but 8 were given

An internal error occurred while processing your request. You can report this issue to the Checkmk team to help fix it. Please open the crash report page and use the form to report the problem.

Should I now revert to version 2.0.0p8 of the master instance and start by upgrading the remaining instances (remote on Raspberry Pi), and only then proceed to update the master version to a newer version?

Or should I first update the remote instances, following the advice in the previous post to upgrade them by only one major version, then update the master instance in the same way, and repeat this process for the remaining remote instances but only with 1 major update?

PablooS · June 20, 2023, 12:21pm

Okay, I have reverted the master instance from backup to the original version (2.0.0p8). In the distributed monitoring, I edited the remote instances without changing anything, saved the changes, and released the configuration in the pending changes, which fixed the error mentioned in the previous post (internal error). Now I am back to where I started, and I will begin the update process as recommended in post #2. Thank you for all the guidance.

scotsie · June 20, 2023, 12:59pm

Thank goodness for those backups and the additional info definitely paints a better picture.

I mentioned the upgrade language change because I was reviewing it preparing to test the 2.2 release in a lab setup. The method you mention is what I’ve been following since 1.5 when we first began using CheckMK and I took over the care and feeding. Thankfully no insurmountable issues so far.

Please follow up and let me know how it goes.

Sincerely,
Scotsie

PablooS · June 26, 2023, 9:20am

Now I have updated one of the remote instances (from version 2.0.0p8 to 2.0.0p17) and encountered the following error after pushing the new configuration from the master to this site:
EDIT: The same error is, when I updated from version p8 to p9.

Got invalid data:

Internal automation error: The parameter "replication_paths" is missing.
Traceback (most recent call last):
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/wato/pages/automation.py", line 170, in _execute_automation_command
    html.write(repr(automation.execute(automation.get_request())))
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/watolib/activate_changes.py", line 2152, in get_request
    for e in ast.literal_eval(_request.get_ascii_input_mandatory("replication_paths"))
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/http.py", line 280, in get_ascii_input_mandatory
    return mandatory_parameter(varname, self.get_ascii_input(varname, deflt))
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/http.py", line 232, in mandatory_parameter
    raise MKUserError(varname, _("The parameter \"%s\" is missing.") % varname)
cmk.gui.exceptions.MKUserError: The parameter "replication_paths" is missing.

Should I upgrade all my remote instances and then upgrade my master version, so that this error doesn’t occur or what I can to do?

EDIT: From the documentation, I read that a higher version remote site is compatible with a lower version master site. However, in my case, it does not work.

PablooS · June 27, 2023, 9:05am

@scotsie can u help me? Or anyone else?

EDIT: For tests I upgraded one of remote sites to version 2.0.0p20 and master too - got error:

Got invalid data:

Missing secret for automation command.

scotsie · June 27, 2023, 12:46pm

Honestly I’m at a loss with my limited exposure. The original error refers to the same ‘folder_attributes’.

I’m hoping someone might recognize the error or traceback information. There’s a single post in german that (with the help of google translate) refers to the same error and upgrading via CLI for a higher release number but doesn’t go into detail.

Barring any other feedback from someone with a bit more experience and given that you have your backups, I’d build out the exact environment versions in a lab and try the “all in one” upgrade method and bring all releases to the latest version of 2.0.0 to see if that makes a difference.

PablooS · June 27, 2023, 1:22pm

That documentation mentioned by you is for version lower as 2.0.0 (its update to version 2.0.0).
I have backups from version 2.0.0p8 and now i tried restore that, but i still have the same error like in post #6

What is that upgrading via CLI? Installation higher version and upgrade with “omd update MYSITE”?
If yes I made that from beginning.

scotsie · June 27, 2023, 1:30pm

I understand. I’m referring to getting your 2.0.0p8 central and distributed nodes up to the latest release check-mk-raw-2.0.0p36 as there are some bug fixes that could be applicable. At this point my advice is mostly from a troubleshooting, process of elimination now though and not from experiencing the exact error you’re getting.

My environment is also Redhat/CentOS based so I don’t know if the OS upgrade could be a contributing factor.

Sincerely,
Scotsie

PablooS · June 30, 2023, 8:24am

I found a solution to my problem with that error ( Internal automation error: The parameter "replication_paths" is missing.):
In Setup > Distributed Monitoring > Edit Site, the “http” needs to be changed to “https”.