Need help after update from 2.2.0p7 to 2.2.0p8: service "Check_MK": [agent] invalid load key, ‘{’.(?) on most systems

Troubleshooting
1

Hi all,
may anyone help me get out of this?

CMK version:
2.2.0p8

OS version:
Debian bookworm 12.1

Error message:
[agent] invalid load key, ‘{’.UNKN, [piggyback] Success (but no data found for this host), Missing monitoring data for all pluginsWARN,

Output of “cmk --debug -vvn hostname”:

Checkmk version 2.2.0p8
+ FETCHING DATA
  Source: SourceInfo(hostname='hostname.domain.local', ipaddress='192.168.0.1', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f1d564c42d0]
Read from cache: AgentFileCache(hostname.domain.local, path_template=/omd/sites/kohly/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 10 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 192.168.0.1:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Reading data from agent
Closing TCP connection to 192.168.0.1:6556
Write data to cache file /omd/sites/kohly/tmp/check_mk/cache/hostname.domain.local
Trying to acquire lock on /omd/sites/kohly/tmp/check_mk/cache/hostname.domain.local
Got lock on /omd/sites/kohly/tmp/check_mk/cache/hostname.domain.local
Releasing lock on /omd/sites/kohly/tmp/check_mk/cache/hostname.domain.local
Released lock on /omd/sites/kohly/tmp/check_mk/cache/hostname.domain.local
[cpu_tracking] Stop [7f1d564c42d0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=4.900000002235174))]
  Source: SourceInfo(hostname='hostname.domain.local', ipaddress='192.168.0.1', ident='special_proxmox_ve', fetcher_type=<FetcherType.SPECIAL_AGENT: 6>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f1d571facd0]
Read from cache: AgentFileCache(hostname.domain.local, path_template=/omd/sites/kohly/tmp/check_mk/data_source_cache/special_proxmox_ve/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 13 sec, allowed is 0 sec)
[ProgramFetcher] Execute data source
Calling: /omd/sites/kohly/share/check_mk/agents/special/agent_proxmox_ve -u checkmk@pve -p UMRMrwlCK0Y --no-cert-check hostname.domain.local
Write data to cache file /omd/sites/kohly/tmp/check_mk/data_source_cache/special_proxmox_ve/hostname.domain.local
Trying to acquire lock on /omd/sites/kohly/tmp/check_mk/data_source_cache/special_proxmox_ve/hostname.domain.local
Got lock on /omd/sites/kohly/tmp/check_mk/data_source_cache/special_proxmox_ve/hostname.domain.local
Releasing lock on /omd/sites/kohly/tmp/check_mk/data_source_cache/special_proxmox_ve/hostname.domain.local
Released lock on /omd/sites/kohly/tmp/check_mk/data_source_cache/special_proxmox_ve/hostname.domain.local
[cpu_tracking] Stop [7f1d571facd0 - Snapshot(process=posix.times_result(user=0.0, system=0.010000000000000009, children_user=0.77, children_system=0.14, elapsed=1.300000000745058))]
  Source: SourceInfo(hostname='hostname.domain.local', ipaddress='192.168.0.1', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f1d5712b050]
Read from cache: NoCache(hostname.domain.local, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for 'hostname.domain.local'. Skip processing.
No piggyback files for '192.168.0.1'. Skip processing.
[cpu_tracking] Stop [7f1d5712b050 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<ovs_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool_status>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool>>> / Transition HostSectionParser -> HostSectionParser
<<<fileinfo:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_thermal:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_nodes>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1692793755,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<dmi_sysinfo>>> / Transition HostSectionParser -> HostSectionParser
<<<sshd_config:cached(1692793511,300)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1692793395,3600):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<apcaccess:cached(1692793762,60):sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<cpufreq:cached(1692793756,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe:cached(1692793756,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<lvm_vgs:cached(1692793756,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<lvm_lvs:cached(1692793756,60):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<filehandler:cached(1692793756,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<iptables:cached(1692793756,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<logins:cached(1692793756,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch:cached(1692793757,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsiostat:cached(1692793757,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<netstat:cached(1692793757,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<smart:cached(1692793757,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd:cached(1692793757,60)>>> / Transition HostSectionParser -> HostSectionParser
############## Piggyback information removed ##############
Received piggyback data for 13 hosts
[cpu_tracking] Start [7f1d56172d90]
value store: synchronizing
Trying to acquire lock on /omd/sites/kohly/tmp/check_mk/counters/hostname.domain.local
Got lock on /omd/sites/kohly/tmp/check_mk/counters/hostname.domain.local
value store: loading from disk
Releasing lock on /omd/sites/kohly/tmp/check_mk/counters/hostname.domain.local
Released lock on /omd/sites/kohly/tmp/check_mk/counters/hostname.domain.local
APC apcupsd.conf Status PEND Check plugin received no monitoring data
APT Updates          PEND Check plugin received no monitoring data
CPU load             PEND Check plugin received no monitoring data
CPU utilization      PEND Check plugin received no monitoring data
Check_MK Agent       PEND Check plugin received no monitoring data
DMI Sysinfo          PEND Check plugin received no monitoring data
Disk IO LVM pve-root PEND Check plugin received no monitoring data
Disk IO LVM pve-swap PEND Check plugin received no monitoring data
Disk IO SUMMARY      PEND Check plugin received no monitoring data
Disk IO sda          PEND Check plugin received no monitoring data
File group kernel config PEND Check plugin received no monitoring data
File group kernel image PEND Check plugin received no monitoring data
File group kernel initrd PEND Check plugin received no monitoring data
Filecount_/boot      PEND Check plugin received no monitoring data
Filecount_/tmp       PEND Check plugin received no monitoring data
Filecount_/var/cache/apt/archives PEND Check plugin received no monitoring data
Filecount_/var/log   PEND Check plugin received no monitoring data
Filehandler          PEND Check plugin received no monitoring data
Interface eth0       PEND Check plugin received no monitoring data
Interface eth1       PEND Check plugin received no monitoring data
Interface lan        PEND Check plugin received no monitoring data
Interface lo         PEND Check plugin received no monitoring data
Interface ovs-system PEND Check plugin received no monitoring data
Interface vmbr0      PEND Check plugin received no monitoring data
Iptables             PEND Check plugin received no monitoring data
Kernel Performance   PEND Check plugin received no monitoring data
LOG /var/log/apcupsd.events PEND Check plugin received no monitoring data
LOG /var/log/auth.log PEND Check plugin received no monitoring data
LOG /var/log/kern.log PEND Check plugin received no monitoring data
LOG /var/log/messages PEND Check plugin received no monitoring data
LOG /var/log/syslog  PEND Check plugin received no monitoring data
LVM VG pve           PEND Check plugin received no monitoring data
Logins               PEND Check plugin received no monitoring data
Memory               PEND Check plugin received no monitoring data
Mount options of /   PEND Check plugin received no monitoring data
Mount options of /boot PEND Check plugin received no monitoring data
Mount options of /etc/pve PEND Check plugin received no monitoring data
NTP Time             PEND Check plugin received no monitoring data
Number of threads    PEND Check plugin received no monitoring data
Postfix Queue        PEND Check plugin received no monitoring data
Postfix status       PEND Check plugin received no monitoring data
Proxmox VE Memory Usage Usage: 76.49% - 11.9 GiB of 15.5 GiB
Proxmox VE Node Info Status: online, Subscription: invalid, Version: 8.0.4, Hosted VMs: 0x LXC, 11x Qemu
Resolve_gw01.kohly.de PEND Check plugin received no monitoring data
SMART Samsung_SSD_840_EVO_500GB_mSATA_S1KMNSAF501737Z Stats PEND Check plugin received no monitoring data
SSH daemon configuration PEND Check plugin received no monitoring data
System_info_cpu      PEND Check plugin received no monitoring data
System_info_debian_version PEND Check plugin received no monitoring data
System_info_hardware PEND Check plugin received no monitoring data
System_info_os-release PEND Check plugin received no monitoring data
System_info_uname-a  PEND Check plugin received no monitoring data
System_info_uname-r  PEND Check plugin received no monitoring data
System_info_uname-v  PEND Check plugin received no monitoring data
Systemd Service Summary PEND Check plugin received no monitoring data
Systemd Socket Summary PEND Check plugin received no monitoring data
TCP Connections      PEND Check plugin received no monitoring data
Temperature Zone 0   PEND Check plugin received no monitoring data
Temperature Zone 1   PEND Check plugin received no monitoring data
Temperature Zone 2   PEND Check plugin received no monitoring data
Uptime               Up since Aug 18 2023 14:46:51, Uptime: 4 days 23 hours
fs_/                 PEND Check plugin received no monitoring data
fs_/boot             PEND Check plugin received no monitoring data
fs_/etc/pve          PEND Check plugin received no monitoring data
reboot               PEND Check plugin received no monitoring data
sshsessions          PEND Check plugin received no monitoring data
No piggyback files for 'hostname.domain.local'. Skip processing.
No piggyback files for '192.168.0.1'. Skip processing.
[cpu_tracking] Stop [7f1d56172d90 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.010000001639127731))]
**[agent] invalid load key, '{'.(?)**, [special_proxmox_ve] Success, [piggyback] Success (but no data found for this host), Missing monitoring data for plugins: apcaccess, apt, checkmk_agent, chrony, cpu_loads, cpu_threads, df, diskstat, dmi_sysinfo, filehandler, fileinfo_groups, iptables, kernel_performance, kernel_util, lnx_if, lnx_thermal, local, logins, logwatch, lvm_vgs, mem_linux, mounts, mrpe, nfsiostat, nfsmounts, postfix_mailq, postfix_mailq_status, smart_stats, sshd_config, systemd_units_services_summary, systemd_units_sockets_summary, tcp_conn_stats(!), execution time 6.2 sec | execution_time=6.210 user_time=0.020 system_time=0.010 children_user_time=0.770 children_system_time=0.140 cmk_time_agent=4.890 cmk_time_ds=0.380
2

I can confirm this problem. I have the problem on both installations that I maintain. Not all monitored hosts were affected though (some hosts were using older cmk-agents, some the up-to-date version; I tried disabling plugins which also didn’t help).

I tried a downgrade to 2.2.0p7, but this resulted in other errors …

I hope that the problem will soon be fixed …

3

just worked around by restoring a backup from before the update to get the site up and running properly.

4

Same here … :frowning:
Let’s hope the backup restores properly…

5

It looks like there is a bit of invalid data causing Checkmk to crash. It is not obvious to me though, where this might originate. Did it work on 2.2.0p7? Is it just affecting Proxmox hosts, or others affected as well?

6

I don’t have any Proxmox hosts, almost all of them are Ubuntu 22.04 Server. 2.2.0p7 worked as expected, the problems came with the upgrade of the server to 2.2.0p8. Client-side upgrade of the agents to 2.2.0p8 does not solve the problem.

Here the output of one of my affected hosts (I modified the output to hide IP address and other sensitive data), using agent 2.2.0p7:

OMD[checkmk]:~$ cmk --debug -vvn host1.local
Checkmk version 2.2.0p8
+ FETCHING DATA
  Source: SourceInfo(hostname='host1.local', ipaddress='192.168.0.12', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f73056d2610]
Read from cache: AgentFileCache(host1.local, path_template=/omd/sites/checkmk/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 35 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 192.168.0.12:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Closing TCP connection to 192.168.0.12:6556
Write data to cache file /omd/sites/checkmk/tmp/check_mk/cache/host1.local
Trying to acquire lock on /omd/sites/checkmk/tmp/check_mk/cache/host1.local
Got lock on /omd/sites/checkmk/tmp/check_mk/cache/host1.local
Releasing lock on /omd/sites/checkmk/tmp/check_mk/cache/host1.local
Released lock on /omd/sites/checkmk/tmp/check_mk/cache/host1.local
[cpu_tracking] Stop [7f73056d2610 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=5.0))]
  Source: SourceInfo(hostname='host1.local', ipaddress='192.168.0.12', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f7305d414d0]
Read from cache: NoCache(host1.local, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for 'host1.local'. Skip processing.
No piggyback files for '192.168.0.12'. Skip processing.
[cpu_tracking] Stop [7f7305d414d0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<multipath>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<ntp:cached(1693294441,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<lvm_vgs>>> / Transition HostSectionParser -> HostSectionParser
<<<lvm_lvs:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<qemu>>> / Transition HostSectionParser -> HostSectionParser
<<<smart>>> / Transition HostSectionParser -> HostSectionParser
  HostKey(hostname='host1.local', source_type=<SourceType.HOST: 1>)  -> Not adding sections: invalid load key, '{'.
  HostKey(hostname='host1.local', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
[cpu_tracking] Start [7f7304adb990]
value store: synchronizing
Trying to acquire lock on /omd/sites/checkmk/tmp/check_mk/counters/host1.local
Got lock on /omd/sites/checkmk/tmp/check_mk/counters/host1.local
value store: loading from disk
Releasing lock on /omd/sites/checkmk/tmp/check_mk/counters/host1.local
Released lock on /omd/sites/checkmk/tmp/check_mk/counters/host1.local
APT Updates          PEND Check plugin received no monitoring data
CPU load             PEND Check plugin received no monitoring data
CPU utilization      PEND Check plugin received no monitoring data
Check_MK Agent       PEND Check plugin received no monitoring data
Disk IO SUMMARY      PEND Check plugin received no monitoring data
Filesystem /         PEND Check plugin received no monitoring data
Interface 02         PEND Check plugin received no monitoring data
Interface 03         PEND Check plugin received no monitoring data
Interface 04         PEND Check plugin received no monitoring data
Interface 05         PEND Check plugin received no monitoring data
Interface 06         PEND Check plugin received no monitoring data
Interface 07         PEND Check plugin received no monitoring data
Kernel Performance   PEND Check plugin received no monitoring data
LVM VG lvm           PEND Check plugin received no monitoring data
LVM VG ssd           PEND Check plugin received no monitoring data
MD Softraid md0      PEND Check plugin received no monitoring data
MD Softraid md1      PEND Check plugin received no monitoring data
Memory               PEND Check plugin received no monitoring data
Mount options of /   PEND Check plugin received no monitoring data
NTP Time             PEND Check plugin received no monitoring data
Number of threads    PEND Check plugin received no monitoring data
Postfix Queue        PEND Check plugin received no monitoring data
Postfix status       PEND Check plugin received no monitoring data
SMART /dev/nvme0n1 Stats PEND Check plugin received no monitoring data
SMART /dev/nvme1n1 Stats PEND Check plugin received no monitoring data
SMART 0ATA_HGST_HUS726040AL_NHG91U7Y Stats PEND Check plugin received no monitoring data
SMART 0ATA_HGST_HUS726040AL_NHG97RMK Stats PEND Check plugin received no monitoring data
SMART 0ATA_HGST_HUS726040AL_NHG992AY Stats PEND Check plugin received no monitoring data
SMART 0ATA_WDC_WD20EFRX-68E_WD-WCC4M5JR20H9 Stats PEND Check plugin received no monitoring data
Systemd Service Summary PEND Check plugin received no monitoring data
Systemd Socket Summary PEND Check plugin received no monitoring data
TCP Connections      PEND Check plugin received no monitoring data
Temperature SMART /dev/nvme0n1 PEND Check plugin received no monitoring data
Temperature SMART /dev/nvme1n1 PEND Check plugin received no monitoring data
Temperature SMART 0ATA_HGST_HUS726040AL_NHG91U7Y PEND Check plugin received no monitoring data
Temperature SMART 0ATA_HGST_HUS726040AL_NHG97RMK PEND Check plugin received no monitoring data
Temperature SMART 0ATA_HGST_HUS726040AL_NHG992AY PEND Check plugin received no monitoring data
Temperature SMART 0ATA_WDC_WD20EFRX-68E_WD-WCC4M5JR20H9 PEND Check plugin received no monitoring data
Uptime               PEND Check plugin received no monitoring data
VM host2           PEND Check plugin received no monitoring data
VM host3        PEND Check plugin received no monitoring data
VM host4       PEND Check plugin received no monitoring data
VM host5            PEND Check plugin received no monitoring data
No piggyback files for 'host1.local'. Skip processing.
No piggyback files for '192.168.0.12'. Skip processing.
[cpu_tracking] Stop [7f7304adb990 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[agent] invalid load key, '{'.(?), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 5.0 sec | execution_time=5.000 user_time=0.000 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=5.000
7

2.2.0p7 works like a charm.
(some) proxmox hosts are also affected as (some) non proxmox hosts.

8

tried the update again (took a snapshot of the vm before :wink:)

during the update, i got the folling error(s):

Creating temporary filesystem /omd/sites/sitename/tmp...OK
Executing update-pre-hooks script "01_mkp-disable-outdated"...
-| [apcaccess 5.2.1]: Not uninstalling
-| [checkmk_update 0.1.3-20230710]: Not uninstalling
-| [fail2ban 1.9.1]: Not uninstalling
-| [apcaccess 5.2.1]: Updating from 5.2.1
-| [apcaccess 5.2.1]: Installed
-| [checkmk_update 0.1.3-20230710]: Updating from 0.1.3-20230710
-| [checkmk_update 0.1.3-20230710]: Installed
-| [fail2ban 1.9.1]: Updating from 1.9.1
-| [fail2ban 1.9.1]: Installed
OK
Executing update-pre-hooks script "02_cmk-update-config"...
-| ATTENTION
-|   Some steps may take a long time depending on your installation.
-|   Please be patient.
-| 
-| Verifying Checkmk configuration...
-|  01/04 Rulesets...
-|  02/04 UI extensions...
-|  03/04 Agent based plugins...
-|  04/04 Deprecated .mk configuration of plugins...
-| Done (success)
-| 
-| Updating Checkmk configuration...
-|  01/17 Validate user IDs...
-|  02/17 Update views...
-|  03/17 Update dashboards...
-|  04/17 User attributes...
-|  05/17 Global settings...
-|  06/17 Rulesets...
-|  + "Rulesets" failed              <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
-| Traceback (most recent call last):
-|   File "/omd/sites/sitename/lib/python3/cmk/update_config/main.py", line 223, in __call__
-|     action(self._logger, self.update_state.setdefault(action.name))
-|   File "/omd/sites/sitename/lib/python3/cmk/update_config/plugins/actions/rulesets.py", line 54, in __call__
-|     _transform_wato_rulesets_params(
-|   File "/omd/sites/sitename/lib/python3/cmk/update_config/plugins/actions/rulesets.py", line 160, in _transform_wato_rulesets_params
-|     valuespec = ruleset.valuespec()
-|                 ^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/sitename/lib/python3/cmk/gui/watolib/rulesets.py", line 879, in valuespec
-|     return self.rulespec.valuespec
-|            ^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/sitename/lib/python3/cmk/gui/watolib/rulespecs.py", line 348, in valuespec
-|     return self._valuespec()
-|            ^^^^^^^^^^^^^^^^^
-|   File "<string>", line 97, in _valuespec_apcaccess_inventory
-| NameError: name 'DropdownChoice' is not defined
-|  07/17 Autochecks...
-|  08/17 Remove unused host attributes....
-|  09/17 Convert persisted sections...
-|  + "Convert persisted sections" failed              <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
-| Traceback (most recent call last):
-|   File "/omd/sites/sitename/lib/python3/cmk/update_config/main.py", line 223, in __call__
-|     action(self._logger, self.update_state.setdefault(action.name))
-|   File "/omd/sites/sitename/lib/python3/cmk/update_config/plugins/actions/persisted_sections.py", line 23, in __call__
-|     self._convert_file(os.path.join(dirpath, filename))
-|   File "/omd/sites/sitename/lib/python3/cmk/update_config/plugins/actions/persisted_sections.py", line 35, in _convert_file
-|     data = store.load_object_from_file(filename, default=None)
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/sitename/lib/python3/cmk/utils/store/__init__.py", line 186, in load_object_from_file
-|     return ObjectStore(Path(path), serializer=DimSerializer()).read_obj(default=default)
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/sitename/lib/python3/cmk/utils/store/_file.py", line 136, in read_obj
-|     return self._serializer.deserialize(raw) if raw else default
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/sitename/lib/python3/cmk/utils/store/_file.py", line 75, in deserialize
-|     return literal_eval(raw.decode("utf-8"))
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/sitename/lib/python3.11/ast.py", line 64, in literal_eval
-|     node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
-|                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-|   File "/omd/sites/sitename/lib/python3.11/ast.py", line 50, in parse
-|     return compile(source, filename, mode, flags,
-|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-| ValueError: source code string cannot contain null bytes
-|  10/17 Cleanup version specific caches...
-|  11/17 Background jobs...
-|  12/17 Extract remote sites CAs...
-|  13/17 Add a rule_id to each notification rule...
-|  14/17 Change absolute paths in registered hosts symlinks to relative...
-|  15/17 Remove old custom logos...
-|  16/17 Check for incompatible password hashes...
-|  Users with outdated, no longer supported password hashes have been found. These users will be unable to log in.
-| Please manually reset these users' passwords either in Setup > Users or on the commandline using the cmk-passwd command.
-| The following users are affected:
-| dani
-| petra
-|  17/17 Update backup config...
-| Done (with errors)
ERROR (exit code: 1)

mayby this is related to the ‘invalid load key’ problem?

9

I am wondering if this is caused by a broken host.mk or host.pkl file and/or autochecks file.

Can you check the corresponding host.mk and host.pkl file and autochecks file please or send them by PM to me ?

find etc/check_mk/ -name "hosts.*"

var/check_mk/autochecks/<hostname>.mk
1 Like
10

The update error is caused by the incompatible extension apcaccess.
you need to uninstall or upgraded it to a version that is for checkmk 2.2

1 Like
11

thanks for the hint, i will try this and report back.
btw: how do you saw it comes from the outdatet plugin?

12

Hi,

in the update log there is this message about missing functions that have to be explicitly imported with checkmk 2.2.
I don´t know if that is the cause for your problem but it is at least another problem, that you need to solve :slight_smile:
I think @r.sander already create a new apcaccess mkp for 2.2 : Checkmk Exchange

2 Likes
13

updating the plugin removes the error from the “06/17 Rulesets…” section.
thank you!

the other error in “09/17 Convert persisted sections…” still persists.
any clue on this?

14

problem solved:
after digging around and with a help from an other member i removed all files from var/check_mk/persisted/
afterwards the update works like expected and the ‘invalid load key’ was gone.

3 Likes