CMK version: check-mk-raw:2.4.0p10
OS version: Ubuntu 24.04
Error message:
/omd/sites/checkmkprod/share/check_mk/agents/special/agent_proxmox_ve --username audit@pve --password 'XXXXXX' --port 8006 --no-cert-check --timeout 120 --log-cutoff-weeks 1 fakehost.fakedomain.com
ERROR 2025-09-02 12:31:04 root: Error in log at line 167: LogParseWarning(167, "End of VM '12609' while still information is missing (we have: {'started_time', 'total_duration'})")
ERROR 2025-09-02 12:31:04 root: Error in log at line 778: LogParseWarning(778, "End of VM '13505' while still information is missing (we have: {'started_time', 'total_duration'})")
ERROR 2025-09-02 12:31:04 agent_proxmox_ve: Parsing the log for UPID='UPID:mkclu3:0021072D:071436B9:68B65DB3:vzdump::root@pam:' resulted in a error(s) - write log content to '/omd/sites/checkmkprod/tmp/check_mk/special_agents/agent_proxmox_ve/erroneous-UPID:mkclu3:0021072D:071436B9:68B65DB3:vzdump::root@pam:.log'
Agent failed - please submit a crash report! (Crash-ID: ee0a93d0-87e7-11f0-a753-005056009492)
Traceback (most recent call last):
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 151, in _special_agent_main_core
return main_fn(args)
^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 480, in agent_proxmox_ve_main
logged_backup_data = fetch_backup_data(args, session, data["nodes"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 425, in fetch_backup_data
return collect_vm_backup_info(
^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 378, in collect_vm_backup_info
for task in backup_tasks:
^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 430, in <genexpr>
for _timestamp, backup_log in (fetch_backup_log(task, node["node"]),)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 412, in fetch_backup_log
timestamp, logs = cached(
^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/v0_unstable/misc.py", line 389, in setdefault
return cache.setdefault(key, value_fn())
^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 416, in <lambda>
session.get_tree({"nodes": {n: {"tasks": {t["upid"]: {"log": []}}}}})["nodes"][
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 896, in get_tree
return rec_get_tree(None, requested_structure, [])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 862, in rec_get_tree
key: rec_get_tree(key, subtree[key], next_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 873, in rec_get_tree
{key: rec_get_tree(key, subtree[key], next_path) for key in subtree}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 862, in rec_get_tree
key: rec_get_tree(key, subtree[key], next_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 873, in rec_get_tree
{key: rec_get_tree(key, subtree[key], next_path) for key in subtree}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 862, in rec_get_tree
key: rec_get_tree(key, subtree[key], next_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 839, in rec_get_tree
response = self._session.get_api_element("/".join(map(str, next_path)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 704, in get_api_element
return self._get_raw("api2/json/" + path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 714, in _get_raw
self._get_logs_or_tasks_paginated(sub_url)
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 742, in _get_logs_or_tasks_paginated
assert isinstance(response_data, Sequence)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Output of “cmk --debug -vvn fakehost.fakedomain.com”:
value store: loading from disk
Checkmk version 2.4.0p10
+ FETCHING DATA
Source: SourceInfo(hostname='fakehost.fakedomain.com', ipaddress='XXX.XX.XXX.XX', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7b8e1fca4680]
Read from cache: AgentFileCache(path_template=/omd/sites/checkmkprod/tmp/check_mk/cache/fakehost.fakedomain.com, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 17 sec, allowed is 0 sec)
Connecting via TCP to XXX.XX.XXX.XX:6556 (10.0s timeout)
Detected transport protocol: TransportProtocol.PBKDF2_MAC
Reading data from agent
Try to decrypt output
Closing TCP connection to XXX.XX.XXX.XX:6556
Write data to cache file /omd/sites/checkmkprod/tmp/check_mk/cache/fakehost.fakedomain.com
Trying to acquire lock on /omd/sites/checkmkprod/tmp/check_mk/cache/fakehost.fakedomain.com
Got lock on /omd/sites/checkmkprod/tmp/check_mk/cache/fakehost.fakedomain.com
Releasing lock on /omd/sites/checkmkprod/tmp/check_mk/cache/fakehost.fakedomain.com
Released lock on /omd/sites/checkmkprod/tmp/check_mk/cache/fakehost.fakedomain.com
[cpu_tracking] Stop [7b8e1fca4680 - Snapshot(process=posix.times_result(user=0.4099999999999999, system=0.0, children_user=0.0, children_system=0.0, elapsed=2.959999999962747))]
Source: SourceInfo(hostname='fakehost.fakedomain.com', ipaddress='XXX.XX.XXX.XX', ident='special_proxmox_ve', fetcher_type=<FetcherType.SPECIAL_AGENT: 6>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7b8e1fca7050]
Read from cache: AgentFileCache(path_template=/omd/sites/checkmkprod/tmp/check_mk/data_source_cache/special_proxmox_ve/fakehost.fakedomain.com, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (does not exist)
Calling: /omd/sites/checkmkprod/share/check_mk/agents/special/agent_proxmox_ve --pwstore=4@0@/omd/sites/checkmkprod/var/check_mk/passwords_merged@uuidf1a1d9fd-9a13-44f4-a8ec-fd2ca54f4ab4 -u audit@pve -p '**************' --port 8006 --no-cert-check --timeout 240 --log-cutoff-weeks 1 fakehost.fakedomain.com
Get data from program
[cpu_tracking] Stop [7b8e1fca7050 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=3.76, children_system=0.18, elapsed=14.700000000186265))]
Source: SourceInfo(hostname='fakehost.fakedomain.com', ipaddress='XXX.XX.XXX.XX', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7b8e1fe06210]
Read from cache: NoCache(path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
0 piggyback files for 'fakehost.fakedomain.com'.
0 piggyback files for 'XXX.XX.XXX.XX'.
Get piggybacked data
[cpu_tracking] Stop [7b8e1fe06210 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7b8e2093cf80]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool_status>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_thermal:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_nodes>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1756808695,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<ipmi:cached(1756808700,300):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<ipmi_discrete:cached(1756808697,300):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<ipmi_sensors:cached(1756808635,300):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<local>>> / Transition HostSectionParser -> HostSectionParser
<<<local>>> / Transition HostSectionParser -> HostSectionParser
<<<lvm_vgs>>> / Transition HostSectionParser -> HostSectionParser
<<<lvm_lvs:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<logins>>> / Transition HostSectionParser -> HostSectionParser
<<<smart:cached(1756808334,1200)>>> / Transition HostSectionParser -> HostSectionParser
<<<apache_status:cached(1756808334,600):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<zfs_arc_cache:cached(1756808334,600)>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe:cached(1756808334,600)>>> / Transition HostSectionParser -> HostSectionParser
<<<netstat:cached(1756808334,600)>>> / Transition HostSectionParser -> HostSectionParser
<<<nginx_status:cached(1756808334,600)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1756741413,86400):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<dir_size:cached(1756731993,86400)>>> / Transition HostSectionParser -> HostSectionParser
Using persisted section SectionName('lnx_packages')
Using persisted section SectionName('lnx_distro')
Using persisted section SectionName('lnx_cpuinfo')
Using persisted section SectionName('dmidecode')
Using persisted section SectionName('lnx_uname')
Using persisted section SectionName('lnx_video')
Using persisted section SectionName('lnx_ip_r')
Using persisted section SectionName('lnx_sysctl')
Using persisted section SectionName('lnx_block_devices')
HostKey(hostname='fakehost.fakedomain.com', source_type=<SourceType.HOST: 1>) -> Add sections: ['apache_status', 'apt', 'check_mk', 'checkmk_agent_plugins_lnx', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df_v2', 'dir_size', 'diskstat', 'dmidecode', 'ipmi', 'ipmi_discrete', 'ipmi_sensors', 'job', 'kernel', 'labels', 'lnx_block_devices', 'lnx_bonding', 'lnx_cpuinfo', 'lnx_distro', 'lnx_if', 'lnx_ip_r', 'lnx_packages', 'lnx_sysctl', 'lnx_thermal', 'lnx_uname', 'lnx_video', 'local', 'logins', 'lvm_lvs', 'lvm_vgs', 'md', 'mem', 'mounts', 'mrpe', 'netstat', 'nfsmounts_v2', 'nginx_status', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'pvecm_nodes', 'pvecm_status', 'smart', 'systemd_units', 'tcp_conn_stats', 'uptime', 'zfs_arc_cache', 'zfsget', 'zpool', 'zpool_status']
HostKey(hostname='fakehost.fakedomain.com', source_type=<SourceType.HOST: 1>) -> Add sections: []
Received no piggyback data
APT Updates 10 normal updates (fluent-bit, libpve-common-perl, proxmox-backup-client, proxmox-backup-file-restore, libpve-storage-perl, proxmox-kernel-helper, proxmox-widget-toolkit, pve-container, pve-docs, pve-manager), 2 security updates (libunbound8, libxml2)
APTSOURCE nothing todo, Cache generated 6 minutes 59 seconds ago, cache interval: 20 minutes 0 seconds, elapsed cache lifespan: 34.92%
Bonding Interface bond0 Status: up, Mode: IEEE 802.3ad Dynamic link aggregation, eno12399np0/94:6D:AE:9B:73:44 up, eno12409np1/94:6D:AE:9B:73:45 up
CPU load 15 min load: 4.53, 15 min load per core: 0.14 (32 cores)
CPU utilization Total CPU: 33.89%
CRON OK: no errors found
Check_MK Agent Version: 2.4.0p10, OS: linux, TLS is not activated on monitored host (see details), Agent plug-ins: 20, Local checks: 52
Disk IO SUMMARY Read: 1.04 MB/s, Write: 8.55 MB/s, Latency: 16 microseconds
Filesystem / Used: 33.59% - 31.6 GiB of 93.9 GiB, trend per 1 day 0 hours: +518 MiB, trend per 1 day 0 hours: +0.54%, Time left until disk full: 123 days 7 hours
Filesystem /boot/efi Used: 0.03% - 356 KiB of 1022 MiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Filesystem /etc/pve Used: 0.35% - 464 KiB of 128 MiB, trend per 1 day 0 hours: -2.74 KiB, trend per 1 day 0 hours: -0.00%
Filesystem /sys/firmware/efi/efivars Used: 28.95% - 88.0 KiB of 304 KiB, trend per 1 day 0 hours: +10.3 KiB, trend per 1 day 0 hours: +3.38%, Time left until disk full: 21 days 1 hour
IPMI Sensor Exhaust_Temp Status: OK, 29.00 C
IPMI Sensor Fan1A Status: OK, 16560.00 RPM
IPMI Sensor Fan1B Status: OK, 15720.00 RPM
IPMI Sensor Fan1C Status: OK, 16920.00 RPM
IPMI Sensor Fan1D Status: OK, 15720.00 RPM
IPMI Sensor Fan2A Status: OK, 16680.00 RPM
IPMI Sensor Fan2B Status: OK, 15600.00 RPM
IPMI Sensor Fan2C Status: OK, 16920.00 RPM
IPMI Sensor Fan2D Status: OK, 15600.00 RPM
IPMI Sensor Fan3A Status: OK, 16920.00 RPM
IPMI Sensor Fan3B Status: OK, 15600.00 RPM
IPMI Sensor Fan3C Status: OK, 16680.00 RPM
IPMI Sensor Fan3D Status: OK, 15360.00 RPM
IPMI Sensor Fan4A Status: OK, 17040.00 RPM
IPMI Sensor Fan4B Status: OK, 15600.00 RPM
IPMI Sensor Fan4C Status: OK, 17040.00 RPM
IPMI Sensor Fan4D Status: OK, 15600.00 RPM
IPMI Sensor Fan_Redundancy Status: Fully Redundant
IPMI Sensor Inlet_Temp Status: OK, 23.00 C
IPMI Sensor Summary 210 sensors in total, 142 sensors ok, 68 sensors skipped
IPMI Sensor Temp Status: OK, 54.00 C
Interface 02 [bond0], (up), MAC: 94:6D:AE:9B:73:44, Speed: 50 GBit/s, In: 19.6 MB/s (0.31%), Out: 22.8 MB/s (0.36%)
Interface 04 [vmbr0], (up), MAC: C4:CB:E1:A6:9E:80, Speed: 10 GBit/s, In: 49.2 kB/s (<0.01%), Out: 97.2 kB/s (<0.01%)
Interface 05 [vmbr1], (up), MAC: 94:6D:AE:9B:73:44, Speed: 50 GBit/s, In: 164 kB/s (<0.01%), Out: 162 kB/s (<0.01%)
Interface 06 [vmbr2], (up), MAC: 94:6D:AE:9B:73:44, Speed: 50 GBit/s, In: 36.9 kB/s (<0.01%), Out: 37.5 kB/s (<0.01%)
Interface 07 [bond0.479], (up), MAC: 94:6D:AE:9B:73:44, Speed: 50 GBit/s, In: 17.8 MB/s (0.28%), Out: 21.7 MB/s (0.35%)
Interface 08 [eno12399np0], (up), MAC: 94:6D:AE:9B:73:44, Speed: 25 GBit/s, In: 16.8 MB/s (0.54%), Out: 11.3 MB/s (0.36%)
Interface 09 [eno12409np1], (up), MAC: 94:6D:AE:9B:73:45, Speed: 25 GBit/s, In: 2.83 MB/s (0.09%), Out: 11.4 MB/s (0.37%)
Interface 10 [eno8303], (up), MAC: C4:CB:E1:A6:9E:80, Speed: 1 GBit/s, In: 373 kB/s (0.30%), Out: 377 kB/s (0.30%)
Interface 12 [fwbr12600i0], (up), MAC: 32:42:C9:C7:22:69, Speed: 10 GBit/s, In: 21.2 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 13 [fwbr12608i0], (up), MAC: 5A:B9:DE:6D:FE:3B, Speed: 10 GBit/s, In: 21.2 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 14 [fwbr12618i0], (up), MAC: 42:E2:C3:02:02:CA, Speed: 10 GBit/s, In: 21.2 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 15 [fwbr12622i0], (up), MAC: 76:AC:F2:44:25:90, Speed: 10 GBit/s, In: 21.2 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 16 [fwbr12624i0], (up), MAC: 72:CE:1C:2F:20:9D, Speed: 10 GBit/s, In: 21.2 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 17 [fwbr12629i0], (up), MAC: 96:3F:5C:26:10:8B, Speed: 10 GBit/s, In: 21.2 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 18 [fwbr12644i0], (up), MAC: 12:07:82:65:97:8A, Speed: 10 GBit/s, In: 21.1 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 19 [fwbr12649i0], (up), MAC: AA:3D:A1:4B:98:5A, Speed: 10 GBit/s, In: 21.1 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 20 [fwbr12652i0], (up), MAC: 9A:A7:16:38:94:23, Speed: 10 GBit/s, In: 21.1 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 21 [fwbr12655i0], (up), MAC: EA:71:9B:84:9A:46, Speed: 10 GBit/s, In: 6.71 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 22 [fwbr12655i1], (up), MAC: E6:A7:94:5B:57:3A, Speed: 10 GBit/s, In: 21.1 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 23 [fwbr12902i0], (up), MAC: 3E:17:87:B3:7F:D8, Speed: 10 GBit/s, In: 21.1 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 24 [fwbr13305i0], (up), MAC: AE:48:9E:9C:74:12, Speed: 10 GBit/s, In: 6.71 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 25 [fwbr13305i1], (up), MAC: E2:48:08:DC:D1:0A, Speed: 10 GBit/s, In: 21.1 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 26 [fwbr13312i0], (up), MAC: F2:5C:F8:C0:81:06, Speed: 10 GBit/s, In: 21.1 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 27 [fwbr13601i0], (up), MAC: 22:C5:F1:CF:82:8A, Speed: 10 GBit/s, In: 6.71 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 28 [fwbr13604i0], (up), MAC: 5E:DE:08:C3:27:F7, Speed: 10 GBit/s, In: 6.71 kB/s (<0.01%), Out: 0.00 B/s (0%)
Interface 29 [fwln12600i0], (up), MAC: 32:42:C9:C7:22:69, Speed: 10 GBit/s, In: 58.6 kB/s (<0.01%), Out: 49.6 kB/s (<0.01%)
Interface 30 [fwln12608i0], (up), MAC: 5A:B9:DE:6D:FE:3B, Speed: 10 GBit/s, In: 1.20 MB/s (0.10%), Out: 378 kB/s (0.03%)
Interface 31 [fwln12618i0], (up), MAC: 42:E2:C3:02:02:CA, Speed: 10 GBit/s, In: 26.9 kB/s (<0.01%), Out: 9.09 kB/s (<0.01%)
Interface 32 [fwln12622i0], (up), MAC: 76:AC:F2:44:25:90, Speed: 10 GBit/s, In: 23.6 kB/s (<0.01%), Out: 4.40 kB/s (<0.01%)
Interface 33 [fwln12624i0], (up), MAC: 72:CE:1C:2F:20:9D, Speed: 10 GBit/s, In: 70.4 kB/s (<0.01%), Out: 56.9 kB/s (<0.01%)
Interface 34 [fwln12629i0], (up), MAC: 96:3F:5C:26:10:8B, Speed: 10 GBit/s, In: 33.1 kB/s (<0.01%), Out: 9.11 kB/s (<0.01%)
Interface 35 [fwln12644i0], (up), MAC: 12:07:82:65:97:8A, Speed: 10 GBit/s, In: 68.0 kB/s (<0.01%), Out: 60.9 kB/s (<0.01%)
Interface 36 [fwln12649i0], (up), MAC: AA:3D:A1:4B:98:5A, Speed: 10 GBit/s, In: 10.6 MB/s (0.84%), Out: 8.38 MB/s (0.67%)
Interface 37 [fwln12652i0], (up), MAC: 9A:A7:16:38:94:23, Speed: 10 GBit/s, In: 27.2 kB/s (<0.01%), Out: 50.8 kB/s (<0.01%)
Interface 38 [fwln12655i0], (up), MAC: EA:71:9B:84:9A:46, Speed: 10 GBit/s, In: 11.3 kB/s (<0.01%), Out: 10.3 kB/s (<0.01%)
Interface 39 [fwln12655i1], (up), MAC: E6:A7:94:5B:57:3A, Speed: 10 GBit/s, In: 23.5 kB/s (<0.01%), Out: 6.77 kB/s (<0.01%)
Interface 40 [fwln12902i0], (up), MAC: 3E:17:87:B3:7F:D8, Speed: 10 GBit/s, In: 701 kB/s (0.06%), Out: 1.65 MB/s (0.13%)
Interface 41 [fwln13305i0], (up), MAC: AE:48:9E:9C:74:12, Speed: 10 GBit/s, In: 9.54 kB/s (<0.01%), Out: 14.7 kB/s (<0.01%)
Interface 42 [fwln13305i1], (up), MAC: E2:48:08:DC:D1:0A, Speed: 10 GBit/s, In: 21.7 kB/s (<0.01%), Out: 153 B/s (<0.01%)
Interface 43 [fwln13312i0], (up), MAC: F2:5C:F8:C0:81:06, Speed: 10 GBit/s, In: 289 kB/s (0.02%), Out: 4.27 MB/s (0.34%)
Interface 44 [fwln13601i0], (up), MAC: 22:C5:F1:CF:82:8A, Speed: 10 GBit/s, In: 157 kB/s (0.01%), Out: 103 kB/s (<0.01%)
Interface 45 [fwln13604i0], (up), MAC: 5E:DE:08:C3:27:F7, Speed: 10 GBit/s, In: 176 kB/s (0.01%), Out: 148 kB/s (0.01%)
Interface 46 [fwpr12600p0], (up), MAC: A6:F4:84:63:70:4D, Speed: 10 GBit/s, In: 49.6 kB/s (<0.01%), Out: 58.6 kB/s (<0.01%)
Interface 47 [fwpr12608p0], (up), MAC: 4A:5E:29:4A:57:2F, Speed: 10 GBit/s, In: 378 kB/s (0.03%), Out: 1.20 MB/s (0.10%)
Interface 48 [fwpr12618p0], (up), MAC: 2A:64:D7:61:A7:DB, Speed: 10 GBit/s, In: 9.09 kB/s (<0.01%), Out: 26.9 kB/s (<0.01%)
Interface 49 [fwpr12622p0], (up), MAC: EE:5D:BD:A1:FA:69, Speed: 10 GBit/s, In: 4.40 kB/s (<0.01%), Out: 23.6 kB/s (<0.01%)
Interface 50 [fwpr12624p0], (up), MAC: 82:F5:C9:9C:9F:FA, Speed: 10 GBit/s, In: 56.9 kB/s (<0.01%), Out: 70.4 kB/s (<0.01%)
Interface 51 [fwpr12629p0], (up), MAC: 36:EB:63:AC:87:D2, Speed: 10 GBit/s, In: 9.10 kB/s (<0.01%), Out: 33.1 kB/s (<0.01%)
Interface 52 [fwpr12644p0], (up), MAC: FA:89:69:EA:F7:09, Speed: 10 GBit/s, In: 60.9 kB/s (<0.01%), Out: 68.0 kB/s (<0.01%)
Interface 53 [fwpr12649p0], (up), MAC: DA:BE:FB:F3:42:BF, Speed: 10 GBit/s, In: 8.38 MB/s (0.67%), Out: 10.6 MB/s (0.84%)
Interface 54 [fwpr12652p0], (up), MAC: 82:C8:29:4F:77:57, Speed: 10 GBit/s, In: 50.8 kB/s (<0.01%), Out: 27.2 kB/s (<0.01%)
Interface 55 [fwpr12655p0], (up), MAC: 5E:F5:51:13:B8:50, Speed: 10 GBit/s, In: 10.3 kB/s (<0.01%), Out: 11.3 kB/s (<0.01%)
Interface 56 [fwpr12655p1], (up), MAC: A2:62:DF:2C:1E:D4, Speed: 10 GBit/s, In: 6.77 kB/s (<0.01%), Out: 23.5 kB/s (<0.01%)
Interface 57 [fwpr12902p0], (up), MAC: BA:15:FE:08:BF:49, Speed: 10 GBit/s, In: 1.65 MB/s (0.13%), Out: 701 kB/s (0.06%)
Interface 58 [fwpr13305p0], (up), MAC: 1A:C4:12:F2:09:1F, Speed: 10 GBit/s, In: 14.7 kB/s (<0.01%), Out: 9.54 kB/s (<0.01%)
Interface 59 [fwpr13305p1], (up), MAC: 96:01:71:78:AD:3A, Speed: 10 GBit/s, In: 153 B/s (<0.01%), Out: 21.7 kB/s (<0.01%)
Interface 60 [fwpr13312p0], (up), MAC: 06:57:2A:C8:05:AB, Speed: 10 GBit/s, In: 4.27 MB/s (0.34%), Out: 289 kB/s (0.02%)
Interface 61 [fwpr13601p0], (up), MAC: 4A:1C:8E:39:D3:27, Speed: 10 GBit/s, In: 103 kB/s (<0.01%), Out: 157 kB/s (0.01%)
Interface 62 [fwpr13604p0], (up), MAC: 3E:1A:F6:AC:D8:B3, Speed: 10 GBit/s, In: 148 kB/s (0.01%), Out: 176 kB/s (0.01%)
Interface 63 [tap12600i0], (up), MAC: 4E:4B:2F:A5:50:D7, Speed: 10 GBit/s, In: 49.6 kB/s (<0.01%), Out: 38.3 kB/s (<0.01%)
Interface 64 [tap12608i0], (up), MAC: FE:D7:94:8B:4B:F2, Speed: 10 GBit/s, In: 378 kB/s (0.03%), Out: 1.20 MB/s (0.10%)
Interface 65 [tap12608i1], (up), MAC: CE:65:66:A5:A2:A3, Speed: 10 GBit/s, In: 0.00 B/s (0%), Out: 32.3 B/s (<0.01%)
Interface 66 [tap12618i0], (up), MAC: D6:CC:CC:5E:5E:28, Speed: 10 GBit/s, In: 9.09 kB/s (<0.01%), Out: 6.63 kB/s (<0.01%)
Interface 67 [tap12624i0], (up), MAC: 1A:B6:A7:D0:2B:D2, Speed: 10 GBit/s, In: 56.9 kB/s (<0.01%), Out: 50.2 kB/s (<0.01%)
Interface 68 [tap12624i1], (up), MAC: AE:47:ED:91:76:DD, Speed: 10 GBit/s, In: 158 kB/s (0.01%), Out: 285 kB/s (0.02%)
Interface 69 [tap12649i0], (up), MAC: 52:63:79:A4:53:BF, Speed: 10 GBit/s, In: 8.38 MB/s (0.67%), Out: 10.9 MB/s (0.87%)
Interface 70 [tap12652i0], (up), MAC: F6:74:4F:FB:58:EB, Speed: 10 GBit/s, In: 50.8 kB/s (<0.01%), Out: 27.2 kB/s (<0.01%)
Interface 71 [tap12655i0], (up), MAC: AE:D5:A6:9B:C1:FD, Speed: 10 GBit/s, In: 10.3 kB/s (<0.01%), Out: 11.3 kB/s (<0.01%)
Interface 72 [tap12655i1], (up), MAC: 86:A9:DD:47:11:58, Speed: 10 GBit/s, In: 6.77 kB/s (<0.01%), Out: 3.23 kB/s (<0.01%)
Interface 73 [tap12902i0], (up), MAC: E6:67:43:B9:C8:96, Speed: 10 GBit/s, In: 1.65 MB/s (0.13%), Out: 682 kB/s (0.05%)
Kernel Performance Process Creations: 114.40/s, Context Switches: 182877.55/s, Major Page Faults: 0.00/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
LVM LV Pool pve/data Data usage: 43.04%, Meta usage: 1.87%
LVM VG pve Used: 99.08% - 1.73 TiB of 1.74 TiB, trend: 0 B / 24 hours
Logins On system: 0
MAILTRANSPORT OK, no stuck mails, Cache generated 19 minutes 19 seconds ago, cache interval: 1 hour 0 minutes, elapsed cache lifespan: 32.20%
Memory Total virtual memory: 43.30% - 54.3 GiB of 125 GiB, 8 additional details available
Mount options of / Mount options exactly as expected
Mount options of /boot/efi Mount options exactly as expected
Mount options of /etc/pve Mount options exactly as expected
NTP Time Offset: 0.0236 ms, Stratum: 3, Time since last sync: 1 minute 3 seconds
Number of threads 1791, Usage: 0.17%
OOM everything okay, Cache generated 6 minutes 59 seconds ago, cache interval: 10 minutes 0 seconds, elapsed cache lifespan: 69.84%
PROXMOX_DEFAULTS Everything okay
PVEFW_fakehost OK 12902, Cache generated 42 minutes 14 seconds ago, cache interval: 1 hour 0 minutes, elapsed cache lifespan: 70.39%
PVE_UnusedDisks OK: All disks are assigned to VMs, Cache generated 5 hours 16 minutes ago, cache interval: 6 hours 0 minutes, elapsed cache lifespan: 87.99%
Postfix Queue default Deferred queue length: 0, Active queue length: 0
Postfix status default Status: the Postfix mail system is running, PID: 2010
Process CRON Processes: 5, Virtual memory: 26.5 MiB, Resident memory: 13.2 MiB, CPU: 0%, Youngest running for: 12 minutes 6 seconds, Oldest running for: 12 minutes 41 seconds
Process RSYSLOGD Processes: 5, Virtual memory: 849 MiB, Resident memory: 24.8 MiB, CPU: 0%, Youngest running for: 12 minutes 7 seconds, Oldest running for: 12 minutes 47 seconds
Process ZOMBIE Processes: 0
Proxmox VE Memory Usage PEND Check plug-in received no monitoring data
Proxmox VE Node Info PEND Check plug-in received no monitoring data
Size of /var/log Folder size: 363 MiB
Systemd Service Summary Total: 177, Disabled: 9, Failed: 0
Systemd Socket Summary Total: 23, Disabled: 5, Failed: 0
TCP Connections Established: 13
Temperature Zone 0 Temperature: 33.0 °C
Temperature Zone 1 Temperature: 57.0 °C
Temperature Zone 2 Temperature: 58.0 °C
Uptime Up since 2025-09-02 12:12:55, Uptime: 12 minutes 57 seconds
0 piggyback files for 'fakehost.fakedomain.com'.
[cpu_tracking] Stop [7b8e2093cf80 - Snapshot(process=posix.times_result(user=0.31000000000000005, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.3200000002980232))]
value store: updating
Trying to acquire lock on /omd/sites/checkmkprod/tmp/check_mk/counters/fakehost.fakedomain.com
Got lock on /omd/sites/checkmkprod/tmp/check_mk/counters/fakehost.fakedomain.com
value store: already loaded
value store: writing to disk
Releasing lock on /omd/sites/checkmkprod/tmp/check_mk/counters/fakehost.fakedomain.com
Released lock on /omd/sites/checkmkprod/tmp/check_mk/counters/fakehost.fakedomain.com
[agent] Success, [special_proxmox_ve] AssertionError(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for plugins, proxmox_ve_mem_usage(!), proxmox_ve_node_info(!), execution time 18.0 sec | execution_time=17.980 user_time=0.720 system_time=0.010 children_user_time=3.760 children_system_time=0.180 cmk_time_agent=2.550 cmk_time_ds=10.760
Agent exited with code 1: ERROR 2025-09-02 12:25:51 root: Error in log at line 24: LogParseWarning(24, "End of VM '12600' while still information is missing (we have: {'total_duration', 'started_time'})")
ERROR 2025-09-02 12:25:51 root: Error in log at line 473: LogParseWarning(473, "End of VM '12902' while still information is missing (we have: {'total_duration', 'started_time'})")
ERROR 2025-09-02 12:25:51 agent_proxmox_ve: Parsing the log for UPID='UPID:fakehost:003D0605:07001E75:68B63383:vzdump::root@pam:' resulted in a error(s) - write log content to '/omd/sites/checkmkprod/tmp/check_mk/special_agents/agent_proxmox_ve/erroneous-UPID:fakehost:003D0605:07001E75:68B63383:vzdump::root@pam:.log'
Agent failed - please submit a crash report! (Crash-ID: 336a64ec-87e7-11f0-a753-005056009492)
Traceback (most recent call last):
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 151, in _special_agent_main_core
return main_fn(args)
^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 480, in agent_proxmox_ve_main
logged_backup_data = fetch_backup_data(args, session, data["nodes"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 425, in fetch_backup_data
return collect_vm_backup_info(
^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 378, in collect_vm_backup_info
for task in backup_tasks:
^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 430, in <genexpr>
for _timestamp, backup_log in (fetch_backup_log(task, node["node"]),)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 412, in fetch_backup_log
timestamp, logs = cached(
^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/v0_unstable/misc.py", line 389, in setdefault
return cache.setdefault(key, value_fn())
^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 416, in <lambda>
session.get_tree({"nodes": {n: {"tasks": {t["upid"]: {"log": []}}}}})["nodes"][
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 896, in get_tree
return rec_get_tree(None, requested_structure, [])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 862, in rec_get_tree
key: rec_get_tree(key, subtree[key], next_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 873, in rec_get_tree
{key: rec_get_tree(key, subtree[key], next_path) for key in subtree}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 862, in rec_get_tree
key: rec_get_tree(key, subtree[key], next_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 873, in rec_get_tree
{key: rec_get_tree(key, subtree[key], next_path) for key in subtree}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 862, in rec_get_tree
key: rec_get_tree(key, subtree[key], next_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 839, in rec_get_tree
response = self._session.get_api_element("/".join(map(str, next_path)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 704, in get_api_element
return self._get_raw("api2/json/" + path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 714, in _get_raw
self._get_logs_or_tasks_paginated(sub_url)
File "/omd/sites/checkmkprod/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 742, in _get_logs_or_tasks_paginated
assert isinstance(response_data, Sequence)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError(!!)
I came across the following Quickfix, but unfortunately it didn’t work for me:
Last Friday, I updated both of my Checkmk instances from patch level p8 to p10. Everything seemed to be working fine at first — I suspect Checkmk was still using cached data at that point.
However, when I checked again on Monday, the issue became apparent.
We also have a second, much smaller Proxmox cluster where the agent is working correctly.
When I run the following command:
/omd/sites/checkmkprod/share/check_mk/agents/special/agent_proxmox_ve \
--username audit@pve \
--password 'XXXXX' \
--port 8006 \
--no-cert-check \
--timeout 120 \
--log-cutoff-weeks 1 \
fakehost.fakedomain.com \
--vcrtrace /opt/omd/sites/checkmkprod/tmp/debug/test
I receive an output of about 1MB, which looks fine at first glance.