I’m facing an issue with a fresh install of a Proxmox cluster and Checkmk RAW. I can’t get it working and I’m seeing the errors shown below
Could someone please help me identify the cause and get this working?
I already checked that the VM is not locked (VMID 112).
CMK version: 2.4.0p18.cre - latest
OS version: debian 12.12 - all updates applied
Error message:
CHECKMK
[agent] Success, [special_proxmox_ve] StopIterationCRIT, [piggyback] Success (but no data found for this host), execution time 3.5 sec
CHECKMK DISCOVERY
Services: all up to date, Host labels: all up to date, [special_proxmox_ve] StopIterationCRIT, execution time 1.6 sec
Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
cmk --debug -vvn host01.example.invalid
value store: loading from disk
Checkmk version X.Y.ZpNN
+ FETCHING DATA
Source: SourceInfo(hostname='host01.example.invalid', ipaddress='10.0.0.21', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [THREAD_ID_1]
Read from cache: AgentFileCache(path_template=/omd/sites/SITE/tmp/check_mk/cache/host01.example.invalid, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is NN sec, allowed is 0 sec)
Connecting via TCP to 10.0.0.21:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.PLAIN
Reading data from agent
Closing TCP connection to 10.0.0.21:6556
Write data to cache file /omd/sites/SITE/tmp/check_mk/cache/host01.example.invalid
Trying to acquire lock on /omd/sites/SITE/tmp/check_mk/cache/host01.example.invalid
Got lock on /omd/sites/SITE/tmp/check_mk/cache/host01.example.invalid
Releasing lock on /omd/sites/SITE/tmp/check_mk/cache/host01.example.invalid
Released lock on /omd/sites/SITE/tmp/check_mk/cache/host01.example.invalid
[cpu_tracking] Stop [THREAD_ID_1 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=1.87))]
Source: SourceInfo(hostname='host01.example.invalid', ipaddress='10.0.0.21', ident='special_proxmox_ve', fetcher_type=<FetcherType.SPECIAL_AGENT: 6>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [THREAD_ID_2]
Read from cache: AgentFileCache(path_template=/omd/sites/SITE/tmp/check_mk/data_source_cache/special_proxmox_ve/host01.example.invalid, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (does not exist)
Calling: /omd/sites/SITE/share/check_mk/agents/special/agent_proxmox_ve --pwstore=REDACTED@/omd/sites/SITE/var/check_mk/passwords_merged@UUID_REDACTED -u user@realm -p '********' --no-cert-check host01.example.invalid
Get data from program
[cpu_tracking] Stop [THREAD_ID_2 - Snapshot(process=posix.times_result(user=0.01, system=0.0, children_user=1.07, children_system=0.03, elapsed=1.48))]
Source: SourceInfo(hostname='host01.example.invalid', ipaddress='10.0.0.21', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [THREAD_ID_3]
Read from cache: NoCache(path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
0 piggyback files for 'host01.example.invalid'.
0 piggyback files for '10.0.0.21'.
Get piggybacked data
[cpu_tracking] Stop [THREAD_ID_3 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [THREAD_ID_4]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool_status>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_thermal:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_nodes>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(EPOCH_REDACTED,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
HostKey(hostname='host01.example.invalid', source_type=<SourceType.HOST: 1>) -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df_v2', 'diskstat', 'job', 'kernel', 'labels', 'lnx_bonding', 'lnx_if', 'lnx_thermal', 'local', 'md', 'mem', 'mounts', 'nfsmounts_v2', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'pvecm_nodes', 'pvecm_status', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest', 'zfsget', 'zpool', 'zpool_status']
HostKey(hostname='host01.example.invalid', source_type=<SourceType.HOST: 1>) -> Add sections: []
Received no piggyback data
Bonding Interface bond0 Status: up, Mode: fault-tolerance (active-backup), Primary: nicX, nicX/MAC_REDACTED up, nicY/MAC_REDACTED down(!)
Bonding Interface bond1 Status: up, Mode: fault-tolerance (active-backup), Primary: nicA, nicA/MAC_REDACTED up, nicB/MAC_REDACTED up
CPU load 15 min load: 2.40, 15 min load per core: 0.04 (NN cores)
CPU utilization Total CPU: 4.43%
Check_MK Agent Version: X.Y.ZpNN, OS: linux, TLS is not activated on monitored host (see details)(!), Agent plug-ins: 0, Local checks: 0
Disk IO SUMMARY Read: 120 MB/s, Write: 3.99 MB/s, Latency: 103 microseconds
Filesystem / Used: 0.70% - 3.03 GiB of 430 GiB, trend per 1 day 0 hours: +18.0 MiB, trend per 1 day 0 hours: +<0.01%
Filesystem /etc/pve Used: 0.03% - 44.0 KiB of 128 MiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Filesystem /rpool Used: 0.71% - 3.05 GiB of 430 GiB, trend per 1 day 0 hours: +16.3 MiB, trend per 1 day 0 hours: +<0.01%
Filesystem /rpool/ROOT Used: 0.70% - 3.03 GiB of 430 GiB, trend per 1 day 0 hours: +18.0 MiB, trend per 1 day 0 hours: +<0.01%
Filesystem /rpool/data Used: <0.01% - 96.0 KiB of 427 GiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Filesystem /sys/firmware/efi/efivars Used: 31.05% - 159 KiB of 512 KiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Filesystem /var/lib/vz Used: <0.01% - 96.0 KiB of 427 GiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Interface 02 [nic0], (up), MAC: MAC_REDACTED_01, Speed: 25 GBit/s, In: 785 kB/s (0.03%), Out: 1.45 MB/s (0.05%)
Interface 03 [nic1], (up), MAC: MAC_REDACTED_02, Speed: 25 GBit/s, In: 1.16 MB/s (0.04%), Out: 1.79 MB/s (0.06%)
Interface 04 [nic2], (up), MAC: MAC_REDACTED_03, Speed: 10 GBit/s, In: 20.3 kB/s (<0.01%), Out: 14.5 kB/s (<0.01%)
Interface 05 [nic3], (up), MAC: MAC_REDACTED_04, Speed: 1 GBit/s, In: 143 B/s (<0.01%), Out: 0.00 B/s (0%)
Interface 06 [nic4], (up), MAC: MAC_REDACTED_05, Speed: 1 GBit/s, In: 196 kB/s (0.16%), Out: 112 kB/s (0.09%)
Interface 08 [bond0], (up), MAC: MAC_REDACTED_05, Speed: 1 GBit/s, In: 196 kB/s (0.16%), Out: 112 kB/s (0.09%)
Interface 09 [bond1], (up), MAC: MAC_REDACTED_03, Speed: 10 GBit/s, In: 20.4 kB/s (<0.01%), Out: 14.5 kB/s (<0.01%)
Interface 10 [vmbr0], (up), MAC: MAC_REDACTED_05, Speed: 10 GBit/s, In: 33.3 kB/s (<0.01%), Out: 49.6 kB/s (<0.01%)
Interface 11 [vmbr1], (up), MAC: MAC_REDACTED_03, Speed: 10 GBit/s, In: 647 B/s (<0.01%), Out: 602 B/s (<0.01%)
Interface 12 [dummy_ceph], (up), MAC: MAC_REDACTED_06, Speed: unknown, In: 0.00 B/s, Out: 0.00 B/s
Interface 14 [fwbrNNNi0], (up), MAC: MAC_REDACTED_07, Speed: 10 GBit/s, In: 241 B/s (<0.01%), Out: 0.00 B/s (0%)
Interface 15 [fwbrNNNi0], (up), MAC: MAC_REDACTED_08, Speed: 10 GBit/s, In: 207 B/s (<0.01%), Out: 0.00 B/s (0%)
Interface 16 [fwbrNNNi0], (up), MAC: MAC_REDACTED_09, Speed: 10 GBit/s, In: 94.0 B/s (<0.01%), Out: 0.00 B/s (0%)
Interface 17 [fwlnNNNi0], (up), MAC: MAC_REDACTED_07, Speed: 10 GBit/s, In: 43.9 kB/s (<0.01%), Out: 32.1 kB/s (<0.01%)
Interface 18 [fwlnNNNi0], (up), MAC: MAC_REDACTED_08, Speed: 10 GBit/s, In: 6.95 kB/s (<0.01%), Out: 736 B/s (<0.01%)
Interface 19 [fwlnNNNi0], (up), MAC: MAC_REDACTED_09, Speed: 10 GBit/s, In: 145 B/s (<0.01%), Out: 101 B/s (<0.01%)
Interface 20 [fwprNNNp0], (up), MAC: MAC_REDACTED_10, Speed: 10 GBit/s, In: 32.1 kB/s (<0.01%), Out: 43.9 kB/s (<0.01%)
Interface 21 [fwprNNNp0], (up), MAC: MAC_REDACTED_11, Speed: 10 GBit/s, In: 736 B/s (<0.01%), Out: 6.95 kB/s (<0.01%)
Interface 22 [fwprNNNp0], (up), MAC: MAC_REDACTED_12, Speed: 10 GBit/s, In: 101 B/s (<0.01%), Out: 145 B/s (<0.01%)
Interface 23 [tapNNNi0], (up), MAC: MAC_REDACTED_13, Speed: 10 GBit/s, In: 2.46 B/s (<0.01%), Out: 281 B/s (<0.01%)
Interface 24 [tapNNNi0], (up), MAC: MAC_REDACTED_14, Speed: 10 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface 25 [tapNNNi0], (up), MAC: MAC_REDACTED_15, Speed: 10 GBit/s, In: 493 B/s (<0.01%), Out: 18.2 kB/s (<0.01%)
Interface 26 [tapNNNi0], (up), MAC: MAC_REDACTED_16, Speed: 10 GBit/s, In: 20.9 kB/s (<0.01%), Out: 103 kB/s (<0.01%)
Interface 27 [tapNNNi0], (up), MAC: MAC_REDACTED_17, Speed: 10 GBit/s, In: 32.1 kB/s (<0.01%), Out: 43.9 kB/s (<0.01%)
Interface 28 [tapNNNi0], (up), MAC: MAC_REDACTED_18, Speed: 10 GBit/s, In: 736 B/s (<0.01%), Out: 6.95 kB/s (<0.01%)
Interface 29 [tapNNNi0], (up), MAC: MAC_REDACTED_19, Speed: 10 GBit/s, In: 7.55 kB/s (<0.01%), Out: 4.10 kB/s (<0.01%)
Interface 30 [tapNNNi0], (up), MAC: MAC_REDACTED_20, Speed: 10 GBit/s, In: 101 B/s (<0.01%), Out: 145 B/s (<0.01%)
Interface 31 [tapNNNi0], (up), MAC: MAC_REDACTED_21, Speed: 10 GBit/s, In: 0.00 B/s (0%), Out: 114 B/s (<0.01%)
Interface 32 [tapNNNi0], (up), MAC: MAC_REDACTED_22, Speed: 10 GBit/s, In: 0.00 B/s (0%), Out: 114 B/s (<0.01%)
Interface 33 [tapNNNi0], (up), MAC: MAC_REDACTED_23, Speed: 10 GBit/s, In: 13.8 B/s (<0.01%), Out: 0.00 B/s (0%)
Interface 34 [tapNNNi0], (up), MAC: MAC_REDACTED_24, Speed: 10 GBit/s, In: 3.28 kB/s (<0.01%), Out: 395 B/s (<0.01%)
Interface 35 [tapNNNi0], (up), MAC: MAC_REDACTED_25, Speed: 10 GBit/s, In: 9.40 kB/s (<0.01%), Out: 494 B/s (<0.01%)
Interface 36 [vmbr0.VLAN], (up), MAC: MAC_REDACTED_05, Speed: 10 GBit/s, In: 32.8 kB/s (<0.01%), Out: 49.6 kB/s (<0.01%)
Interface 37 [vmbr1.VLAN], (up), MAC: MAC_REDACTED_03, Speed: 10 GBit/s, In: 544 B/s (<0.01%), Out: 602 B/s (<0.01%)
Kernel Performance Process Creations: 31.46/s, Context Switches: 41906.52/s, Major Page Faults: 0.03/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Memory Total virtual memory: 55.56% - 105 GiB of 189 GiB, 8 additional details available
Mount options of /etc/pve Mount options exactly as expected
NTP Time Offset: 0.0433 ms, Stratum: 3, Time since last sync: 10 minutes 45 seconds
Number of threads 1860, Usage: 0.12%
PVE Cluster State Name: CLUSTER_NAME_REDACTED, Nodes: 3, No faults
PVE Node host01 (local) ID: 1, Votes: 1
PVE Node host02 ID: 2, Votes: 1
PVE Node host03 ID: 3, Votes: 1
Postfix Queue default Deferred queue length: 0, Active queue length: 0
Postfix status default Status: the Postfix mail system is running, PID: PID_REDACTED
Storage Pool rpool Used: 0.68% - 3.00 GiB of 444 GiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Systemd Service Summary Total: 195, Disabled: 8, Failed: 0
Systemd Socket Summary Total: 24, Disabled: 7, Failed: 0
TCP Connections Established: 711
Temperature Zone 0 Temperature: 50.0 °C
Temperature Zone 1 Temperature: 53.0 °C
Uptime Up since 2025-12-16 16:57:01, Uptime: 6 days 0 hours
zpool status All pools are healthy
0 piggyback files for 'host01.example.invalid'.
[cpu_tracking] Stop [THREAD_ID_4 - Snapshot(process=posix.times_result(user=0.06, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.06))]
[agent] Success, [special_proxmox_ve] StopIteration(!!), [piggyback] Success (but no data found for this host), execution time 3.4 sec | execution_time=3.410 user_time=0.070 system_time=0.000 children_user_time=1.070 children_system_time=0.030 cmk_time_agent=1.870 cmk_time_ds=0.370
Agent exited with code 1: WARNING 2025-12-22 17:39:37 agent_proxmox_ve: Found error for VM 'VMID_REDACTED': "can't lock file '/var/lock/qemu-server/lock-VMID_REDACTED.conf' - got timeout"
Agent failed - please submit a crash report! (Crash-ID: CRASH_ID_REDACTED)
Traceback (most recent call last):
File "/omd/sites/SITE/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 532, in agent_proxmox_ve_main
tz = node_timezones[all_vms[vmid]["node"]]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'host01'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/omd/sites/SITE/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 151, in _special_agent_main_core
return main_fn(args)
^^^^^^^^^^^^^
File "/omd/sites/SITE/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 535, in agent_proxmox_ve_main
tz = next(iter(node_timezones.values()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration(!!)