System unable to find Proxmox servers after running fine, update to 2.3 made it worse

CMK version:
2.3.0p9.cre
OS version:
Debian GNU/Linux 12 (bookworm)
Error message:
[special_proxmox_ve]: ValueError: could not convert string to float: '8.41 s (average 17.11 MiB/'CRIT

So I have a checkmk instance that’s been running for a few months, I did some rearranging of my network, however everything but CMK was totally fine. I dug into why it wasn’t discovering 2 Proxmox servers through these steps:

systemctl status check-mk-agent.socket

systemctl status check-mk-agent-async.service

systemctl status cmk-agent-ctl-daemon.service

all enabled and active/running!

From checkMK cli:

ping hostnames to ensure resolving

NAS →

  user@checkMK:~$ ping NAS
  PING NAS (10.20.20.235): 56 data bytes
  64 bytes from 10.20.20.235: icmp_seq=0 ttl=64 time=0.207 ms
  64 bytes from 10.20.20.235: icmp_seq=1 ttl=64 time=0.404 ms
  64 bytes from 10.20.20.235: icmp_seq=2 ttl=64 time=0.399 ms

Check listening on port:

ss -tulpn | grep 6556

tcp   LISTEN 0  4096   *:6556   *:*    users:(("cmk-agent-ctl",pid=2518,fd=9))

Check relationship status:

root@NAS:~# cmk-agent-ctl status
Version: 2.2.0p25
Agent socket: operational
IP allowlist: any
Legacy mode: enabled
No connections

When I run Discovery:

[special_proxmox_ve]: Agent exited with code 1: Agent failed - please submit a crash report! (Crash-ID: 48b91b26-3fa0-11ef-ab00-0bd704d6ebe2)

I didn’t find anything on that Crash-ID, so I removed all hosts and tried readding PM and started getting the ValueError. I updated from 2.2 to 2.3 to see if there were dependency changes that broke it, and that didn’t work. I destroyed the LXC and rebuilt with 2.3 directly, and still getting the ValueError. I really don’t know what else to try.

Thanks for taking the time to help a newb!

Wanted to add, I did follow the checkmk tutorial video on initial setup and this time to verify I did everything the same.

This output comes not from the “normal” agent but from the Proxmox special agent.
It looks like your Proxmox host is only configured to communicate with the special agent.
Can you pls check the host configuration of this host.
It should be configured this way


Then it will use the Proxmox special agent and the normal CheckMK agent.
But i would also inspect why your special agent is not working.
You can get the command line of the special agent with “cmk -D proxmoxhost”

Thank you, I didn’t reinstall the linux agent when I recreated the LXC, and I had the host CMK agent / API integration set to default. The interesting question for me is also, why isn’t the special agent receiving anything by itself? I have the cmkadmin user set with user permissions on the proxmox host correctly, and the special agent rule set up as well.

I don’t think the first time I installed this that I had to install the agent on my proxmox hosts, and the CMK video doesn’t cover installing the agent either. I am receiving data through the agent at this point though, so thank you again.

ETA: I am also still receiving these 2 critical errors on my dashboard for the ValueErrors:

agent] Success, [special_proxmox_ve] ValueError: could not convert string to float: '8.41 s (average 17.11 MiB/'**CRIT** , [piggyback] Success (but no data found for this host), execution time 2.4 sec

Services: all up to date, Host labels: all up to date, [special_proxmox_ve] ValueError: could not convert string to float: '8.41 s (average 17.11 MiB/'CRIT

Same here since the upgrade.

ERROR 2024-07-17 01:35:24 root: Error in log at line 241: LogParseWarning(241, "End of VM '117' while still information is missing (we have: {'total_duration', 'started_time'})")
ERROR 2024-07-17 01:35:24 agent_proxmox_ve: Parsing the log for UPID='UPID:hv2:002536BB:0312C68A:669571B8:vzdump::root@pam:' resulted in a error(s) - write log content to '/omd/sites/monitor/tmp/check_mk/special_agents/agent_proxmox_ve/erroneous-UPID:hv2:002536BB:0312C68A:669571B8:vzdump::root@pam:.log'

[agent] Success, [mgmt_ipmi] Success, [special_proxmox_ve] Agent exited with code 1: Agent failed - please submit a crash report! (Crash-ID: 5b6587ce-43cd-11ef-a000-bc241138d6a4)**CRIT**, [piggyback] Success (but no data found for this host), execution time 6.6 sec

i have the same issue. I think it occurred after I did a proxmox update.
Seems that some Variables have other values now and cant be parsed.
[special_proxmox_ve]: ValueError: could not convert string to float: '91.26 s (average 2.502 MiB/'CRIT
Still there with 2.3.0p10

Hi all!

Werk #16863: proxmox: Fix log parsing crash for Proxmox versions 3.2.4 and newer – this might be a helpful Werk for the situation.
Should be available in the next release.

1 Like

here is a workaround