Proxmox Special Agent crashing since proxmox host

Troubleshooting
1

CMK version:
2.3.0p6
OS version:
Debian 12

Error message:
Since updating proxmox backup client to latest version the proxmox special agent is crashing:

ValueError (could not convert string to float: ‘53.75 s (average 18.894 MiB/’)

It seems like they changed the output for completed backups. The running time has now an “average” value and so parsing this value into checkmk (and convert it to float) make it crash.

File “/omd/sites/cmk/lib/python3/cmk/special_agents/agent_proxmox_ve.py”, line 358, in _extract_logs
current_dataset[“backup_time”] = float(backup_time)

Crash Log is committed ( 83d1a922-2875-11ef-a408-f29af79d99ec )

2

Nobody else with this problem?

3

Hi @ampfinger,

which version of Proxmox are you using and which version does your proxmox-backup-client have after the update.

I cannot reproduce the issue in Proxmix 8.2.4 and with the proxmox-backup-client 3.2.3-1.

4

Hi @sebkir

In difference to your installation I have got latest proxmox-backup-client version 3.2.5-1
There is an output change in backup logs.

Example version 3.2.3-1:

INFO: root.pxar: had to backup 962.93 MiB of 5.444 GiB (compressed 200.638 MiB) in 37.10s

Example version 3.2.5-1:

INFO: root.pxar: had to backup 1004.756 MiB of 5.434 GiB (compressed 221.297 MiB) in 58.86 s (average 17.07 MiB/s)

Problematic is the proxmox-backup-client package.
The change in the client was made her:
https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=7401be7e963703fa0a65f23b4411766bdc923265

I think the proxmox special agent didn’t expect the change in the output.

5

Thanks. That clarifies it. I will forward this to our developers.

6

Just one more question: Which repo did you get the 3.2.5-1 from? Are you using pvetest perhaps? Because in deb Index of /debian/pve/ bookworm pve-no-subscription the backup client is still on 3.2.4-1.

7

Hi,

Fair enough I got pvetest repo.
I thought this change was made in 3.2.4-1.

8

It’s just good to know. Sooner or later this line will be in the stable release of Proxmox. That way we still have a few days before more users will be affected by this.

So: Thanks for reporting this - let’s say - ahead of time. I will post here again, when the check is adapted to this new output.

9

Hi @sebkir

After installing 2.3.0p7 bug still exists. The Agent isn’t crashing anymore (Werk #16988) but piggyback data still can’t be read (float error as before).

Find a solution and I think it will work for old and new version.

File:
lib/python3/cmk/special_agents/agent_proxmox_ve.py

Line:
214

Before:
r"^INFO: (.*): had to backup (.*) of (.*) \(compressed (.*)\) in (.*)s",

Afterwards:
r"^INFO: (.*): had to backup (.*) of (.*) \(compressed (.*)\) in (.*)s (.*)",

After this change piggyback data can be read and backup state will be correctly shown.

Maybe I can help with this.

3 Likes
10

Our fix will look like this:

r"^INFO: (.*): had to backup (.*) of (.*) \(compressed (.*)\) in ([\d.]+)[\s]*s.*",

2 Likes
11

Thats fine for me, might be better regex in this case.
Thanks for your efford!

12

Hi @ampfinger ,

would it be possible for you to the test the following .mkp in your Checkmk site? We just need to confirm this works, before we can add it to the next patch release.

Since the change in Proxmox only seems to apply in very special circumstances (“split pxar archives”), I am not able to reproduce it in a timely manner. Your help would be appreciated.

fix-proxmox-log-parsing-1.0.0.mkp (9.9 KB)

2 Likes
13

Hi @sebkir ,

I have installed it over the gui and activated it on the site.
Then I got the message:
[special_proxmox_ve] ModuleNotFoundError: No module named 'cmk.special_agents.utils.agent_common

Did I miss something?

Edit: Line 38 ff.
Before:
from cmk.special_agents.v0_unstable.agent_common [...] from cmk.special_agents.v0_unstable.argument_parsing [...] from cmk.special_agents.v0_unstable.misc [...]

Afterwards:
from cmk.special_agents.utils.agent_common [...] from cmk.special_agents.utils.argument_parsing [...] from cmk.special_agents.utils.misc [...]

Some internal changes? Changing those lines everything works.

14

My bad… Might be that the fix only applies to Checkmk 2.2.0+ and not to 2.3.0. I will ask our dev.

15

I think you don’t need to.
I reverted my changes - broken (as expected)
Then I copied these three lines into your fix and it works :slight_smile:

So fix is approved :slight_smile:

16

Can you please mark the answer as the solution if it solved your problem?

mark-as-soln
mark-as-soln776×136 9.67 KB

17

Hi @sebkir

As I can see there is a werk online:

This will be release for 2.3.0p9 (today 2.3.0p8).
Is there a chance to get this werk pre-release in some way?

18

Unfortunately these werk descriptions are not linked to any git/github commit id or something like this, so this makes tracking changes quite challenging for third-party developers. However, see here for the corresponding werk commit on github:

19

Thanks for the great work :top:
Now waiting for 2.3.0p11 :blush:

20

Hi @ampfinger , I have the same issue but it’s unclear to me how to solve it. I used the mkp provided, changed the import lines, but I still get the ModuleNotFoundError: No module named 'cmk.special_agents.utils.agent_common error. And I see 2.3.0p11 is not out yet (using p10 now). Can you advise how you solved it in the end? Thanks!