ProxMox backup extension incorrectly reports backups too old

Hi,

I installed the plugin that monitors the backup age of VMs on ProxMox. However, for some reason I find that the backup of two of the VMs is reported incorrectly:

Looking at the backup age in ProxMox itself shows that the backups are less then 7 days old:

Looking at the agent output it’s reported correctly as well:

cmk -d digprox01
...
<<<proxmox_qemu_backup>>>
QEMU-MACHINE;;;;;/etc/pve/qemu-server/109.conf;;;;;digsvl007
QEMU-MACHINE;;;;;/etc/pve/qemu-server/102.conf;;;;;digsvl003
QEMU-MACHINE;;;;;/etc/pve/qemu-server/108.conf;;;;;digsvl006
QEMU-MACHINE;;;;;/etc/pve/qemu-server/110.conf;;;;;digsvw001
QEMU-MACHINE;;;;;/etc/pve/qemu-server/115.conf;;;;;tmp001
QEMU-MACHINE;;;;;/etc/pve/qemu-server/105.conf;;;;;digsvl218
QEMU-MACHINE;;;;;/etc/pve/qemu-server/114.conf;;;;;digsvl012
QEMU-MACHINE;;;;;/etc/pve/qemu-server/113.conf;;;;;digsvl011
/var/log/vzdump/qemu-102.log 2021-12-04 00:00:02 INFO: Starting Backup of VM 102 (qemu)
/var/log/vzdump/qemu-102.log 2021-12-04 00:00:02 INFO: status = running
/var/log/vzdump/qemu-102.log 2021-12-04 00:00:02 INFO: VM Name: digsvl003
/var/log/vzdump/qemu-102.log 2021-12-04 00:00:02 INFO: include disk 'scsi0' 'VM-Storage:102/vm-102-disk-0.qcow2' 32G
/var/log/vzdump/qemu-102.log 2021-12-04 00:00:16 INFO: backup mode: snapshot
/var/log/vzdump/qemu-102.log 2021-12-04 00:00:16 INFO: ionice priority: 7
/var/log/vzdump/qemu-102.log 2021-12-04 00:00:16 INFO: creating vzdump archive '/mnt/pve/Remote_Backup/dump/vzdump-qemu-102-2021_12_04-00_00_02.vma.zst'
/var/log/vzdump/qemu-102.log 2021-12-04 00:00:16 INFO: issuing guest-agent 'fs-freeze' command
/var/log/vzdump/qemu-102.log 2021-12-04 00:02:37 INFO: issuing guest-agent 'fs-thaw' command
/var/log/vzdump/qemu-102.log 2021-12-04 00:02:38 INFO: started backup task 'a59a5854-81df-47a0-9358-74c8fef6fb4b'
/var/log/vzdump/qemu-102.log 2021-12-04 00:02:38 INFO: resuming VM again
/var/log/vzdump/qemu-102.log 2021-12-04 00:02:41 INFO:   0% (13.5 MiB of 32.0 GiB) in 3s, read: 4.5 MiB/s, write: 3.5 MiB/s
/var/log/vzdump/qemu-102.log 2021-12-04 00:03:28 INFO:   1% (328.8 MiB of 32.0 GiB) in 50s, read: 6.7 MiB/s, write: 6.1 MiB/s
/var/log/vzdump/qemu-102.log 2021-12-04 00:04:35 INFO:   2% (668.9 MiB of 32.0 GiB) in 1m 57s, read: 5.1 MiB/s, write: 4.6 MiB/s
/var/log/vzdump/qemu-102.log 2021-12-04 00:06:08 INFO:   3% (986.6 MiB of 32.0 GiB) in 3m 30s, read: 3.4 MiB/s, write: 3.2 MiB/s
...

Anyone else experiencing this issue? I only have this for these two VMs. The rest is reported correctly.

Kind regards,
Louis

Hi,

which Proxmox plugin is it? It doesn’t seem to be the Proxmox special agent that is build in Checkmk 2.0:

1 Like

Indeed it isn’t. That’s why I posted in the CheckMK Exchange group. It’s the one that checks the backup:
https://exchange.checkmk.com/p/proxmox-qemu-backup-1

I see. Is it better than our special agent?

Ah good question. I followed the instructions in the link you provided earlier. That allowed me to setup monitoring of the clustered services properly. But I didn’t see the option of monitoring the status and age of the last backup in the special agent. Did I miss something there? If so, I’d be happy to use the special agent for monitoring of the backups as well.

You will not see it in the special agent/at the node level. Instead, it generates piggybacked data for the monitored VM.

So if the VM is called for example vm1.domain.com in Proxmox and has the same name in Checkmk, then the data is attached to it automatically. (case sensitive!)
If the names do not match, you will have to write a rule of the type “Hostname translation for piggybacked hosts” - but I prefer to have simply the same names - makes life much easier.

Well, my hostnames do match, so that shouldn’t be an issue. (I just verified to make sure). However, the backup is not showing without the agent from the CheckMK exchange.

In Proxmox I have for instance the VM called digsvl003:
image

Which matches with what it’s called in in CheckMK:

But to be honest, I don’t see any services generated by the piggybacked hosts. Looking at the documentation I should see more than just the backup, but there’s none.

I do have the rule for the proxmox hosts:

And setup the hosts to use both API integrations and the CheckMK agent:

I verified that I can login to the Proxmox cluster using the checkmk user and the password specified in the rule I created.

So obviously I’m missing something, but what?

Kind regards,
Louis

Look into /omd/sites/SITENAME/tmp/check_mk/piggyback/, there should be folders named like your VMs/LXCs. If these names are matching to the VM hostname in Checkmk, then do a rediscovery of such a VM/LXC host and it should find new services.

I checked that directory. It’s empty. Which explains why I don’t have any of the services. Just to be sure I checked piggyback_sources as well; that’s also empty.

run a cmk -D ProxmoxHostname . It will output the configuration and close to the top you will see configured the special agent call.
You can copy that and run it manually and check if you run into problems.

Well, that explains why it’s not working I guess:

OMD[digio]:~$ /omd/sites/digio/share/check_mk/agents/special/agent_proxmox_ve '-u' 'checkmk' '-p' '<password> '--no-cert-check' 'digprox01'
Unhandled exception: RuntimeError("Couldn't authenticate 'checkmk' @ 'https://digprox01:8006/api2/json/access/ticket'")

Copy / paste username from the command let’s me login to the ProxMox web interface, so that’s working; no typo’s there at least.

Try checkmk@pve as username. @pve is the authentication realm. It can be @pam (Linux) or @pve (Proxmox user)

1 Like

Thanks, that nailed it. Something so stupid to overlook.

1 Like