Proxmox VE special agent - unhandled Exception

Pixelpoint · September 5, 2022, 8:06am

Dear CheckMK Community, we experience a problem with the Proxmox VE Special Agent trying to monitor a Proxmox VE Hypervisor.

CMK version: Checkmk Enterprise Edition 2.1.0p11
OS version: Ubuntu 20.04

Error message:

[agent] Success, [special_proxmox_ve] Agent exited with code 1: Caught unhandled TypeError('NoneType' object is not iterable) in /omd/sites/mysite/lib/python3/cmk/special_agents/utils/agent_common.py:135(!!), execution time 2.9 sec | execution_time=2.880 user_time=0.070 system_time=0.020 children_user_time=0.370 children_system_time=0.040 cmk_time_agent=1.780 cmk_time_ds=0.600

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

OMD[mysite]:~$ cmk --debug -vvn pveiloon
Checkmk version 2.1.0p11
Try license usage history update.
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Got lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Got lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Released lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Released lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7fbbb7840820]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(pveiloon, base_path=/omd/sites/mysite/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 52 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 83.64.35.147:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Write data to cache file /omd/sites/mysite/tmp/check_mk/cache/pveiloon
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/cache/pveiloon
Got lock on /omd/sites/mysite/tmp/check_mk/cache/pveiloon
Releasing lock on /omd/sites/mysite/tmp/check_mk/cache/pveiloon
Released lock on /omd/sites/mysite/tmp/check_mk/cache/pveiloon
Closing TCP connection to 83.64.35.147:6556
[cpu_tracking] Stop [7fbbb7840820 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.0, children_user=0.0, children_system=0.0, elapsed=1.9299999997019768))]
  Source: SourceType.HOST/FetcherType.PROGRAM
[cpu_tracking] Start [7fbbb7868a60]
[ProgramFetcher] Fetch with cache settings: DefaultAgentFileCache(pveiloon, base_path=/omd/sites/mysite/tmp/check_mk/data_source_cache/special_proxmox_ve, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 295127 sec, allowed is 0 sec)
[ProgramFetcher] Execute data source
Calling: /omd/sites/mysite/share/check_mk/agents/special/agent_proxmox_ve '-u' 'checkmk@pve' '-p' 'PASSWORD' '--port' 8006 '--no-cert-check' '--timeout' 60 'pveiloon'
[cpu_tracking] Stop [7fbbb7868a60 - Snapshot(process=posix.times_result(user=0.0, system=0.010000000000000009, children_user=0.31, children_system=0.09, elapsed=1.469999998807907))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7fbbb78681f0]
[PiggybackFetcher] Fetch with cache settings: NoCache(pveiloon, base_path=/omd/sites/mysite/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'pveiloon'. Skip processing.
No piggyback files for '83.64.35.147'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7fbbb78681f0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool_status>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_thermal:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_nodes>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1662364689,30)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<check_mk:cached(1662364269,10800)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1662354118,10800):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['apt', 'check_mk', 'checkmk_agent_plugins_lnx', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df', 'diskstat', 'job', 'kernel', 'labels', 'lnx_bonding', 'lnx_if', 'lnx_thermal', 'local', 'md', 'mem', 'mounts', 'nfsmounts', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'pvecm_nodes', 'pvecm_status', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest', 'zfsget', 'zpool', 'zpool_status']
  Source: SourceType.HOST/FetcherType.PROGRAM
  -> Not adding sections: Agent exited with code 1: Caught unhandled TypeError('NoneType' object is not iterable) in /omd/sites/mysite/lib/python3/cmk/special_agents/utils/agent_common.py:135
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7fbbb7841730]
value store: synchronizing
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/counters/pveiloon
Got lock on /omd/sites/mysite/tmp/check_mk/counters/pveiloon
value store: loading from disk
Releasing lock on /omd/sites/mysite/tmp/check_mk/counters/pveiloon
Released lock on /omd/sites/mysite/tmp/check_mk/counters/pveiloon
APT Updates          No updates pending for installation
CPU load             15 min load: 0.22, 15 min load per core: 0.01 (20 cores)
CPU utilization      Total CPU: 0.92%
Check_MK Agent       Version: 2.1.0p11, OS: linux, Last update: Sep 01 2022 12:47:42, Agent plugins: 2, Local checks: 3
Disk IO SUMMARY      Read: 76.1 kB/s, Write: 125 kB/s, Latency: 228 microseconds
Filesystem /         10.34% used (9.71 of 93.93 GB), trend: +114.45 MB / 24 hours - time left until disk full: more than a year
Filesystem /boot/efi 0.06% used (328.00 kB of 510.98 MB), trend: 0.00 B / 24 hours
Filesystem /etc/pve  0.02% used (24.00 kB of 128.00 MB), trend: +1.53 kB / 24 hours - time left until disk full: more than a year
Interface 2          [eno1], (up), MAC: B4:7A:F1:4A:C1:80, Speed: 1 GBit/s, In: 4.27 kB/s (<0.01%), Out: 4.16 kB/s (<0.01%)
Interface 6          [vmbr0], (up), MAC: B4:7A:F1:4A:C1:80, Speed: 1 GBit/s, In: 3.25 kB/s (<0.01%), Out: 1.54 kB/s (<0.01%)
Interface 7          [tap100i0], (up), MAC: 42:E1:CE:92:D3:5A, Speed: 10 MBit/s, In: 1.32 kB/s (0.11%), Out: 2.77 kB/s (0.22%)
Interface 8          [tap101i0], (up), MAC: 62:DD:DD:B0:B1:08, Speed: 10 MBit/s, In: 2.69 kB/s (0.22%), Out: 3.81 kB/s (0.30%)
Job proxmox-backup   Latest exit code: 0, Real time: 3 seconds, Job age: 3 hours 33 minutes
Memory               Total virtual memory: 44.99% - 31.70 GB of 70.47 GB, 9 additional details available
Mount options of /   Mount options exactly as expected
Mount options of /boot/efi Mount options exactly as expected
Mount options of /etc/pve Mount options exactly as expected
NTP Time             Offset: 0.0769 ms, Stratum: 3, Time since last sync: 17 minutes 58 seconds
Postfix Queue        Deferred queue length: 0, Active queue length: 0
Postfix status       Status: the Postfix mail system is running, PID: 1304
Reboot               OK
Systemd Service Summary Total: 147, Disabled: 5, Failed: 0
Temperature Zone 0   48.0 °C
Temperature Zone 1   39.0 °C
UFW                  running
Uptime               Up since Sep 04 2022 13:32:37, Uptime: 20 hours 26 minutes
No piggyback files for 'pveiloon'. Skip processing.
No piggyback files for '83.64.35.147'. Skip processing.
[cpu_tracking] Stop [7fbbb7841730 - Snapshot(process=posix.times_result(user=0.04999999999999982, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.05999999865889549))]
[agent] Success, [special_proxmox_ve] Agent exited with code 1: Caught unhandled TypeError('NoneType' object is not iterable) in /omd/sites/mysite/lib/python3/cmk/special_agents/utils/agent_common.py:135(!!), execution time 3.5 sec | execution_time=3.460 user_time=0.060 system_time=0.020 children_user_time=0.310 children_system_time=0.090 cmk_time_agent=1.920 cmk_time_ds=1.060

Short explanation
The server crashed last week and I had to reinstall everything.
After restoring the backups everything seemed to work fine except for the Proxmox VE Special Agent.
IP addresses or firewall configuration have not changed since ages.

Does somebody know what’s wrong here?

Best regards,
pixelpoint

andreas-doehler · September 5, 2022, 8:34am

What happens if you execute the special agent manually?

The proxmox agent has no debug

Pixelpoint · September 5, 2022, 9:23am

When executing the Proxmox VE special agent manually, I get the following error:

OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_proxmox_ve '-u' 'checkmk@pve' '-p' 'PASSWORD' '--port' 8006 '--no-cert-check' '--timeout' 60 'pveiloon'
Caught unhandled TypeError('NoneType' object is not iterable) in /omd/sites/mysite/lib/python3/cmk/special_agents/utils/agent_common.py:135

andreas-doehler · September 5, 2022, 9:30am

The command line option “-v” should give a little bit more output, I hope.

Pixelpoint · September 5, 2022, 9:48am

Here is the output when executing with -vv

OMD[mysite]:~$ /omd/sites/mysite/share/check_mk/agents/special/agent_proxmox_ve -vv '-u' 'checkmk@pve' '-p' 'PASSWORD' '--port' 8006 '--no-cert-check' '--timeout' 60 'pveiloon'
INFO 2022-09-05 11:46:39 root: running file /omd/sites/mysite/lib/python3/cmk/special_agents/utils/agent_common.py
INFO 2022-09-05 11:46:39 root: using Python interpreter v3.9.10.final.0 at /omd/sites/mysite/bin/python3
DEBUG 2022-09-05 11:46:39 root: args: {'debug': False, 'verbose': 2, 'vcrtrace': False, 'timeout': 60, 'port': 8006, 'username': 'checkmk@pve', 'password': 'PASSWORD', 'log_cutoff_weeks': 2, 'dump_logs': False, 'no_cert_check': True, 'hostname': 'pveiloon'}
INFO 2022-09-05 11:46:39 agent_proxmox_ve: Establish connection to Proxmox VE host 'pveiloon'
INFO 2022-09-05 11:46:39 agent_proxmox_ve: Fetch general cluster and node information..
INFO 2022-09-05 11:46:40 agent_proxmox_ve: Fetch and process backup logs..
DEBUG 2022-09-05 11:46:40 cmk.special_agents.utils.misc: Cache: loaded 6 elements
INFO 2022-09-05 11:46:40 agent_proxmox_ve: BackupTask('vzdump', t='2022.09.04-20:30:01', vms=('101',))
DEBUG 2022-09-05 11:46:40 agent_proxmox_ve: {'101': {'started_time': '2022-09-04 20:30:01', 'transfer_size': 1429365116109, 'transfer_time': 1501, 'total_duration': 1505.0}}
Caught unhandled TypeError('NoneType' object is not iterable) in /omd/sites/mysite/lib/python3/cmk/special_agents/utils/agent_common.py:135

Thanks for trying to help

Best regards,
pixelpoint

andreas-doehler · September 5, 2022, 10:08am

That means one off the backup jobs has not correct data.
Now you can only try to insert some extra code to get to know what backup job has a problem.

Pixelpoint · September 15, 2022, 8:53am

Thank you for your answer and please excuse long reaction time.
I couldn’t really find out what is wrong here, even after looking through content of the REST API answer.

On this Proxmox VE there’s only 2 VMs.

ID 100: Windows Server
ID 101: Proxmox Backup Server (no backup of course, as this is the backup machine)

I had to re-install everything and accidently gave Proxmox Backup Server VM ID 100 and Windows Server VM ID 101.
This cannot be changed after creation, except for cloning the VM and deleting the old one etc.

I suspect the problem is the Backup of ID 100 does not look like it’s a Backup of VM ID 100.

I will keep the backups locally for a while.
After everything seems to be okay, I will delete the backups of VM ID 100 from the local Proxmox Backup Server (we have one in the cloud too, for offsite-backup purposes) and re-activate the Proxmox VE Check again to see if this fixed the problem.

I will post another comment when I have done that.

Thank you for your help and patience.

Best regards,
pixelpoint

system · September 15, 2023, 8:54am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.