Hello dear CheckMK Community,
Lately (I think after Upgrade to 2.1.0) the APC Temperature service crashes.
The APC is directly connected (via USB) to our Hypervisor (Proxmox VE, Hostname: pve).
APC Status Services is OK (still works, never stopped).
I don’t exactly know why this happens but I guess it has to do with CheckMK APC check, as we have not done any Firmware Upgrades on the APC.
Executing the Check apcaccess on the host directly yields the following output (which seems normal to me):
<<<apcaccess:sep(58)>>>
[[apcupsd.conf]]
APC : 001,043,1008
DATE : 2022-06-22 09:49:20 +0200
HOSTNAME : pve
VERSION : 3.14.14 (31 May 2016) debian
UPSNAME : pve
CABLE : USB Cable
DRIVER : USB UPS Driver
UPSMODE : Stand Alone
STARTTIME: 2022-06-21 20:25:18 +0200
MODEL : Smart-UPS 1500 RM
STATUS : ONLINE
LINEV : 233.2 Volts
LOADPCT : 19.5 Percent
BCHARGE : 100.0 Percent
TIMELEFT : 66.0 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME : 0 Seconds
OUTPUTV : 233.2 Volts
SENSE : High
DWAKE : -1 Seconds
DSHUTD : 630 Seconds
LOTRANS : 208.0 Volts
HITRANS : 253.0 Volts
RETPCT : 0.0 Percent
ITEMP : 31.9 C
ALARMDEL : 30 Seconds
BATTV : 27.4 Volts
LINEFREQ : 50.0 Hz
LASTXFER : Automatic or explicit self test
NUMXFERS : 0
TONBATT : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STESTI : 14 days
STATFLAG : 0x05000008
MANDATE : 2006-09-16
SERIALNO : AS0638331142
BATTDATE : 2021-12-14
NOMOUTV : 230 Volts
NOMBATTV : 24.0 Volts
FIRMWARE : 617.3.I USB FW:8.1
END APC : 2022-06-22 09:49:24 +0200
CMK version: Checkmk Enterprise Edition 2.1.0p2
OS version: Ubuntu 20.04
Exception
TypeError (check_temperature() missing 1 required keyword-only argument: 'value_store')
Traceback
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 469, in get_aggregated_result
result = _aggregate_results(
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 577, in _aggregate_results
perfdata, results = _consume_and_dispatch_result_types(subresults)
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 621, in _consume_and_dispatch_result_types
for subr in subresults:
File "/omd/sites/mysite/lib/python3/cmk/base/api/agent_based/register/check_plugins.py", line 94, in filtered_generator
for element in generator(*args, **kwargs):
File "/omd/sites/mysite/local/lib/python3/cmk/base/plugins/agent_based/apcaccess.py", line 163, in check_apcaccess_temp
yield from temperature.check_temperature(float(itemp[0]),
Local Variables
{'item': 'apcupsd.conf',
'itemp': ['31.9', 'C'],
'params': Parameters({'levels': (40, 50)}),
'section': {'apcupsd.conf': {'ALARMDEL': '30 Seconds',
'APC': '001,043,1008',
'BATTDATE': '2021-12-14',
'BATTV': '27.4 Volts',
'BCHARGE': '100.0 Percent',
'CABLE': 'USB Cable',
'CUMONBATT': '0 Seconds',
'DATE': '2022-06-22 09:36:40 +0200',
'DRIVER': 'USB UPS Driver',
'DSHUTD': '630 Seconds',
'DWAKE': '-1 Seconds',
'END APC': '2022-06-22 09:36:41 +0200',
'FIRMWARE': '617.3.I USB FW:8.1',
'HITRANS': '253.0 Volts',
'HOSTNAME': 'pve',
'ITEMP': '31.9 C',
'LASTXFER': 'Automatic or explicit self test',
'LINEFREQ': '50.0 Hz',
'LINEV': '231.8 Volts',
'LOADPCT': '20.1 Percent',
'LOTRANS': '208.0 Volts',
'MANDATE': '2006-09-16',
'MAXTIME': '0 Seconds',
'MBATTCHG': '5 Percent',
'MINTIMEL': '3 Minutes',
'MODEL': 'Smart-UPS 1500 RM',
'NOMBATTV': '24.0 Volts',
'NOMOUTV': '230 Volts',
'NUMXFERS': '0',
'OUTPUTV': '233.2 Volts',
'RETPCT': '0.0 Percent',
'SELFTEST': 'NO',
'SENSE': 'High',
'SERIALNO': 'AS0638331142',
'STARTTIME': '2022-06-21 20:25:18 +0200',
'STATFLAG': '0x05000008',
'STATUS': 'ONLINE',
'STESTI': '14 days',
'TIMELEFT': '64.0 Minutes',
'TONBATT': '0 Seconds',
'UPSMODE': 'Stand Alone',
'UPSNAME': 'pve',
'VERSION': '3.14.14 (31 May 2016) debian',
'XOFFBATT': 'N/A'}}}
Output of “cmk --debug -vvn hostname”
cmk --debug -vvn pve
Checkmk version 2.1.0p2
Try license usage history update.
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Got lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Got lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Released lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Released lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
+ FETCHING DATA
Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7fb08bfac730]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(pve, base_path=/omd/sites/mysite/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 95 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 172.16.14.30:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Write data to cache file /omd/sites/mysite/tmp/check_mk/cache/pve
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/cache/pve
Got lock on /omd/sites/mysite/tmp/check_mk/cache/pve
Releasing lock on /omd/sites/mysite/tmp/check_mk/cache/pve
Released lock on /omd/sites/mysite/tmp/check_mk/cache/pve
Closing TCP connection to 172.16.14.30:6556
[cpu_tracking] Stop [7fb08bfac730 - Snapshot(process=posix.times_result(user=0.010000000000000231, system=0.0, children_user=0.0, children_system=0.0, elapsed=2.5700000002980232))]
Source: SourceType.HOST/FetcherType.PROGRAM
[cpu_tracking] Start [7fb08bfac430]
[ProgramFetcher] Fetch with cache settings: DefaultAgentFileCache(pve, base_path=/omd/sites/mysite/tmp/check_mk/data_source_cache/special_proxmox_ve, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 96 sec, allowed is 0 sec)
[ProgramFetcher] Execute data source
Calling: /omd/sites/mysite/share/check_mk/agents/special/agent_proxmox_ve '-u' 'checkmk@pve' '-p' 'crackundlsd' '--port' 8006 '--no-cert-check' '--timeout' 60 'pve'
Write data to cache file /omd/sites/mysite/tmp/check_mk/data_source_cache/special_proxmox_ve/pve
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/data_source_cache/special_proxmox_ve/pve
Got lock on /omd/sites/mysite/tmp/check_mk/data_source_cache/special_proxmox_ve/pve
Releasing lock on /omd/sites/mysite/tmp/check_mk/data_source_cache/special_proxmox_ve/pve
Released lock on /omd/sites/mysite/tmp/check_mk/data_source_cache/special_proxmox_ve/pve
[cpu_tracking] Stop [7fb08bfac430 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.010000000000000009, children_user=0.65, children_system=0.1, elapsed=1.130000002682209))]
Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7fb08bfaca00]
[PiggybackFetcher] Fetch with cache settings: NoCache(pve, base_path=/omd/sites/mysite/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'pve'. Skip processing.
No piggyback files for '172.16.14.30'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7fb08bfaca00 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<zfsget>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool_status>>> / Transition HostSectionParser -> HostSectionParser
<<<zpool>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_thermal:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<pvecm_nodes>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<timesyncd>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<apcaccess:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<check_mk:cached(1655881751,10800)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1655883374,10800):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
Trying to acquire lock on /omd/sites/mysite/var/check_mk/persisted/pve
Got lock on /omd/sites/mysite/var/check_mk/persisted/pve
Releasing lock on /omd/sites/mysite/var/check_mk/persisted/pve
Released lock on /omd/sites/mysite/var/check_mk/persisted/pve
Stored persisted sections: lnx_packages, lnx_distro, lnx_cpuinfo, dmidecode, lnx_uname, lnx_video, lnx_ip_r, lnx_sysctl, lnx_block_devices
Using persisted section SectionName('lnx_packages')
Using persisted section SectionName('lnx_distro')
Using persisted section SectionName('lnx_cpuinfo')
Using persisted section SectionName('dmidecode')
Using persisted section SectionName('lnx_uname')
Using persisted section SectionName('lnx_video')
Using persisted section SectionName('lnx_ip_r')
Using persisted section SectionName('lnx_sysctl')
Using persisted section SectionName('lnx_block_devices')
-> Add sections: ['apcaccess', 'apt', 'check_mk', 'checkmk_agent_plugins_lnx', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df', 'diskstat', 'dmidecode', 'job', 'kernel', 'labels', 'lnx_block_devices', 'lnx_cpuinfo', 'lnx_distro', 'lnx_if', 'lnx_ip_r', 'lnx_packages', 'lnx_sysctl', 'lnx_thermal', 'lnx_uname', 'lnx_video', 'local', 'md', 'mem', 'mounts', 'nfsmounts', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'pvecm_nodes', 'pvecm_status', 'systemd_units', 'tcp_conn_stats', 'timesyncd', 'uptime', 'vbox_guest', 'zfsget', 'zpool', 'zpool_status']
Source: SourceType.HOST/FetcherType.PROGRAM
<<<proxmox_ve_node_info:sep(0)>>> / Transition NOOPParser -> HostSectionParser
<<<proxmox_ve_mem_usage:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
Transition HostSectionParser -> NOOPParser
[...] I removed all the piggyback stuff (this is a hypervisor)
[...] I removed all the "trying to get lock / got lock / released lock" stuff (no problems there)
APC apcupsd.conf Status UPSNAME: pve, MODEL: Smart-UPS 1500 RM, SERIALNO: AS0638331142, FIRMWARE: 617.3.I USB FW:8.1, UPSMODE: Stand Alone, Output Voltage: 232V, Output Load: 20.10%, Battery Capacity: 100.00%, Time Left: 1 hour 2 minutes
[cpu_tracking] Stop [7fb08be409d0 - Snapshot(process=posix.times_result(user=0.10999999999999988, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.09999999776482582))]
Trying to acquire lock on /omd/sites/mysite/var/check_mk/crashes/base/951ed8b2-f1fe-11ec-b007-173c58e74ab3/crash.info
Got lock on /omd/sites/mysite/var/check_mk/crashes/base/951ed8b2-f1fe-11ec-b007-173c58e74ab3/crash.info
Releasing lock on /omd/sites/mysite/var/check_mk/crashes/base/951ed8b2-f1fe-11ec-b007-173c58e74ab3/crash.info
Released lock on /omd/sites/mysite/var/check_mk/crashes/base/951ed8b2-f1fe-11ec-b007-173c58e74ab3/crash.info
Traceback (most recent call last):
File "/omd/sites/mysite/bin/cmk", line 98, in <module>
exit_status = modes.call("--check", None, opts, args)
File "/omd/sites/mysite/lib/python3/cmk/base/modes/__init__.py", line 69, in call
return handler(*handler_args)
File "/omd/sites/mysite/lib/python3/cmk/base/modes/check_mk.py", line 1804, in mode_check
checking.commandline_checking(
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/decorator.py", line 43, in wrapped_check_func
status, output_text = _combine_texts(check_func(hostname, *args, **kwargs))
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 121, in commandline_checking
return _execute_checkmk_checks(
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 174, in _execute_checkmk_checks
num_success, plugins_missing_data = check_host_services(
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 322, in check_host_services
success = _execute_check(
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 382, in _execute_check
submittable = get_aggregated_result(
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 469, in get_aggregated_result
result = _aggregate_results(
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 577, in _aggregate_results
perfdata, results = _consume_and_dispatch_result_types(subresults)
File "/omd/sites/mysite/lib/python3/cmk/base/agent_based/checking/__init__.py", line 621, in _consume_and_dispatch_result_types
for subr in subresults:
File "/omd/sites/mysite/lib/python3/cmk/base/api/agent_based/register/check_plugins.py", line 94, in filtered_generator
for element in generator(*args, **kwargs):
File "/omd/sites/mysite/local/lib/python3/cmk/base/plugins/agent_based/apcaccess.py", line 163, in check_apcaccess_temp
yield from temperature.check_temperature(float(itemp[0]),
TypeError: check_temperature() missing 1 required keyword-only argument: 'value_store'
Is anybody else having this (or a) problem with APC devices?
Best regards,
pixelpoint
