CMK version: 2.2.0p9.cee
OS version: Debian Bullseye
Error message:
ValueError (Cannot render negative timespan)
Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
t[cpu_tracking] Start [7fb803019910]
Read from cache: AgentFileCache(Domain.name, path_template=/omd/sites/monitoring/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 35 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to NODEIP:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Closing TCP connection to NODEIP:6556
Write data to cache file /omd/sites/monitoring/tmp/check_mk/cache/domain.name
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/cache/domain.name
Got lock on /omd/sites/monitoring/tmp/check_mk/cache/domain.name
Releasing lock on /omd/sites/monitoring/tmp/check_mk/cache/domain.name
Released lock on /omd/sites/monitoring/tmp/check_mk/cache/domain.name
[cpu_tracking] Stop [7fb803019910 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.7699999958276749))]
Source: SourceInfo(hostname='domain.name', ipaddress='NODEIP', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fb8034391d0]
Read from cache: NoCache(domain.name, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
Piggyback file '/omd/sites/monitoring/tmp/check_mk/piggyback/domain.name/host.domain': Successfully processed from source 'host.domain'
No piggyback files for 'NODEIP'. Skip processing.
[cpu_tracking] Stop [7fb8034391d0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1696847755,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_instances>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_sessions>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_stat_database:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_locks:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_query_duration:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_connections:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_stats:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_version:sep(1)>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_conn_time>>> / Transition HostSectionParser -> HostSectionParser
<<<postgres_bloat:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<sshd_config>>> / Transition HostSectionParser -> HostSectionParser
<<<nginx_status>>> / Transition HostSectionParser -> HostSectionParser
<<<cmk_update_agent_status:cached(1696844993,3600):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1696844213,86400):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<proxmox_ve_vm_info:cached(1696847702,90):sep(0)>>> / Transition NOOPParser -> HostSectionParser
<<<proxmox_ve_mem_usage:cached(1696847702,90):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<proxmox_ve_vm_backup_status:cached(1696847702,90):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<proxmox_ve_vm_snapshot_age:cached(1696847702,90):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
HostKey(hostname='domain.name', source_type=<SourceType.HOST: 1>) -> Add sections: ['apt', 'check_mk', 'checkmk_agent_plugins_lnx', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cmk_update_agent_status', 'cpu', 'df_v2', 'diskstat', 'job', 'kernel', 'labels', 'lnx_if', 'local', 'logwatch', 'md', 'mem', 'mounts', 'nfsmounts_v2', 'nginx_status', 'postgres_bloat', 'postgres_conn_time', 'postgres_connections', 'postgres_instances', 'postgres_locks', 'postgres_query_duration', 'postgres_sessions', 'postgres_stat_database', 'postgres_stats', 'postgres_version', 'ps_lnx', 'sshd_config', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
HostKey(hostname='domain.name', source_type=<SourceType.HOST: 1>) -> Add sections: ['labels', 'proxmox_ve_mem_usage', 'proxmox_ve_vm_backup_status', 'proxmox_ve_vm_info', 'proxmox_ve_vm_snapshot_age']
Received no piggyback data
[cpu_tracking] Start [7fb802724590]
value store: synchronizing
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/domain.name
Got lock on /omd/sites/monitoring/tmp/check_mk/counters/domain.name
value store: loading from disk
Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/domain.name
Released lock on /omd/sites/monitoring/tmp/check_mk/counters/domain.name
[cpu_tracking] Stop [7fb802724590 - Snapshot(process=posix.times_result(user=0.029999999999999805, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.03999999910593033))]
Trying to acquire lock on /omd/sites/monitoring/var/check_mk/crashes/base/a6832ee4-668f-11ee-bb31-9600019c2722/crash.info
Got lock on /omd/sites/monitoring/var/check_mk/crashes/base/a6832ee4-668f-11ee-bb31-9600019c2722/crash.info
Releasing lock on /omd/sites/monitoring/var/check_mk/crashes/base/a6832ee4-668f-11ee-bb31-9600019c2722/crash.info
Released lock on /omd/sites/monitoring/var/check_mk/crashes/base/a6832ee4-668f-11ee-bb31-9600019c2722/crash.info
Traceback (most recent call last):
File "/omd/sites/monitoring/bin/cmk", line 118, in <module>
exit_status = modes.call("--check", None, opts, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/monitoring/lib/python3/cmk/base/modes/__init__.py", line 68, in call
return handler(*handler_args)
^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/monitoring/lib/python3/cmk/base/modes/check_mk.py", line 1998, in mode_check
with error_handler:
File "/omd/sites/monitoring/lib/python3/cmk/checkers/error_handling.py", line 59, in __exit__
self._result = _handle_failure(
^^^^^^^^^^^^^^^^
File "/omd/sites/monitoring/lib/python3/cmk/checkers/error_handling.py", line 95, in _handle_failure
raise exc
File "/omd/sites/monitoring/lib/python3/cmk/base/modes/check_mk.py", line 2001, in mode_check
check_result = checking.execute_checkmk_checks(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/monitoring/lib/python3/cmk/base/agent_based/checking/_checking.py", line 117, in execute_checkmk_checks
service_results = check_host_services(
^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/monitoring/lib/python3/cmk/base/agent_based/checking/_checking.py", line 293, in check_host_services
submittables = [
^
File "/omd/sites/monitoring/lib/python3/cmk/base/agent_based/checking/_checking.py", line 303, in <listcomp>
else get_aggregated_result(
^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/monitoring/lib/python3/cmk/base/agent_based/checking/_checking.py", line 413, in get_aggregated_result
consume_check_results(
File "/omd/sites/monitoring/lib/python3/cmk/base/api/agent_based/checking_classes.py", line 484, in consume_check_results
for subr in subresults:
File "/omd/sites/monitoring/lib/python3/cmk/base/api/agent_based/register/check_plugins.py", line 93, in filtered_generator
for element in generator(*args, **kwargs):
File "/omd/sites/monitoring/lib/python3/cmk/base/plugins/agent_based/proxmox_ve_backup_status.py", line 226, in check_proxmox_ve_vm_backup_status_unpure
yield from check_proxmox_ve_vm_backup_status(datetime.now(tz=timezone.utc), params, section)
File "/omd/sites/monitoring/lib/python3/cmk/base/plugins/agent_based/proxmox_ve_backup_status.py", line 154, in check_proxmox_ve_vm_backup_status
yield from check_levels(
File "/omd/sites/monitoring/lib/python3/cmk/base/api/agent_based/utils.py", line 337, in check_levels
info_text = str(render_func(value)) # forgive wrong output type
^^^^^^^^^^^^^^^^^^
File "/omd/sites/monitoring/lib/python3/cmk/base/api/agent_based/render.py", line 107, in timespan
ts = " ".join(_gen_timespan_chunks(float(seconds), nchunks=2))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/omd/sites/monitoring/lib/python3/cmk/base/api/agent_based/render.py", line 73, in _gen_timespan_chunks
raise ValueError("Cannot render negative timespan")
ValueError: Cannot render negative timespan
Hello everyone,
I have the Issue shown above. In Short: A single VM in our Proxmox Cluster won’t show the proper VM Backup Status. Each Update (every 2h) the check turns “OK” right after the backups has been created, only to turn back to “UNKNOWN” shortly after.
I’m using the exact same setup to monitor Backups for a couple of other VMs on the same cluster that work without issues, it’s only this single one not working.
I tried updating checkmk, removing the host and re-adding it, wait a bit… Nothing helped so far. Since the error points toward time issues of some sort I checked the system time of all involved systems and they are all correctly using UTC and are synced to a time server.
I also did submit the error using the Checkmk interface if that’s important/helpful.
It’s not that big of an issue since I can manuelly check the backup status every now and then, but the regular Notifications about the unknow status are getting annoying.
Hope anyone got any idea.