Problem with Aggregation options for clustered services: Additional results from: [other node]

Now the service in question is a mount on a DRBD device managed by Pacemaker/Corosync and it really only is active on the first node, so I don’t see why this check complains that it gets additional results.

I’d like to set the aggregation option to Failover, but at the moment this doesn’t work because of this problem.

CMK version: 2.1.0p9.cre
OS version: Debian GNU/Linux 10 (buster)

Error message:
Best: [nfsrefer-01], 5.77% used (228.59 MB of 3.87 GB), trend: +16.91 B / 24 hours, Additional results from: [nfsrefer-02]

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Checkmk version 2.1.0p9
Try license usage history update.
Trying to acquire lock on /omd/sites/zdv/var/check_mk/license_usage/next_run
Got lock on /omd/sites/zdv/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/zdv/var/check_mk/license_usage/history.json
Got lock on /omd/sites/zdv/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/zdv/var/check_mk/license_usage/history.json
Released lock on /omd/sites/zdv/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/zdv/var/check_mk/license_usage/next_run
Released lock on /omd/sites/zdv/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f880ecadeb0]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(nfsrefer-01, base_path=/omd/sites/zdv/tmp/check_mk/cache, max_age=MaxAge(checking=90, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Using data from cache file /omd/sites/zdv/tmp/check_mk/cache/nfsrefer-01
Got 106381 bytes data from cache
[TCPFetcher] Use cached data
[cpu_tracking] Stop [7f880ecadeb0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f880e940820]
[PiggybackFetcher] Fetch with cache settings: NoCache(nfsrefer-01, base_path=/omd/sites/zdv/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=90, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
Piggyback file '/omd/sites/zdv/tmp/check_mk/piggyback/nfsrefer-01/backup-01': Successfully processed from source 'backup-01'
No piggyback files for '10.94.23.114'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f880e940820 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f880e9400a0]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(nfsrefer-02, base_path=/omd/sites/zdv/tmp/check_mk/cache, max_age=MaxAge(checking=90, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Using data from cache file /omd/sites/zdv/tmp/check_mk/cache/nfsrefer-02
Got 86322 bytes data from cache
[TCPFetcher] Use cached data
[cpu_tracking] Stop [7f880e9400a0 - Snapshot(process=posix.times_result(user=0.010000000000000231, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f880e940e50]
[PiggybackFetcher] Fetch with cache settings: NoCache(nfsrefer-02, base_path=/omd/sites/zdv/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=90, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
Piggyback file '/omd/sites/zdv/tmp/check_mk/piggyback/nfsrefer-02/backup-01': Successfully processed from source 'backup-01'
No piggyback files for '10.94.23.115'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f880e940e50 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<drbd>>> / Transition HostSectionParser -> HostSectionParser
<<<heartbeat_crm>>> / Transition HostSectionParser -> HostSectionParser
<<<fileinfo:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1660041746,30)>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe:cached(1660041569,300)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cfengine3>>> / Transition HostSectionParser -> HostSectionParser
<<<debian_lts_status>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch>>> / Transition HostSectionParser -> HostSectionParser
<<<netstat>>> / Transition HostSectionParser -> HostSectionParser
<<<logfiles:cached(1660041569,300)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1660039388,3600):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
Trying to acquire lock on /omd/sites/zdv/var/check_mk/persisted/nfsrefer-01
Got lock on /omd/sites/zdv/var/check_mk/persisted/nfsrefer-01
Releasing lock on /omd/sites/zdv/var/check_mk/persisted/nfsrefer-01
Released lock on /omd/sites/zdv/var/check_mk/persisted/nfsrefer-01
Stored persisted sections: lnx_packages, lnx_distro, lnx_cpuinfo, dmidecode, lnx_uname, lnx_video, lnx_ip_r, lnx_sysctl, lnx_block_devices
Using persisted section SectionName('lnx_packages')
Using persisted section SectionName('lnx_distro')
Using persisted section SectionName('lnx_cpuinfo')
Using persisted section SectionName('dmidecode')
Using persisted section SectionName('lnx_uname')
Using persisted section SectionName('lnx_video')
Using persisted section SectionName('lnx_ip_r')
Using persisted section SectionName('lnx_sysctl')
Using persisted section SectionName('lnx_block_devices')
  -> Add sections: ['apt', 'cfengine3', 'check_mk', 'chrony', 'cifsmounts', 'cpu', 'debian_lts_status', 'df', 'diskstat', 'dmidecode', 'drbd', 'fileinfo', 'heartbeat_crm', 'kernel', 'labels', 'lnx_block_devices', 'lnx_cpuinfo', 'lnx_distro', 'lnx_if', 'lnx_ip_r', 'lnx_packages', 'lnx_sysctl', 'lnx_uname', 'lnx_video', 'local', 'logfiles', 'logwatch', 'md', 'mem', 'mounts', 'mrpe', 'netstat', 'nfsmounts', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
<<<burp:cached(1660041510,300)>>> / Transition NOOPParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['burp', 'labels']
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<drbd>>> / Transition HostSectionParser -> HostSectionParser
<<<heartbeat_crm>>> / Transition HostSectionParser -> HostSectionParser
<<<fileinfo:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1660041716,30)>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe:cached(1660041655,300)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cfengine3>>> / Transition HostSectionParser -> HostSectionParser
<<<debian_lts_status>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch>>> / Transition HostSectionParser -> HostSectionParser
<<<netstat>>> / Transition HostSectionParser -> HostSectionParser
<<<logfiles:cached(1660041475,300)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1660041184,3600):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
Trying to acquire lock on /omd/sites/zdv/var/check_mk/persisted/nfsrefer-02
Got lock on /omd/sites/zdv/var/check_mk/persisted/nfsrefer-02
Releasing lock on /omd/sites/zdv/var/check_mk/persisted/nfsrefer-02
Released lock on /omd/sites/zdv/var/check_mk/persisted/nfsrefer-02
Stored persisted sections: lnx_packages, lnx_distro, lnx_cpuinfo, dmidecode, lnx_uname, lnx_video, lnx_ip_r, lnx_sysctl, lnx_block_devices
Using persisted section SectionName('lnx_packages')
Using persisted section SectionName('lnx_distro')
Using persisted section SectionName('lnx_cpuinfo')
Using persisted section SectionName('dmidecode')
Using persisted section SectionName('lnx_uname')
Using persisted section SectionName('lnx_video')
Using persisted section SectionName('lnx_ip_r')
Using persisted section SectionName('lnx_sysctl')
Using persisted section SectionName('lnx_block_devices')
  -> Add sections: ['apt', 'cfengine3', 'check_mk', 'chrony', 'cifsmounts', 'cpu', 'debian_lts_status', 'df', 'diskstat', 'dmidecode', 'drbd', 'fileinfo', 'heartbeat_crm', 'kernel', 'labels', 'lnx_block_devices', 'lnx_cpuinfo', 'lnx_distro', 'lnx_if', 'lnx_ip_r', 'lnx_packages', 'lnx_sysctl', 'lnx_uname', 'lnx_video', 'local', 'logfiles', 'logwatch', 'md', 'mem', 'mounts', 'mrpe', 'netstat', 'nfsmounts', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
<<<burp:cached(1660041510,300)>>> / Transition NOOPParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['burp', 'labels']
Received no piggyback data
Received no piggyback data
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7f880dd48b50]
value store: synchronizing
Trying to acquire lock on /omd/sites/zdv/tmp/check_mk/counters/nfsrefer
Got lock on /omd/sites/zdv/tmp/check_mk/counters/nfsrefer
value store: loading from disk
Releasing lock on /omd/sites/zdv/tmp/check_mk/counters/nfsrefer
Released lock on /omd/sites/zdv/tmp/check_mk/counters/nfsrefer
Filesystem /data     Best: [nfsrefer-01], 5.77% used (228.59 MB of 3.87 GB), trend: +16.92 B / 24 hours, Additional results from: [nfsrefer-02]
Filesystem /uni-mainz.de Best: [nfsrefer-01], 11.63% used (453.05 MB of 3.80 GB), trend: +41.01 kB / 24 hours, Additional results from: [nfsrefer-02]
[cpu_tracking] Stop [7f880dd48b50 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.019999999552965164))]
Trying to acquire lock on /omd/sites/zdv/var/check_mk/crashes/base/250762ae-17d0-11ed-9298-a0369ff0d398/crash.info
Got lock on /omd/sites/zdv/var/check_mk/crashes/base/250762ae-17d0-11ed-9298-a0369ff0d398/crash.info
Releasing lock on /omd/sites/zdv/var/check_mk/crashes/base/250762ae-17d0-11ed-9298-a0369ff0d398/crash.info
Released lock on /omd/sites/zdv/var/check_mk/crashes/base/250762ae-17d0-11ed-9298-a0369ff0d398/crash.info
Traceback (most recent call last):
  File "/omd/sites/zdv/bin/cmk", line 98, in <module>
    exit_status = modes.call("--check", None, opts, args)
  File "/omd/sites/zdv/lib/python3/cmk/base/modes/__init__.py", line 69, in call
    return handler(*handler_args)
  File "/omd/sites/zdv/lib/python3/cmk/base/modes/check_mk.py", line 1804, in mode_check
    checking.commandline_checking(
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/decorator.py", line 43, in wrapped_check_func
    status, output_text = _combine_texts(check_func(hostname, *args, **kwargs))
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/__init__.py", line 121, in commandline_checking
    return _execute_checkmk_checks(
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/__init__.py", line 174, in _execute_checkmk_checks
    num_success, plugins_missing_data = check_host_services(
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/__init__.py", line 322, in check_host_services
    success = _execute_check(
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/__init__.py", line 382, in _execute_check
    submittable = get_aggregated_result(
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/__init__.py", line 470, in get_aggregated_result
    result = _aggregate_results(
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/__init__.py", line 578, in _aggregate_results
    perfdata, results = _consume_and_dispatch_result_types(subresults)
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/__init__.py", line 622, in _consume_and_dispatch_result_types
    for subr in subresults:
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/_cluster_modes.py", line 145, in _cluster_check
    yield from summarizer.secondary_results(
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/_cluster_modes.py", line 225, in secondary_results
    yield from (
  File "/omd/sites/zdv/lib/python3/cmk/base/agent_based/checking/_cluster_modes.py", line 226, in <genexpr>
    Result(
  File "/omd/sites/zdv/lib/python3/cmk/base/api/agent_based/checking_classes.py", line 367, in __new__
    state, summary, details = _create_result_fields(**kwargs)
  File "/omd/sites/zdv/lib/python3/cmk/base/api/agent_based/checking_classes.py", line 405, in _create_result_fields
    raise ValueError(f"'{name}' must be non-empty str or None, got {var}")
ValueError: 'notice' must be non-empty str or None, got 

(Ignore the crash at the end - that’s a different problem: Cluster Checks crashing (Apache and MRPE) - #5 by schlarbm)

Thanks,
Moritz

I saw this too, let me double-check that.

Yesterday i had the same problem. It looks like that the option “Failover” is not working correctly.
Only best or worst is usable at the moment.

Thank you two for confirming :wink:

I opened an internal ticket on this. Let’s see, what the experts think. :wink:

1 Like

Hello @andreas-doehler and @robin.gierse, has there been an update to this issue? I know it’s a bit stale but I haven’t been able to get any hits in the forum regarding “Failover” being fixed yet.

I realize I’m late to the game. What I did just now, on a daily build of the 2.2 and the 2.1:

Set up two hosts, one (“old”) with the regular Checkmk agent, and a second one (“old2”) with the agent output piped through grep -v boot. I then clustered the "Filesystem .*" services with the aggregation mode “failover”. This is what I see:

This is what I expected. If I understood the issue correctly, it seems fixed.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.