Checkmk unknow data

Troubleshooting
1

CMK version:1.6
OS version:Oracle Enterprise Linux 7

We are seeing many checks are getting converted into unknow and again coming back to okay frequently , its local plugin based checked , script is running fine and returning all checks on console

image
image901×655 38.5 KB

/usr/lib/check_mk_agent/local
[root@c1zlcc2a local]# ls
600 check_mcollective.py check_puppet.py check_sssd_auth.py mk_vts_app_ebs sqlnet.log
[root@c1zlcc2a local]# ./mk_vts_app_ebs
0 EBS-conc_mgr_cebsprd2_CRPINQMGR - name=CRPINQMGR,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_FNDCPOPP - name=FNDCPOPP,target=4,actual=4,pending=0
2 EBS-conc_mgr_cebsprd2_FNDCRM - name=FNDCRM,target=1,actual=1,pending=26
0 EBS-conc_mgr_cebsprd2_FNDICM - name=FNDICM,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_FNDIM_C1ZLCC2A - name=FNDIM_C1ZLCC2A,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_FNDMBLPNTFSVC - name=FNDMBLPNTFSVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_FNDSCH - name=FNDSCH,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_FNDSM_C1ZLCC2A - name=FNDSM_C1ZLCC2A,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_FTE_TXN_MANAGER - name=FTE_TXN_MANAGER,target=10,actual=10,pending=0
0 EBS-conc_mgr_cebsprd2_ICUCONCMNGSHP - name=ICUCONCMNGSHP,target=30,actual=30,pending=2
0 EBS-conc_mgr_cebsprd2_ICUFMCONMGR - name=ICUFMCONMGR,target=20,actual=20,pending=0
0 EBS-conc_mgr_cebsprd2_ICUQM - name=ICUQM,target=20,actual=20,pending=0
0 EBS-conc_mgr_cebsprd2_ICUSRVPRG - name=ICUSRVPRG,target=10,actual=10,pending=0
0 EBS-conc_mgr_cebsprd2_INVMGR - name=INVMGR,target=50,actual=50,pending=0
0 EBS-conc_mgr_cebsprd2_INVTMRPM - name=INVTMRPM,target=5,actual=5,pending=0
0 EBS-conc_mgr_cebsprd2_MRPMGR - name=MRPMGR,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_PASMGR - name=PASMGR,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_PODAMGR - name=PODAMGR,target=3,actual=3,pending=0
0 EBS-conc_mgr_cebsprd2_RCVOLTM - name=RCVOLTM,target=5,actual=5,pending=0
0 EBS-conc_mgr_cebsprd2_STANDARD - name=STANDARD,target=80,actual=80,pending=26
0 EBS-conc_mgr_cebsprd2_WFALSNRSVC - name=WFALSNRSVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_WFMLRSVC - name=WFMLRSVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_WFWSSVC - name=WFWSSVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_APPL_SVC - name=XDP_APPL_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_CTRL_SVC - name=XDP_CTRL_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_Q_EVENT_SVC - name=XDP_Q_EVENT_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_Q_FA_SVC - name=XDP_Q_FA_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_Q_FE_READY_SVC - name=XDP_Q_FE_READY_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_Q_IN_MSG_SVC - name=XDP_Q_IN_MSG_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_Q_ORDER_SVC - name=XDP_Q_ORDER_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_Q_TIMER_SVC - name=XDP_Q_TIMER_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XDP_Q_WI_SVC - name=XDP_Q_WI_SVC,target=1,actual=1,pending=0
0 EBS-conc_mgr_cebsprd2_XXDAYSM - name=XXDAYSM,target=15,actual=15,pending=0

what could be issue can anyone help

OMD[default]:~$ cmk --debug -vvn c1zlcc2a.ora.vtscloud.io
WARNING: Found rules for deprecated ruleset ‘inventory_services’. These rules are not applied anymore. In case you still need them, you need to migrate them by hand. Otherwise you can remove them from your configuration.
[cpu_tracking] Start with phase ‘busy’
Check_MK version 1.6.0p20
Try aquire lock on /omd/sites/default/tmp/check_mk/counters/c1zlcc2a.ora.vtscloud.io
Got lock on /omd/sites/default/tmp/check_mk/counters/c1zlcc2a.ora.vtscloud.io
Releasing lock on /omd/sites/default/tmp/check_mk/counters/c1zlcc2a.ora.vtscloud.io
Released lock on /omd/sites/default/tmp/check_mk/counters/c1zlcc2a.ora.vtscloud.io
Loading autochecks from /omd/sites/default/var/check_mk/autochecks/c1zlcc2a.ora.vtscloud.io.mk

  • FETCHING DATA
    [cpu_tracking] Push phase ‘agent’ (Stack: [‘busy’])
    [agent] No persisted sections loaded
    [agent] Not using cache (Don’t try it)
    [agent] Execute data source
    [agent] Connecting via TCP to 10.162.31.19:6556 (40.0s timeout)
    [agent] Reading data from agent
    [agent] Write data to cache file /omd/sites/default/tmp/check_mk/cache/c1zlcc2a.ora.vtscloud.io
    Try aquire lock on /omd/sites/default/tmp/check_mk/cache/c1zlcc2a.ora.vtscloud.io
    Got lock on /omd/sites/default/tmp/check_mk/cache/c1zlcc2a.ora.vtscloud.io
    Releasing lock on /omd/sites/default/tmp/check_mk/cache/c1zlcc2a.ora.vtscloud.io
    Released lock on /omd/sites/default/tmp/check_mk/cache/c1zlcc2a.ora.vtscloud.io
    [cpu_tracking] Pop phase ‘agent’ (Stack: [‘busy’, ‘agent’])
    [cpu_tracking] Push phase ‘agent’ (Stack: [‘busy’])
    [piggyback] No persisted sections loaded
    [piggyback] Execute data source
    No piggyback files for ‘c1zlcc2a.ora.vtscloud.io’. Skip processing.
    No piggyback files for ‘10.162.31.19’. Skip processing.
    [cpu_tracking] Pop phase ‘agent’ (Stack: [‘busy’, ‘agent’])
    CPU load OK - 15 min load: 0.63 at 16 Cores (0.04 per Core)
    CPU utilization OK - User: 4.25%, System: 1.49%, Wait: 0.01%, Steal: 0.001%, Total CPU: 5.75%
    Check_Dns OK - 0 check_dns - OK - velocity.vcamp.io is resolving correctly
    Check_SSSD_Auth OK - vtsmonitoring@velocity.m.vtscloud.io OK
    Disk IO LVM vg_shared-lv_shared OK - Utilization: 0.14%, Read: 0 B/s, Write: 498.67 kB/s, Average Wait: 1.27 ms, Average Read Wait: 0.00 ms, Average Write Wait: 1.27 ms, Latency: 0.23 ms, Average Queue Length: 0.00, Read operations: 0.00 1/s, Write operations: 6.22 1/s
    Disk IO sda OK - Utilization: 0.62%, Read: 42.67 B/s, Write: 1.38 MB/s, Average Wait: 1.51 ms, Average Read Wait: 1.00 ms, Average Write Wait: 1.51 ms, Latency: 0.12 ms, Average Queue Length: 0.00, Read operations: 0.01 1/s, Write operations: 52.08 1/s
    Disk IO sdb OK - Utilization: 0.14%, Read: 0 B/s, Write: 498.67 kB/s, Average Wait: 1.19 ms, Average Read Wait: 0.00 ms, Average Write Wait: 1.19 ms, Latency: 0.26 ms, Average Queue Length: 0.00, Read operations: 0.00 1/s, Write operations: 5.36 1/s
    EBS-EBS-opmn_EBSPRD_balance_OHS OK - name=OHS,status=Alive
    EBS-MS-AdminServer_EBSPRD_balance OK - AdminServer is RUNNING
    EBS-MS-oacore_server11_EBSPRD_balance OK - oacore_server11 is RUNNING
    EBS-MS-oacore_server13_EBSPRD_balance OK - oacore_server13 is RUNNING
    EBS-MS-oacore_server14_EBSPRD_balance OK - oacore_server14 is RUNNING
    Traceback (most recent call last):
    File “/omd/sites/default/bin/cmk”, line 100, in
    exit_status = modes.call(“–check”, None, opts, args)
    File “/omd/sites/default/lib/python/cmk_base/modes/init.py”, line 72, in call
    return mode.handler_function(*handler_args)
    File “/omd/sites/default/lib/python/cmk_base/modes/check_mk.py”, line 1579, in mode_check
    return checking.do_check(hostname, ipaddress, options.get(“checks”))
    File “/omd/sites/default/lib/python/cmk_base/decorator.py”, line 58, in wrapped_check_func
    status, infotexts, long_infotexts, perfdata = check_func(hostname, *args, **kwargs)
    File “/omd/sites/default/lib/python/cmk_base/checking.py”, line 111, in do_check
    _do_all_checks_on_host(sources, host_config, ipaddress, only_check_plugin_names)
    File “/omd/sites/default/lib/python/cmk_base/checking.py”, line 247, in _do_all_checks_on_host
    service.description)
    File “/omd/sites/default/lib/python/cmk_base/checking.py”, line 322, in execute_check
    cmk_base.check_utils.is_snmp_check(check_plugin_name))
    File “/omd/sites/default/lib/python/cmk_base/checking.py”, line 444, in sanitize_check_result
    return _sanitize_yield_check_result(result, is_snmp)
    File “/omd/sites/default/lib/python/cmk_base/checking.py”, line 450, in _sanitize_yield_check_result
    subresults = list(result)
    File “/omd/sites/default/share/check_mk/checks/local”, line 232, in check_local
    if params is not None and params.get(“outcome_on_cluster”, “worst”) == “best”:
    AttributeError: ‘tuple’ object has no attribute ‘get’
    OMD[default]:~$
2

ok that should be no problem

I don’t think so. It looks like the script does not finish intime. Every time you have a runtime problem with your script your get the UNKN message as no data is delivered.
Please set the local check for async execution with some cache time and also a higher timeout setting than it is now.

Please update your system. This old agent cannot handle async local checks with timeout correctly.

3

When I am running script locally, it’s running fine , also I am seeing all data on doing telnet on port 6556,

Also agent running fine within 40 seconds ,
Don’t know what and how i need to update to get it fixed

4

Hi @imbalram

Don’t know what and how i need to update to get it fixed

Your entire Checkmk environment should be updated. 1.6 has been EOL for a long time now.

Please refer to this post for all the documentation: Upgrade checkmk from 1.6.0p8.cre to latest version - #2 by elias.voelker

1 Like
5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.