CMK version:
Version: 2.1.0p34, Edition: cre
OS version:
CentOS Linux release 7.9.2009
Description of the problem :
Our Cisco 3850 L3 switch encounter high cpu caused by bcast traffic for a long period. On switch CLI we see all the 5s, 1min and 5 mins CPU usage are above 94%; but at CheckMK monitoring, it only showed the cpu usage around 40 - 50%.
Thus we only notice when we logged into the switch and sense the slowness and check. Is the SNMP OID inaccurate or any reason on that? How may we have the accurate cpu usage stats report?
Output of “cmk --debug -vvn hostname”:
OMD[sha2_netwk_stage]:~$ cmk --debug -vvn sha2-core
Checkmk version 2.1.0p34
Try license usage history update.
Trying to acquire lock on /omd/sites/sha2_netwk_stage/var/check_mk/license_usage/next_run
Got lock on /omd/sites/sha2_netwk_stage/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/sha2_netwk_stage/var/check_mk/license_usage/history.json
Got lock on /omd/sites/sha2_netwk_stage/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/sha2_netwk_stage/var/check_mk/license_usage/history.json
Released lock on /omd/sites/sha2_netwk_stage/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/sha2_netwk_stage/var/check_mk/license_usage/next_run
Released lock on /omd/sites/sha2_netwk_stage/var/check_mk/license_usage/next_run
+ FETCHING DATA
Source: SourceType.HOST/FetcherType.SNMP
[cpu_tracking] Start [7efd9d5e31c0]
[SNMPFetcher] Fetch with cache settings: SNMPFileCache(sha2-core, base_path=/omd/sites/sha2_netwk_stage/tmp/check_mk/data_source_cache/snmp, max_age=MaxAge(checking=0, discovery=450, inventory=450), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 169 sec, allowed is 0 sec)
[SNMPFetcher] Execute data source
Loading .1.3.6.1.2.1.47.1.1.1.1.2 from walk cache /omd/sites/sha2_netwk_stage/var/check_mk/snmp_cache/sha2-core/OID.1.3.6.1.2.1.47.1.1.1.1.2
Loading .1.3.6.1.2.1.2.2.1.2 from walk cache /omd/sites/sha2_netwk_stage/var/check_mk/snmp_cache/sha2-core/OID.1.3.6.1.2.1.2.2.1.2
Loading .1.3.6.1.2.1.2.2.1.7 from walk cache /omd/sites/sha2_netwk_stage/var/check_mk/snmp_cache/sha2-core/OID.1.3.6.1.2.1.2.2.1.7
cisco_cpu_memory: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.109.1.1.1.1.2'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.109.1.1.1.1.12'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.109.1.1.1.1.13'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.109.1.1.1.1.14'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.2.1.47.1.1.1.1.7'
cisco_cpu_multiitem: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Already fetched OID: .1.3.6.1.4.1.9.9.109.1.1.1.1.2
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.109.1.1.1.1.8'
Already fetched OID: .1.3.6.1.2.1.47.1.1.1.1.7
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.2.1.47.1.1.1.1.5'
cisco_oldcpu: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.2.1.57'
cisco_fan: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.13.1.4.1.2'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.13.1.4.1.3'
cisco_mem: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.48.1.1.1.2'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.48.1.1.1.5'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.48.1.1.1.6'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.48.1.1.1.7'
cisco_power: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.13.1.5.1.2'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.13.1.5.1.3'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.13.1.5.1.4'
cisco_stack: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.500.1.2.1.1.1'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.500.1.2.1.1.3'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.4.1.9.9.500.1.2.1.1.6'
snmp_info: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.2.1.1.1'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.2.1.1.4'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.2.1.1.5'
Running 'snmpbulkwalk -Cr10 -v2c -c dida^uwh5r5 -m "" -M "" -Cc -OQ -OU -On -Ot 10.70.0.3 .1.3.6.1.2.1.1.6'
Write data to cache file /omd/sites/sha2_netwk_stage/tmp/check_mk/data_source_cache/snmp/checking/sha2-core
Trying to acquire lock on /omd/sites/sha2_netwk_stage/tmp/check_mk/data_source_cache/snmp/checking/sha2-core
Got lock on /omd/sites/sha2_netwk_stage/tmp/check_mk/data_source_cache/snmp/checking/sha2-core
Releasing lock on /omd/sites/sha2_netwk_stage/tmp/check_mk/data_source_cache/snmp/checking/sha2-core
Released lock on /omd/sites/sha2_netwk_stage/tmp/check_mk/data_source_cache/snmp/checking/sha2-core
[cpu_tracking] Stop [7efd9d5e31c0 - Snapshot(process=posix.times_result(user=0.10999999999999988, system=0.09, children_user=0.05, children_system=0.16, elapsed=0.8299999982118607))]
Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7efd9d5e3be0]
[PiggybackFetcher] Fetch with cache settings: NoCache(sha2-core, base_path=/omd/sites/sha2_netwk_stage/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=450, inventory=450), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'sha2-core'. Skip processing.
No piggyback files for '10.70.0.3'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7efd9d5e3be0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
Source: SourceType.HOST/FetcherType.SNMP
No persisted sections
-> Add sections: ['cisco_cpu_memory', 'cisco_cpu_multiitem', 'cisco_fan', 'cisco_mem', 'cisco_oldcpu', 'cisco_power', 'cisco_stack', 'snmp_info']
Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
-> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7efd9d5e3d90]
value store: synchronizing
Trying to acquire lock on /omd/sites/sha2_netwk_stage/tmp/check_mk/counters/sha2-core
Got lock on /omd/sites/sha2_netwk_stage/tmp/check_mk/counters/sha2-core
value store: loading from disk
Releasing lock on /omd/sites/sha2_netwk_stage/tmp/check_mk/counters/sha2-core
Released lock on /omd/sites/sha2_netwk_stage/tmp/check_mk/counters/sha2-core
CPU Memory utilization Switch 1 Usage: 48.45% - 1.82 GB of 3.75 GB
CPU Memory utilization Switch 2 Usage: 45.89% - 1.72 GB of 3.75 GB
CPU Memory utilization Switch 3 Usage: 30.56% - 1.15 GB of 3.75 GB
CPU Memory utilization Switch 4 Usage: 31.54% - 1.18 GB of 3.75 GB
CPU Memory utilization Switch 5 Usage: 32.14% - 1.21 GB of 3.75 GB
CPU utilization Total CPU: 15.0%
CPU utilization Switch 1 Utilization in the last 5 minutes: 14.00%
CPU utilization Switch 2 Utilization in the last 5 minutes: 7.00%
CPU utilization Switch 3 Utilization in the last 5 minutes: 4.00%
CPU utilization Switch 4 Utilization in the last 5 minutes: 3.00%
CPU utilization Switch 5 Utilization in the last 5 minutes: 3.00%
FAN Switch 1 - FAN - T1 1 Status: normal
FAN Switch 1 - FAN - T1 2 Status: normal
FAN Switch 1 - FAN - T1 3 Status: normal
FAN Switch 2 - FAN - T1 1 Status: normal
FAN Switch 2 - FAN - T1 2 Status: normal
FAN Switch 2 - FAN - T1 3 Status: normal
FAN Switch 3 - FAN - T1 1 Status: normal
FAN Switch 3 - FAN - T1 2 Status: normal
FAN Switch 3 - FAN - T1 3 Status: normal
FAN Switch 4 - FAN - T1 1 Status: normal
FAN Switch 4 - FAN - T1 2 Status: normal
FAN Switch 4 - FAN - T1 3 Status: normal
FAN Switch 5 - FAN - T1 1 Status: normal
FAN Switch 5 - FAN - T1 2 Status: normal
FAN Switch 5 - FAN - T1 3 Status: normal
Mem used Processor Usage: 36.44% - 282.76 MB of 776.06 MB
Power Switch 1 - Power Supply A 1015 Status: normal, Source: AC
Power Switch 1 - Power Supply B 1016 Status: normal, Source: AC
Power Switch 2 - Power Supply A 2015 Status: normal, Source: AC
Power Switch 2 - Power Supply B 2016 Status: normal, Source: AC
Power Switch 3 - Power Supply A 3015 Status: normal, Source: AC
Power Switch 3 - Power Supply B 3016 Status: normal, Source: AC
Power Switch 4 - Power Supply A 4015 Status: normal, Source: AC
Power Switch 4 - Power Supply B 4016 Status: normal, Source: AC
Power Switch 5 - Power Supply A 5015 Status: normal, Source: AC
Power Switch 5 - Power Supply B 5016 Status: normal, Source: AC
SNMP Info Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.12.8, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2022 by Cisco Systems, Inc. Compiled Thu 15-Sep-22 06:, sha2-core.Synaptics.com, Shanghai Pudong, it-network@synaptics.com
Switch stack status 1 Switch state: Ready (ready), switch role: master
Switch stack status 2 Switch state: Ready (ready), switch role: standby
Switch stack status 3 Switch state: Ready (ready), switch role: member
Switch stack status 4 Switch state: Ready (ready), switch role: member
Switch stack status 5 Switch state: Ready (ready), switch role: member
No piggyback files for 'sha2-core'. Skip processing.
No piggyback files for '10.70.0.3'. Skip processing.
[cpu_tracking] Stop [7efd9d5e3d90 - Snapshot(process=posix.times_result(user=0.020000000000000018, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
[snmp] Success, execution time 0.8 sec | execution_time=0.840 user_time=0.130 system_time=0.090 children_user_time=0.050 children_system_time=0.160 cmk_time_snmp=0.420 cmk_time_agent=0.000