Brocade / Ruckus Switch Stacks and SNMP Service Check Timeouts

dave10x1 · July 1, 2024, 10:55am

CMK version: 2.2.0p18.cre
OS version: Ubuntu Server 22.04.4 LTS

Error message: (Service Check Timed Out)

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Checkmk version 2.2.0p18

FETCHING DATA
Source: SourceInfo(hostname=‘Hirst-Stack–2’, ipaddress=‘10.1.4.29’, ident=‘snmp’, fetcher_type=<FetcherType.SNMP: 7>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f343e87e850]
Read from cache: SNMPFileCache(Hirst-Stack–2, path_template=/omd/sites/monitoring/tmp/check_mk/data_source_cache/snmp/{mode}/{hostname}, max_age=MaxAge(checking=0, discovery=450.0, inventory=450.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 1688 sec, allowed is 0 sec)
[SNMPFetcher] Execute data source
SNMP scan:
Getting OID .1.3.6.1.2.1.1.1.0: Running ‘snmpget -v2c -c public -m “” -M “” -r 10 -On -OQ -Oe -Ot 10.1.4.29 .1.3.6.1.2.1.1.1.0’
SNMP answer: ==> [“Ruckus Wireless, Inc. ICX8200-48, IronWare Version 10.0.10cT253 Compiled on Dec 24 2023 at 00:43:34 labeled as RDR10010c”]
b’Ruckus Wireless, Inc. ICX8200-48, IronWare Version 10.0.10cT253 Compiled on Dec 24 2023 at 00:43:34 labeled as RDR10010c’
Getting OID .1.3.6.1.2.1.1.2.0: Running ‘snmpget -v2c -c public -m “” -M “” -r 10 -On -OQ -Oe -Ot 10.1.4.29 .1.3.6.1.2.1.1.2.0’
SNMP answer: ==> [.1.3.6.1.4.1.1991.1.3.48.14.2]
b’.1.3.6.1.4.1.1991.1.3.48.14.2’
Using cached OID .1.3.6.1.2.1.1.1.0: ‘Ruckus Wireless, Inc. ICX8200-48, IronWare Version 10.0.10cT253 Compiled on Dec 24 2023 at 00:43:34 labeled as RDR10010c’
SNMP scan found snmp_uptime
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/snmp_scan_cache/Hirst-Stack–2.10.1.4.29
Got lock on /omd/sites/monitoring/tmp/check_mk/snmp_scan_cache/Hirst-Stack–2.10.1.4.29
Releasing lock on /omd/sites/monitoring/tmp/check_mk/snmp_scan_cache/Hirst-Stack–2.10.1.4.29
Released lock on /omd/sites/monitoring/tmp/check_mk/snmp_scan_cache/Hirst-Stack–2.10.1.4.29
brocade_mlx: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running ‘snmpbulkwalk -Cr20 -v2c -c public -m “” -M “” -r 10 -Cc -OQ -OU -On -Ot 10.1.4.29 .1.3.6.1.4.1.1991.1.1.2.2.1.1.1’
Running ‘snmpbulkwalk -Cr20 -v2c -c public -m “” -M “” -r 10 -Cc -OQ -OU -On -Ot 10.1.4.29 .1.3.6.1.4.1.1991.1.1.2.2.1.1.2’
Running ‘snmpbulkwalk -Cr20 -v2c -c public -m “” -M “” -r 10 -Cc -OQ -OU -On -Ot 10.1.4.29 .1.3.6.1.4.1.1991.1.1.2.2.1.1.12’
Running ‘snmpbulkwalk -Cr20 -v2c -c public -m “” -M “” -r 10 -Cc -OQ -OU -On -Ot 10.1.4.29 .1.3.6.1.4.1.1991.1.1.2.2.1.1.24’
Running ‘snmpbulkwalk -Cr20 -v2c -c public -m “” -M “” -r 10 -Cc -OQ -OU -On -Ot 10.1.4.29 .1.3.6.1.4.1.1991.1.1.2.2.1.1.25’
^C[cpu_tracking] Stop [7f343e87e850 - Snapshot(process=posix.times_result(user=0.010000000000000231, system=0.009999999999999953, children_user=0.06, children_system=0.06, elapsed=5.849999997764826))]
Source: SourceInfo(hostname=‘Hirst-Stack–2’, ipaddress=‘10.1.4.29’, ident=‘piggyback’, fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f343e884c10]
Read from cache: NoCache(Hirst-Stack–2, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for ‘Hirst-Stack–2’. Skip processing.
No piggyback files for ‘10.1.4.29’. Skip processing.
[cpu_tracking] Stop [7f343e884c10 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
PARSE FETCHER RESULTS
HostKey(hostname=‘Hirst-Stack–2’, source_type=<SourceType.HOST: 1>) → Not adding sections: MKTerminate
HostKey(hostname=‘Hirst-Stack–2’, source_type=<SourceType.HOST: 1>) → Add sections:
Received no piggyback data
[cpu_tracking] Start [7f343e886410]
value store: synchronizing
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/Hirst-Stack–2
Got lock on /omd/sites/monitoring/tmp/check_mk/counters/Hirst-Stack–2
value store: loading from disk
Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/Hirst-Stack–2
Released lock on /omd/sites/monitoring/tmp/check_mk/counters/Hirst-Stack–2
Fan 1 Fan 1 (Rear Side Right) PEND Check plugin received no monitoring data
Interface 00000065 PEND Check plugin received no monitoring data
Interface 00000066 PEND Check plugin received no monitoring data
Interface 00000067 PEND Check plugin received no monitoring data
Interface 00000321 PEND Check plugin received no monitoring data
Interface 00000322 PEND Check plugin received no monitoring data
Interface 00000323 PEND Check plugin received no monitoring data
Interface 00000557 PEND Check plugin received no monitoring data
Interface 00000577 PEND Check plugin received no monitoring data
Interface 00000579 PEND Check plugin received no monitoring data
Interface 00000833 PEND Check plugin received no monitoring data
Interface 00000835 PEND Check plugin received no monitoring data
Interface 00001089 PEND Check plugin received no monitoring data
Interface 00001090 PEND Check plugin received no monitoring data
Interface 00001091 PEND Check plugin received no monitoring data
Interface 00001345 PEND Check plugin received no monitoring data
Interface 00001346 PEND Check plugin received no monitoring data
Interface 00001347 PEND Check plugin received no monitoring data
Power supply 1 PEND Check plugin received no monitoring data
Power supply 2 PEND Check plugin received no monitoring data
SNMP Info PEND Check plugin received no monitoring data
Status Module 1 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 10 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 13 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 14 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 17 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 18 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 2 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 21 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 22 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 5 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 6 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 9 ICX8200-48ZP2 POE 48-port Management PEND Check plugin received no monitoring data
Temperature Sensor 1 Temperature PEND Check plugin received no monitoring data
Temperature Sensor 2 Temperature PEND Check plugin received no monitoring data
Temperature Sensor 3 Temperature PEND Check plugin received no monitoring data
Temperature Sensor 4 Temperature PEND Check plugin received no monitoring data
Uptime PEND Check plugin received no monitoring data
No piggyback files for ‘Hirst-Stack–2’. Skip processing.
No piggyback files for ‘10.1.4.29’. Skip processing.
[cpu_tracking] Stop [7f343e886410 - Snapshot(process=posix.times_result(user=0.18999999999999995, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.19999999925494194))]
[snmp] MKTerminate(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 6.0 sec | execution_time=6.050 user_time=0.200 system_time=0.010 children_user_time=0.060 children_system_time=0.060 cmk_time_snmp=5.710 cmk_time_agent=0.000

real 0m8.974s
user 0m2.708s
sys 0m0.557s
OMD[monitoring]:~$ ^C
OMD[monitoring]:~$ time cmk -vvn Hirst-Stack–2
Checkmk version 2.2.0p18

FETCHING DATA
Source: SourceInfo(hostname=‘Hirst-Stack–2’, ipaddress=‘10.1.4.29’, ident=‘snmp’, fetcher_type=<FetcherType.SNMP: 7>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7eff4d3de810]
Read from cache: SNMPFileCache(Hirst-Stack–2, path_template=/omd/sites/monitoring/tmp/check_mk/data_source_cache/snmp/{mode}/{hostname}, max_age=MaxAge(checking=0, discovery=450.0, inventory=450.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 1702 sec, allowed is 0 sec)
[SNMPFetcher] Execute data source
SNMP scan:
Getting OID .1.3.6.1.2.1.1.1.0: Running ‘snmpget -v2c -c public -m “” -M “” -r 10 -On -OQ -Oe -Ot 10.1.4.29 .1.3.6.1.2.1.1.1.0’
ERROR: SNMP error
Timeout: No Response from 10.1.4.29.

failed.
[cpu_tracking] Stop [7eff4d3de810 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.01, elapsed=11.03999999910593))]
Source: SourceInfo(hostname=‘Hirst-Stack–2’, ipaddress=‘10.1.4.29’, ident=‘piggyback’, fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7eff4d3dfd90]
Read from cache: NoCache(Hirst-Stack–2, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for ‘Hirst-Stack–2’. Skip processing.
No piggyback files for ‘10.1.4.29’. Skip processing.
[cpu_tracking] Stop [7eff4d3dfd90 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]

PARSE FETCHER RESULTS
HostKey(hostname=‘Hirst-Stack–2’, source_type=<SourceType.HOST: 1>) → Not adding sections: Cannot fetch system description OID .1.3.6.1.2.1.1.1.0. Please check your SNMP configuration. Possible reason might be: Wrong credentials, wrong SNMP version, Firewall rules, etc.
HostKey(hostname=‘Hirst-Stack–2’, source_type=<SourceType.HOST: 1>) → Add sections:
Received no piggyback data
[cpu_tracking] Start [7eff4d3e5750]
value store: synchronizing
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/Hirst-Stack–2
Got lock on /omd/sites/monitoring/tmp/check_mk/counters/Hirst-Stack–2
value store: loading from disk
Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/Hirst-Stack–2
Released lock on /omd/sites/monitoring/tmp/check_mk/counters/Hirst-Stack–2
Fan 1 Fan 1 (Rear Side Right) PEND Check plugin received no monitoring data
Interface 00000065 PEND Check plugin received no monitoring data
Interface 00000066 PEND Check plugin received no monitoring data
Interface 00000067 PEND Check plugin received no monitoring data
Interface 00000321 PEND Check plugin received no monitoring data
Interface 00000322 PEND Check plugin received no monitoring data
Interface 00000323 PEND Check plugin received no monitoring data
Interface 00000557 PEND Check plugin received no monitoring data
Interface 00000577 PEND Check plugin received no monitoring data
Interface 00000579 PEND Check plugin received no monitoring data
Interface 00000833 PEND Check plugin received no monitoring data
Interface 00000835 PEND Check plugin received no monitoring data
Interface 00001089 PEND Check plugin received no monitoring data
Interface 00001090 PEND Check plugin received no monitoring data
Interface 00001091 PEND Check plugin received no monitoring data
Interface 00001345 PEND Check plugin received no monitoring data
Interface 00001346 PEND Check plugin received no monitoring data
Interface 00001347 PEND Check plugin received no monitoring data
Power supply 1 PEND Check plugin received no monitoring data
Power supply 2 PEND Check plugin received no monitoring data
SNMP Info PEND Check plugin received no monitoring data
Status Module 1 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 10 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 13 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 14 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 17 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 18 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 2 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 21 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 22 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 5 ICX8200-48 48-port Management PEND Check plugin received no monitoring data
Status Module 6 ICX8200-4x25G 4-port 100G PEND Check plugin received no monitoring data
Status Module 9 ICX8200-48ZP2 POE 48-port Management PEND Check plugin received no monitoring data
Temperature Sensor 1 Temperature PEND Check plugin received no monitoring data
Temperature Sensor 2 Temperature PEND Check plugin received no monitoring data
Temperature Sensor 3 Temperature PEND Check plugin received no monitoring data
Temperature Sensor 4 Temperature PEND Check plugin received no monitoring data
Uptime PEND Check plugin received no monitoring data
No piggyback files for ‘Hirst-Stack–2’. Skip processing.
No piggyback files for ‘10.1.4.29’. Skip processing.
[cpu_tracking] Stop [7eff4d3e5750 - Snapshot(process=posix.times_result(user=0.17999999999999972, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.16999999806284904))]
[snmp] Cannot fetch system description OID .1.3.6.1.2.1.1.1.0. Please check your SNMP configuration. Possible reason might be: Wrong credentials, wrong SNMP version, Firewall rules, etc.(!!), [piggyback] Success (but no data found for this host), Missing monitoring data for all plugins(!), execution time 11.2 sec | execution_time=11.210 user_time=0.180 system_time=0.000 children_user_time=0.000 children_system_time=0.010 cmk_time_snmp=11.030 cmk_time_agent=0.000

real 0m14.161s
user 0m2.761s
sys 0m0.389s

dave10x1 · July 3, 2024, 7:56am

Hi we are having problems monitoring some of our switches Ruckus ICX8200, I have so far tried SNMP v1, v2c, v3.

Increased time outs for the service_checks_timeout in the tuning.cfg, only made the duff checks run longer and longer.

Changing the “Timing settings for SNMP access” didn’t seem to help either.

I’ve tried adding a rule to use inline SNMP.

The main problem I can see is when troubleshooting using the “time cmk -vvn HOSTNAME” Check snmpwalk takes ages on some OIDS.

Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.2.2.1.1.1’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.2.2.1.1.2’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.2.2.1.1.12’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.2.2.1.1.24’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.2.2.1.1.25’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.2.11.1.1.5’
brocade_mlx_fan: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.3.1.1.1’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.3.1.1.2’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.3.1.1.3’
brocade_mlx_power: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.2.1.1.1’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.2.1.1.2’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.2.1.1.3’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.2.2.1.2’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.2.2.1.3’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.1.2.2.1.4’
brocade_mlx_temp: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.2.13.1.1.3’
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.4.1.1991.1.1.2.13.1.1.4’
if: Fetching data (SNMP walk cache is enabled: Use any locally cached information)
Running ‘snmpwalk -v1 -c public -m “” -M “” -t 60.00 -r 10 -Cc -OQ -OU -On -Ot 10.1.4.28 .1.3.6.1.2.1.2.2.1.1’